2024-03-22T02:04:47.740 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-22T02:04:47.741 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797 -- /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/orig.config.yaml 2024-03-22T02:04:47.796 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T02:04:47.898 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797 branch: wip-batrick-testing-20240321.172515 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7615797' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240321.172515 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240321.172515 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 316 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e sleep_before_teardown: 0 subset: 420/512 suite: fs suite_branch: wip-batrick-testing-20240321.172515 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e targets: smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9gYB2JLCkhhT7tRIg+YLlsEkMATFzPQQ8SJHSXQFHF6cKHd1tOP43ruhbCek5CtFhzw1V8mRnW5ArGBuEzOtk= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI0jM9zBtKu98jQ4JaAW1AjLMKw+ByZWirpv/cqAd7JP3LSLTbheXI3hQoSBW7mH4qnef52ht82Qro/uCBqaaO0= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_01:06:30 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-22T02:04:47.898 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa; will attempt to use it 2024-03-22T02:04:47.899 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks 2024-03-22T02:04:47.899 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-22T02:04:47.901 INFO:teuthology.task.internal:Checking packages... 2024-03-22T02:04:47.924 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e' 2024-03-22T02:04:47.924 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:04:47.925 INFO:teuthology.packaging:ref: None 2024-03-22T02:04:47.925 INFO:teuthology.packaging:tag: None 2024-03-22T02:04:47.925 INFO:teuthology.packaging:branch: wip-batrick-testing-20240321.172515 2024-03-22T02:04:47.925 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:04:47.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240321.172515 2024-03-22T02:04:48.147 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2344.gc3a04c0e 2024-03-22T02:04:48.148 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-22T02:04:48.164 INFO:teuthology.task.internal:no buildpackages task found 2024-03-22T02:04:48.164 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-22T02:04:48.211 INFO:teuthology.task.internal:Saving configuration 2024-03-22T02:04:48.234 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-22T02:04:48.264 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T02:04:48.295 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:59:08.689238', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9gYB2JLCkhhT7tRIg+YLlsEkMATFzPQQ8SJHSXQFHF6cKHd1tOP43ruhbCek5CtFhzw1V8mRnW5ArGBuEzOtk='} 2024-03-22T02:04:48.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:59:08.691214', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI0jM9zBtKu98jQ4JaAW1AjLMKw+ByZWirpv/cqAd7JP3LSLTbheXI3hQoSBW7mH4qnef52ht82Qro/uCBqaaO0='} 2024-03-22T02:04:48.319 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-22T02:04:48.330 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-22T02:04:48.330 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-22T02:04:48.330 INFO:teuthology.run_tasks:Running task console_log... 2024-03-22T02:04:48.419 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f16bd459160>, signals=[15]) 2024-03-22T02:04:48.420 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-22T02:04:48.425 INFO:teuthology.task.internal:Opening connections... 2024-03-22T02:04:48.426 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:04:48.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:04:48.559 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-03-22T02:04:48.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:04:48.653 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-22T02:04:48.660 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-03-22T02:04:48.677 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-03-22T02:04:48.677 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-03-22T02:04:48.734 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T02:04:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-03-22T02:04:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T02:04:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T02:04:48.735 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T02:04:48.735 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-03-22T02:04:48.760 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-03-22T02:04:48.778 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-03-22T02:04:48.778 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-03-22T02:04:48.834 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-03-22T02:04:48.834 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T02:04:48.835 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T02:04:48.836 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-03-22T02:04:48.860 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-22T02:04:48.870 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-22T02:04:48.931 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-22T02:04:48.932 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-03-22T02:04:48.934 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-03-22T02:04:48.950 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-22T02:04:48.977 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-22T02:04:48.977 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-03-22T02:04:48.990 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-03-22T02:04:49.079 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-22T02:04:49.137 INFO:teuthology.run_tasks:Running task kernel... 2024-03-22T02:04:49.153 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-22T02:04:49.153 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-22T02:04:49.154 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T02:04:49.154 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T02:04:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:04:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:04:49.172 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-22T02:04:49.172 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-22T02:04:49.229 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-03-22T02:04:49.230 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-361.el9.x86_64 2024-03-22T02:04:49.230 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-03-22T02:04:49.231 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-03-22T02:04:49.231 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-03-22T02:04:49.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-03-22T02:04:51.473 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-22T02:04:51.491 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-22T02:04:52.941 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-22T02:04:53.166 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-22T02:04:53.765 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-22T02:04:54.273 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-22T02:04:56.167 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-22T02:05:00.186 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-22T02:05:02.468 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-22T02:05:03.262 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-22T02:05:08.144 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-22T02:05:10.243 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T02:05:10.243 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T02:05:10.332 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:05:10.339 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:05:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-22T02:05:10.339 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:05:10.339 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:05:10.339 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-03-22T02:05:10.340 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:05:10.341 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 76 M 2024-03-22T02:05:10.341 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 123 M 2024-03-22T02:05:10.341 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:05:10.738 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 379 kB/s | 44 kB 00:00 2024-03-22T02:05:12.154 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 9.7 MB/s | 15 MB 00:01 2024-03-22T02:05:13.338 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-22T02:05:13.863 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 10 MB/s | 33 MB 00:03 2024-03-22T02:05:13.864 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:05:13.864 INFO:teuthology.orchestra.run.smithi103.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-22T02:05:14.528 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:05:14.875 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:05:14.875 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:05:15.911 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:05:15.912 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:05:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:05:19.012 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T02:05:19.099 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:05:21.849 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:05:21.978 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:05:26.686 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:05:26.720 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:05:30.371 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 542 kB/s | 19 MB 00:35 2024-03-22T02:05:31.445 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:05:34.972 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.1 MB 00:01 2024-03-22T02:05:36.431 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-22T02:05:38.452 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-22T02:05:43.425 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-22T02:05:45.548 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T02:05:45.548 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T02:05:45.647 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:05:45.653 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:05:45.653 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-22T02:05:45.653 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-03-22T02:05:45.654 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:05:45.655 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 76 M 2024-03-22T02:05:45.655 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-03-22T02:05:45.655 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:05:46.218 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 185 kB/s | 44 kB 00:00 2024-03-22T02:05:52.701 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 2.2 MB/s | 15 MB 00:06 2024-03-22T02:05:54.959 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 3.7 MB/s | 33 MB 00:08 2024-03-22T02:05:55.477 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 3.0 MB/s | 27 MB 00:09 2024-03-22T02:05:55.477 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:05:55.477 INFO:teuthology.orchestra.run.smithi122.stdout:Total 7.7 MB/s | 76 MB 00:09 2024-03-22T02:05:56.134 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:05:56.519 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:05:56.519 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:05:57.509 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:05:57.509 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:05:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:06:00.568 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:00.568 INFO:teuthology.orchestra.run.smithi103.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T02:06:00.568 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:06:00.592 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T02:06:00.612 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:00.704 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:06:00.945 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:00.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T02:06:00.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:06:00.945 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:06:01.364 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:06:01.693 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T02:06:01.800 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.800 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T02:06:01.800 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T02:06:01.800 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T02:06:01.800 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T02:06:01.800 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-22T02:06:01.801 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T02:06:01.801 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T02:06:01.880 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:01.880 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T02:06:01.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-03-22T02:06:01.914 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-03-22T02:06:01.914 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-03-22T02:06:01.915 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:06:01.916 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T02:06:01.917 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T02:06:01.917 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-430.el9.x86_64 2024-03-22T02:06:01.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T02:06:02.485 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-03-22T02:06:03.415 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:06:03.569 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:06:03.857 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T02:06:03.892 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-03-22T02:06:03.894 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-03-22T02:06:03.909 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.w8KKW00BqS 2024-03-22T02:06:03.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w8KKW00BqS 2024-03-22T02:06:03.978 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.w8KKW00BqS 2024-03-22T02:06:04.115 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.w8KKW00BqS is 6KB 2024-03-22T02:06:04.135 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.w8KKW00BqS 2024-03-22T02:06:04.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T02:06:04.218 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T02:06:04.218 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T02:06:04.218 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T02:06:04.218 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T02:06:04.219 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T02:06:04.379 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-03-22T02:06:08.298 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:06:08.445 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:13.153 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:34.382 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-22T02:06:34.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:06:34.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:06:42.655 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:42.655 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T02:06:42.655 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:06:42.699 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T02:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T02:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T02:06:43.525 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T02:06:43.526 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:06:43.526 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:06:43.867 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T02:06:43.968 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:43.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T02:06:43.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T02:06:43.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T02:06:43.969 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T02:06:43.969 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-22T02:06:43.969 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T02:06:43.969 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T02:06:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T02:06:44.049 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T02:06:44.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T02:06:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:06:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T02:06:44.085 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T02:06:44.085 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-430.el9.x86_64 2024-03-22T02:06:44.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T02:06:44.679 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-03-22T02:06:46.107 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T02:06:46.134 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-03-22T02:06:46.135 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-22T02:06:46.151 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.tj2uTYXfBn 2024-03-22T02:06:46.151 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tj2uTYXfBn 2024-03-22T02:06:46.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.tj2uTYXfBn 2024-03-22T02:06:46.373 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.tj2uTYXfBn is 6KB 2024-03-22T02:06:46.422 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.tj2uTYXfBn 2024-03-22T02:06:46.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T02:06:46.501 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T02:06:46.501 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T02:06:46.501 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T02:06:46.502 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T02:06:46.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T02:06:46.647 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-03-22T02:06:46.751 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-22T02:06:52.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-22T02:07:01.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:07:01.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:07:04.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-22T02:07:16.660 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-22T02:07:16.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:07:16.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:07:16.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:07:16.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:07:19.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-22T02:07:34.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:07:34.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:07:35.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-03-22T02:07:44.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:07:44.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:07:47.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-03-22T02:07:53.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-22T02:07:59.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:07:59.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:08:11.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:08:11.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:08:11.753 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-22T02:08:12.156 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:08:12.156 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-22T02:08:12.157 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-22T02:08:12.172 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-430.el9.x86_64 2024-03-22T02:08:12.172 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T02:08:12.172 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T02:08:12.172 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T02:08:13.174 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-22T02:08:13.174 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-22T02:08:13.234 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-03-22T02:08:13.271 DEBUG:teuthology.parallel:result is None 2024-03-22T02:08:31.167 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-22T02:08:31.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:08:31.595 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-22T02:08:31.595 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-22T02:08:31.609 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-430.el9.x86_64 2024-03-22T02:08:31.609 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T02:08:31.609 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T02:08:31.609 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T02:08:32.610 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-22T02:08:32.610 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-22T02:08:32.714 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-03-22T02:08:32.750 DEBUG:teuthology.parallel:result is None 2024-03-22T02:08:32.751 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-22T02:08:32.759 INFO:teuthology.task.internal:Creating test directory... 2024-03-22T02:08:32.759 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T02:08:32.761 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T02:08:32.777 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-22T02:08:32.806 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-22T02:08:32.895 INFO:teuthology.task.internal:Creating archive directory... 2024-03-22T02:08:32.896 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T02:08:32.898 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T02:08:32.937 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-22T02:08:32.952 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-22T02:08:32.952 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T02:08:32.974 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T02:08:33.001 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T02:08:33.011 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T02:08:33.014 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T02:08:33.025 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T02:08:33.027 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-22T02:08:33.034 INFO:teuthology.task.internal:Configuring sudo... 2024-03-22T02:08:33.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T02:08:33.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T02:08:33.099 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-22T02:08:33.109 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-22T02:08:33.109 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T02:08:33.142 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T02:08:33.156 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T02:08:33.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T02:08:33.288 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T02:08:33.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T02:08:33.413 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:08:33.413 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T02:08:33.481 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T02:08:33.531 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T02:08:33.603 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T02:08:33.656 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T02:08:33.713 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:08:33.714 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T02:08:33.758 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-03-22T02:08:33.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-03-22T02:08:33.804 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T02:08:33.864 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T02:08:34.318 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-22T02:08:34.326 INFO:teuthology.task.internal:Starting timer... 2024-03-22T02:08:34.327 INFO:teuthology.run_tasks:Running task pcp... 2024-03-22T02:08:34.367 INFO:teuthology.run_tasks:Running task selinux... 2024-03-22T02:08:34.407 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-22T02:08:34.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-03-22T02:08:34.489 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-03-22T02:08:34.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-03-22T02:08:34.550 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-03-22T02:08:34.551 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-22T02:08:34.552 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-03-22T02:08:34.578 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-03-22T02:08:34.578 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-03-22T02:08:34.619 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-03-22T02:08:34.619 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-03-22T02:08:34.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T02:08:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:08:34.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T02:08:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:08:34.677 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-22T02:08:34.677 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-03-22T02:08:34.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-03-22T02:08:34.879 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-22T02:08:34.889 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-22T02:08:34.895 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-22T02:08:35.022 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-22T02:08:35.039 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-22T02:08:35.040 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-22T02:13:56.140 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-03-22T02:13:56.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:13:56.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:13:56.233 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-22T02:13:56.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-22T02:13:56.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:13:56.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:13:56.410 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-22T02:13:56.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-22T02:13:56.491 INFO:teuthology.run_tasks:Running task clock... 2024-03-22T02:13:56.511 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-22T02:13:56.511 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T02:13:56.512 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:13:56.514 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T02:13:56.514 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:13:56.542 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T02:13:56.558 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T02:13:56.564 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T02:13:56.581 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T02:13:56.587 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-03-22T02:13:56.601 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-03-22T02:13:56.611 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-03-22T02:13:56.617 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T02:13:56.625 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-03-22T02:13:56.633 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T02:13:56.641 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T02:13:56.658 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T02:13:56.684 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.688 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.710 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.713 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:13:56.714 INFO:teuthology.run_tasks:Running task install... 2024-03-22T02:13:56.723 DEBUG:teuthology.task.install:project ceph 2024-03-22T02:13:56.723 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'}} 2024-03-22T02:13:56.723 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'} 2024-03-22T02:13:56.723 INFO:teuthology.task.install:Using flavor: default 2024-03-22T02:13:56.729 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-22T02:13:56.729 INFO:teuthology.task.install:extra packages: [] 2024-03-22T02:13:56.730 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T02:13:56.730 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:13:56.730 INFO:teuthology.packaging:ref: None 2024-03-22T02:13:56.730 INFO:teuthology.packaging:tag: None 2024-03-22T02:13:56.730 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:13:56.730 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:56.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-22T02:13:56.733 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T02:13:56.733 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:13:56.733 INFO:teuthology.packaging:ref: None 2024-03-22T02:13:56.733 INFO:teuthology.packaging:tag: None 2024-03-22T02:13:56.733 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:13:56.733 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:56.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-22T02:13:57.001 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/ 2024-03-22T02:13:57.001 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-22T02:13:57.062 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/ 2024-03-22T02:13:57.062 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-22T02:13:57.121 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T02:13:57.121 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:13:57.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T02:13:57.150 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-22T02:13:57.150 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:13:57.150 INFO:teuthology.packaging:ref: None 2024-03-22T02:13:57.150 INFO:teuthology.packaging:tag: None 2024-03-22T02:13:57.150 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:13:57.150 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:57.150 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T02:13:57.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T02:13:57.229 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T02:13:57.229 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:13:57.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T02:13:57.257 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-22T02:13:57.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:13:57.257 INFO:teuthology.packaging:ref: None 2024-03-22T02:13:57.257 INFO:teuthology.packaging:tag: None 2024-03-22T02:13:57.258 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:13:57.258 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:57.258 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T02:13:57.298 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T02:13:57.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T02:13:57.364 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-03-22T02:13:57.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-03-22T02:13:57.406 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T02:13:57.473 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-03-22T02:13:57.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-03-22T02:13:57.601 INFO:teuthology.orchestra.run.smithi122.stdout:74 files removed 2024-03-22T02:13:57.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-03-22T02:13:57.731 INFO:teuthology.orchestra.run.smithi103.stdout:74 files removed 2024-03-22T02:13:57.758 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-03-22T02:13:58.234 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 249 kB/s | 77 kB 00:00 2024-03-22T02:13:58.369 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 248 kB/s | 77 kB 00:00 2024-03-22T02:13:58.659 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2024-03-22T02:13:58.803 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2024-03-22T02:13:59.001 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-03-22T02:13:59.162 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.6 kB/s | 1.8 kB 00:00 2024-03-22T02:13:59.587 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-22T02:14:00.089 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-22T02:14:01.154 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-22T02:14:01.639 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-22T02:14:02.480 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-22T02:14:10.595 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 903 kB/s | 8.0 MB 00:09 2024-03-22T02:14:25.296 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 886 kB/s | 19 MB 00:21 2024-03-22T02:14:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-22T02:14:30.706 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-22T02:14:33.702 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 874 kB/s | 19 MB 00:22 2024-03-22T02:14:37.988 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-22T02:14:39.437 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-03-22T02:14:40.630 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-22T02:14:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-22T02:14:47.882 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:14:47.902 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:14:47.902 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:14:47.902 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:14:47.902 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:14:47.902 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 14 M 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.1 M 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 5.0 M 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 23 M 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 636 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T02:14:47.903 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 457 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 4.9 M 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 45 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 117 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 156 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 320 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 293 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 99 k 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:14:47.904 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:14:47.905 INFO:teuthology.orchestra.run.smithi122.stdout:Install 22 Packages 2024-03-22T02:14:47.905 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-03-22T02:14:47.905 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:14:47.905 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 58 M 2024-03-22T02:14:47.905 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:14:48.523 INFO:teuthology.orchestra.run.smithi122.stdout:(1/24): ceph-base-17.2.7-726.g9c4e0c34.el9.x86_ 9.3 MB/s | 5.0 MB 00:00 2024-03-22T02:14:48.615 INFO:teuthology.orchestra.run.smithi122.stdout:(2/24): ceph-selinux-17.2.7-726.g9c4e0c34.el9.x 275 kB/s | 25 kB 00:00 2024-03-22T02:14:48.740 INFO:teuthology.orchestra.run.smithi122.stdout:(3/24): libcephfs2-17.2.7-726.g9c4e0c34.el9.x86 5.0 MB/s | 636 kB 00:00 2024-03-22T02:14:48.807 INFO:teuthology.orchestra.run.smithi122.stdout:(4/24): libradosstriper1-17.2.7-726.g9c4e0c34.e 6.7 MB/s | 457 kB 00:00 2024-03-22T02:14:49.149 INFO:teuthology.orchestra.run.smithi122.stdout:(5/24): librgw2-17.2.7-726.g9c4e0c34.el9.x86_64 14 MB/s | 4.9 MB 00:00 2024-03-22T02:14:49.216 INFO:teuthology.orchestra.run.smithi122.stdout:(6/24): python3-ceph-argparse-17.2.7-726.g9c4e0 680 kB/s | 45 kB 00:00 2024-03-22T02:14:49.283 INFO:teuthology.orchestra.run.smithi122.stdout:(7/24): python3-ceph-common-17.2.7-726.g9c4e0c3 1.7 MB/s | 117 kB 00:00 2024-03-22T02:14:49.391 INFO:teuthology.orchestra.run.smithi122.stdout:(8/24): python3-cephfs-17.2.7-726.g9c4e0c34.el9 1.4 MB/s | 156 kB 00:00 2024-03-22T02:14:49.475 INFO:teuthology.orchestra.run.smithi122.stdout:(9/24): python3-rados-17.2.7-726.g9c4e0c34.el9. 3.7 MB/s | 320 kB 00:00 2024-03-22T02:14:49.542 INFO:teuthology.orchestra.run.smithi122.stdout:(10/24): python3-rbd-17.2.7-726.g9c4e0c34.el9.x 4.3 MB/s | 293 kB 00:00 2024-03-22T02:14:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:(11/24): python3-rgw-17.2.7-726.g9c4e0c34.el9.x 1.2 MB/s | 99 kB 00:00 2024-03-22T02:14:49.809 INFO:teuthology.orchestra.run.smithi122.stdout:(12/24): ceph-radosgw-17.2.7-726.g9c4e0c34.el9. 7.7 MB/s | 14 MB 00:01 2024-03-22T02:14:49.885 INFO:teuthology.orchestra.run.smithi122.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 615 kB/s | 159 kB 00:00 2024-03-22T02:14:49.901 INFO:teuthology.orchestra.run.smithi122.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 489 kB/s | 44 kB 00:00 2024-03-22T02:14:49.960 INFO:teuthology.orchestra.run.smithi122.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-22T02:14:49.985 INFO:teuthology.orchestra.run.smithi122.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-22T02:14:50.019 INFO:teuthology.orchestra.run.smithi122.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2024-03-22T02:14:50.428 INFO:teuthology.orchestra.run.smithi122.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 119 kB/s | 49 kB 00:00 2024-03-22T02:14:50.578 INFO:teuthology.orchestra.run.smithi122.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 450 kB/s | 67 kB 00:00 2024-03-22T02:14:50.820 INFO:teuthology.orchestra.run.smithi122.stdout:(20/24): ceph-common-17.2.7-726.g9c4e0c34.el9.x 7.9 MB/s | 23 MB 00:02 2024-03-22T02:14:50.845 INFO:teuthology.orchestra.run.smithi122.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 359 kB/s | 309 kB 00:00 2024-03-22T02:14:51.170 INFO:teuthology.orchestra.run.smithi122.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-03-22T02:14:51.229 INFO:teuthology.orchestra.run.smithi122.stdout:(23/24): librados2-17.2.7-726.g9c4e0c34.el9.x86 7.5 MB/s | 3.1 MB 00:00 2024-03-22T02:14:51.287 INFO:teuthology.orchestra.run.smithi122.stdout:(24/24): librbd1-17.2.7-726.g9c4e0c34.el9.x86_6 6.6 MB/s | 2.9 MB 00:00 2024-03-22T02:14:51.291 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:14:51.291 INFO:teuthology.orchestra.run.smithi122.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-22T02:14:51.292 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:14:51.321 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:14:51.321 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:14:51.782 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:14:51.782 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:14:52.084 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 997 kB/s | 21 MB 00:21 2024-03-22T02:14:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:14:53.343 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-22T02:14:53.389 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:14:53.481 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:14:53.560 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-22T02:14:53.583 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:14:53.653 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:14:53.720 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-22T02:14:53.762 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-22T02:14:54.197 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-22T02:14:54.227 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-22T02:14:54.273 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-22T02:14:54.302 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-22T02:14:54.344 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 10/26 2024-03-22T02:14:54.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-22T02:14:54.444 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-22T02:14:54.472 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:14:54.542 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:14:54.585 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-22T02:14:54.621 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-22T02:14:54.654 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-22T02:14:54.952 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-22T02:14:54.984 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-22T02:14:55.057 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-22T02:14:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 19/26 2024-03-22T02:14:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 20/26 2024-03-22T02:14:57.031 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-22T02:14:57.560 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:14:57.608 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:14:58.161 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:14:58.194 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-22T02:14:58.269 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-22T02:14:58.269 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T02:14:58.269 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T02:14:58.269 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:14:58.312 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:14:59.459 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:14:59.475 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:14:59.475 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 14 M 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.1 M 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 5.0 M 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 23 M 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 636 k 2024-03-22T02:14:59.476 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 457 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 4.9 M 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 45 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 117 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 156 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T02:14:59.477 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 320 k 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 293 k 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 99 k 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:Install 22 Packages 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 58 M 2024-03-22T02:14:59.478 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:15:00.641 INFO:teuthology.orchestra.run.smithi103.stdout:(1/24): ceph-radosgw-17.2.7-726.g9c4e0c34.el9.x 13 MB/s | 14 MB 00:01 2024-03-22T02:15:00.691 INFO:teuthology.orchestra.run.smithi103.stdout:(2/24): ceph-selinux-17.2.7-726.g9c4e0c34.el9.x 505 kB/s | 25 kB 00:00 2024-03-22T02:15:00.783 INFO:teuthology.orchestra.run.smithi103.stdout:(3/24): libcephfs2-17.2.7-726.g9c4e0c34.el9.x86 6.8 MB/s | 636 kB 00:00 2024-03-22T02:15:00.849 INFO:teuthology.orchestra.run.smithi103.stdout:(4/24): ceph-base-17.2.7-726.g9c4e0c34.el9.x86_ 3.9 MB/s | 5.0 MB 00:01 2024-03-22T02:15:00.875 INFO:teuthology.orchestra.run.smithi103.stdout:(5/24): libradosstriper1-17.2.7-726.g9c4e0c34.e 4.9 MB/s | 457 kB 00:00 2024-03-22T02:15:00.917 INFO:teuthology.orchestra.run.smithi103.stdout:(6/24): python3-ceph-argparse-17.2.7-726.g9c4e0 1.1 MB/s | 45 kB 00:00 2024-03-22T02:15:00.967 INFO:teuthology.orchestra.run.smithi103.stdout:(7/24): python3-ceph-common-17.2.7-726.g9c4e0c3 2.3 MB/s | 117 kB 00:00 2024-03-22T02:15:01.034 INFO:teuthology.orchestra.run.smithi103.stdout:(8/24): python3-cephfs-17.2.7-726.g9c4e0c34.el9 2.3 MB/s | 156 kB 00:00 2024-03-22T02:15:01.109 INFO:teuthology.orchestra.run.smithi103.stdout:(9/24): python3-rados-17.2.7-726.g9c4e0c34.el9. 4.2 MB/s | 320 kB 00:00 2024-03-22T02:15:01.176 INFO:teuthology.orchestra.run.smithi103.stdout:(10/24): python3-rbd-17.2.7-726.g9c4e0c34.el9.x 4.3 MB/s | 293 kB 00:00 2024-03-22T02:15:01.234 INFO:teuthology.orchestra.run.smithi103.stdout:(11/24): python3-rgw-17.2.7-726.g9c4e0c34.el9.x 1.7 MB/s | 99 kB 00:00 2024-03-22T02:15:01.360 INFO:teuthology.orchestra.run.smithi103.stdout:(12/24): librgw2-17.2.7-726.g9c4e0c34.el9.x86_6 9.6 MB/s | 4.9 MB 00:00 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi103.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 865 kB/s | 159 kB 00:00 2024-03-22T02:15:01.444 INFO:teuthology.orchestra.run.smithi103.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 529 kB/s | 44 kB 00:00 2024-03-22T02:15:01.586 INFO:teuthology.orchestra.run.smithi103.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 661 kB 00:00 2024-03-22T02:15:01.611 INFO:teuthology.orchestra.run.smithi103.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-22T02:15:01.636 INFO:teuthology.orchestra.run.smithi103.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-22T02:15:01.804 INFO:teuthology.orchestra.run.smithi103.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-03-22T02:15:02.054 INFO:teuthology.orchestra.run.smithi103.stdout:(19/24): ceph-common-17.2.7-726.g9c4e0c34.el9.x 9.0 MB/s | 23 MB 00:02 2024-03-22T02:15:02.079 INFO:teuthology.orchestra.run.smithi103.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 661 kB/s | 309 kB 00:00 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi103.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 224 kB/s | 67 kB 00:00 2024-03-22T02:15:02.163 INFO:teuthology.orchestra.run.smithi103.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-22T02:15:02.363 INFO:teuthology.orchestra.run.smithi103.stdout:(23/24): librbd1-17.2.7-726.g9c4e0c34.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-03-22T02:15:02.480 INFO:teuthology.orchestra.run.smithi103.stdout:(24/24): librados2-17.2.7-726.g9c4e0c34.el9.x86 8.2 MB/s | 3.1 MB 00:00 2024-03-22T02:15:02.485 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:02.485 INFO:teuthology.orchestra.run.smithi103.stdout:Total 19 MB/s | 58 MB 00:03 2024-03-22T02:15:02.485 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:15:02.517 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:15:02.517 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:15:02.970 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:15:02.970 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:15:04.214 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:15:04.543 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-22T02:15:04.599 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:15:04.690 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:15:04.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-22T02:15:04.800 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:15:04.872 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:15:04.931 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-22T02:15:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-22T02:15:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-22T02:15:05.495 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-22T02:15:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-22T02:15:05.578 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-22T02:15:05.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 10/26 2024-03-22T02:15:05.649 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-22T02:15:05.711 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-22T02:15:05.739 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:15:05.788 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:15:05.828 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-22T02:15:05.864 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-22T02:15:05.888 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-22T02:15:06.192 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-22T02:15:06.218 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-22T02:15:06.291 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-22T02:15:06.364 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 19/26 2024-03-22T02:15:06.387 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 20/26 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-03-22T02:15:07.877 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:08.777 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:15:08.817 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:15:09.191 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:15:09.253 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:09.255 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-22T02:15:09.344 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-22T02:15:09.352 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-22T02:15:09.352 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:09.378 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-22T02:15:09.455 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-22T02:15:09.455 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T02:15:09.455 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T02:15:09.455 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:09.496 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:15:10.595 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:10.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/26 2024-03-22T02:15:10.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:15:10.595 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 7/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 8/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 9/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 10/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:15:10.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-22T02:15:10.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 25/26 2024-03-22T02:15:11.484 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T02:15:11.485 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T02:15:11.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:11.487 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:11.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-03-22T02:15:12.397 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:27 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:12.544 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 48 M 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:15:12.545 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 163 k 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-03-22T02:15:12.546 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:12.547 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 49 M 2024-03-22T02:15:12.547 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 196 M 2024-03-22T02:15:12.547 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:12.981 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-22T02:15:13.057 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 300 kB 00:00 2024-03-22T02:15:13.082 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): libcephsqlite-17.2.7-726.g9c4e0c34.el9.x 623 kB/s | 163 kB 00:00 2024-03-22T02:15:13.107 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-22T02:15:13.307 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-22T02:15:16.832 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el9.x86_6 12 MB/s | 48 MB 00:04 2024-03-22T02:15:16.835 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:16.835 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 49 MB 00:04 2024-03-22T02:15:16.849 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:16.870 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:16.870 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:17.248 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:17.248 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:17.834 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:17.907 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-22T02:15:17.943 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-22T02:15:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-22T02:15:18.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-22T02:15:18.046 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-03-22T02:15:19.117 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:20.418 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-22T02:15:20.496 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-22T02:15:20.496 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:20.497 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T02:15:20.497 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:15:20.497 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:15:20.497 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:20.498 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-22T02:15:20.580 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-22T02:15:20.580 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:21.863 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:21.863 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 7/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 8/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 9/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 10/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-22T02:15:21.864 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-22T02:15:21.865 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 25/26 2024-03-22T02:15:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-22T02:15:22.552 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.854 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.855 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:22.856 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:15:23.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-03-22T02:15:23.714 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:26 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:15:23.866 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:15:23.867 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:15:23.867 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:15:23.867 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:15:23.867 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 48 M 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 163 k 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-03-22T02:15:23.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:23.869 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 49 M 2024-03-22T02:15:23.869 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 196 M 2024-03-22T02:15:23.869 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:15:24.112 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-22T02:15:24.163 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-22T02:15:24.204 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-22T02:15:24.638 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): libcephsqlite-17.2.7-726.g9c4e0c34.el9.x 233 kB/s | 163 kB 00:00 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/6 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/6 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-22T02:15:24.896 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-22T02:15:25.155 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 197 kB/s | 187 kB 00:00 2024-03-22T02:15:25.298 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-16.el9.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:25.299 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:25.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-03-22T02:15:26.187 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:41 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:26.356 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 6.5 k 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.1 M 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 1.4 M 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 240 k 2024-03-22T02:15:26.359 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.7 M 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 15 M 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T02:15:26.360 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T02:15:26.361 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout:Install 33 Packages 2024-03-22T02:15:26.362 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:26.363 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 27 M 2024-03-22T02:15:26.363 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 100 M 2024-03-22T02:15:26.363 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:27.003 INFO:teuthology.orchestra.run.smithi122.stdout:(1/33): ceph-17.2.7-726.g9c4e0c34.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-03-22T02:15:27.245 INFO:teuthology.orchestra.run.smithi122.stdout:(2/33): ceph-mds-17.2.7-726.g9c4e0c34.el9.x86_6 5.1 MB/s | 2.1 MB 00:00 2024-03-22T02:15:28.104 INFO:teuthology.orchestra.run.smithi122.stdout:(3/33): ceph-mgr-17.2.7-726.g9c4e0c34.el9.x86_6 1.1 MB/s | 1.4 MB 00:01 2024-03-22T02:15:28.254 INFO:teuthology.orchestra.run.smithi122.stdout:(4/33): ceph-osd-17.2.7-726.g9c4e0c34.el9.x86_6 15 MB/s | 15 MB 00:01 2024-03-22T02:15:28.279 INFO:teuthology.orchestra.run.smithi122.stdout:(5/33): ceph-mgr-modules-core-17.2.7-726.g9c4e0 1.3 MB/s | 240 kB 00:00 2024-03-22T02:15:28.363 INFO:teuthology.orchestra.run.smithi122.stdout:(6/33): ceph-mon-17.2.7-726.g9c4e0c34.el9.x86_6 2.7 MB/s | 3.7 MB 00:01 2024-03-22T02:15:28.480 INFO:teuthology.orchestra.run.smithi122.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 158 kB/s | 32 kB 00:00 2024-03-22T02:15:28.513 INFO:teuthology.orchestra.run.smithi122.stdout:(8/33): python3-mako-1.1.4-6.el9.noarch.rpm 557 kB/s | 144 kB 00:00 2024-03-22T02:15:28.538 INFO:teuthology.orchestra.run.smithi122.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 312 kB/s | 47 kB 00:00 2024-03-22T02:15:28.564 INFO:teuthology.orchestra.run.smithi122.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 462 kB/s | 39 kB 00:00 2024-03-22T02:15:28.722 INFO:teuthology.orchestra.run.smithi122.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 241 kB 00:00 2024-03-22T02:15:28.998 INFO:teuthology.orchestra.run.smithi122.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 237 kB/s | 103 kB 00:00 2024-03-22T02:15:29.039 INFO:teuthology.orchestra.run.smithi122.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 2.3 MB/s | 1.1 MB 00:00 2024-03-22T02:15:29.065 INFO:teuthology.orchestra.run.smithi122.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 362 kB/s | 124 kB 00:00 2024-03-22T02:15:29.090 INFO:teuthology.orchestra.run.smithi122.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 187 kB 00:00 2024-03-22T02:15:29.190 INFO:teuthology.orchestra.run.smithi122.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 831 kB/s | 125 kB 00:00 2024-03-22T02:15:29.249 INFO:teuthology.orchestra.run.smithi122.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 237 kB/s | 43 kB 00:00 2024-03-22T02:15:29.341 INFO:teuthology.orchestra.run.smithi122.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-22T02:15:29.374 INFO:teuthology.orchestra.run.smithi122.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 604 kB/s | 172 kB 00:00 2024-03-22T02:15:29.425 INFO:teuthology.orchestra.run.smithi122.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-22T02:15:29.458 INFO:teuthology.orchestra.run.smithi122.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-22T02:15:29.508 INFO:teuthology.orchestra.run.smithi122.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-22T02:15:29.534 INFO:teuthology.orchestra.run.smithi122.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-22T02:15:29.609 INFO:teuthology.orchestra.run.smithi122.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-03-22T02:15:29.634 INFO:teuthology.orchestra.run.smithi122.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2024-03-22T02:15:29.660 INFO:teuthology.orchestra.run.smithi122.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 764 kB/s | 358 kB 00:00 2024-03-22T02:15:29.693 INFO:teuthology.orchestra.run.smithi122.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-22T02:15:29.727 INFO:teuthology.orchestra.run.smithi122.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-22T02:15:29.761 INFO:teuthology.orchestra.run.smithi122.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-22T02:15:29.786 INFO:teuthology.orchestra.run.smithi122.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-22T02:15:29.811 INFO:teuthology.orchestra.run.smithi122.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-22T02:15:29.845 INFO:teuthology.orchestra.run.smithi122.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-22T02:15:29.945 INFO:teuthology.orchestra.run.smithi122.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-03-22T02:15:29.948 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:29.948 INFO:teuthology.orchestra.run.smithi122.stdout:Total 7.5 MB/s | 27 MB 00:03 2024-03-22T02:15:29.962 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:30.000 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:30.001 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:30.337 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:30.337 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:31.575 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:31.648 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-22T02:15:31.688 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-22T02:15:31.730 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-22T02:15:31.822 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-22T02:15:31.900 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-22T02:15:31.941 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-22T02:15:32.021 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-22T02:15:32.075 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-22T02:15:32.113 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-22T02:15:32.162 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-22T02:15:32.226 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-22T02:15:32.410 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-22T02:15:32.467 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-22T02:15:32.506 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-22T02:15:32.540 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-22T02:15:32.570 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-22T02:15:32.647 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-22T02:15:32.681 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-22T02:15:32.714 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-22T02:15:32.754 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-22T02:15:32.790 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-22T02:15:32.824 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-22T02:15:32.913 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-22T02:15:32.955 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-22T02:15:33.034 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-22T02:15:33.170 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-22T02:15:33.289 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-22T02:15:33.446 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 28/33 2024-03-22T02:15:33.477 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:15:33.502 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-22T02:15:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-22T02:15:34.879 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:34.879 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T02:15:34.879 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:15:34.879 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:15:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:35.270 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:15:35.293 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:35.496 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:15:35.519 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-22T02:15:37.214 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 6/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-22T02:15:37.215 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-22T02:15:37.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-22T02:15:37.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-22T02:15:37.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-22T02:15:37.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-22T02:15:37.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-22T02:15:37.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:38.220 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T02:15:38.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T02:15:38.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:38.223 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:38.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-03-22T02:15:39.143 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:54 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:39.239 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:15:39.291 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:39.292 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:15:39.292 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:39.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-03-22T02:15:39.929 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:54 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:40.077 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 82 k 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:40.078 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:15:40.079 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:40.079 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-03-22T02:15:40.079 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 372 k 2024-03-22T02:15:40.079 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:40.529 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-17.2.7-726.g9c4e0c34.el9.noarch.rpm 182 kB/s | 82 kB 00:00 2024-03-22T02:15:40.530 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:40.530 INFO:teuthology.orchestra.run.smithi122.stdout:Total 182 kB/s | 82 kB 00:00 2024-03-22T02:15:40.531 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:40.535 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:40.535 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:40.537 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:40.537 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:40.615 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:41.051 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:15:41.124 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:15:41.421 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el9.x86_6 2.7 MB/s | 48 MB 00:17 2024-03-22T02:15:41.423 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:41.424 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.8 MB/s | 49 MB 00:17 2024-03-22T02:15:41.438 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:15:41.457 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:15:41.458 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:15:41.504 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:41.772 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:41.841 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:15:41.841 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:15:41.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-03-22T02:15:42.418 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:15:42.496 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-22T02:15:42.530 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-22T02:15:42.538 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:57 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:42.562 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-22T02:15:42.605 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-22T02:15:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-22T02:15:42.690 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:42.691 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:42.691 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-22T02:15:42.691 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:42.691 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:42.691 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 137 k 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 137 k 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 379 k 2024-03-22T02:15:42.692 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:43.036 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 400 kB/s | 137 kB 00:00 2024-03-22T02:15:43.037 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:43.037 INFO:teuthology.orchestra.run.smithi122.stdout:Total 398 kB/s | 137 kB 00:00 2024-03-22T02:15:43.038 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:43.045 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:43.045 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:43.097 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:43.097 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:43.231 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:43.304 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:15:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:15:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T02:15:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:44.308 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:44.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-03-22T02:15:45.116 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:45.213 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:15:45.264 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:45.265 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:15:45.265 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:45.341 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-03-22T02:15:45.896 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:46.045 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:46.046 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:46.046 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:15:46.046 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:46.046 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:46.046 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 1.7 M 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 23 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 15 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout:Install 7 Packages 2024-03-22T02:15:46.047 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:46.048 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.0 M 2024-03-22T02:15:46.048 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 8.1 M 2024-03-22T02:15:46.048 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:46.457 INFO:teuthology.orchestra.run.smithi122.stdout:(1/7): ceph-prometheus-alerts-17.2.7-726.g9c4e0 68 kB/s | 15 kB 00:00 2024-03-22T02:15:46.482 INFO:teuthology.orchestra.run.smithi122.stdout:(2/7): ceph-grafana-dashboards-17.2.7-726.g9c4e 97 kB/s | 23 kB 00:00 2024-03-22T02:15:46.591 INFO:teuthology.orchestra.run.smithi122.stdout:(3/7): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 4.9 MB/s | 1.7 MB 00:00 2024-03-22T02:15:46.783 INFO:teuthology.orchestra.run.smithi122.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 28 kB/s | 9.0 kB 00:00 2024-03-22T02:15:46.825 INFO:teuthology.orchestra.run.smithi122.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 132 kB/s | 31 kB 00:00 2024-03-22T02:15:46.858 INFO:teuthology.orchestra.run.smithi122.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 109 kB/s | 41 kB 00:00 2024-03-22T02:15:46.976 INFO:teuthology.orchestra.run.smithi122.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 980 kB/s | 188 kB 00:00 2024-03-22T02:15:46.979 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:46.979 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-22T02:15:46.980 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:46.987 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:46.987 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:47.049 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:47.049 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:47.061 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-22T02:15:47.132 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-22T02:15:47.333 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:47.393 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-22T02:15:47.427 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-22T02:15:47.473 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-22T02:15:47.505 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-22T02:15:47.543 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 5/7 2024-03-22T02:15:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 6/7 2024-03-22T02:15:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-22T02:15:48.361 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-22T02:15:48.361 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 1/7 2024-03-22T02:15:48.362 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2/7 2024-03-22T02:15:48.362 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 3/7 2024-03-22T02:15:48.362 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-22T02:15:48.362 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-22T02:15:48.362 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-22T02:15:48.678 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:48.679 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:15:48.919 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-22T02:15:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-22T02:15:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/6 2024-03-22T02:15:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/6 2024-03-22T02:15:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-22T02:15:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-22T02:15:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-22T02:15:49.474 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:04 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:15:49.623 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:15:49.624 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-03-22T02:15:49.624 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:15:49.624 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 7.4 M 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T02:15:49.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 37 M 2024-03-22T02:15:49.626 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 209 M 2024-03-22T02:15:49.627 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-16.el9.x86_64 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T02:15:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T02:15:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:49.789 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:15:50.104 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-03-22T02:15:50.216 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 187 kB/s | 32 kB 00:00 2024-03-22T02:15:50.258 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 417 kB/s | 17 kB 00:00 2024-03-22T02:15:50.300 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-22T02:15:50.367 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.3 MB/s | 3.0 MB 00:00 2024-03-22T02:15:50.609 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-22T02:15:50.673 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:53 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:15:50.675 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-22T02:15:50.701 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-22T02:15:50.801 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-72 9.8 MB/s | 7.4 MB 00:00 2024-03-22T02:15:50.834 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:15:50.836 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-22T02:15:50.836 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:15:50.836 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-22T02:15:50.836 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 6.5 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.1 M 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 1.4 M 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 240 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.7 M 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 15 M 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T02:15:50.837 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T02:15:50.838 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T02:15:50.839 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout:Install 33 Packages 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 27 M 2024-03-22T02:15:50.840 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 100 M 2024-03-22T02:15:50.841 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:15:51.135 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-22T02:15:51.177 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 524 kB/s | 195 kB 00:00 2024-03-22T02:15:51.227 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-22T02:15:51.241 INFO:teuthology.orchestra.run.smithi103.stdout:(1/33): ceph-17.2.7-726.g9c4e0c34.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-03-22T02:15:51.616 INFO:teuthology.orchestra.run.smithi103.stdout:(2/33): ceph-mon-17.2.7-726.g9c4e0c34.el9.x86_6 9.9 MB/s | 3.7 MB 00:00 2024-03-22T02:15:51.667 INFO:teuthology.orchestra.run.smithi103.stdout:(3/33): ceph-mds-17.2.7-726.g9c4e0c34.el9.x86_6 3.5 MB/s | 2.1 MB 00:00 2024-03-22T02:15:51.709 INFO:teuthology.orchestra.run.smithi103.stdout:(4/33): ceph-mgr-17.2.7-726.g9c4e0c34.el9.x86_6 2.2 MB/s | 1.4 MB 00:00 2024-03-22T02:15:51.768 INFO:teuthology.orchestra.run.smithi103.stdout:(5/33): ceph-mgr-modules-core-17.2.7-726.g9c4e0 2.3 MB/s | 240 kB 00:00 2024-03-22T02:15:51.834 INFO:teuthology.orchestra.run.smithi103.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-22T02:15:51.860 INFO:teuthology.orchestra.run.smithi103.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 348 kB/s | 32 kB 00:00 2024-03-22T02:15:51.885 INFO:teuthology.orchestra.run.smithi103.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-03-22T02:15:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-22T02:15:51.936 INFO:teuthology.orchestra.run.smithi103.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-22T02:15:51.986 INFO:teuthology.orchestra.run.smithi103.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-22T02:15:52.011 INFO:teuthology.orchestra.run.smithi103.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-22T02:15:52.037 INFO:teuthology.orchestra.run.smithi103.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-22T02:15:52.062 INFO:teuthology.orchestra.run.smithi103.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-22T02:15:52.069 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 263 kB/s | 245 kB 00:00 2024-03-22T02:15:52.074 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:52.078 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-22T02:15:52.118 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:15:52.129 INFO:teuthology.orchestra.run.smithi103.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-22T02:15:52.142 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:15:52.142 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:15:52.346 INFO:teuthology.orchestra.run.smithi103.stdout:(16/33): ceph-osd-17.2.7-726.g9c4e0c34.el9.x86_ 21 MB/s | 15 MB 00:00 2024-03-22T02:15:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-03-22T02:15:52.481 INFO:teuthology.orchestra.run.smithi103.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-22T02:15:52.505 INFO:teuthology.orchestra.run.smithi103.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 457 kB/s | 172 kB 00:00 2024-03-22T02:15:52.603 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:15:52.603 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:15:52.630 INFO:teuthology.orchestra.run.smithi103.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 117 kB/s | 18 kB 00:00 2024-03-22T02:15:52.647 INFO:teuthology.orchestra.run.smithi103.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-03-22T02:15:52.673 INFO:teuthology.orchestra.run.smithi103.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-03-22T02:15:52.781 INFO:teuthology.orchestra.run.smithi103.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-03-22T02:15:52.798 INFO:teuthology.orchestra.run.smithi103.stdout:(24/33): python3-jaraco-functools-3.5.0-2.el9.n 116 kB/s | 19 kB 00:00 2024-03-22T02:15:52.848 INFO:teuthology.orchestra.run.smithi103.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 264 kB/s | 46 kB 00:00 2024-03-22T02:15:52.991 INFO:teuthology.orchestra.run.smithi103.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 116 kB/s | 16 kB 00:00 2024-03-22T02:15:53.016 INFO:teuthology.orchestra.run.smithi103.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 337 kB/s | 79 kB 00:00 2024-03-22T02:15:53.075 INFO:teuthology.orchestra.run.smithi103.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 987 kB/s | 272 kB 00:00 2024-03-22T02:15:53.166 INFO:teuthology.orchestra.run.smithi103.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 513 kB/s | 90 kB 00:00 2024-03-22T02:15:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 194 kB/s | 36 kB 00:00 2024-03-22T02:15:53.308 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:15:53.359 INFO:teuthology.orchestra.run.smithi103.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 126 kB/s | 20 kB 00:00 2024-03-22T02:15:53.384 INFO:teuthology.orchestra.run.smithi103.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 743 kB/s | 230 kB 00:00 2024-03-22T02:15:53.393 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-22T02:15:53.423 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-22T02:15:53.517 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-22T02:15:53.526 INFO:teuthology.orchestra.run.smithi103.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.2 MB/s | 427 kB 00:00 2024-03-22T02:15:53.530 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:15:53.530 INFO:teuthology.orchestra.run.smithi103.stdout:Total 9.9 MB/s | 27 MB 00:02 2024-03-22T02:15:53.544 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:15:53.560 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-22T02:15:53.584 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:15:53.584 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:15:53.738 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-22T02:15:53.767 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-22T02:15:53.844 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-22T02:15:53.930 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:15:53.930 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:15:54.503 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-22T02:15:54.622 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-22T02:15:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:15:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-22T02:15:55.245 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-22T02:15:55.277 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-22T02:15:55.370 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-22T02:15:55.448 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-22T02:15:55.489 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-22T02:15:55.560 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-22T02:15:55.618 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-22T02:15:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-22T02:15:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-22T02:15:55.776 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-22T02:15:55.872 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-22T02:15:55.961 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-22T02:15:56.007 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-22T02:15:56.051 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-22T02:15:56.101 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-22T02:15:56.136 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-22T02:15:56.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-22T02:15:56.246 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-22T02:15:56.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-22T02:15:56.320 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-22T02:15:56.355 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-22T02:15:56.390 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-22T02:15:56.508 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-22T02:15:56.554 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-22T02:15:56.627 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-22T02:15:56.771 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-22T02:15:56.888 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-22T02:15:56.906 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-22T02:15:56.933 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-22T02:15:57.036 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 28/33 2024-03-22T02:15:57.068 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-22T02:15:57.092 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-22T02:15:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:57.092 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T02:15:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:15:57.093 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:15:57.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:58.427 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:15:58.453 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:58.828 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-22T02:15:58.850 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-22T02:15:58.850 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:58.850 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T02:15:58.850 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:15:58.850 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:15:58.851 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:15:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:15:59.142 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 1/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-22T02:15:59.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-22T02:15:59.572 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-22T02:15:59.572 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-22T02:15:59.572 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-22T02:15:59.980 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T02:15:59.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T02:15:59.982 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:15:59.982 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:00.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-03-22T02:16:00.833 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 6/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-22T02:16:00.834 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-22T02:16:00.835 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-22T02:16:00.836 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-22T02:16:00.837 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-22T02:16:00.837 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-22T02:16:00.998 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:15 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:01.148 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:01.149 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 47 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T02:16:01.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-03-22T02:16:01.151 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:01.152 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.0 M 2024-03-22T02:16:01.152 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-03-22T02:16:01.152 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:01.659 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 24 kB 00:00 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:01.680 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T02:16:01.681 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T02:16:01.682 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T02:16:01.683 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:01.683 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:01.684 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-22T02:16:01.735 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-22T02:16:01.760 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el9. 198 kB/s | 47 kB 00:00 2024-03-22T02:16:01.785 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-22T02:16:01.810 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-22T02:16:01.835 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-22T02:16:01.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-03-22T02:16:02.261 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2024-03-22T02:16:02.286 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 64 kB/s | 32 kB 00:00 2024-03-22T02:16:02.478 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-22T02:16:02.554 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 282 kB/s | 203 kB 00:00 2024-03-22T02:16:02.563 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:05 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:02.659 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:02.670 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 468 kB/s | 90 kB 00:00 2024-03-22T02:16:02.710 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:02.711 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:16:02.711 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:02.773 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-03-22T02:16:03.354 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:06 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:03.500 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:03.501 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:16:03.501 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:03.501 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 82 k 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 82 k 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 372 k 2024-03-22T02:16:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:03.630 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 763 kB/s | 1.0 MB 00:01 2024-03-22T02:16:03.635 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:03.636 INFO:teuthology.orchestra.run.smithi122.stdout:Total 843 kB/s | 2.0 MB 00:02 2024-03-22T02:16:03.636 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:03.650 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:03.651 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:03.703 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-17.2.7-726.g9c4e0c34.el9.noarch.rpm 408 kB/s | 82 kB 00:00 2024-03-22T02:16:03.703 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:03.703 INFO:teuthology.orchestra.run.smithi103.stdout:Total 406 kB/s | 82 kB 00:00 2024-03-22T02:16:03.704 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:16:03.709 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:16:03.709 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:16:03.710 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:16:03.711 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:16:03.745 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:03.745 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:03.784 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:16:04.234 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:16:04.264 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:04.326 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:16:04.427 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-22T02:16:04.489 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-22T02:16:04.543 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-22T02:16:04.581 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-22T02:16:04.622 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-22T02:16:04.705 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:16:04.725 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-22T02:16:04.841 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-22T02:16:04.891 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-22T02:16:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-22T02:16:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:04.982 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:04.983 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:05.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-03-22T02:16:05.619 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-22T02:16:05.671 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-22T02:16:05.704 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-22T02:16:05.751 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-22T02:16:05.775 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-22T02:16:05.778 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:08 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:05.929 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 137 k 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:05.930 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:16:05.931 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:05.931 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 137 k 2024-03-22T02:16:05.931 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 379 k 2024-03-22T02:16:05.931 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:06.292 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 381 kB/s | 137 kB 00:00 2024-03-22T02:16:06.292 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:06.293 INFO:teuthology.orchestra.run.smithi103.stdout:Total 379 kB/s | 137 kB 00:00 2024-03-22T02:16:06.293 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:16:06.300 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:16:06.300 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:16:06.355 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:16:06.355 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:16:06.500 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:16:06.551 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:16:06.575 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:16:06.575 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:16:06.575 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T02:16:06.575 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 1/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-22T02:16:06.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-22T02:16:06.597 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-22T02:16:06.944 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-22T02:16:06.944 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:06.944 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T02:16:06.945 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:06.946 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:07.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-03-22T02:16:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-22T02:16:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:07.479 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:07.479 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:07.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-03-22T02:16:07.775 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:22 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 123 k 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 6.8 M 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:08.264 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-22T02:16:08.281 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:11 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:08.378 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:08.431 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:08.432 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:16:08.432 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:08.472 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-22T02:16:08.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-03-22T02:16:08.573 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 282 kB/s | 123 kB 00:00 2024-03-22T02:16:09.058 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:12 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:09.207 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 1.7 M 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 23 k 2024-03-22T02:16:09.208 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 15 k 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout:Install 7 Packages 2024-03-22T02:16:09.209 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:09.210 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2024-03-22T02:16:09.210 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 8.1 M 2024-03-22T02:16:09.210 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:09.564 INFO:teuthology.orchestra.run.smithi103.stdout:(1/7): ceph-grafana-dashboards-17.2.7-726.g9c4e 118 kB/s | 23 kB 00:00 2024-03-22T02:16:09.589 INFO:teuthology.orchestra.run.smithi103.stdout:(2/7): ceph-prometheus-alerts-17.2.7-726.g9c4e0 66 kB/s | 15 kB 00:00 2024-03-22T02:16:09.656 INFO:teuthology.orchestra.run.smithi103.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2024-03-22T02:16:09.681 INFO:teuthology.orchestra.run.smithi103.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-03-22T02:16:09.706 INFO:teuthology.orchestra.run.smithi103.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 611 kB/s | 31 kB 00:00 2024-03-22T02:16:09.765 INFO:teuthology.orchestra.run.smithi103.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-22T02:16:09.807 INFO:teuthology.orchestra.run.smithi103.stdout:(7/7): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 3.8 MB/s | 1.7 MB 00:00 2024-03-22T02:16:09.808 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:09.809 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.3 MB/s | 2.0 MB 00:00 2024-03-22T02:16:09.809 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:16:09.816 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:16:09.816 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:16:09.874 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:16:09.874 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:16:10.178 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:16:10.241 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-22T02:16:10.275 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-22T02:16:10.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-22T02:16:10.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-22T02:16:10.391 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 5/7 2024-03-22T02:16:10.620 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 6/7 2024-03-22T02:16:10.653 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-22T02:16:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-22T02:16:11.175 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 1/7 2024-03-22T02:16:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2/7 2024-03-22T02:16:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 3/7 2024-03-22T02:16:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-22T02:16:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-22T02:16:11.176 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-22T02:16:11.471 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-22T02:16:11.471 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:11.471 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:11.471 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:11.472 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:11.606 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 164 kB/s | 548 kB 00:03 2024-03-22T02:16:11.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-22T02:16:11.940 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 17 kB/s | 58 kB 00:03 2024-03-22T02:16:12.073 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 15 kB/s | 51 kB 00:03 2024-03-22T02:16:12.076 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:12.076 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.6 MB/s | 6.8 MB 00:04 2024-03-22T02:16:12.091 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:12.102 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:12.102 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:12.193 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:12.194 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:12.260 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:15 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:12.409 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:12.410 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-03-22T02:16:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:16:12.410 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-03-22T02:16:12.410 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 7.4 M 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T02:16:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 37 M 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-03-22T02:16:12.412 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:12.498 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:12.679 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-22T02:16:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-22T02:16:12.881 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-03-22T02:16:12.932 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 343 kB/s | 17 kB 00:00 2024-03-22T02:16:13.057 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 274 kB/s | 34 kB 00:00 2024-03-22T02:16:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-22T02:16:13.307 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-22T02:16:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-22T02:16:13.374 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-22T02:16:13.408 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-22T02:16:13.437 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-22T02:16:13.466 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-72 10 MB/s | 7.4 MB 00:00 2024-03-22T02:16:13.591 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-22T02:16:13.617 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 904 kB/s | 217 kB 00:00 2024-03-22T02:16:14.092 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-22T02:16:14.117 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 392 kB/s | 195 kB 00:00 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/6 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-22T02:16:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-22T02:16:14.451 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 938 kB/s | 803 kB 00:00 2024-03-22T02:16:14.484 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 626 kB/s | 245 kB 00:00 2024-03-22T02:16:14.486 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:14.487 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:14.492 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:14.528 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:16:14.553 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:16:14.553 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:16:14.750 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-03-22T02:16:14.978 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:16:14.978 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:16:15.311 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:30 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:15.460 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:15.460 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:15.460 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-22T02:16:15.460 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 812 k 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-03-22T02:16:15.461 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:15.462 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 891 k 2024-03-22T02:16:15.462 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.6 M 2024-03-22T02:16:15.462 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:15.592 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-03-22T02:16:15.677 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:16:15.762 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-22T02:16:15.814 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-22T02:16:15.825 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el9.x86_6 2.2 MB/s | 812 kB 00:00 2024-03-22T02:16:15.826 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:15.827 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.4 MB/s | 891 kB 00:00 2024-03-22T02:16:15.827 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:15.846 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:15.847 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:15.907 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-22T02:16:15.918 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:15.919 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:15.951 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-22T02:16:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:16.128 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-22T02:16:16.158 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-22T02:16:16.234 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-22T02:16:16.237 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-22T02:16:16.308 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-22T02:16:16.903 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-22T02:16:16.994 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-22T02:16:16.994 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/2 2024-03-22T02:16:17.013 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-22T02:16:17.275 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-22T02:16:17.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:17.276 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:17.276 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-22T02:16:17.276 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:17.276 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:17.570 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-03-22T02:16:18.148 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:33 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-22T02:16:18.297 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 126 k 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:18.298 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 126 k 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 448 k 2024-03-22T02:16:18.299 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:18.530 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-17.2.7-726.g9c4e0c34.el9.x86_64. 547 kB/s | 126 kB 00:00 2024-03-22T02:16:18.531 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:18.531 INFO:teuthology.orchestra.run.smithi122.stdout:Total 544 kB/s | 126 kB 00:00 2024-03-22T02:16:18.531 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:18.536 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:18.537 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:18.591 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:18.591 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:18.734 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:18.844 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-22T02:16:19.334 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-22T02:16:19.570 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:19.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:19.570 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:19.570 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:19.570 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:19.571 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:19.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-03-22T02:16:20.457 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:35 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:20.556 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:20.610 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:20.611 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:20.611 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:20.687 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-03-22T02:16:21.246 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:36 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-22T02:16:21.394 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 25 k 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 110 k 2024-03-22T02:16:21.395 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:21.642 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el9.x86_64 101 kB/s | 25 kB 00:00 2024-03-22T02:16:21.643 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:21.643 INFO:teuthology.orchestra.run.smithi122.stdout:Total 100 kB/s | 25 kB 00:00 2024-03-22T02:16:21.643 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:21.647 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:21.647 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:21.649 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:21.649 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:21.738 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:21.893 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-22T02:16:21.893 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 1/12 2024-03-22T02:16:21.893 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-22T02:16:21.893 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-22T02:16:21.894 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-22T02:16:22.212 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:22.380 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T02:16:22.381 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T02:16:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T02:16:22.382 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:22.382 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:22.473 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:22.685 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-03-22T02:16:22.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-03-22T02:16:23.245 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:38 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:23.341 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:26 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:23.393 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:23.394 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:23.394 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:23.467 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-03-22T02:16:23.504 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 47 k 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T02:16:23.505 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:23.506 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:23.507 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-03-22T02:16:23.507 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:23.507 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2024-03-22T02:16:23.507 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-03-22T02:16:23.508 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:23.703 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 154 kB/s | 18 kB 00:00 2024-03-22T02:16:23.908 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 182 kB/s | 24 kB 00:00 2024-03-22T02:16:23.908 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-22T02:16:23.908 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-22T02:16:23.909 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-22T02:16:23.909 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-22T02:16:23.909 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el9. 153 kB/s | 47 kB 00:00 2024-03-22T02:16:24.026 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:39 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:24.072 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-03-22T02:16:24.088 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-22T02:16:24.123 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:24.175 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:24.176 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:24.176 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:24.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-03-22T02:16:24.247 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-22T02:16:24.356 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 441 kB/s | 203 kB 00:00 2024-03-22T02:16:24.414 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-03-22T02:16:24.456 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-22T02:16:24.458 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:24.458 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-22T02:16:24.458 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:16:24.471 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:16:24.471 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:16:24.561 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:16:24.562 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:16:24.793 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:39 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:24.890 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:24.941 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:24.942 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:24.943 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:25.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-03-22T02:16:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:16:25.234 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-22T02:16:25.290 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-22T02:16:25.353 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-22T02:16:25.400 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-22T02:16:25.441 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-22T02:16:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-22T02:16:25.574 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:40 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:25.643 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-22T02:16:25.670 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:25.701 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-22T02:16:25.723 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:25.724 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:25.724 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:25.791 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-03-22T02:16:26.352 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:41 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:26.410 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-22T02:16:26.448 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:26.466 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-22T02:16:26.499 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-22T02:16:26.501 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:26.502 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:26.502 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:26.545 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-22T02:16:26.570 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-22T02:16:26.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-03-22T02:16:27.134 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:27.231 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:16:27.282 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:27.283 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-22T02:16:27.284 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:27.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-03-22T02:16:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-22T02:16:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 1/13 2024-03-22T02:16:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-22T02:16:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T02:16:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:27.722 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:16:27.932 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:27.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 82 k 2024-03-22T02:16:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 222 k 2024-03-22T02:16:28.095 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:28.298 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 397 kB/s | 82 kB 00:00 2024-03-22T02:16:28.299 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:28.299 INFO:teuthology.orchestra.run.smithi122.stdout:Total 394 kB/s | 82 kB 00:00 2024-03-22T02:16:28.299 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:28.307 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:28.307 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:28.363 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:28.363 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:28.499 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:31 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:16:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:28.649 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:16:28.650 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:28.650 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:16:28.650 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:28.650 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 123 k 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-03-22T02:16:28.651 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:16:28.652 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 6.8 M 2024-03-22T02:16:28.652 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-03-22T02:16:28.652 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:16:29.027 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:29.306 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:29.310 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-22T02:16:29.376 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 280 kB/s | 123 kB 00:00 2024-03-22T02:16:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-03-22T02:16:29.594 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 762 kB/s | 51 kB 00:00 2024-03-22T02:16:29.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-03-22T02:16:30.166 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:30.321 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:30.321 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.9 M 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 12 M 2024-03-22T02:16:30.322 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:30.353 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 175 kB/s | 248 kB 00:01 2024-03-22T02:16:31.418 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 2.7 MB/s | 2.9 MB 00:01 2024-03-22T02:16:31.419 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:31.419 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.7 MB/s | 2.9 MB 00:01 2024-03-22T02:16:31.419 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:31.428 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:31.428 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:31.499 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:31.499 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:31.903 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:31.960 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:31.985 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:31.985 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:16:31.985 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T02:16:31.985 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:16:31.986 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:16:31.986 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:33.108 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:33.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-03-22T02:16:33.900 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:48 ago on Fri 22 Mar 2024 02:14:45 AM UTC. 2024-03-22T02:16:34.054 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-22T02:16:34.055 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:34.055 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-22T02:16:34.055 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:34.055 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 165 k 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 165 k 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 477 k 2024-03-22T02:16:34.056 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-22T02:16:34.413 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 462 kB/s | 165 kB 00:00 2024-03-22T02:16:34.414 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:16:34.414 INFO:teuthology.orchestra.run.smithi122.stdout:Total 461 kB/s | 165 kB 00:00 2024-03-22T02:16:34.414 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-22T02:16:34.421 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-22T02:16:34.422 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-22T02:16:34.476 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-22T02:16:34.476 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-22T02:16:34.635 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-22T02:16:34.728 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:35.437 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:16:35.438 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:35.438 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-22T02:16:35.438 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:16:35.438 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:16:35.438 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-22T02:16:35.675 DEBUG:teuthology.parallel:result is None 2024-03-22T02:17:33.599 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.5 kB/s | 548 kB 01:04 2024-03-22T02:17:33.600 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:33.600 INFO:teuthology.orchestra.run.smithi103.stdout:Total 107 kB/s | 6.8 MB 01:04 2024-03-22T02:17:33.610 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:33.622 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:33.622 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:33.712 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:33.712 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:34.024 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:34.185 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-22T02:17:34.238 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-22T02:17:34.746 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-22T02:17:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-22T02:17:34.897 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-22T02:17:34.926 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-22T02:17:35.675 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-22T02:17:35.978 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:35.979 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:36.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-03-22T02:17:36.822 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:39 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:36.975 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:36.975 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-22T02:17:36.975 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:36.975 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 812 k 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-03-22T02:17:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:36.977 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 891 k 2024-03-22T02:17:36.977 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.6 M 2024-03-22T02:17:36.977 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:37.101 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 632 kB/s | 79 kB 00:00 2024-03-22T02:17:37.443 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el9.x86_6 1.7 MB/s | 812 kB 00:00 2024-03-22T02:17:37.444 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:37.445 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.9 MB/s | 891 kB 00:00 2024-03-22T02:17:37.445 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:37.466 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:37.466 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:37.539 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:37.539 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:37.713 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:37.855 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-22T02:17:37.926 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-22T02:17:38.601 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-22T02:17:38.601 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/2 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:38.891 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:39.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-03-22T02:17:39.747 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:42 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:39.896 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 126 k 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 126 k 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 448 k 2024-03-22T02:17:39.897 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-17.2.7-726.g9c4e0c34.el9.x86_64. 265 kB/s | 126 kB 00:00 2024-03-22T02:17:40.375 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:40.375 INFO:teuthology.orchestra.run.smithi103.stdout:Total 264 kB/s | 126 kB 00:00 2024-03-22T02:17:40.375 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:40.380 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:40.381 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:40.435 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:40.435 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:40.576 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:40.672 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:41.108 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:41.409 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:41.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-03-22T02:17:42.268 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:45 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:42.367 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:42.420 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:42.421 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:42.421 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:42.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-03-22T02:17:43.050 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:46 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:43.202 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:43.203 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:43.204 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:43.205 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:17:43.205 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:43.205 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 25 k 2024-03-22T02:17:43.205 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 110 k 2024-03-22T02:17:43.205 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:43.454 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el9.x86_64 99 kB/s | 25 kB 00:00 2024-03-22T02:17:43.455 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:43.456 INFO:teuthology.orchestra.run.smithi103.stdout:Total 99 kB/s | 25 kB 00:00 2024-03-22T02:17:43.456 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:43.459 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:43.459 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:43.460 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:43.461 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:43.661 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:44.044 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:44.291 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:44.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:44.291 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:44.292 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:17:44.292 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:44.292 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:44.472 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-03-22T02:17:45.036 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:48 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:45.133 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:45.186 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:45.187 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:45.187 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:45.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-03-22T02:17:45.814 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:48 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:45.911 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:45.963 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:45.964 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:45.964 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:46.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-03-22T02:17:46.597 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:49 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:46.694 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:46.746 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:46.747 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:46.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-03-22T02:17:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:50 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:47.471 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:47.524 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:47.525 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:47.525 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:47.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-03-22T02:17:48.154 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:51 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:48.302 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:48.303 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:48.303 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:48.379 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-03-22T02:17:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:51 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:49.033 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-22T02:17:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:49.085 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-22T02:17:49.086 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:49.158 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-03-22T02:17:49.714 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:52 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:49.862 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 82 k 2024-03-22T02:17:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 82 k 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 222 k 2024-03-22T02:17:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:50.333 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 174 kB/s | 82 kB 00:00 2024-03-22T02:17:50.334 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:50.334 INFO:teuthology.orchestra.run.smithi103.stdout:Total 174 kB/s | 82 kB 00:00 2024-03-22T02:17:50.334 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:50.340 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:50.341 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:50.395 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:50.395 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:51.020 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:51.307 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:51.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-03-22T02:17:52.137 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:55 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:52.292 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:52.293 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.9 M 2024-03-22T02:17:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 12 M 2024-03-22T02:17:52.294 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:54.178 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 1.6 MB/s | 2.9 MB 00:01 2024-03-22T02:17:54.179 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:54.180 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.6 MB/s | 2.9 MB 00:01 2024-03-22T02:17:54.180 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:54.190 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:54.191 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:54.265 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:54.265 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:54.671 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:17:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:55.898 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:55.899 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:55.899 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:17:55.899 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:55.899 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:56.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-03-22T02:17:56.726 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:59 ago on Fri 22 Mar 2024 02:14:57 AM UTC. 2024-03-22T02:17:56.873 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 165 k 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-22T02:17:56.874 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:56.875 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 165 k 2024-03-22T02:17:56.875 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 477 k 2024-03-22T02:17:56.875 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-22T02:17:57.233 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 461 kB/s | 165 kB 00:00 2024-03-22T02:17:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:17:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Total 459 kB/s | 165 kB 00:00 2024-03-22T02:17:57.234 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-22T02:17:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-22T02:17:57.241 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-22T02:17:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-22T02:17:57.298 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-22T02:17:57.445 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-22T02:17:57.548 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:57.968 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:17:58.249 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-22T02:17:58.529 DEBUG:teuthology.parallel:result is None 2024-03-22T02:17:58.530 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:17:58.530 INFO:teuthology.packaging:ref: None 2024-03-22T02:17:58.530 INFO:teuthology.packaging:tag: None 2024-03-22T02:17:58.530 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:17:58.530 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:17:58.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-22T02:17:58.762 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T02:17:58.783 INFO:teuthology.orchestra.run.smithi103.stdout:17.2.7-726.g9c4e0c34.el9 2024-03-22T02:17:58.783 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el9 2024-03-22T02:17:58.783 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-22T02:17:58.785 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T02:17:58.785 INFO:teuthology.packaging:ref: None 2024-03-22T02:17:58.785 INFO:teuthology.packaging:tag: None 2024-03-22T02:17:58.785 INFO:teuthology.packaging:branch: quincy 2024-03-22T02:17:58.785 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:17:58.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-22T02:17:59.016 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T02:17:59.040 INFO:teuthology.orchestra.run.smithi122.stdout:17.2.7-726.g9c4e0c34.el9 2024-03-22T02:17:59.041 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el9 2024-03-22T02:17:59.041 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-22T02:17:59.043 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-22T02:17:59.043 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:17:59.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T02:17:59.070 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:17:59.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T02:17:59.111 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-22T02:17:59.111 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:17:59.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T02:17:59.138 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T02:17:59.208 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:17:59.208 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T02:17:59.237 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T02:17:59.306 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-22T02:17:59.306 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:17:59.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T02:17:59.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T02:17:59.399 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:17:59.399 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T02:17:59.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T02:17:59.496 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-22T02:17:59.497 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:17:59.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T02:17:59.523 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T02:17:59.590 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:17:59.590 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T02:17:59.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T02:17:59.685 INFO:teuthology.run_tasks:Running task print... 2024-03-22T02:17:59.695 INFO:teuthology.task.print:**** done install task... 2024-03-22T02:17:59.695 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-22T02:17:59.787 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'} 2024-03-22T02:17:59.787 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-22T02:17:59.787 INFO:tasks.cephadm:Cluster fsid is 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:17:59.788 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-22T02:17:59.788 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-22T02:17:59.788 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi103': '172.21.15.103', 'mon.smithi122': '172.21.15.122'} 2024-03-22T02:17:59.788 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-22T02:17:59.788 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-03-22T02:17:59.815 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-03-22T02:17:59.845 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-22T02:17:59.845 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.076 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 22 02:18 /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.076 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.312 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 22 02:18 /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.313 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.328 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T02:18:00.353 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-22T02:18:00.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-22T02:18:00.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-22T02:18:01.195 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-22T02:18:01.314 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-22T02:18:28.344 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-22T02:18:32.612 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-03-22T02:18:32.613 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:18:32.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-03-22T02:18:32.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-03-22T02:18:32.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-03-22T02:18:32.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-03-22T02:18:32.790 INFO:tasks.cephadm:Writing seed config... 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-22T02:18:32.791 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-22T02:18:32.792 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-22T02:18:32.793 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-22T02:18:32.794 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-22T02:18:32.795 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:18:32.795 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-22T02:18:32.809 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 67659b4a-e7f2-11ee-95cd-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-22T02:18:32.810 DEBUG:teuthology.orchestra.run.smithi103:mon.smithi103> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service 2024-03-22T02:18:32.852 INFO:tasks.cephadm:Bootstrapping... 2024-03-22T02:18:32.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.103 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:18:33.056 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:18:33.057 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '67659b4a-e7f2-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.103', '--skip-admin-label'] 2024-03-22T02:18:33.081 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T02:18:33.081 INFO:teuthology.orchestra.run.smithi103.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-22T02:18:33.082 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying podman|docker is present... 2024-03-22T02:18:33.105 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T02:18:33.105 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying lvm2 is present... 2024-03-22T02:18:33.105 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying time synchronization is in place... 2024-03-22T02:18:33.113 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T02:18:33.113 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T02:18:33.119 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T02:18:33.119 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:18:33.126 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-03-22T02:18:33.132 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-03-22T02:18:33.132 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-03-22T02:18:33.132 INFO:teuthology.orchestra.run.smithi103.stdout:Repeating the final host check... 2024-03-22T02:18:33.157 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T02:18:33.157 INFO:teuthology.orchestra.run.smithi103.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-22T02:18:33.157 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl is present 2024-03-22T02:18:33.157 INFO:teuthology.orchestra.run.smithi103.stdout:lvcreate is present 2024-03-22T02:18:33.164 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T02:18:33.164 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T02:18:33.170 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T02:18:33.170 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:18:33.176 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:Host looks OK 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:Cluster fsid: 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:Acquiring lock 140309492888672 on /run/cephadm/67659b4a-e7f2-11ee-95cd-87774f69a715.lock 2024-03-22T02:18:33.183 INFO:teuthology.orchestra.run.smithi103.stdout:Lock 140309492888672 acquired on /run/cephadm/67659b4a-e7f2-11ee-95cd-87774f69a715.lock 2024-03-22T02:18:33.184 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 3300 ... 2024-03-22T02:18:33.184 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 6789 ... 2024-03-22T02:18:33.184 INFO:teuthology.orchestra.run.smithi103.stdout:Base mon IP(s) is [172.21.15.103:3300, 172.21.15.103:6789], mon addrv is [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-03-22T02:18:33.187 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.103 metric 100 2024-03-22T02:18:33.187 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.103 metric 100 2024-03-22T02:18:33.190 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-22T02:18:33.190 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-22T02:18:33.190 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1724sec hoplimit 64 pref medium 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-03-22T02:18:33.193 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T02:18:33.194 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-03-22T02:18:33.194 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-03-22T02:18:33.194 INFO:teuthology.orchestra.run.smithi103.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-22T02:18:33.194 INFO:teuthology.orchestra.run.smithi103.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-22T02:18:33.195 INFO:teuthology.orchestra.run.smithi103.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Getting image source signatures 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:601ec453fe24458667695dda21742a4f3ea719c31ad8aa8ec48bf585cc42a7d7 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying config sha256:837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-22T02:18:33.680 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-22T02:18:34.594 INFO:teuthology.orchestra.run.smithi103.stdout:ceph: stdout ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-22T02:18:34.594 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph version: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-22T02:18:34.594 INFO:teuthology.orchestra.run.smithi103.stdout:Extracting ceph user uid/gid from container image... 2024-03-22T02:18:35.558 INFO:teuthology.orchestra.run.smithi103.stdout:stat: stdout 167 167 2024-03-22T02:18:35.558 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial keys... 2024-03-22T02:18:36.466 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQB76vxlNTVgNhAAayIcxnLomYi6rSI1mwjFCw== 2024-03-22T02:18:37.513 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQB86vxlGjOOMBAAaAi1wpHdVWXhyY5EkG17cg== 2024-03-22T02:18:38.435 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQB96vxlZbvYMRAAF0rjUa9b7zlyTTDsJ4dorw== 2024-03-22T02:18:38.436 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial monmap... 2024-03-22T02:18:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T02:18:39.391 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool for smithi103 [v2:172.21.15.103:3300,v1:172.21.15.103:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = octopus 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: set fsid to 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:18:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mon... 2024-03-22T02:18:40.530 INFO:teuthology.orchestra.run.smithi103.stdout:create mon.smithi103 on 2024-03-22T02:18:40.731 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-22T02:18:40.873 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-22T02:18:41.022 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target. 2024-03-22T02:18:41.022 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target. 2024-03-22T02:18:41.189 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103 2024-03-22T02:18:41.190 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service: Unit ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service not loaded. 2024-03-22T02:18:41.354 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target.wants/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service → /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@.service. 2024-03-22T02:18:41.820 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:41 smithi103 podman[28864]: 2024-03-22 02:18:41.759832301 +0000 UTC m=+0.278105443 container start b2007e87afb38757a9fb72e320993aa02c71bd91c968be1684fd3351daeccac8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-22T02:18:41.869 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:18:41.869 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-03-22T02:18:41.876 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:18:41.876 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:18:41.876 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-03-22T02:18:41.876 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T02:18:41.876 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon to start... 2024-03-22T02:18:41.877 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon... 2024-03-22T02:18:42.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:41 smithi103 ceph-mon[28883]: mkfs 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:42.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:41 smithi103 ceph-mon[28883]: mon.smithi103 is new leader, mons smithi103 in quorum (ranks 0) 2024-03-22T02:18:42.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:41 smithi103 bash[28864]: b2007e87afb38757a9fb72e320993aa02c71bd91c968be1684fd3351daeccac8 2024-03-22T02:18:42.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:41 smithi103 systemd[1]: Started Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:18:42.924 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout cluster: 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout id: 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout services: 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi103 (age 0.561055s) 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout data: 2024-03-22T02:18:42.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pgs: 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc2860a2700 1 Processor -- start 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc2860a2700 1 -- start start 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc2860a2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc2860a2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2801026e0 con 0x7fc280101d90 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59274/0 (socket says 172.21.15.103:59274) 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.304+0000 7fc27f7fe700 1 -- 172.21.15.103:0/864958672 learned_addr learned my addr 172.21.15.103:0/864958672 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc27f7fe700 1 -- 172.21.15.103:0/864958672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280102820 con 0x7fc280101d90 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc27f7fe700 1 --2- 172.21.15.103:0/864958672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc27000bb80 tx=0x7fc27000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a0f3c05b4513c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc27e7fc700 1 -- 172.21.15.103:0/864958672 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc27000dd00 con 0x7fc280101d90 2024-03-22T02:18:42.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc27e7fc700 1 -- 172.21.15.103:0/864958672 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27000de60 con 0x7fc280101d90 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc27e7fc700 1 -- 172.21.15.103:0/864958672 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2700105f0 con 0x7fc280101d90 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc2860a2700 1 -- 172.21.15.103:0/864958672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 msgr2=0x7fc2801021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc2860a2700 1 --2- 172.21.15.103:0/864958672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc27000bb80 tx=0x7fc27000b4f0 comp rx=0 tx=0).stop 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc2860a2700 1 -- 172.21.15.103:0/864958672 shutdown_connections 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc2860a2700 1 --2- 172.21.15.103:0/864958672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc2801021a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.305+0000 7fc2860a2700 1 -- 172.21.15.103:0/864958672 >> 172.21.15.103:0/864958672 conn(0x7fc2800fc700 msgr2=0x7fc2800feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 -- 172.21.15.103:0/864958672 shutdown_connections 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 -- 172.21.15.103:0/864958672 wait complete. 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 Processor -- start 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 -- start start 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc2860a2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28010f210 con 0x7fc280101d90 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc27f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59276/0 (socket says 172.21.15.103:59276) 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.306+0000 7fc27f7fe700 1 -- 172.21.15.103:0/3287233594 learned_addr learned my addr 172.21.15.103:0/3287233594 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:42.932 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc27f7fe700 1 -- 172.21.15.103:0/3287233594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27000b790 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc27f7fe700 1 --2- 172.21.15.103:0/3287233594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc27000b530 tx=0x7fc27001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2700189b0 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27001d070 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28010f410 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc270018cb0 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28010ff80 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc27002e460 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.307+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc270014070 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.308+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28005f5f0 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.310+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc27001f050 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.378+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc280110a50 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.378+0000 7fc267fff700 1 -- 172.21.15.103:0/3287233594 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fc270018e60 con 0x7fc280101d90 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 msgr2=0x7fc28010ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 --2- 172.21.15.103:0/3287233594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc27000b530 tx=0x7fc27001a000 comp rx=0 tx=0).stop 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 shutdown_connections 2024-03-22T02:18:42.933 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 --2- 172.21.15.103:0/3287233594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d90 0x7fc28010ecd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:42.934 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 >> 172.21.15.103:0/3287233594 conn(0x7fc2800fc700 msgr2=0x7fc2800fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:42.934 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 shutdown_connections 2024-03-22T02:18:42.934 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:42.379+0000 7fc2860a2700 1 -- 172.21.15.103:0/3287233594 wait complete. 2024-03-22T02:18:42.934 INFO:teuthology.orchestra.run.smithi103.stdout:mon is available 2024-03-22T02:18:42.934 INFO:teuthology.orchestra.run.smithi103.stdout:Assimilating anything we can from ceph.conf... 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: mon.smithi103 is new leader, mons smithi103 in quorum (ranks 0) 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: monmap e1: 1 mons at {smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: fsmap 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: mgrmap e1: no daemons active 2024-03-22T02:18:43.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:42 smithi103 ceph-mon[28883]: from='client.? 172.21.15.103:0/3287233594' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T02:18:44.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:43 smithi103 ceph-mon[28883]: from='client.? 172.21.15.103:0/3429121728' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T02:18:44.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:43 smithi103 ceph-mon[28883]: from='client.? 172.21.15.103:0/3429121728' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T02:18:44.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a7270700 1 Processor -- start 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a7270700 1 -- start start 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a7270700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a7270700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a0103270 con 0x7fb1a0102920 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a500c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:44.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a500c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59286/0 (socket says 172.21.15.103:59286) 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a500c700 1 -- 172.21.15.103:0/565536097 learned_addr learned my addr 172.21.15.103:0/565536097 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.378+0000 7fb1a500c700 1 -- 172.21.15.103:0/565536097 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a01033b0 con 0x7fb1a0102920 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a500c700 1 --2- 172.21.15.103:0/565536097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb190009e50 tx=0x7fb190009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d39299b52e8a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb19ffff700 1 -- 172.21.15.103:0/565536097 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1900092e0 con 0x7fb1a0102920 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb19ffff700 1 -- 172.21.15.103:0/565536097 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19000ba20 con 0x7fb1a0102920 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb19ffff700 1 -- 172.21.15.103:0/565536097 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19000bd30 con 0x7fb1a0102920 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 -- 172.21.15.103:0/565536097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 msgr2=0x7fb1a0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 --2- 172.21.15.103:0/565536097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb190009e50 tx=0x7fb190009840 comp rx=0 tx=0).stop 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 -- 172.21.15.103:0/565536097 shutdown_connections 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 --2- 172.21.15.103:0/565536097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0102d30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 -- 172.21.15.103:0/565536097 >> 172.21.15.103:0/565536097 conn(0x7fb1a00fe0a0 msgr2=0x7fb1a01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 -- 172.21.15.103:0/565536097 shutdown_connections 2024-03-22T02:18:44.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.379+0000 7fb1a7270700 1 -- 172.21.15.103:0/565536097 wait complete. 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a7270700 1 Processor -- start 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a7270700 1 -- start start 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a7270700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a7270700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a0111420 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a500c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a500c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59292/0 (socket says 172.21.15.103:59292) 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a500c700 1 -- 172.21.15.103:0/3429121728 learned_addr learned my addr 172.21.15.103:0/3429121728 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.380+0000 7fb1a500c700 1 -- 172.21.15.103:0/3429121728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190009ae0 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb1a500c700 1 --2- 172.21.15.103:0/3429121728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0103660 tx=0x7fb190017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb190012b90 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb190012d10 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb19001e400 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a0111620 con 0x7fb1a0102920 2024-03-22T02:18:44.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a0112190 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb190019030 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.381+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb19001ed00 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.382+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a005f5f0 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.384+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb190017070 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.450+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb1a0103d30 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.453+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb19001e620 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.453+0000 7fb19dffb700 1 -- 172.21.15.103:0/3429121728 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fb1a0103d30 con 0x7fb1a0102920 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.454+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 msgr2=0x7fb1a0110ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.454+0000 7fb1a7270700 1 --2- 172.21.15.103:0/3429121728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0103660 tx=0x7fb190017000 comp rx=0 tx=0).stop 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.454+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 shutdown_connections 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.454+0000 7fb1a7270700 1 --2- 172.21.15.103:0/3429121728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1a0102920 0x7fb1a0110ee0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.454+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 >> 172.21.15.103:0/3429121728 conn(0x7fb1a00fe0a0 msgr2=0x7fb1a0107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:44.085 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.455+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 shutdown_connections 2024-03-22T02:18:44.085 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:43.455+0000 7fb1a7270700 1 -- 172.21.15.103:0/3429121728 wait complete. 2024-03-22T02:18:44.085 INFO:teuthology.orchestra.run.smithi103.stdout:Generating new minimal ceph.conf... 2024-03-22T02:18:45.110 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:44 smithi103 ceph-mon[28883]: from='client.? 172.21.15.103:0/3609465199' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7be1dc8700 1 Processor -- start 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7be1dc8700 1 -- start start 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7be1dc8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7be1dc8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1020f0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49090/0 (socket says 172.21.15.103:49090) 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7bdb7fe700 1 -- 172.21.15.103:0/2999708111 learned_addr learned my addr 172.21.15.103:0/2999708111 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.491+0000 7f7bdb7fe700 1 -- 172.21.15.103:0/2999708111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc102230 con 0x7f7bdc1017a0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7bdb7fe700 1 --2- 172.21.15.103:0/2999708111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0098e0 tx=0x7f7bcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1df17eb73bcde28 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7bdaffd700 1 -- 172.21.15.103:0/2999708111 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bcc00b860 con 0x7f7bdc1017a0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7bdaffd700 1 -- 172.21.15.103:0/2999708111 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7bcc00be50 con 0x7f7bdc1017a0 2024-03-22T02:18:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7bdaffd700 1 -- 172.21.15.103:0/2999708111 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bcc010380 con 0x7f7bdc1017a0 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7be1dc8700 1 -- 172.21.15.103:0/2999708111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 msgr2=0x7f7bdc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7be1dc8700 1 --2- 172.21.15.103:0/2999708111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0098e0 tx=0x7f7bcc009250 comp rx=0 tx=0).stop 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7be1dc8700 1 -- 172.21.15.103:0/2999708111 shutdown_connections 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7be1dc8700 1 --2- 172.21.15.103:0/2999708111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc101bb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.492+0000 7f7be1dc8700 1 -- 172.21.15.103:0/2999708111 >> 172.21.15.103:0/2999708111 conn(0x7f7bdc0fcf40 msgr2=0x7f7bdc0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 -- 172.21.15.103:0/2999708111 shutdown_connections 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 -- 172.21.15.103:0/2999708111 wait complete. 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 Processor -- start 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 -- start start 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7be1dc8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1143c0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:45.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7bdb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49098/0 (socket says 172.21.15.103:49098) 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7bdb7fe700 1 -- 172.21.15.103:0/3609465199 learned_addr learned my addr 172.21.15.103:0/3609465199 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.493+0000 7f7bdb7fe700 1 -- 172.21.15.103:0/3609465199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0094f0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bdb7fe700 1 --2- 172.21.15.103:0/3609465199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00b440 tx=0x7f7bcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bcc010760 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7bcc010d50 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bcc01ae20 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc114500 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc1104a0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7bcc02e6c0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.494+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7bcc024900 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.495+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc04bed0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.497+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7bcc029070 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.563+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7bdc103570 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7bd8ff9700 1 -- 172.21.15.103:0/3609465199 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f7bcc01a6a0 con 0x7f7bdc1017a0 2024-03-22T02:18:45.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 msgr2=0x7f7bdc076b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 --2- 172.21.15.103:0/3609465199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00b440 tx=0x7f7bcc009250 comp rx=0 tx=0).stop 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 shutdown_connections 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 --2- 172.21.15.103:0/3609465199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bdc1017a0 0x7f7bdc076b20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 >> 172.21.15.103:0/3609465199 conn(0x7f7bdc0fcf40 msgr2=0x7f7bdc0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 shutdown_connections 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:44.564+0000 7f7be1dc8700 1 -- 172.21.15.103:0/3609465199 wait complete. 2024-03-22T02:18:45.120 INFO:teuthology.orchestra.run.smithi103.stdout:Restarting the monitor... 2024-03-22T02:18:45.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 systemd[1]: Stopping Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:18:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[28879]: 2024-03-22T02:18:45.388+0000 7f0ed2877700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi103 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:18:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[28879]: 2024-03-22T02:18:45.388+0000 7f0ed2877700 -1 mon.smithi103@0(leader) e1 *** Got Signal Terminated *** 2024-03-22T02:18:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 podman[29125]: 2024-03-22 02:18:45.634379637 +0000 UTC m=+0.356819270 container died b2007e87afb38757a9fb72e320993aa02c71bd91c968be1684fd3351daeccac8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5) 2024-03-22T02:18:46.155 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 podman[29125]: 2024-03-22 02:18:45.979776351 +0000 UTC m=+0.702215992 container cleanup b2007e87afb38757a9fb72e320993aa02c71bd91c968be1684fd3351daeccac8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=quincy-9c4e0c3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T02:18:46.155 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:45 smithi103 bash[29125]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103 2024-03-22T02:18:46.406 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 podman[29139]: 2024-03-22 02:18:46.155838903 +0000 UTC m=+0.511983077 container remove b2007e87afb38757a9fb72e320993aa02c71bd91c968be1684fd3351daeccac8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-9c4e0c3, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:18:46.406 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service: Deactivated successfully. 2024-03-22T02:18:46.406 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 systemd[1]: Stopped Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:18:46.406 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 systemd[1]: Starting Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:18:46.720 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 podman[29238]: 2024-03-22 02:18:46.606613967 +0000 UTC m=+0.101849306 container create 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-22T02:18:46.721 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 podman[29238]: 2024-03-22 02:18:46.522648097 +0000 UTC m=+0.017883462 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph:quincy 2024-03-22T02:18:46.888 INFO:teuthology.orchestra.run.smithi103.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 podman[29238]: 2024-03-22 02:18:46.80907299 +0000 UTC m=+0.304308328 container init 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 podman[29238]: 2024-03-22 02:18:46.812382244 +0000 UTC m=+0.307617584 container start 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3) 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable), process ceph-mon, pid 2 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: pidfile_write: ignore empty --pid-file 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: load: jerasure load: lrc 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: RocksDB version: 6.15.5 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Compile date Mar 21 2024 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: DB SUMMARY 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: DB Session ID: YT5IGRKGNOPVHMSYVSEP 2024-03-22T02:18:47.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi103/store.db dir, Total Num: 1, files: 000008.sst 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi103/store.db: 000010.log size: 98178 ; 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.env: 0x5640a52dda80 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.fs: Posix File System 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.info_log: 0x5640a7203c00 2024-03-22T02:18:47.011 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:18:47.012 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.statistics: (nil) 2024-03-22T02:18:47.012 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.use_fsync: 0 2024-03-22T02:18:47.012 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:18:47.013 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:18:47.013 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.db_log_dir: 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi103/store.db 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:18:47.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.write_buffer_manager: 0x5640a7f78a50 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:18:47.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.unordered_write: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.row_cache: None 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.wal_filter: None 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.preserve_deletes: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:18:47.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:18:47.017 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_open_files: -1 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Compression algorithms supported: 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kZSTD supported: 0 2024-03-22T02:18:47.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000009 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.merge_operator: 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_filter: None 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:18:47.019 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5640a7258f58) 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_type: 0 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_index_type: 0 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_shortening: 1 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: hash_index_allow_collision: 1 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: checksum: 1 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: no_block_cache: 0 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache: 0x5640a728b090 2024-03-22T02:18:47.020 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_options: 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: capacity : 536870912 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: num_shard_bits : 4 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: strict_capacity_limit : 0 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_compressed: (nil) 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: persistent_cache: (nil) 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size: 4096 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size_deviation: 10 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_restart_interval: 16 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_block_restart_interval: 1 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: metadata_block_size: 4096 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: partition_filters: 0 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: use_delta_encoding: 1 2024-03-22T02:18:47.021 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: whole_key_filtering: 1 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: verify_compression: 0 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: format_version: 4 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_index_compression: 1 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_align: 0 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression: NoCompression 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.num_levels: 7 2024-03-22T02:18:47.022 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:18:47.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:18:47.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.arena_block_size: 4194304 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:18:47.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.ttl: 2592000 2024-03-22T02:18:47.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073926838324, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-22T02:18:47.027 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073926839182, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93384, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91496, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84060, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711073926, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9990aa1b-a7d4-407c-ba6c-57d7353e84cf", "db_session_id": "YT5IGRKGNOPVHMSYVSEP"}} 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073926839543, "job": 1, "event": "recovery_finished"} 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi103/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5640a721f180 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: DB pointer 0x5640a72f4000 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** DB Stats ** 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-22T02:18:47.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L0 2/0 92.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Sum 2/0 92.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.029 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative compaction: 0.00 GB write, 29.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval compaction: 0.00 GB write, 29.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:18:47.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L0 2/0 92.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Sum 2/0 92.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:18:47.031 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative compaction: 0.00 GB write, 28.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: starting mon.smithi103 rank 0 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi103 fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???) e1 preinit fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).mds e1 new map 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).mds e1 print_map 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: e1 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: legacy client fscid: -1 2024-03-22T02:18:47.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: No filesystems configured 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 bash[29238]: 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 systemd[1]: Started Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mon.smithi103 is new leader, mons smithi103 in quorum (ranks 0) 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: monmap e1: 1 mons at {smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: fsmap 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T02:18:47.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:46 smithi103 ceph-mon[29258]: mgrmap e1: no daemons active 2024-03-22T02:18:48.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc218723700 1 Processor -- start 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc218723700 1 -- start start 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc218723700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc218723700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2101031b0 con 0x7fc210102860 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc2164bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc2164bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49108/0 (socket says 172.21.15.103:49108) 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.377+0000 7fc2164bf700 1 -- 172.21.15.103:0/87742975 learned_addr learned my addr 172.21.15.103:0/87742975 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.378+0000 7fc2164bf700 1 -- 172.21.15.103:0/87742975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2101032f0 con 0x7fc210102860 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.378+0000 7fc2164bf700 1 --2- 172.21.15.103:0/87742975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0098e0 tx=0x7fc20c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6d9ebbe3ec8465 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.378+0000 7fc2154bd700 1 -- 172.21.15.103:0/87742975 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc20c00b860 con 0x7fc210102860 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.378+0000 7fc2154bd700 1 -- 172.21.15.103:0/87742975 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc20c00be50 con 0x7fc210102860 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.378+0000 7fc2154bd700 1 -- 172.21.15.103:0/87742975 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc20c010380 con 0x7fc210102860 2024-03-22T02:18:48.023 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- 172.21.15.103:0/87742975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 msgr2=0x7fc210102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 --2- 172.21.15.103:0/87742975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0098e0 tx=0x7fc20c009250 comp rx=0 tx=0).stop 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- 172.21.15.103:0/87742975 shutdown_connections 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 --2- 172.21.15.103:0/87742975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210102c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- 172.21.15.103:0/87742975 >> 172.21.15.103:0/87742975 conn(0x7fc2100fe000 msgr2=0x7fc210100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- 172.21.15.103:0/87742975 shutdown_connections 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- 172.21.15.103:0/87742975 wait complete. 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 Processor -- start 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.379+0000 7fc218723700 1 -- start start 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc218723700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc218723700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc210111bd0 con 0x7fc210102860 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc2164bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc2164bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49112/0 (socket says 172.21.15.103:49112) 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc2164bf700 1 -- 172.21.15.103:0/2569403219 learned_addr learned my addr 172.21.15.103:0/2569403219 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:48.024 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc2164bf700 1 -- 172.21.15.103:0/2569403219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0094f0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc2164bf700 1 --2- 172.21.15.103:0/2569403219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc20c000c00 tx=0x7fc20c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc20c0107c0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc20c010db0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc20c01ad00 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc210111dd0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc210112940 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.380+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc20c024440 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.381+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc20c019710 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.381+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21005f5f0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.383+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc20c01a530 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.448+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc2101045e0 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.450+0000 7fc206ffd700 1 -- 172.21.15.103:0/2569403219 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc20c010920 con 0x7fc210102860 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 msgr2=0x7fc210111690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.025 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 --2- 172.21.15.103:0/2569403219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc20c000c00 tx=0x7fc20c009250 comp rx=0 tx=0).stop 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 shutdown_connections 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 --2- 172.21.15.103:0/2569403219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc210102860 0x7fc210111690 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 >> 172.21.15.103:0/2569403219 conn(0x7fc2100fe000 msgr2=0x7fc2100feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 shutdown_connections 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:47.451+0000 7fc218723700 1 -- 172.21.15.103:0/2569403219 wait complete. 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mgr... 2024-03-22T02:18:48.026 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 9283 ... 2024-03-22T02:18:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mgr.smithi103.pfvmui 2024-03-22T02:18:48.193 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mgr.smithi103.pfvmui.service: Unit ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mgr.smithi103.pfvmui.service not loaded. 2024-03-22T02:18:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715.target.wants/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mgr.smithi103.pfvmui.service → /etc/systemd/system/ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@.service. 2024-03-22T02:18:48.760 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2569403219' entity='client.admin' 2024-03-22T02:18:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:18:48.875 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-03-22T02:18:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:18:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:18:48.882 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-03-22T02:18:48.883 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T02:18:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:18:48.891 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-03-22T02:18:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:18:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:18:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-03-22T02:18:48.898 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-22T02:18:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr to start... 2024-03-22T02:18:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr... 2024-03-22T02:18:49.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:49 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2524891271' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "67659b4a-e7f2-11ee-95cd-87774f69a715", 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "smithi103" 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:18:49.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:18:49.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:49.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:18:49.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:18:49.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:18:49.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:18:41.819129+0000", 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.337+0000 7fdd81dee700 1 Processor -- start 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.337+0000 7fdd81dee700 1 -- start start 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.337+0000 7fdd81dee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.337+0000 7fdd81dee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c1031b0 con 0x7fdd7c102860 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.337+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49130/0 (socket says 172.21.15.103:49130) 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7b7fe700 1 -- 172.21.15.103:0/3841129994 learned_addr learned my addr 172.21.15.103:0/3841129994 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7b7fe700 1 -- 172.21.15.103:0/3841129994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c1032f0 con 0x7fdd7c102860 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7b7fe700 1 --2- 172.21.15.103:0/3841129994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd700098e0 tx=0x7fdd70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f53b92cc633ee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7a7fc700 1 -- 172.21.15.103:0/3841129994 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7000b860 con 0x7fdd7c102860 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7a7fc700 1 -- 172.21.15.103:0/3841129994 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdd7000be50 con 0x7fdd7c102860 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.338+0000 7fdd7a7fc700 1 -- 172.21.15.103:0/3841129994 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd70010380 con 0x7fdd7c102860 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 -- 172.21.15.103:0/3841129994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 msgr2=0x7fdd7c102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 --2- 172.21.15.103:0/3841129994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd700098e0 tx=0x7fdd70009250 comp rx=0 tx=0).stop 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 -- 172.21.15.103:0/3841129994 shutdown_connections 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 --2- 172.21.15.103:0/3841129994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c102c70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 -- 172.21.15.103:0/3841129994 >> 172.21.15.103:0/3841129994 conn(0x7fdd7c0fe000 msgr2=0x7fdd7c100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 -- 172.21.15.103:0/3841129994 shutdown_connections 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 -- 172.21.15.103:0/3841129994 wait complete. 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.339+0000 7fdd81dee700 1 Processor -- start 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd81dee700 1 -- start start 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd81dee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd81dee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c111c20 con 0x7fdd7c102860 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49136/0 (socket says 172.21.15.103:49136) 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd7b7fe700 1 -- 172.21.15.103:0/2524891271 learned_addr learned my addr 172.21.15.103:0/2524891271 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd7b7fe700 1 -- 172.21.15.103:0/2524891271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd700094f0 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.340+0000 7fdd7b7fe700 1 --2- 172.21.15.103:0/2524891271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000b440 tx=0x7fdd70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd70010760 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdd70010d50 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c111e20 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd7001ae20 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c112990 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdd7002e6c0 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.341+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdd700248c0 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.342+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c05f5f0 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.345+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fdd70029070 con 0x7fdd7c102860 2024-03-22T02:18:49.970 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.417+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdd7c1045e0 con 0x7fdd7c102860 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.418+0000 7fdd63fff700 1 -- 172.21.15.103:0/2524891271 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fdd7001a6a0 con 0x7fdd7c102860 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 msgr2=0x7fdd7c1116e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 --2- 172.21.15.103:0/2524891271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000b440 tx=0x7fdd70009250 comp rx=0 tx=0).stop 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 shutdown_connections 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 --2- 172.21.15.103:0/2524891271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7c102860 0x7fdd7c1116e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 >> 172.21.15.103:0/2524891271 conn(0x7fdd7c0fe000 msgr2=0x7fdd7c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 shutdown_connections 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:49.419+0000 7fdd81dee700 1 -- 172.21.15.103:0/2524891271 wait complete. 2024-03-22T02:18:49.971 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (1/15)... 2024-03-22T02:18:52.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:52 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3351375885' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:18:53.020 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:53.020 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:18:53.020 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "67659b4a-e7f2-11ee-95cd-87774f69a715", 2024-03-22T02:18:53.020 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:18:53.020 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "smithi103" 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:18:53.021 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:18:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:18:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:18:53.022 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:18:53.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:18:41.819129+0000", 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.350+0000 7f9c0673e700 1 Processor -- start 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.350+0000 7f9c0673e700 1 -- start start 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.350+0000 7f9c0673e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.350+0000 7f9c0673e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00071660 con 0x7f9c00071250 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.350+0000 7f9bfffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49150/0 (socket says 172.21.15.103:49150) 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfffff700 1 -- 172.21.15.103:0/2892738908 learned_addr learned my addr 172.21.15.103:0/2892738908 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfffff700 1 -- 172.21.15.103:0/2892738908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c00105a80 con 0x7f9c00071250 2024-03-22T02:18:53.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfffff700 1 --2- 172.21.15.103:0/2892738908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40098e0 tx=0x7f9bf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0034b06908fda6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfeffd700 1 -- 172.21.15.103:0/2892738908 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf400b860 con 0x7f9c00071250 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfeffd700 1 -- 172.21.15.103:0/2892738908 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9bf400be50 con 0x7f9c00071250 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.351+0000 7f9bfeffd700 1 -- 172.21.15.103:0/2892738908 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf4010380 con 0x7f9c00071250 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 -- 172.21.15.103:0/2892738908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 msgr2=0x7f9c00105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 --2- 172.21.15.103:0/2892738908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40098e0 tx=0x7f9bf4009250 comp rx=0 tx=0).stop 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 -- 172.21.15.103:0/2892738908 shutdown_connections 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 --2- 172.21.15.103:0/2892738908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00105540 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 -- 172.21.15.103:0/2892738908 >> 172.21.15.103:0/2892738908 conn(0x7f9c000fcfb0 msgr2=0x7f9c000ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 -- 172.21.15.103:0/2892738908 shutdown_connections 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 -- 172.21.15.103:0/2892738908 wait complete. 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.352+0000 7f9c0673e700 1 Processor -- start 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9c0673e700 1 -- start start 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9c0673e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9c0673e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c00111ca0 con 0x7f9c00071250 2024-03-22T02:18:53.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49158/0 (socket says 172.21.15.103:49158) 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfffff700 1 -- 172.21.15.103:0/3351375885 learned_addr learned my addr 172.21.15.103:0/3351375885 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfffff700 1 -- 172.21.15.103:0/3351375885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40094f0 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfffff700 1 --2- 172.21.15.103:0/3351375885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4006fd0 tx=0x7f9bf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf4010760 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.353+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c00111ea0 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.354+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9bf4010d50 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.354+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf401adc0 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.354+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c00112a10 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.354+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9bf4024690 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.354+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9bf40198a0 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.355+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0005f5f0 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.357+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9bf4029060 con 0x7f9c00071250 2024-03-22T02:18:53.034 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.429+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9c001068c0 con 0x7f9c00071250 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.429+0000 7f9bfcff9700 1 -- 172.21.15.103:0/3351375885 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f9bf401a5f0 con 0x7f9c00071250 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 msgr2=0x7f9c00111760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 --2- 172.21.15.103:0/3351375885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4006fd0 tx=0x7f9bf4009250 comp rx=0 tx=0).stop 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 shutdown_connections 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 --2- 172.21.15.103:0/3351375885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c00071250 0x7f9c00111760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 >> 172.21.15.103:0/3351375885 conn(0x7f9c000fcfb0 msgr2=0x7f9c000fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 shutdown_connections 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:52.430+0000 7f9c0673e700 1 -- 172.21.15.103:0/3351375885 wait complete. 2024-03-22T02:18:53.035 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (2/15)... 2024-03-22T02:18:55.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: mgrmap e2: smithi103.pfvmui(active, starting, since 0.00291269s) 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:18:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:18:55.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:18:55.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:18:55.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' 2024-03-22T02:18:55.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' 2024-03-22T02:18:55.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:55 smithi103 ceph-mon[29258]: from='mgr.14100 172.21.15.103:0/596546428' entity='mgr.smithi103.pfvmui' 2024-03-22T02:18:56.109 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:18:56.109 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:18:56.109 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "67659b4a-e7f2-11ee-95cd-87774f69a715", 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "smithi103" 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-22T02:18:56.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:18:56.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:18:41.819129+0000", 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f2365731700 1 Processor -- start 2024-03-22T02:18:56.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f2365731700 1 -- start start 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f2365731700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f2365731700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360103270 con 0x7f2360102920 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f235effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f235effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54680/0 (socket says 172.21.15.103:54680) 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f235effd700 1 -- 172.21.15.103:0/1305012477 learned_addr learned my addr 172.21.15.103:0/1305012477 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.428+0000 7f235effd700 1 -- 172.21.15.103:0/1305012477 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23601033b0 con 0x7f2360102920 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f235effd700 1 --2- 172.21.15.103:0/1305012477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f23500098e0 tx=0x7f2350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28628e819b6a22a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f235dffb700 1 -- 172.21.15.103:0/1305012477 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f235000b860 con 0x7f2360102920 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f235dffb700 1 -- 172.21.15.103:0/1305012477 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f235000be50 con 0x7f2360102920 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f235dffb700 1 -- 172.21.15.103:0/1305012477 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2350010380 con 0x7f2360102920 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 -- 172.21.15.103:0/1305012477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 msgr2=0x7f2360102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 --2- 172.21.15.103:0/1305012477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f23500098e0 tx=0x7f2350009250 comp rx=0 tx=0).stop 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 -- 172.21.15.103:0/1305012477 shutdown_connections 2024-03-22T02:18:56.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 --2- 172.21.15.103:0/1305012477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f2360102d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 -- 172.21.15.103:0/1305012477 >> 172.21.15.103:0/1305012477 conn(0x7f23600fe0a0 msgr2=0x7f23601004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 -- 172.21.15.103:0/1305012477 shutdown_connections 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.429+0000 7f2365731700 1 -- 172.21.15.103:0/1305012477 wait complete. 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f2365731700 1 Processor -- start 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f2365731700 1 -- start start 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f2365731700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f2365731700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360111be0 con 0x7f2360102920 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f235effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f235effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54690/0 (socket says 172.21.15.103:54690) 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f235effd700 1 -- 172.21.15.103:0/3523833301 learned_addr learned my addr 172.21.15.103:0/3523833301 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.430+0000 7f235effd700 1 -- 172.21.15.103:0/3523833301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23500094f0 con 0x7f2360102920 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f235effd700 1 --2- 172.21.15.103:0/3523833301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f235000b440 tx=0x7f2350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2350010760 con 0x7f2360102920 2024-03-22T02:18:56.115 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2350010d50 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2360111de0 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f235001ae20 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.431+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2360112950 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.432+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f23500108c0 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.432+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236005f5f0 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.432+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f235004a0e0 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.434+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f235001a430 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.503+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f236004bed0 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.503+0000 7f23477fe700 1 -- 172.21.15.103:0/3523833301 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f2350049090 con 0x7f2360102920 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 msgr2=0x7f23601116a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 --2- 172.21.15.103:0/3523833301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f235000b440 tx=0x7f2350009250 comp rx=0 tx=0).stop 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 shutdown_connections 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 --2- 172.21.15.103:0/3523833301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2360102920 0x7f23601116a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 >> 172.21.15.103:0/3523833301 conn(0x7f23600fe0a0 msgr2=0x7f23600fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 shutdown_connections 2024-03-22T02:18:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:18:55.504+0000 7f2365731700 1 -- 172.21.15.103:0/3523833301 wait complete. 2024-03-22T02:18:56.117 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (3/15)... 2024-03-22T02:18:56.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:56 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3523833301' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:18:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:56 smithi103 ceph-mon[29258]: mgrmap e3: smithi103.pfvmui(active, since 1.00643s) 2024-03-22T02:18:58.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:18:58 smithi103 ceph-mon[29258]: mgrmap e4: smithi103.pfvmui(active, since 2s) 2024-03-22T02:19:01.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:01 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1513668534' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:19:07.798 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "67659b4a-e7f2-11ee-95cd-87774f69a715", 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:19:07.799 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "smithi103" 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:19:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:19:07.809 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:19:07.810 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:18:41.819129+0000", 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.017+0000 7f9a924c4700 1 Processor -- start 2024-03-22T02:19:07.811 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.017+0000 7f9a924c4700 1 -- start start 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.017+0000 7f9a924c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.017+0000 7f9a924c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c102180 con 0x7f9a8c101830 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.017+0000 7f9a8bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54702/0 (socket says 172.21.15.103:54702) 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8bfff700 1 -- 172.21.15.103:0/2632812214 learned_addr learned my addr 172.21.15.103:0/2632812214 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8bfff700 1 -- 172.21.15.103:0/2632812214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c1022c0 con 0x7f9a8c101830 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8bfff700 1 --2- 172.21.15.103:0/2632812214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098e0 tx=0x7f9a7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb48776b783dd2e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8affd700 1 -- 172.21.15.103:0/2632812214 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c00b860 con 0x7f9a8c101830 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8affd700 1 -- 172.21.15.103:0/2632812214 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a7c00be50 con 0x7f9a8c101830 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.018+0000 7f9a8affd700 1 -- 172.21.15.103:0/2632812214 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c010380 con 0x7f9a8c101830 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- 172.21.15.103:0/2632812214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 msgr2=0x7f9a8c101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 --2- 172.21.15.103:0/2632812214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098e0 tx=0x7f9a7c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- 172.21.15.103:0/2632812214 shutdown_connections 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 --2- 172.21.15.103:0/2632812214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c101c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:07.812 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- 172.21.15.103:0/2632812214 >> 172.21.15.103:0/2632812214 conn(0x7f9a8c0fcfb0 msgr2=0x7f9a8c0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- 172.21.15.103:0/2632812214 shutdown_connections 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- 172.21.15.103:0/2632812214 wait complete. 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 Processor -- start 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.019+0000 7f9a924c4700 1 -- start start 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a924c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a924c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c114400 con 0x7f9a8c101830 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a8bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a8bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54718/0 (socket says 172.21.15.103:54718) 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a8bfff700 1 -- 172.21.15.103:0/1513668534 learned_addr learned my addr 172.21.15.103:0/1513668534 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a8bfff700 1 -- 172.21.15.103:0/1513668534 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0094f0 con 0x7f9a8c101830 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a8bfff700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006b00 tx=0x7f9a7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c010760 con 0x7f9a8c101830 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a7c010d50 con 0x7f9a8c101830 2024-03-22T02:19:07.813 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c114540 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c01ad40 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.020+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c1149d0 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.021+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f9a7c02e440 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.021+0000 7f9a88ff9700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 0x7f9a74038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.021+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8c04bed0 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.022+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9a7c04a6c0 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.022+0000 7f9a8b7fe700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 0x7f9a74038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.022+0000 7f9a8b7fe700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 0x7f9a74038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009990 tx=0x7f9a80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.027+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9a7c018940 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.287+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9a8c103560 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.288+0000 7f9a88ff9700 1 -- 172.21.15.103:0/1513668534 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f9a7c01bd10 con 0x7f9a8c101830 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 msgr2=0x7f9a74038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 0x7f9a74038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009990 tx=0x7f9a80006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:07.814 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 msgr2=0x7f9a8c071250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006b00 tx=0x7f9a7c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 shutdown_connections 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f9a74036740 0x7f9a74038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 --2- 172.21.15.103:0/1513668534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a8c101830 0x7f9a8c071250 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 >> 172.21.15.103:0/1513668534 conn(0x7f9a8c0fcfb0 msgr2=0x7f9a8c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 shutdown_connections 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:01.290+0000 7f9a924c4700 1 -- 172.21.15.103:0/1513668534 wait complete. 2024-03-22T02:19:07.815 INFO:teuthology.orchestra.run.smithi103.stdout:mgr is available 2024-03-22T02:19:09.317 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:08 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1999567609' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T02:19:09.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:19:09.522 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T02:19:09.523 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.705+0000 7fca3147f700 1 Processor -- start 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.705+0000 7fca3147f700 1 -- start start 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.705+0000 7fca3147f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.705+0000 7fca3147f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c103260 con 0x7fca2c102910 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37924/0 (socket says 172.21.15.103:37924) 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca2affd700 1 -- 172.21.15.103:0/3962878225 learned_addr learned my addr 172.21.15.103:0/3962878225 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca2affd700 1 -- 172.21.15.103:0/3962878225 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c1033a0 con 0x7fca2c102910 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca2affd700 1 --2- 172.21.15.103:0/3962878225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca200098e0 tx=0x7fca20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcd2bf36ece1a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca29ffb700 1 -- 172.21.15.103:0/3962878225 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca2000b860 con 0x7fca2c102910 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca29ffb700 1 -- 172.21.15.103:0/3962878225 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fca2000be50 con 0x7fca2c102910 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.706+0000 7fca29ffb700 1 -- 172.21.15.103:0/3962878225 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca20010380 con 0x7fca2c102910 2024-03-22T02:19:09.524 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 -- 172.21.15.103:0/3962878225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 msgr2=0x7fca2c102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 --2- 172.21.15.103:0/3962878225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca200098e0 tx=0x7fca20009250 comp rx=0 tx=0).stop 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 -- 172.21.15.103:0/3962878225 shutdown_connections 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 --2- 172.21.15.103:0/3962878225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c102d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 -- 172.21.15.103:0/3962878225 >> 172.21.15.103:0/3962878225 conn(0x7fca2c0fe070 msgr2=0x7fca2c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 -- 172.21.15.103:0/3962878225 shutdown_connections 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.707+0000 7fca3147f700 1 -- 172.21.15.103:0/3962878225 wait complete. 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.708+0000 7fca3147f700 1 Processor -- start 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.708+0000 7fca3147f700 1 -- start start 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.708+0000 7fca3147f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.708+0000 7fca3147f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c111c40 con 0x7fca2c102910 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca2affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37932/0 (socket says 172.21.15.103:37932) 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca2affd700 1 -- 172.21.15.103:0/1999567609 learned_addr learned my addr 172.21.15.103:0/1999567609 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca2affd700 1 -- 172.21.15.103:0/1999567609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200094f0 con 0x7fca2c102910 2024-03-22T02:19:09.525 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca2affd700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca2000b440 tx=0x7fca20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca20010760 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fca20010d50 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c111e40 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c1129b0 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.709+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca2001ae20 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.710+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fca20024450 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.710+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c05f5f0 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.710+0000 7fca137fe700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 0x7fca14038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.710+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fca2004a5a0 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.711+0000 7fca2a7fc700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 0x7fca14038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.711+0000 7fca2a7fc700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 0x7fca14038bd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009990 tx=0x7fca1c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.719+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fca20018940 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.911+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fca2c04bed0 con 0x7fca2c102910 2024-03-22T02:19:09.526 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.913+0000 7fca137fe700 1 -- 172.21.15.103:0/1999567609 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fca20018350 con 0x7fca2c102910 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 msgr2=0x7fca14038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 0x7fca14038bd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009990 tx=0x7fca1c006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 msgr2=0x7fca2c111700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca2000b440 tx=0x7fca20009250 comp rx=0 tx=0).stop 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 shutdown_connections 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fca14036720 0x7fca14038bd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 --2- 172.21.15.103:0/1999567609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca2c102910 0x7fca2c111700 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 >> 172.21.15.103:0/1999567609 conn(0x7fca2c0fe070 msgr2=0x7fca2c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 shutdown_connections 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:08.916+0000 7fca3147f700 1 -- 172.21.15.103:0/1999567609 wait complete. 2024-03-22T02:19:09.527 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling cephadm module... 2024-03-22T02:19:10.569 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:10 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3404482764' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f7bcfc700 1 Processor -- start 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f7bcfc700 1 -- start start 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f7bcfc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f7bcfc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f741015d0 con 0x7f5f74100c80 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f79a98700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f79a98700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37946/0 (socket says 172.21.15.103:37946) 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f79a98700 1 -- 172.21.15.103:0/3617500684 learned_addr learned my addr 172.21.15.103:0/3617500684 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.966+0000 7f5f79a98700 1 -- 172.21.15.103:0/3617500684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74101710 con 0x7f5f74100c80 2024-03-22T02:19:10.862 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f79a98700 1 --2- 172.21.15.103:0/3617500684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5f640098e0 tx=0x7f5f64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aee2932a7ef6c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f78a96700 1 -- 172.21.15.103:0/3617500684 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f6400b860 con 0x7f5f74100c80 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f78a96700 1 -- 172.21.15.103:0/3617500684 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5f6400be50 con 0x7f5f74100c80 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f78a96700 1 -- 172.21.15.103:0/3617500684 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f64010380 con 0x7f5f74100c80 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3617500684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 msgr2=0x7f5f74101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3617500684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5f640098e0 tx=0x7f5f64009250 comp rx=0 tx=0).stop 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3617500684 shutdown_connections 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3617500684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74101090 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3617500684 >> 172.21.15.103:0/3617500684 conn(0x7f5f740fc700 msgr2=0x7f5f740feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3617500684 shutdown_connections 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.967+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3617500684 wait complete. 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f7bcfc700 1 Processor -- start 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f7bcfc700 1 -- start start 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f7bcfc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f7bcfc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f74111bd0 con 0x7f5f74100c80 2024-03-22T02:19:10.863 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f79a98700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f79a98700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37958/0 (socket says 172.21.15.103:37958) 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f79a98700 1 -- 172.21.15.103:0/3404482764 learned_addr learned my addr 172.21.15.103:0/3404482764 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f79a98700 1 -- 172.21.15.103:0/3404482764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640094f0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.968+0000 7f5f79a98700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000c00 tx=0x7f5f64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f640107c0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5f64010db0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f6401ad00 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f74111dd0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f74112940 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.969+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f5f64010920 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.970+0000 7f5f6a7fc700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 0x7f5f60038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.970+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7405f5f0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.970+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5f6404a5b0 con 0x7f5f74100c80 2024-03-22T02:19:10.864 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.970+0000 7f5f79297700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 0x7f5f60038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.970+0000 7f5f79297700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 0x7f5f60038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009990 tx=0x7f5f70006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:09.974+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f64010bd0 con 0x7f5f74100c80 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.214+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5f7404bed0 con 0x7f5f74100c80 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.267+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5f64017db0 con 0x7f5f74100c80 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.268+0000 7f5f6a7fc700 1 -- 172.21.15.103:0/3404482764 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f5f64023810 con 0x7f5f74100c80 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 msgr2=0x7f5f60038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 0x7f5f60038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009990 tx=0x7f5f70006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 msgr2=0x7f5f74111690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000c00 tx=0x7f5f64009250 comp rx=0 tx=0).stop 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 shutdown_connections 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f5f60036790 0x7f5f60038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 --2- 172.21.15.103:0/3404482764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f74100c80 0x7f5f74111690 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 >> 172.21.15.103:0/3404482764 conn(0x7f5f740fc700 msgr2=0x7f5f740fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 shutdown_connections 2024-03-22T02:19:10.865 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:10.270+0000 7f5f7bcfc700 1 -- 172.21.15.103:0/3404482764 wait complete. 2024-03-22T02:19:11.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:11 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3404482764' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-22T02:19:11.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:11 smithi103 ceph-mon[29258]: mgrmap e5: smithi103.pfvmui(active, since 14s) 2024-03-22T02:19:12.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:19:12.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-22T02:19:12.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:19:12.219 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "smithi103.pfvmui", 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.328+0000 7fc5c1ef2700 1 Processor -- start 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.328+0000 7fc5c1ef2700 1 -- start start 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.328+0000 7fc5c1ef2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.328+0000 7fc5c1ef2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc102180 con 0x7fc5bc101830 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5bb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5bb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37996/0 (socket says 172.21.15.103:37996) 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5bb7fe700 1 -- 172.21.15.103:0/3920732361 learned_addr learned my addr 172.21.15.103:0/3920732361 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5bb7fe700 1 -- 172.21.15.103:0/3920732361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc1022c0 con 0x7fc5bc101830 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5bb7fe700 1 --2- 172.21.15.103:0/3920732361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00098e0 tx=0x7fc5b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2831e513294ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5ba7fc700 1 -- 172.21.15.103:0/3920732361 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b000b860 con 0x7fc5bc101830 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5ba7fc700 1 -- 172.21.15.103:0/3920732361 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc5b000be50 con 0x7fc5bc101830 2024-03-22T02:19:12.220 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.329+0000 7fc5ba7fc700 1 -- 172.21.15.103:0/3920732361 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b0010380 con 0x7fc5bc101830 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/3920732361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 msgr2=0x7fc5bc101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/3920732361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00098e0 tx=0x7fc5b0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/3920732361 shutdown_connections 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/3920732361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc101c40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/3920732361 >> 172.21.15.103:0/3920732361 conn(0x7fc5bc0fcfb0 msgr2=0x7fc5bc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/3920732361 shutdown_connections 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/3920732361 wait complete. 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.330+0000 7fc5c1ef2700 1 Processor -- start 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5c1ef2700 1 -- start start 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5c1ef2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5c1ef2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc111c40 con 0x7fc5bc101830 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5bb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:12.221 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5bb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37998/0 (socket says 172.21.15.103:37998) 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5bb7fe700 1 -- 172.21.15.103:0/1255550094 learned_addr learned my addr 172.21.15.103:0/1255550094 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5bb7fe700 1 -- 172.21.15.103:0/1255550094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00094f0 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5bb7fe700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006b00 tx=0x7fc5b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.331+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b0010760 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.332+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc5b0010d50 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.332+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5b001ad40 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.332+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc111e40 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.332+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc1129b0 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.332+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc5b002e440 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.333+0000 7fc5a3fff700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 0x7fc5a4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.333+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc5b004a540 con 0x7fc5bc101830 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.333+0000 7fc5baffd700 1 -- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 msgr2=0x7fc5a4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.333+0000 7fc5baffd700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 0x7fc5a4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.333+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc05f5f0 con 0x7fc5bc101830 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.338+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc5b0018940 con 0x7fc5bc101830 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.534+0000 7fc5baffd700 1 -- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 msgr2=0x7fc5a4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.534+0000 7fc5baffd700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 0x7fc5a4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.599+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc5bc113480 con 0x7fc5bc101830 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.600+0000 7fc5a3fff700 1 -- 172.21.15.103:0/1255550094 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc5b0018350 con 0x7fc5bc101830 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 msgr2=0x7fc5a4038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 0x7fc5a4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 msgr2=0x7fc5bc111700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006b00 tx=0x7fc5b0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 shutdown_connections 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7fc5a40367e0 0x7fc5a4038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 --2- 172.21.15.103:0/1255550094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc101830 0x7fc5bc111700 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:12.223 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 >> 172.21.15.103:0/1255550094 conn(0x7fc5bc0fcfb0 msgr2=0x7fc5bc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:12.224 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 shutdown_connections 2024-03-22T02:19:12.224 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:11.603+0000 7fc5c1ef2700 1 -- 172.21.15.103:0/1255550094 wait complete. 2024-03-22T02:19:12.224 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-03-22T02:19:12.224 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 5... 2024-03-22T02:19:12.369 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:12 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1255550094' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: Active manager daemon smithi103.pfvmui restarted 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: osdmap e2: 0 total, 0 up, 0 in 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: mgrmap e6: smithi103.pfvmui(active, starting, since 0.0031894s) 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:16.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:16.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:16.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:16 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:18.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:17 smithi103 ceph-mon[29258]: Found migration_current of "None". Setting to last migration. 2024-03-22T02:19:18.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:17 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:19:18.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:17 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:19:18.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:17 smithi103 ceph-mon[29258]: mgrmap e7: smithi103.pfvmui(active, since 1.00514s) 2024-03-22T02:19:18.297 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4faef700 1 Processor -- start 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4faef700 1 -- start start 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4faef700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4faef700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48076f90 con 0x7f7c48076b80 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4d88b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4d88b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38004/0 (socket says 172.21.15.103:38004) 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.709+0000 7f7c4d88b700 1 -- 172.21.15.103:0/3334972881 learned_addr learned my addr 172.21.15.103:0/3334972881 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:18.298 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.710+0000 7f7c4d88b700 1 -- 172.21.15.103:0/3334972881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48105a60 con 0x7f7c48076b80 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.710+0000 7f7c4d88b700 1 --2- 172.21.15.103:0/3334972881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7c380098e0 tx=0x7f7c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04a25f8e6e026ff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.710+0000 7f7c4c889700 1 -- 172.21.15.103:0/3334972881 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c3800b860 con 0x7f7c48076b80 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.710+0000 7f7c4c889700 1 -- 172.21.15.103:0/3334972881 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7c3800be50 con 0x7f7c48076b80 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.710+0000 7f7c4c889700 1 -- 172.21.15.103:0/3334972881 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c38010380 con 0x7f7c48076b80 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- 172.21.15.103:0/3334972881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 msgr2=0x7f7c48105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 --2- 172.21.15.103:0/3334972881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7c380098e0 tx=0x7f7c38009250 comp rx=0 tx=0).stop 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- 172.21.15.103:0/3334972881 shutdown_connections 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 --2- 172.21.15.103:0/3334972881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48105520 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- 172.21.15.103:0/3334972881 >> 172.21.15.103:0/3334972881 conn(0x7f7c480fcf90 msgr2=0x7f7c480ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- 172.21.15.103:0/3334972881 shutdown_connections 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- 172.21.15.103:0/3334972881 wait complete. 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 Processor -- start 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- start start 2024-03-22T02:19:18.299 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.711+0000 7f7c4faef700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c481033b0 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4d88b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4d88b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38020/0 (socket says 172.21.15.103:38020) 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4d88b700 1 -- 172.21.15.103:0/2132209111 learned_addr learned my addr 172.21.15.103:0/2132209111 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4d88b700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380094f0 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4d88b700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7c38006b00 tx=0x7f7c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c380107c0 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7c38010db0 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c48101420 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c3801acf0 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.712+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c48101910 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7c38010920 con 0x7f7c48076b80 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c3e7fc700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:18.300 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c340393a0 con 0x7f7c340367e0 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c4d08a700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7c3804a4f0 con 0x7f7c48076b80 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.713+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.914+0000 7f7c4d08a700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:12.914+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:13.315+0000 7f7c4d08a700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:13.315+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:14.116+0000 7f7c4d08a700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:14.116+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:15.717+0000 7f7c4d08a700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1160196781 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:15.717+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:16.743+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f7c3800ed40 con 0x7f7c48076b80 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:16.743+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 msgr2=0x7f7c34038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:19:18.301 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:16.743+0000 7f7c3e7fc700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.744+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f7c380193c0 con 0x7f7c48076b80 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.744+0000 7f7c3e7fc700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 0x7f7c3403c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.744+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c340393a0 con 0x7f7c34039fc0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.744+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 0x7f7c3403c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.745+0000 7f7c4d08a700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 0x7f7c3403c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7c440039d0 tx=0x7f7c44009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.745+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f7c340393a0 con 0x7f7c34039fc0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.751+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7c4804bed0 con 0x7f7c34039fc0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.751+0000 7f7c3e7fc700 1 -- 172.21.15.103:0/2132209111 <== mgr.14120 v2:172.21.15.103:6800/1197413654 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7c4804bed0 con 0x7f7c34039fc0 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 msgr2=0x7f7c3403c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 0x7f7c3403c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7c440039d0 tx=0x7f7c44009350 comp rx=0 tx=0).stop 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 msgr2=0x7f7c48102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:18.302 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7c38006b00 tx=0x7f7c38009250 comp rx=0 tx=0).stop 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 shutdown_connections 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7c34039fc0 0x7f7c3403c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48076b80 0x7f7c48102e70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 --2- 172.21.15.103:0/2132209111 >> [v2:172.21.15.103:6800/1160196781,v1:172.21.15.103:6801/1160196781] conn(0x7f7c340367e0 0x7f7c34038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 >> 172.21.15.103:0/2132209111 conn(0x7f7c480fcf90 msgr2=0x7f7c480fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 shutdown_connections 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:17.752+0000 7f7c4faef700 1 -- 172.21.15.103:0/2132209111 wait complete. 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 5 is available 2024-03-22T02:19:18.303 INFO:teuthology.orchestra.run.smithi103.stdout:Setting orchestrator backend to cephadm... 2024-03-22T02:19:18.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:18 smithi103 ceph-mon[29258]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T02:19:18.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:18 smithi103 ceph-mon[29258]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T02:19:18.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:18 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:19.531 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fdd7b9700 1 Processor -- start 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fdd7b9700 1 -- start start 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fdd7b9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fdd7b9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8103250 con 0x7f2fd8102900 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59984/0 (socket says 172.21.15.103:59984) 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.683+0000 7f2fd6ffd700 1 -- 172.21.15.103:0/1072526431 learned_addr learned my addr 172.21.15.103:0/1072526431 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.684+0000 7f2fd6ffd700 1 -- 172.21.15.103:0/1072526431 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8103390 con 0x7f2fd8102900 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.684+0000 7f2fd6ffd700 1 --2- 172.21.15.103:0/1072526431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80098e0 tx=0x7f2fc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5305e1be026f343 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.684+0000 7f2fd67fc700 1 -- 172.21.15.103:0/1072526431 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc800b860 con 0x7f2fd8102900 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.684+0000 7f2fd67fc700 1 -- 172.21.15.103:0/1072526431 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2fc800be50 con 0x7f2fd8102900 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.684+0000 7f2fd67fc700 1 -- 172.21.15.103:0/1072526431 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc8010380 con 0x7f2fd8102900 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/1072526431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 msgr2=0x7f2fd8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/1072526431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80098e0 tx=0x7f2fc8009250 comp rx=0 tx=0).stop 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/1072526431 shutdown_connections 2024-03-22T02:19:19.532 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/1072526431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8102d10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/1072526431 >> 172.21.15.103:0/1072526431 conn(0x7f2fd80fe080 msgr2=0x7f2fd8100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/1072526431 shutdown_connections 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/1072526431 wait complete. 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 Processor -- start 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.685+0000 7f2fdd7b9700 1 -- start start 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fdd7b9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fdd7b9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8111bc0 con 0x7f2fd8102900 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fd6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59992/0 (socket says 172.21.15.103:59992) 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fd6ffd700 1 -- 172.21.15.103:0/4181651961 learned_addr learned my addr 172.21.15.103:0/4181651961 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fd6ffd700 1 -- 172.21.15.103:0/4181651961 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80094f0 con 0x7f2fd8102900 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fd6ffd700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800b440 tx=0x7f2fc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc8010760 con 0x7f2fd8102900 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd8111dc0 con 0x7f2fd8102900 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2fc8010d50 con 0x7f2fd8102900 2024-03-22T02:19:19.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc801ae20 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.686+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd8112930 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.687+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f2fc802e440 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.687+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd805f5f0 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.687+0000 7f2fcffff700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 0x7f2fc4038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.687+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2fc804a430 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.687+0000 7f2fce7fc700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 0x7f2fc4038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.688+0000 7f2fce7fc700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 0x7f2fc4038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009990 tx=0x7f2fc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.692+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2fc8018940 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.797+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2fc80243f0 con 0x7f2fd8102900 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.895+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fd80ff2e0 con 0x7f2fc4036660 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.903+0000 7f2fcffff700 1 -- 172.21.15.103:0/4181651961 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2fd80ff2e0 con 0x7f2fc4036660 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 msgr2=0x7f2fc4038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 0x7f2fc4038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009990 tx=0x7f2fc0006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 msgr2=0x7f2fd8111680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:19.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800b440 tx=0x7f2fc8009250 comp rx=0 tx=0).stop 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 shutdown_connections 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2fc4036660 0x7f2fc4038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 --2- 172.21.15.103:0/4181651961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fd8102900 0x7f2fd8111680 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 >> 172.21.15.103:0/4181651961 conn(0x7f2fd80fe080 msgr2=0x7f2fd80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 shutdown_connections 2024-03-22T02:19:19.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:18.905+0000 7f2fdd7b9700 1 -- 172.21.15.103:0/4181651961 wait complete. 2024-03-22T02:19:20.110 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:18] ENGINE Bus STARTING 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:18] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:18] ENGINE Bus STARTED 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: mgrmap e8: smithi103.pfvmui(active, since 2s) 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:20.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:19 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.914+0000 7f6f574fa700 1 Processor -- start 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.914+0000 7f6f574fa700 1 -- start start 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.914+0000 7f6f574fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.914+0000 7f6f574fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50070db0 con 0x7f6f50103f00 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f55296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f55296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60002/0 (socket says 172.21.15.103:60002) 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f55296700 1 -- 172.21.15.103:0/2822938554 learned_addr learned my addr 172.21.15.103:0/2822938554 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f55296700 1 -- 172.21.15.103:0/2822938554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50104850 con 0x7f6f50103f00 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f55296700 1 --2- 172.21.15.103:0/2822938554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6f400098e0 tx=0x7f6f40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbc57ece5721ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f47fff700 1 -- 172.21.15.103:0/2822938554 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4000b860 con 0x7f6f50103f00 2024-03-22T02:19:20.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f47fff700 1 -- 172.21.15.103:0/2822938554 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6f4000be50 con 0x7f6f50103f00 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.915+0000 7f6f47fff700 1 -- 172.21.15.103:0/2822938554 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f40010380 con 0x7f6f50103f00 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- 172.21.15.103:0/2822938554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 msgr2=0x7f6f50104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2822938554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6f400098e0 tx=0x7f6f40009250 comp rx=0 tx=0).stop 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- 172.21.15.103:0/2822938554 shutdown_connections 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2822938554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f50104310 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- 172.21.15.103:0/2822938554 >> 172.21.15.103:0/2822938554 conn(0x7f6f500fc700 msgr2=0x7f6f500feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- 172.21.15.103:0/2822938554 shutdown_connections 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- 172.21.15.103:0/2822938554 wait complete. 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 Processor -- start 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- start start 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.916+0000 7f6f574fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50111bf0 con 0x7f6f50103f00 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f55296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f55296700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60018/0 (socket says 172.21.15.103:60018) 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f55296700 1 -- 172.21.15.103:0/2485803968 learned_addr learned my addr 172.21.15.103:0/2485803968 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f55296700 1 -- 172.21.15.103:0/2485803968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400094f0 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f55296700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b440 tx=0x7f6f40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f40010760 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6f40010d50 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f50111df0 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4001ae20 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.917+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f50112960 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.918+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6f4002e440 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.918+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5005f5f0 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.918+0000 7f6f45ffb700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 0x7f6f3c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.919+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6f4004a6c0 con 0x7f6f50103f00 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.919+0000 7f6f54a95700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 0x7f6f3c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.919+0000 7f6f54a95700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 0x7f6f3c038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009990 tx=0x7f6f4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:19.923+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6f40018940 con 0x7f6f50103f00 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.109+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f500fd9f0 con 0x7f6f3c0367e0 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.110+0000 7f6f45ffb700 1 -- 172.21.15.103:0/2485803968 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6f500fd9f0 con 0x7f6f3c0367e0 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 msgr2=0x7f6f3c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 0x7f6f3c038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009990 tx=0x7f6f4c006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 msgr2=0x7f6f501116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b440 tx=0x7f6f40009250 comp rx=0 tx=0).stop 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 shutdown_connections 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f6f3c0367e0 0x7f6f3c038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 --2- 172.21.15.103:0/2485803968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f50103f00 0x7f6f501116b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 >> 172.21.15.103:0/2485803968 conn(0x7f6f500fc700 msgr2=0x7f6f500fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 shutdown_connections 2024-03-22T02:19:20.658 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:20.112+0000 7f6f574fa700 1 -- 172.21.15.103:0/2485803968 wait complete. 2024-03-22T02:19:20.659 INFO:teuthology.orchestra.run.smithi103.stdout:Generating ssh key... 2024-03-22T02:19:21.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:20 smithi103 ceph-mon[29258]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:22.103 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:21 smithi103 ceph-mon[29258]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:22.103 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:21 smithi103 ceph-mon[29258]: Generating ssh key... 2024-03-22T02:19:22.104 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:21 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:22.104 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:21 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.103+0000 7f9a01528700 1 Processor -- start 2024-03-22T02:19:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.103+0000 7f9a01528700 1 -- start start 2024-03-22T02:19:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f9a01528700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f9a01528700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc103240 con 0x7f99fc1028f0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60026/0 (socket says 172.21.15.103:60026) 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99faffd700 1 -- 172.21.15.103:0/493653352 learned_addr learned my addr 172.21.15.103:0/493653352 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99faffd700 1 -- 172.21.15.103:0/493653352 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc103380 con 0x7f99fc1028f0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99faffd700 1 --2- 172.21.15.103:0/493653352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098e0 tx=0x7f99f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40e3e8e1624290d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99f9ffb700 1 -- 172.21.15.103:0/493653352 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f000b860 con 0x7f99fc1028f0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99f9ffb700 1 -- 172.21.15.103:0/493653352 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f99f000be50 con 0x7f99fc1028f0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.104+0000 7f99f9ffb700 1 -- 172.21.15.103:0/493653352 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f0010380 con 0x7f99fc1028f0 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 -- 172.21.15.103:0/493653352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 msgr2=0x7f99fc102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 --2- 172.21.15.103:0/493653352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098e0 tx=0x7f99f0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 -- 172.21.15.103:0/493653352 shutdown_connections 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 --2- 172.21.15.103:0/493653352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc102d00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 -- 172.21.15.103:0/493653352 >> 172.21.15.103:0/493653352 conn(0x7f99fc0fe070 msgr2=0x7f99fc100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 -- 172.21.15.103:0/493653352 shutdown_connections 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.105+0000 7f9a01528700 1 -- 172.21.15.103:0/493653352 wait complete. 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f9a01528700 1 Processor -- start 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f9a01528700 1 -- start start 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f9a01528700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f9a01528700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc111bb0 con 0x7f99fc1028f0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60034/0 (socket says 172.21.15.103:60034) 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99faffd700 1 -- 172.21.15.103:0/2722470171 learned_addr learned my addr 172.21.15.103:0/2722470171 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99faffd700 1 -- 172.21.15.103:0/2722470171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00094f0 con 0x7f99fc1028f0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99faffd700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99f0000c00 tx=0x7f99f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f00107c0 con 0x7f99fc1028f0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.106+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f99f0010db0 con 0x7f99fc1028f0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.107+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f001ad00 con 0x7f99fc1028f0 2024-03-22T02:19:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.107+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc111db0 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.107+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc112920 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.107+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f99f00193c0 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.108+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc05f5f0 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.108+0000 7f99e37fe700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 0x7f99e4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.108+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f99f004a770 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.108+0000 7f99fa7fc700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 0x7f99e4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.108+0000 7f99fa7fc700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 0x7f99e4038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009990 tx=0x7f99ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.112+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f99f0022520 con 0x7f99fc1028f0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.297+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f99fc0ff2e0 con 0x7f99e40367e0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.556+0000 7f99e37fe700 1 -- 172.21.15.103:0/2722470171 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f99fc0ff2e0 con 0x7f99e40367e0 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 msgr2=0x7f99e4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 0x7f99e4038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009990 tx=0x7f99ec006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 msgr2=0x7f99fc111670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99f0000c00 tx=0x7f99f0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 shutdown_connections 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f99e40367e0 0x7f99e4038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 --2- 172.21.15.103:0/2722470171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99fc1028f0 0x7f99fc111670 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 >> 172.21.15.103:0/2722470171 conn(0x7f99fc0fe070 msgr2=0x7f99fc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 shutdown_connections 2024-03-22T02:19:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:21.558+0000 7f9a01528700 1 -- 172.21.15.103:0/2722470171 wait complete. 2024-03-22T02:19:23.351 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVwv70uezTxr7yb7DSbnmiDoWU6urHN2UNp1+MpbYPnYKweUVn+/ssZz+Bh6gJS2b+ZcdWzTUcLJKEikNNU+3nvMIHToTsyoaUHTisiJPQZeiuzI0s0tEtwktsnT7xi1LN9/rJFZ+/uIJGtZVuzbWYRrBiPM44D7snnWk5nsllQo4dCrqs4Qen3zinTvrru897fysCLpF+t2UWV6de0EXmia57/TsIBmlyor000jsKyv/+nV6d7yQBXvp9xmSxD3Ni1F+a4jCBzBtr65y7A+6xYyt4R3QwBo3lCSxF3+TuAmp5Ywg7lwMSka3Ihq7hgqbaZCIcZLUMGH/wc5AQUTFqukTjkGGbIP16/LlRJBQSIHk1KWft3elBfSdC79X5V6cgUz4UsxF4GUT5sEkNRZ1ihM08aJrrQGNerkzWgTRkihVcIJYkw06e4+oZccD8PkxaEqmIkefwTYD/mJkqqudqSP390iPZZ1eyPkACxX7sg7WkKUSskMqkUUsMRgnRgNU= ceph-67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:19:23.351 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.579+0000 7ff547e87700 1 Processor -- start 2024-03-22T02:19:23.351 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.579+0000 7ff547e87700 1 -- start start 2024-03-22T02:19:23.351 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.579+0000 7ff547e87700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.579+0000 7ff547e87700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540103210 con 0x7ff5401028c0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff545c23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff545c23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60042/0 (socket says 172.21.15.103:60042) 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff545c23700 1 -- 172.21.15.103:0/1672703871 learned_addr learned my addr 172.21.15.103:0/1672703871 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff545c23700 1 -- 172.21.15.103:0/1672703871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff540103350 con 0x7ff5401028c0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff545c23700 1 --2- 172.21.15.103:0/1672703871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098e0 tx=0x7ff53c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7c5005e2e5034a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff544c21700 1 -- 172.21.15.103:0/1672703871 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53c00b860 con 0x7ff5401028c0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff544c21700 1 -- 172.21.15.103:0/1672703871 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff53c00be50 con 0x7ff5401028c0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.580+0000 7ff544c21700 1 -- 172.21.15.103:0/1672703871 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53c010380 con 0x7ff5401028c0 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 -- 172.21.15.103:0/1672703871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 msgr2=0x7ff540102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 --2- 172.21.15.103:0/1672703871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098e0 tx=0x7ff53c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 -- 172.21.15.103:0/1672703871 shutdown_connections 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 --2- 172.21.15.103:0/1672703871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540102cd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:23.352 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 -- 172.21.15.103:0/1672703871 >> 172.21.15.103:0/1672703871 conn(0x7ff5400fe080 msgr2=0x7ff540100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 -- 172.21.15.103:0/1672703871 shutdown_connections 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 -- 172.21.15.103:0/1672703871 wait complete. 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.581+0000 7ff547e87700 1 Processor -- start 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff547e87700 1 -- start start 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff547e87700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff547e87700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff540111ba0 con 0x7ff5401028c0 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff545c23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff545c23700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60058/0 (socket says 172.21.15.103:60058) 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff545c23700 1 -- 172.21.15.103:0/3352147057 learned_addr learned my addr 172.21.15.103:0/3352147057 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff545c23700 1 -- 172.21.15.103:0/3352147057 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c0094f0 con 0x7ff5401028c0 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff545c23700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00b440 tx=0x7ff53c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53c010760 con 0x7ff5401028c0 2024-03-22T02:19:23.353 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff53c010d50 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53c01ae20 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540111da0 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.582+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540112910 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.583+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff53c0108c0 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.583+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54005f5f0 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.583+0000 7ff5367fc700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 0x7ff52c038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.583+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff53c04a5c0 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.584+0000 7ff545422700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 0x7ff52c038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.584+0000 7ff545422700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 0x7ff52c038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff530009990 tx=0x7ff530006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.588+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff53c01a430 con 0x7ff5401028c0 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.773+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5400ff2e0 con 0x7ff52c036770 2024-03-22T02:19:23.354 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.774+0000 7ff5367fc700 1 -- 172.21.15.103:0/3352147057 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff5400ff2e0 con 0x7ff52c036770 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 msgr2=0x7ff52c038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 0x7ff52c038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff530009990 tx=0x7ff530006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 msgr2=0x7ff540111660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00b440 tx=0x7ff53c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 shutdown_connections 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7ff52c036770 0x7ff52c038c20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 --2- 172.21.15.103:0/3352147057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5401028c0 0x7ff540111660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 >> 172.21.15.103:0/3352147057 conn(0x7ff5400fe080 msgr2=0x7ff5400febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 shutdown_connections 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:22.776+0000 7ff547e87700 1 -- 172.21.15.103:0/3352147057 wait complete. 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:Adding key to root@localhost authorized_keys... 2024-03-22T02:19:23.355 INFO:teuthology.orchestra.run.smithi103.stdout:Adding host smithi103... 2024-03-22T02:19:23.603 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:23 smithi103 ceph-mon[29258]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:24.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:24 smithi103 ceph-mon[29258]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:25.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:25 smithi103 ceph-mon[29258]: Deploying cephadm binary to smithi103 2024-03-22T02:19:26.610 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Added host 'smithi103' with addr '172.21.15.103' 2024-03-22T02:19:26.610 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.803+0000 7f7a58961700 1 Processor -- start 2024-03-22T02:19:26.610 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.803+0000 7f7a58961700 1 -- start start 2024-03-22T02:19:26.610 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a58961700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a58961700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54103260 con 0x7f7a54102910 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37256/0 (socket says 172.21.15.103:37256) 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a5259c700 1 -- 172.21.15.103:0/3354787790 learned_addr learned my addr 172.21.15.103:0/3354787790 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.804+0000 7f7a5259c700 1 -- 172.21.15.103:0/3354787790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a541033a0 con 0x7f7a54102910 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a5259c700 1 --2- 172.21.15.103:0/3354787790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e50 tx=0x7f7a4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1513ed68ca3fb2a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a5159a700 1 -- 172.21.15.103:0/3354787790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c0092e0 con 0x7f7a54102910 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a5159a700 1 -- 172.21.15.103:0/3354787790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7a4c012430 con 0x7f7a54102910 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a5159a700 1 -- 172.21.15.103:0/3354787790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c0075c0 con 0x7f7a54102910 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 -- 172.21.15.103:0/3354787790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 msgr2=0x7f7a54102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 --2- 172.21.15.103:0/3354787790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e50 tx=0x7f7a4c009840 comp rx=0 tx=0).stop 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 -- 172.21.15.103:0/3354787790 shutdown_connections 2024-03-22T02:19:26.611 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 --2- 172.21.15.103:0/3354787790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54102d20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 -- 172.21.15.103:0/3354787790 >> 172.21.15.103:0/3354787790 conn(0x7f7a540fe070 msgr2=0x7f7a541004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 -- 172.21.15.103:0/3354787790 shutdown_connections 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.805+0000 7f7a58961700 1 -- 172.21.15.103:0/3354787790 wait complete. 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a58961700 1 Processor -- start 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a58961700 1 -- start start 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a58961700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a58961700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54111c50 con 0x7f7a54102910 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37270/0 (socket says 172.21.15.103:37270) 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a5259c700 1 -- 172.21.15.103:0/1341820031 learned_addr learned my addr 172.21.15.103:0/1341820031 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.806+0000 7f7a5259c700 1 -- 172.21.15.103:0/1341820031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c009ae0 con 0x7f7a54102910 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a5259c700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009fd0 tx=0x7f7a4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c0092e0 con 0x7f7a54102910 2024-03-22T02:19:26.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7a4c007cc0 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a54111e50 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c01c9a0 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.807+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a541129c0 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7a4c019070 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5405f5f0 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a42ffd700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 0x7f7a3c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7a4c049d50 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a51d9b700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 0x7f7a3c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.808+0000 7f7a51d9b700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 0x7f7a3c038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009990 tx=0x7f7a48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:23.812+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7a4c04d030 con 0x7f7a54102910 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:24.001+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a540ff410 con 0x7f7a3c036790 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.083+0000 7f7a42ffd700 1 -- 172.21.15.103:0/1341820031 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7a540ff410 con 0x7f7a3c036790 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 msgr2=0x7f7a3c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:26.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 0x7f7a3c038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009990 tx=0x7f7a48006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 msgr2=0x7f7a54111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009fd0 tx=0x7f7a4c009840 comp rx=0 tx=0).stop 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 shutdown_connections 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f7a3c036790 0x7f7a3c038c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 --2- 172.21.15.103:0/1341820031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a54102910 0x7f7a54111710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 >> 172.21.15.103:0/1341820031 conn(0x7f7a540fe070 msgr2=0x7f7a540fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 shutdown_connections 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:26.085+0000 7f7a58961700 1 -- 172.21.15.103:0/1341820031 wait complete. 2024-03-22T02:19:26.614 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying mon service with default placement... 2024-03-22T02:19:27.270 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:27 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:27.271 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:27 smithi103 ceph-mon[29258]: Added host smithi103 2024-03-22T02:19:27.271 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:27 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2ce5eb3700 1 Processor -- start 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2ce5eb3700 1 -- start start 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2ce5eb3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2ce5eb3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce0076f40 con 0x7f2ce0076b30 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37278/0 (socket says 172.21.15.103:37278) 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.044+0000 7f2cdf7fe700 1 -- 172.21.15.103:0/2911429555 learned_addr learned my addr 172.21.15.103:0/2911429555 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:27.986 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.045+0000 7f2cdf7fe700 1 -- 172.21.15.103:0/2911429555 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce0105a90 con 0x7f2ce0076b30 2024-03-22T02:19:27.987 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.045+0000 7f2cdf7fe700 1 --2- 172.21.15.103:0/2911429555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00098e0 tx=0x7f2cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2894cf236103b7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:27.987 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.045+0000 7f2cde7fc700 1 -- 172.21.15.103:0/2911429555 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd000b8f0 con 0x7f2ce0076b30 2024-03-22T02:19:27.987 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.045+0000 7f2cde7fc700 1 -- 172.21.15.103:0/2911429555 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2cd0011420 con 0x7f2ce0076b30 2024-03-22T02:19:27.987 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.045+0000 7f2cde7fc700 1 -- 172.21.15.103:0/2911429555 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd0010540 con 0x7f2ce0076b30 2024-03-22T02:19:27.987 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/2911429555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 msgr2=0x7f2ce0105550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/2911429555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00098e0 tx=0x7f2cd0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/2911429555 shutdown_connections 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/2911429555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0105550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/2911429555 >> 172.21.15.103:0/2911429555 conn(0x7f2ce00fd000 msgr2=0x7f2ce00ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/2911429555 shutdown_connections 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/2911429555 wait complete. 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 Processor -- start 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- start start 2024-03-22T02:19:27.988 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.046+0000 7f2ce5eb3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01033e0 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cdf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37282/0 (socket says 172.21.15.103:37282) 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cdf7fe700 1 -- 172.21.15.103:0/3738528005 learned_addr learned my addr 172.21.15.103:0/3738528005 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cdf7fe700 1 -- 172.21.15.103:0/3738528005 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00094f0 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cdf7fe700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0000c00 tx=0x7f2cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd00107b0 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2cd0010da0 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce0101450 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cd001add0 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.047+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce0101940 con 0x7f2ce0076b30 2024-03-22T02:19:27.989 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.048+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2cd0024480 con 0x7f2ce0076b30 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.048+0000 7f2cc7fff700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 0x7f2cc8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.048+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce004bed0 con 0x7f2ce0076b30 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.048+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2cd004ac80 con 0x7f2ce0076b30 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.048+0000 7f2cdeffd700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 0x7f2cc8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.049+0000 7f2cdeffd700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 0x7f2cc8038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009990 tx=0x7f2cd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.053+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2cd002e440 con 0x7f2ce0076b30 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.270+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ce0102050 con 0x7f2cc8036790 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.277+0000 7f2cc7fff700 1 -- 172.21.15.103:0/3738528005 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2ce0102050 con 0x7f2cc8036790 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 msgr2=0x7f2cc8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 0x7f2cc8038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009990 tx=0x7f2cd4006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 msgr2=0x7f2ce0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0000c00 tx=0x7f2cd0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:27.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 shutdown_connections 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f2cc8036790 0x7f2cc8038c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 --2- 172.21.15.103:0/3738528005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce0076b30 0x7f2ce0102ea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 >> 172.21.15.103:0/3738528005 conn(0x7f2ce00fd000 msgr2=0x7f2ce00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 shutdown_connections 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:27.279+0000 7f2ce5eb3700 1 -- 172.21.15.103:0/3738528005 wait complete. 2024-03-22T02:19:27.991 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying mgr service with default placement... 2024-03-22T02:19:28.584 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:28 smithi103 ceph-mon[29258]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:28.584 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:28 smithi103 ceph-mon[29258]: Saving service mon spec with placement count:5 2024-03-22T02:19:28.584 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:28 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:29.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:29 smithi103 ceph-mon[29258]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:29 smithi103 ceph-mon[29258]: Saving service mgr spec with placement count:2 2024-03-22T02:19:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:29 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:29 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:29.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1acee33700 1 Processor -- start 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1acee33700 1 -- start start 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1acee33700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1acee33700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8070db0 con 0x7f1ac8103f30 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1accbcf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1accbcf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37286/0 (socket says 172.21.15.103:37286) 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.385+0000 7f1accbcf700 1 -- 172.21.15.103:0/1615570140 learned_addr learned my addr 172.21.15.103:0/1615570140 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.386+0000 7f1accbcf700 1 -- 172.21.15.103:0/1615570140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac8104880 con 0x7f1ac8103f30 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.386+0000 7f1accbcf700 1 --2- 172.21.15.103:0/1615570140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0098e0 tx=0x7f1abc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9d942ebc64b9e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.386+0000 7f1ac77fe700 1 -- 172.21.15.103:0/1615570140 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc00b860 con 0x7f1ac8103f30 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.386+0000 7f1ac77fe700 1 -- 172.21.15.103:0/1615570140 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1abc00be50 con 0x7f1ac8103f30 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.386+0000 7f1ac77fe700 1 -- 172.21.15.103:0/1615570140 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc010380 con 0x7f1ac8103f30 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- 172.21.15.103:0/1615570140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 msgr2=0x7f1ac8104340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:29.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 --2- 172.21.15.103:0/1615570140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0098e0 tx=0x7f1abc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- 172.21.15.103:0/1615570140 shutdown_connections 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 --2- 172.21.15.103:0/1615570140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac8104340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- 172.21.15.103:0/1615570140 >> 172.21.15.103:0/1615570140 conn(0x7f1ac80fc730 msgr2=0x7f1ac80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- 172.21.15.103:0/1615570140 shutdown_connections 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- 172.21.15.103:0/1615570140 wait complete. 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 Processor -- start 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.387+0000 7f1acee33700 1 -- start start 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1acee33700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1acee33700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8111c20 con 0x7f1ac8103f30 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1accbcf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1accbcf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37300/0 (socket says 172.21.15.103:37300) 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1accbcf700 1 -- 172.21.15.103:0/1304213464 learned_addr learned my addr 172.21.15.103:0/1304213464 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1accbcf700 1 -- 172.21.15.103:0/1304213464 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0094f0 con 0x7f1ac8103f30 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1accbcf700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00ba90 tx=0x7f1abc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.388+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc0107c0 con 0x7f1ac8103f30 2024-03-22T02:19:29.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1abc010db0 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac8111e20 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc01add0 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac8112990 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f1abc02e440 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.389+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac805f5f0 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.390+0000 7f1ac57fa700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 0x7f1ab8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.390+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1abc04a650 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.390+0000 7f1ac7fff700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 0x7f1ab8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.390+0000 7f1ac7fff700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 0x7f1ab8038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009990 tx=0x7f1ab4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.394+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1abc018710 con 0x7f1ac8103f30 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.593+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ac80fda20 con 0x7f1ab8036790 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.600+0000 7f1ac57fa700 1 -- 172.21.15.103:0/1304213464 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ac80fda20 con 0x7f1ab8036790 2024-03-22T02:19:29.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 msgr2=0x7f1ab8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 0x7f1ab8038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009990 tx=0x7f1ab4006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 msgr2=0x7f1ac81116e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00ba90 tx=0x7f1abc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 shutdown_connections 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f1ab8036790 0x7f1ab8038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 --2- 172.21.15.103:0/1304213464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ac8103f30 0x7f1ac81116e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 >> 172.21.15.103:0/1304213464 conn(0x7f1ac80fc730 msgr2=0x7f1ac80fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 shutdown_connections 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:28.602+0000 7f1acee33700 1 -- 172.21.15.103:0/1304213464 wait complete. 2024-03-22T02:19:29.962 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying crash service with default placement... 2024-03-22T02:19:31.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:31 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:31.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:31 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd5a1a5a700 1 Processor -- start 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd5a1a5a700 1 -- start start 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd5a1a5a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd5a1a5a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c1018d0 con 0x7fd59c100f80 2024-03-22T02:19:31.533 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd59affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd59affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37312/0 (socket says 172.21.15.103:37312) 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.406+0000 7fd59affd700 1 -- 172.21.15.103:0/1389343379 learned_addr learned my addr 172.21.15.103:0/1389343379 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.407+0000 7fd59affd700 1 -- 172.21.15.103:0/1389343379 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c101a10 con 0x7fd59c100f80 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.407+0000 7fd59affd700 1 --2- 172.21.15.103:0/1389343379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0098e0 tx=0x7fd58c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23172678fd61b145 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.407+0000 7fd599ffb700 1 -- 172.21.15.103:0/1389343379 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c00b860 con 0x7fd59c100f80 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.407+0000 7fd599ffb700 1 -- 172.21.15.103:0/1389343379 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd58c00be90 con 0x7fd59c100f80 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.407+0000 7fd599ffb700 1 -- 172.21.15.103:0/1389343379 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c0103c0 con 0x7fd59c100f80 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/1389343379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 msgr2=0x7fd59c101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/1389343379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0098e0 tx=0x7fd58c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/1389343379 shutdown_connections 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/1389343379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c101390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/1389343379 >> 172.21.15.103:0/1389343379 conn(0x7fd59c0fc6e0 msgr2=0x7fd59c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/1389343379 shutdown_connections 2024-03-22T02:19:31.534 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.408+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/1389343379 wait complete. 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd5a1a5a700 1 Processor -- start 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd5a1a5a700 1 -- start start 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd5a1a5a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd5a1a5a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c110af0 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd59affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd59affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37314/0 (socket says 172.21.15.103:37314) 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd59affd700 1 -- 172.21.15.103:0/37839784 learned_addr learned my addr 172.21.15.103:0/37839784 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.409+0000 7fd59affd700 1 -- 172.21.15.103:0/37839784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c0094f0 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd59affd700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0090b0 tx=0x7fd58c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c010840 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c110cf0 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd58c010e70 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c111860 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.410+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c024e40 con 0x7fd59c100f80 2024-03-22T02:19:31.535 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.411+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd58c01a510 con 0x7fd59c100f80 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.411+0000 7fd5837fe700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 0x7fd584038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.411+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c05f5f0 con 0x7fd59c100f80 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.411+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd58c04b830 con 0x7fd59c100f80 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.411+0000 7fd59a7fc700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 0x7fd584038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.412+0000 7fd59a7fc700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 0x7fd584038bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd590009990 tx=0x7fd590006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.416+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd58c018940 con 0x7fd59c100f80 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.630+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fd59c0fd420 con 0x7fd584036740 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.634+0000 7fd5837fe700 1 -- 172.21.15.103:0/37839784 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fd59c0fd420 con 0x7fd584036740 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 msgr2=0x7fd584038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 0x7fd584038bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd590009990 tx=0x7fd590006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 msgr2=0x7fd59c1105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0090b0 tx=0x7fd58c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 shutdown_connections 2024-03-22T02:19:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fd584036740 0x7fd584038bf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:31.537 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 --2- 172.21.15.103:0/37839784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd59c100f80 0x7fd59c1105b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:31.537 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 >> 172.21.15.103:0/37839784 conn(0x7fd59c0fc6e0 msgr2=0x7fd59c0fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:31.537 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 shutdown_connections 2024-03-22T02:19:31.537 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:30.636+0000 7fd5a1a5a700 1 -- 172.21.15.103:0/37839784 wait complete. 2024-03-22T02:19:31.537 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying prometheus service with default placement... 2024-03-22T02:19:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:32 smithi103 ceph-mon[29258]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:32 smithi103 ceph-mon[29258]: Saving service crash spec with placement * 2024-03-22T02:19:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:32 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:32 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:32 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:32.839 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.998+0000 7f32a0289700 1 Processor -- start 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.998+0000 7f32a0289700 1 -- start start 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f32a0289700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f32a0289700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981021b0 con 0x7f3298101860 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f329e025700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f329e025700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37318/0 (socket says 172.21.15.103:37318) 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f329e025700 1 -- 172.21.15.103:0/660483997 learned_addr learned my addr 172.21.15.103:0/660483997 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f329e025700 1 -- 172.21.15.103:0/660483997 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32981022f0 con 0x7f3298101860 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:31.999+0000 7f329e025700 1 --2- 172.21.15.103:0/660483997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32940098e0 tx=0x7f3294009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1e979a1c6fb8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.000+0000 7f329d023700 1 -- 172.21.15.103:0/660483997 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329400b8f0 con 0x7f3298101860 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.000+0000 7f329d023700 1 -- 172.21.15.103:0/660483997 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3294011420 con 0x7f3298101860 2024-03-22T02:19:32.840 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.000+0000 7f329d023700 1 -- 172.21.15.103:0/660483997 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3294010540 con 0x7f3298101860 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.000+0000 7f32a0289700 1 -- 172.21.15.103:0/660483997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 msgr2=0x7f3298101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.000+0000 7f32a0289700 1 --2- 172.21.15.103:0/660483997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32940098e0 tx=0x7f3294009250 comp rx=0 tx=0).stop 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- 172.21.15.103:0/660483997 shutdown_connections 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 --2- 172.21.15.103:0/660483997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298101c70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- 172.21.15.103:0/660483997 >> 172.21.15.103:0/660483997 conn(0x7f32980fcfa0 msgr2=0x7f32980ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- 172.21.15.103:0/660483997 shutdown_connections 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- 172.21.15.103:0/660483997 wait complete. 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 Processor -- start 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- start start 2024-03-22T02:19:32.841 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f32a0289700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981121e0 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.001+0000 7f329e025700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f329e025700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37330/0 (socket says 172.21.15.103:37330) 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f329e025700 1 -- 172.21.15.103:0/2246675531 learned_addr learned my addr 172.21.15.103:0/2246675531 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f329e025700 1 -- 172.21.15.103:0/2246675531 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940094f0 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f329e025700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3294006b00 tx=0x7f3294009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3294010800 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3294010e30 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32981167f0 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329401ae40 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.002+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3298116bc0 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.003+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f329402e440 con 0x7f3298101860 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.003+0000 7f328effd700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 0x7f3284038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.003+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f329404a5c0 con 0x7f3298101860 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.003+0000 7f329d824700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 0x7f3284038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.003+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329804bed0 con 0x7f3298101860 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.004+0000 7f329d824700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 0x7f3284038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3288009990 tx=0x7f3288006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.008+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3294018940 con 0x7f3298101860 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.222+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f32980fe2b0 con 0x7f3284036790 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.226+0000 7f328effd700 1 -- 172.21.15.103:0/2246675531 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f32980fe2b0 con 0x7f3284036790 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 msgr2=0x7f3284038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 0x7f3284038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3288009990 tx=0x7f3288006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 msgr2=0x7f3298076b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3294006b00 tx=0x7f3294009250 comp rx=0 tx=0).stop 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 shutdown_connections 2024-03-22T02:19:32.843 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f3284036790 0x7f3284038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 --2- 172.21.15.103:0/2246675531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3298101860 0x7f3298076b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.228+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 >> 172.21.15.103:0/2246675531 conn(0x7f32980fcfa0 msgr2=0x7f32980fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.229+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 shutdown_connections 2024-03-22T02:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:32.229+0000 7f32a0289700 1 -- 172.21.15.103:0/2246675531 wait complete. 2024-03-22T02:19:32.844 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying grafana service with default placement... 2024-03-22T02:19:33.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:33 smithi103 ceph-mon[29258]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:33.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:33 smithi103 ceph-mon[29258]: Saving service prometheus spec with placement count:1 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe71b338700 1 Processor -- start 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe71b338700 1 -- start start 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe71b338700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe71b338700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe714079100 con 0x7fe714078cf0 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe7190d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe7190d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37332/0 (socket says 172.21.15.103:37332) 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe7190d4700 1 -- 172.21.15.103:0/2739876811 learned_addr learned my addr 172.21.15.103:0/2739876811 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.268+0000 7fe7190d4700 1 -- 172.21.15.103:0/2739876811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140777b0 con 0x7fe714078cf0 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe7190d4700 1 --2- 172.21.15.103:0/2739876811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7100098e0 tx=0x7fe710009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e38aed41dda5c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:34.509 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe70bfff700 1 -- 172.21.15.103:0/2739876811 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe71000b860 con 0x7fe714078cf0 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe70bfff700 1 -- 172.21.15.103:0/2739876811 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe71000be90 con 0x7fe714078cf0 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe70bfff700 1 -- 172.21.15.103:0/2739876811 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7100103c0 con 0x7fe714078cf0 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 -- 172.21.15.103:0/2739876811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 msgr2=0x7fe714077270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 --2- 172.21.15.103:0/2739876811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7100098e0 tx=0x7fe710009250 comp rx=0 tx=0).stop 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 -- 172.21.15.103:0/2739876811 shutdown_connections 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 --2- 172.21.15.103:0/2739876811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714077270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 -- 172.21.15.103:0/2739876811 >> 172.21.15.103:0/2739876811 conn(0x7fe7140fe000 msgr2=0x7fe714100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.269+0000 7fe71b338700 1 -- 172.21.15.103:0/2739876811 shutdown_connections 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe71b338700 1 -- 172.21.15.103:0/2739876811 wait complete. 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe71b338700 1 Processor -- start 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe71b338700 1 -- start start 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe71b338700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe71b338700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe714113e80 con 0x7fe714078cf0 2024-03-22T02:19:34.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe7190d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe7190d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37334/0 (socket says 172.21.15.103:37334) 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe7190d4700 1 -- 172.21.15.103:0/464447738 learned_addr learned my addr 172.21.15.103:0/464447738 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.270+0000 7fe7190d4700 1 -- 172.21.15.103:0/464447738 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7100094f0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe7190d4700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe710010680 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714114080 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe710010cb0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe71001acf0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.271+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe714114bf0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe7100193c0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe709ffb700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 0x7fe700038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe710010e20 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe7188d3700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 0x7fe700038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71405f5f0 con 0x7fe714078cf0 2024-03-22T02:19:34.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.272+0000 7fe7188d3700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 0x7fe700038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe704009990 tx=0x7fe704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.277+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe710023940 con 0x7fe714078cf0 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.492+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7140ff2f0 con 0x7fe700036790 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.497+0000 7fe709ffb700 1 -- 172.21.15.103:0/464447738 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe7140ff2f0 con 0x7fe700036790 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 msgr2=0x7fe700038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 0x7fe700038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe704009990 tx=0x7fe704006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 msgr2=0x7fe714113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710009250 comp rx=0 tx=0).stop 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 shutdown_connections 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fe700036790 0x7fe700038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 --2- 172.21.15.103:0/464447738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe714078cf0 0x7fe714113940 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 >> 172.21.15.103:0/464447738 conn(0x7fe7140fe000 msgr2=0x7fe7140feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 shutdown_connections 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:33.499+0000 7fe71b338700 1 -- 172.21.15.103:0/464447738 wait complete. 2024-03-22T02:19:34.512 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying node-exporter service with default placement... 2024-03-22T02:19:34.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:34 smithi103 ceph-mon[29258]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:34.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:34 smithi103 ceph-mon[29258]: Saving service grafana spec with placement count:1 2024-03-22T02:19:34.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:34 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:36.117 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.104+0000 7fa06a6ad700 1 Processor -- start 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.104+0000 7fa06a6ad700 1 -- start start 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa06a6ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa06a6ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0641032c0 con 0x7fa064102970 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa063fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa063fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60230/0 (socket says 172.21.15.103:60230) 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa063fff700 1 -- 172.21.15.103:0/2918884757 learned_addr learned my addr 172.21.15.103:0/2918884757 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa063fff700 1 -- 172.21.15.103:0/2918884757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064103400 con 0x7fa064102970 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa063fff700 1 --2- 172.21.15.103:0/2918884757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0580098e0 tx=0x7fa058009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fa55c1ca407e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:36.118 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa062ffd700 1 -- 172.21.15.103:0/2918884757 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa05800b860 con 0x7fa064102970 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa062ffd700 1 -- 172.21.15.103:0/2918884757 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa05800be90 con 0x7fa064102970 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.105+0000 7fa062ffd700 1 -- 172.21.15.103:0/2918884757 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0580103c0 con 0x7fa064102970 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 -- 172.21.15.103:0/2918884757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 msgr2=0x7fa064102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/2918884757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0580098e0 tx=0x7fa058009250 comp rx=0 tx=0).stop 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 -- 172.21.15.103:0/2918884757 shutdown_connections 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/2918884757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064102d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 -- 172.21.15.103:0/2918884757 >> 172.21.15.103:0/2918884757 conn(0x7fa0640fe090 msgr2=0x7fa064100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 -- 172.21.15.103:0/2918884757 shutdown_connections 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.106+0000 7fa06a6ad700 1 -- 172.21.15.103:0/2918884757 wait complete. 2024-03-22T02:19:36.119 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa06a6ad700 1 Processor -- start 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa06a6ad700 1 -- start start 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa06a6ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa06a6ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064111ca0 con 0x7fa064102970 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa063fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa063fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60236/0 (socket says 172.21.15.103:60236) 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa063fff700 1 -- 172.21.15.103:0/527951111 learned_addr learned my addr 172.21.15.103:0/527951111 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa063fff700 1 -- 172.21.15.103:0/527951111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580094f0 con 0x7fa064102970 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.107+0000 7fa063fff700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa058011c10 tx=0x7fa058009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.108+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa058010890 con 0x7fa064102970 2024-03-22T02:19:36.120 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.108+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa058010ec0 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.108+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa064111ea0 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.108+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0580243f0 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.108+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa064112a10 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa05802d3f0 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06405f5f0 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa060ff9700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 0x7fa04c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa05804a370 con 0x7fa064102970 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa0637fe700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 0x7fa04c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:36.121 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.109+0000 7fa0637fe700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 0x7fa04c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa054009990 tx=0x7fa054006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.113+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa058018b00 con 0x7fa064102970 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.318+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0640ff490 con 0x7fa04c0367e0 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.322+0000 7fa060ff9700 1 -- 172.21.15.103:0/527951111 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa0640ff490 con 0x7fa04c0367e0 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 msgr2=0x7fa04c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 0x7fa04c038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa054009990 tx=0x7fa054006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 msgr2=0x7fa064111760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa058011c10 tx=0x7fa058009250 comp rx=0 tx=0).stop 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 shutdown_connections 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa04c0367e0 0x7fa04c038c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 --2- 172.21.15.103:0/527951111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa064102970 0x7fa064111760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:36.122 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 >> 172.21.15.103:0/527951111 conn(0x7fa0640fe090 msgr2=0x7fa0640fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:36.123 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 shutdown_connections 2024-03-22T02:19:36.123 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:35.324+0000 7fa06a6ad700 1 -- 172.21.15.103:0/527951111 wait complete. 2024-03-22T02:19:36.123 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying alertmanager service with default placement... 2024-03-22T02:19:36.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:36 smithi103 ceph-mon[29258]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:36.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:36 smithi103 ceph-mon[29258]: Saving service node-exporter spec with placement * 2024-03-22T02:19:36.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:36 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d47cc7700 1 Processor -- start 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d47cc7700 1 -- start start 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d47cc7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d47cc7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401032e0 con 0x7f5d40076ba0 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d45a63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:37.328 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d45a63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60246/0 (socket says 172.21.15.103:60246) 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d45a63700 1 -- 172.21.15.103:0/1539823376 learned_addr learned my addr 172.21.15.103:0/1539823376 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.527+0000 7f5d45a63700 1 -- 172.21.15.103:0/1539823376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40107e30 con 0x7f5d40076ba0 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d45a63700 1 --2- 172.21.15.103:0/1539823376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0098e0 tx=0x7f5d3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9db5760f753775 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d44a61700 1 -- 172.21.15.103:0/1539823376 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3c00b860 con 0x7f5d40076ba0 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d44a61700 1 -- 172.21.15.103:0/1539823376 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d3c00be90 con 0x7f5d40076ba0 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d44a61700 1 -- 172.21.15.103:0/1539823376 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3c0103c0 con 0x7f5d40076ba0 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d47cc7700 1 -- 172.21.15.103:0/1539823376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 msgr2=0x7f5d401078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/1539823376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0098e0 tx=0x7f5d3c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d47cc7700 1 -- 172.21.15.103:0/1539823376 shutdown_connections 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/1539823376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d401078f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:37.329 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.528+0000 7f5d47cc7700 1 -- 172.21.15.103:0/1539823376 >> 172.21.15.103:0/1539823376 conn(0x7f5d400fcf40 msgr2=0x7f5d400ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 -- 172.21.15.103:0/1539823376 shutdown_connections 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 -- 172.21.15.103:0/1539823376 wait complete. 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 Processor -- start 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 -- start start 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d47cc7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40113e50 con 0x7f5d40076ba0 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d45a63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d45a63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60248/0 (socket says 172.21.15.103:60248) 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d45a63700 1 -- 172.21.15.103:0/3366071707 learned_addr learned my addr 172.21.15.103:0/3366071707 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.529+0000 7f5d45a63700 1 -- 172.21.15.103:0/3366071707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0094f0 con 0x7f5d40076ba0 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d45a63700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0094c0 tx=0x7f5d3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:37.330 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3c010890 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d3c010ec0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d40114050 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3c0243f0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.530+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d40114bc0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5d3c02d3f0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4005f5f0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d367fc700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 0x7f5d2c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5d3c04a430 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d45262700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 0x7f5d2c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.531+0000 7f5d45262700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 0x7f5d2c038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009990 tx=0x7f5d30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.535+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5d3c02b8e0 con 0x7f5d40076ba0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.729+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d400fe2d0 con 0x7f5d2c0367e0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.736+0000 7f5d367fc700 1 -- 172.21.15.103:0/3366071707 <== mgr.14120 v2:172.21.15.103:6800/1197413654 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f5d400fe2d0 con 0x7f5d2c0367e0 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 msgr2=0x7f5d2c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:37.331 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 0x7f5d2c038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009990 tx=0x7f5d30006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 msgr2=0x7f5d40113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0094c0 tx=0x7f5d3c009250 comp rx=0 tx=0).stop 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 shutdown_connections 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f5d2c0367e0 0x7f5d2c038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 --2- 172.21.15.103:0/3366071707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d40076ba0 0x7f5d40113910 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 >> 172.21.15.103:0/3366071707 conn(0x7f5d400fcf40 msgr2=0x7f5d400fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 shutdown_connections 2024-03-22T02:19:37.332 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:36.738+0000 7f5d47cc7700 1 -- 172.21.15.103:0/3366071707 wait complete. 2024-03-22T02:19:37.996 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:37 smithi103 ceph-mon[29258]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:37.996 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:37 smithi103 ceph-mon[29258]: Saving service alertmanager spec with placement count:1 2024-03-22T02:19:37.996 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:37 smithi103 ceph-mon[29258]: from='mgr.14120 172.21.15.103:0/1183539972' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:37.996 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:37 smithi103 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:19:38.609 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.802+0000 7f418e182700 1 Processor -- start 2024-03-22T02:19:38.653 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.802+0000 7f418e182700 1 -- start start 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f418e182700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f418e182700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188071660 con 0x7f4188071250 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f41877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f41877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60262/0 (socket says 172.21.15.103:60262) 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f41877fe700 1 -- 172.21.15.103:0/2260502250 learned_addr learned my addr 172.21.15.103:0/2260502250 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f41877fe700 1 -- 172.21.15.103:0/2260502250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188103330 con 0x7f4188071250 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f41877fe700 1 --2- 172.21.15.103:0/2260502250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41780098e0 tx=0x7f4178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbef02806161ecbb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f4186ffd700 1 -- 172.21.15.103:0/2260502250 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417800b860 con 0x7f4188071250 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f4186ffd700 1 -- 172.21.15.103:0/2260502250 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f417800be90 con 0x7f4188071250 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.803+0000 7f4186ffd700 1 -- 172.21.15.103:0/2260502250 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41780103c0 con 0x7f4188071250 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 -- 172.21.15.103:0/2260502250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 msgr2=0x7f4188107940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:38.654 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 --2- 172.21.15.103:0/2260502250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41780098e0 tx=0x7f4178009250 comp rx=0 tx=0).stop 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 -- 172.21.15.103:0/2260502250 shutdown_connections 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 --2- 172.21.15.103:0/2260502250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188107940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 -- 172.21.15.103:0/2260502250 >> 172.21.15.103:0/2260502250 conn(0x7f41880fcfb0 msgr2=0x7f41880ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 -- 172.21.15.103:0/2260502250 shutdown_connections 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.804+0000 7f418e182700 1 -- 172.21.15.103:0/2260502250 wait complete. 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f418e182700 1 Processor -- start 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f418e182700 1 -- start start 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f418e182700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f418e182700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188113e40 con 0x7f4188071250 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f41877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f41877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60264/0 (socket says 172.21.15.103:60264) 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f41877fe700 1 -- 172.21.15.103:0/1909123648 learned_addr learned my addr 172.21.15.103:0/1909123648 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f41877fe700 1 -- 172.21.15.103:0/1909123648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41780094f0 con 0x7f4188071250 2024-03-22T02:19:38.655 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.805+0000 7f41877fe700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41780105b0 tx=0x7f4178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.806+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4178010840 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.806+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4178010e70 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.806+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41881140a0 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.806+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417801ae30 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.806+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4188114bb0 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f417802d3f0 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418805f5f0 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f4184ff9700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 0x7f4174038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f417804a270 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f417effd700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 0x7f4174038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.807+0000 7f417effd700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 0x7f4174038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4170009990 tx=0x7f4170006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.812+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4178018d20 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:37.995+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f418804bed0 con 0x7f4188071250 2024-03-22T02:19:38.656 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.001+0000 7f4184ff9700 1 -- 172.21.15.103:0/1909123648 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4178018b40 con 0x7f4188071250 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 msgr2=0x7f4174038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 0x7f4174038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4170009990 tx=0x7f4170006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 msgr2=0x7f4188113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41780105b0 tx=0x7f4178009250 comp rx=0 tx=0).stop 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 shutdown_connections 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f41740367e0 0x7f4174038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 --2- 172.21.15.103:0/1909123648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4188071250 0x7f4188113900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 >> 172.21.15.103:0/1909123648 conn(0x7f41880fcfb0 msgr2=0x7f41880fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 shutdown_connections 2024-03-22T02:19:38.657 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:38.003+0000 7f418e182700 1 -- 172.21.15.103:0/1909123648 wait complete. 2024-03-22T02:19:39.268 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:39 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1909123648' entity='client.admin' 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950eacb700 1 Processor -- start 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950eacb700 1 -- start start 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950eacb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950eacb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508103270 con 0x7f9508102920 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950c867700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950c867700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60266/0 (socket says 172.21.15.103:60266) 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950c867700 1 -- 172.21.15.103:0/267911168 learned_addr learned my addr 172.21.15.103:0/267911168 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950c867700 1 -- 172.21.15.103:0/267911168 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95081033b0 con 0x7f9508102920 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.076+0000 7f950c867700 1 --2- 172.21.15.103:0/267911168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0098e0 tx=0x7f94fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aec252adf4985c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f95077fe700 1 -- 172.21.15.103:0/267911168 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc00b860 con 0x7f9508102920 2024-03-22T02:19:39.907 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f95077fe700 1 -- 172.21.15.103:0/267911168 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94fc00be90 con 0x7f9508102920 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f95077fe700 1 -- 172.21.15.103:0/267911168 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc0103c0 con 0x7f9508102920 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 -- 172.21.15.103:0/267911168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 msgr2=0x7f9508102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 --2- 172.21.15.103:0/267911168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0098e0 tx=0x7f94fc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 -- 172.21.15.103:0/267911168 shutdown_connections 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 --2- 172.21.15.103:0/267911168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508102d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 -- 172.21.15.103:0/267911168 >> 172.21.15.103:0/267911168 conn(0x7f95080fe0a0 msgr2=0x7f95081004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 -- 172.21.15.103:0/267911168 shutdown_connections 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.077+0000 7f950eacb700 1 -- 172.21.15.103:0/267911168 wait complete. 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950eacb700 1 Processor -- start 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950eacb700 1 -- start start 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950eacb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950eacb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508111c40 con 0x7f9508102920 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950c867700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:39.908 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950c867700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60276/0 (socket says 172.21.15.103:60276) 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.078+0000 7f950c867700 1 -- 172.21.15.103:0/3469989503 learned_addr learned my addr 172.21.15.103:0/3469989503 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f950c867700 1 -- 172.21.15.103:0/3469989503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc0094f0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f950c867700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94fc011680 tx=0x7f94fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc010890 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94fc010ec0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94fc0243f0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9508111e40 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.079+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95081129b0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f94fc02d3f0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f95057fa700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 0x7f94f8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950805f5f0 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f94fc04a370 con 0x7f9508102920 2024-03-22T02:19:39.909 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f9507fff700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 0x7f94f8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.080+0000 7f9507fff700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 0x7f94f8038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009990 tx=0x7f94f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.085+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f94fc018d20 con 0x7f9508102920 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.268+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f950804bed0 con 0x7f9508102920 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.273+0000 7f95057fa700 1 -- 172.21.15.103:0/3469989503 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f94fc006d70 con 0x7f9508102920 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 msgr2=0x7f94f8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 0x7f94f8038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009990 tx=0x7f94f4006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 msgr2=0x7f9508111700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f94fc011680 tx=0x7f94fc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 shutdown_connections 2024-03-22T02:19:39.910 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f94f80367e0 0x7f94f8038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 --2- 172.21.15.103:0/3469989503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9508102920 0x7f9508111700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 >> 172.21.15.103:0/3469989503 conn(0x7f95080fe0a0 msgr2=0x7f95080fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 shutdown_connections 2024-03-22T02:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:39.276+0000 7f950eacb700 1 -- 172.21.15.103:0/3469989503 wait complete. 2024-03-22T02:19:39.911 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling the dashboard module... 2024-03-22T02:19:40.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:40 smithi103 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:19:40.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:40 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3469989503' entity='client.admin' 2024-03-22T02:19:41.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:41 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/4293680280' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-22T02:19:41.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.286+0000 7fa339e9a700 1 Processor -- start 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa339e9a700 1 -- start start 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa339e9a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa339e9a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334101680 con 0x7fa334100d30 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60288/0 (socket says 172.21.15.103:60288) 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa3337fe700 1 -- 172.21.15.103:0/4241607672 learned_addr learned my addr 172.21.15.103:0/4241607672 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa3337fe700 1 -- 172.21.15.103:0/4241607672 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3341017c0 con 0x7fa334100d30 2024-03-22T02:19:41.965 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.287+0000 7fa3337fe700 1 --2- 172.21.15.103:0/4241607672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa3240098e0 tx=0x7fa324009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51142a4cc17b7e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa3327fc700 1 -- 172.21.15.103:0/4241607672 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa32400b860 con 0x7fa334100d30 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa3327fc700 1 -- 172.21.15.103:0/4241607672 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa32400be90 con 0x7fa334100d30 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa3327fc700 1 -- 172.21.15.103:0/4241607672 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3240103c0 con 0x7fa334100d30 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 -- 172.21.15.103:0/4241607672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 msgr2=0x7fa334101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4241607672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa3240098e0 tx=0x7fa324009250 comp rx=0 tx=0).stop 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 -- 172.21.15.103:0/4241607672 shutdown_connections 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4241607672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334101140 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 -- 172.21.15.103:0/4241607672 >> 172.21.15.103:0/4241607672 conn(0x7fa3340fc730 msgr2=0x7fa3340feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.288+0000 7fa339e9a700 1 -- 172.21.15.103:0/4241607672 shutdown_connections 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa339e9a700 1 -- 172.21.15.103:0/4241607672 wait complete. 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa339e9a700 1 Processor -- start 2024-03-22T02:19:41.966 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa339e9a700 1 -- start start 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa339e9a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa339e9a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334111c90 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60292/0 (socket says 172.21.15.103:60292) 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.289+0000 7fa3337fe700 1 -- 172.21.15.103:0/4293680280 learned_addr learned my addr 172.21.15.103:0/4293680280 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa3337fe700 1 -- 172.21.15.103:0/4293680280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3240094f0 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa3337fe700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa324011c10 tx=0x7fa324009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa324010890 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa324010ec0 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa334111e90 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3240243f0 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.290+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa334112a00 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.291+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa32401a430 con 0x7fa334100d30 2024-03-22T02:19:41.967 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.291+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33405f5f0 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.291+0000 7fa31bfff700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 0x7fa31c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.291+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa32404a230 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.292+0000 7fa332ffd700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 0x7fa31c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.292+0000 7fa332ffd700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 0x7fa31c038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa328009990 tx=0x7fa328006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.296+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa32402b8c0 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:40.543+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa33404bed0 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.277+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa32402d3f0 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.278+0000 7fa31bfff700 1 -- 172.21.15.103:0/4293680280 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa32400f970 con 0x7fa334100d30 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 msgr2=0x7fa31c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 0x7fa31c038c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa328009990 tx=0x7fa328006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 msgr2=0x7fa334111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:41.968 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa324011c10 tx=0x7fa324009250 comp rx=0 tx=0).stop 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 shutdown_connections 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fa31c036790 0x7fa31c038c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 --2- 172.21.15.103:0/4293680280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa334100d30 0x7fa334111750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.280+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 >> 172.21.15.103:0/4293680280 conn(0x7fa3340fc730 msgr2=0x7fa3340fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.281+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 shutdown_connections 2024-03-22T02:19:41.969 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:41.281+0000 7fa339e9a700 1 -- 172.21.15.103:0/4293680280 wait complete. 2024-03-22T02:19:42.596 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:42 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/4293680280' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-22T02:19:42.596 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:42 smithi103 ceph-mon[29258]: mgrmap e9: smithi103.pfvmui(active, since 24s) 2024-03-22T02:19:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:19:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-22T02:19:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "smithi103.pfvmui", 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.399+0000 7f970879e700 1 Processor -- start 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.399+0000 7f970879e700 1 -- start start 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.399+0000 7f970879e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.399+0000 7f970879e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97000716a0 con 0x7f9700071290 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f970653a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f970653a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60314/0 (socket says 172.21.15.103:60314) 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f970653a700 1 -- 172.21.15.103:0/4014199083 learned_addr learned my addr 172.21.15.103:0/4014199083 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f970653a700 1 -- 172.21.15.103:0/4014199083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700105a40 con 0x7f9700071290 2024-03-22T02:19:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f970653a700 1 --2- 172.21.15.103:0/4014199083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f96f00098e0 tx=0x7f96f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e2caa0173ec953 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f9705538700 1 -- 172.21.15.103:0/4014199083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f000b860 con 0x7f9700071290 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f9705538700 1 -- 172.21.15.103:0/4014199083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96f000be90 con 0x7f9700071290 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.400+0000 7f9705538700 1 -- 172.21.15.103:0/4014199083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f00103c0 con 0x7f9700071290 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 -- 172.21.15.103:0/4014199083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 msgr2=0x7f9700105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 --2- 172.21.15.103:0/4014199083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f96f00098e0 tx=0x7f96f0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 -- 172.21.15.103:0/4014199083 shutdown_connections 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 --2- 172.21.15.103:0/4014199083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700105500 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 -- 172.21.15.103:0/4014199083 >> 172.21.15.103:0/4014199083 conn(0x7f97000fcfb0 msgr2=0x7f97000ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 -- 172.21.15.103:0/4014199083 shutdown_connections 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.401+0000 7f970879e700 1 -- 172.21.15.103:0/4014199083 wait complete. 2024-03-22T02:19:43.282 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970879e700 1 Processor -- start 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970879e700 1 -- start start 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970879e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970879e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9700111cc0 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970653a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970653a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60326/0 (socket says 172.21.15.103:60326) 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970653a700 1 -- 172.21.15.103:0/329774856 learned_addr learned my addr 172.21.15.103:0/329774856 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.402+0000 7f970653a700 1 -- 172.21.15.103:0/329774856 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00094f0 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f970653a700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96f00105b0 tx=0x7f96f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f0010840 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9700111ec0 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96f0010e70 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f001ae30 con 0x7f9700071290 2024-03-22T02:19:43.283 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.403+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9700112a30 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f96f002d3f0 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970005f5f0 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f96f6ffd700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 0x7f96ec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f96f004a270 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f9705d39700 1 -- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 msgr2=0x7f96ec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.404+0000 7f9705d39700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 0x7f96ec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.409+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f96f002b8e0 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.605+0000 7f9705d39700 1 -- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 msgr2=0x7f96ec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.605+0000 7f9705d39700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 0x7f96ec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.698+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9700112d50 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.698+0000 7f96f6ffd700 1 -- 172.21.15.103:0/329774856 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f96f002b2f0 con 0x7f9700071290 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 msgr2=0x7f96ec038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 0x7f96ec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.284 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 msgr2=0x7f9700111780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96f00105b0 tx=0x7f96f0009250 comp rx=0 tx=0).stop 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 shutdown_connections 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7f96ec0367e0 0x7f96ec038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 --2- 172.21.15.103:0/329774856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9700071290 0x7f9700111780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 >> 172.21.15.103:0/329774856 conn(0x7f97000fcfb0 msgr2=0x7f97000fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 shutdown_connections 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:42.701+0000 7f970879e700 1 -- 172.21.15.103:0/329774856 wait complete. 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-03-22T02:19:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 9... 2024-03-22T02:19:43.450 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:43 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/329774856' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: Active manager daemon smithi103.pfvmui restarted 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: osdmap e3: 0 total, 0 up, 0 in 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: mgrmap e10: smithi103.pfvmui(active, starting, since 0.00329678s) 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:19:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:19:48.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:48.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:47 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:49.197 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:48 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:19:49.198 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:48 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:19:49.198 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:48 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:49.198 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:48 smithi103 ceph-mon[29258]: mgrmap e11: smithi103.pfvmui(active, since 1.00632s) 2024-03-22T02:19:49.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:19:49.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0de6a5700 1 Processor -- start 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0de6a5700 1 -- start start 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0de6a5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0de6a5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80715c0 con 0x7fb0d80711b0 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60332/0 (socket says 172.21.15.103:60332) 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.697+0000 7fb0d7fff700 1 -- 172.21.15.103:0/3578558296 learned_addr learned my addr 172.21.15.103:0/3578558296 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.698+0000 7fb0d7fff700 1 -- 172.21.15.103:0/3578558296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8105b20 con 0x7fb0d80711b0 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.698+0000 7fb0d7fff700 1 --2- 172.21.15.103:0/3578558296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0098e0 tx=0x7fb0cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31499b9669e5b2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.698+0000 7fb0d6ffd700 1 -- 172.21.15.103:0/3578558296 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc00b860 con 0x7fb0d80711b0 2024-03-22T02:19:49.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.698+0000 7fb0d6ffd700 1 -- 172.21.15.103:0/3578558296 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0cc00be90 con 0x7fb0d80711b0 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.698+0000 7fb0d6ffd700 1 -- 172.21.15.103:0/3578558296 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc0103c0 con 0x7fb0d80711b0 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 -- 172.21.15.103:0/3578558296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 msgr2=0x7fb0d81055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/3578558296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0098e0 tx=0x7fb0cc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 -- 172.21.15.103:0/3578558296 shutdown_connections 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/3578558296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d81055e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 -- 172.21.15.103:0/3578558296 >> 172.21.15.103:0/3578558296 conn(0x7fb0d80fcfb0 msgr2=0x7fb0d80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 -- 172.21.15.103:0/3578558296 shutdown_connections 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 -- 172.21.15.103:0/3578558296 wait complete. 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.699+0000 7fb0de6a5700 1 Processor -- start 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0de6a5700 1 -- start start 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0de6a5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0de6a5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8111ca0 con 0x7fb0d80711b0 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60334/0 (socket says 172.21.15.103:60334) 2024-03-22T02:19:49.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d7fff700 1 -- 172.21.15.103:0/2476271840 learned_addr learned my addr 172.21.15.103:0/2476271840 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d7fff700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc0094f0 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d7fff700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0105b0 tx=0x7fb0cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.700+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc010840 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.701+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8111ea0 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.701+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d8112a10 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.701+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0cc010e70 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.701+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc01ae30 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb0cc0244c0 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d4ff9700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0c00393f0 con 0x7fb0c0036830 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb0cc04b660 con 0x7fb0d80711b0 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d77fe700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.702+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:19:49.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.903+0000 7fb0d77fe700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:43.903+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:44.304+0000 7fb0d77fe700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:44.304+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:45.105+0000 7fb0d77fe700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:45.105+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:46.706+0000 7fb0d77fe700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1197413654 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:46.706+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:47.881+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fb0cc02e440 con 0x7fb0d80711b0 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:47.882+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 msgr2=0x7fb0c0038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:47.882+0000 7fb0d4ff9700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.883+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fb0cc024770 con 0x7fb0d80711b0 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.884+0000 7fb0d4ff9700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 0x7fb0c003c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.884+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0c00393f0 con 0x7fb0c003a070 2024-03-22T02:19:49.486 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.884+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 0x7fb0c003c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.885+0000 7fb0d77fe700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 0x7fb0c003c450 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80039d0 tx=0x7fb0c8009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.885+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb0c00393f0 con 0x7fb0c003a070 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.891+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb0c40025f0 con 0x7fb0c003a070 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.891+0000 7fb0d4ff9700 1 -- 172.21.15.103:0/2476271840 <== mgr.14162 v2:172.21.15.103:6800/2790950462 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb0c40025f0 con 0x7fb0c003a070 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 msgr2=0x7fb0c003c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 0x7fb0c003c450 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80039d0 tx=0x7fb0c8009350 comp rx=0 tx=0).stop 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 msgr2=0x7fb0d8111760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0105b0 tx=0x7fb0cc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 shutdown_connections 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb0c003a070 0x7fb0c003c450 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0d80711b0 0x7fb0d8111760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 --2- 172.21.15.103:0/2476271840 >> [v2:172.21.15.103:6800/1197413654,v1:172.21.15.103:6801/1197413654] conn(0x7fb0c0036830 0x7fb0c0038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 >> 172.21.15.103:0/2476271840 conn(0x7fb0d80fcfb0 msgr2=0x7fb0d80fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 shutdown_connections 2024-03-22T02:19:49.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:48.892+0000 7fb0de6a5700 1 -- 172.21.15.103:0/2476271840 wait complete. 2024-03-22T02:19:49.488 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 9 is available 2024-03-22T02:19:49.488 INFO:teuthology.orchestra.run.smithi103.stdout:Generating a dashboard self-signed certificate... 2024-03-22T02:19:50.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:48] ENGINE Bus STARTING 2024-03-22T02:19:50.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:48] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:19:50.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: [22/Mar/2024:02:19:48] ENGINE Bus STARTED 2024-03-22T02:19:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T02:19:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T02:19:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:49 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.894+0000 7f53d2d9d700 1 Processor -- start 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d2d9d700 1 -- start start 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d2d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d2d9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc101440 con 0x7f53cc100af0 2024-03-22T02:19:50.882 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54240/0 (socket says 172.21.15.103:54240) 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d1d9b700 1 -- 172.21.15.103:0/2108963111 learned_addr learned my addr 172.21.15.103:0/2108963111 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d1d9b700 1 -- 172.21.15.103:0/2108963111 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc101580 con 0x7f53cc100af0 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d1d9b700 1 --2- 172.21.15.103:0/2108963111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0098e0 tx=0x7f53bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d768554ea7ed282 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.895+0000 7f53d0d99700 1 -- 172.21.15.103:0/2108963111 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc00b860 con 0x7f53cc100af0 2024-03-22T02:19:50.883 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d0d99700 1 -- 172.21.15.103:0/2108963111 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53bc00be90 con 0x7f53cc100af0 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d0d99700 1 -- 172.21.15.103:0/2108963111 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc0103c0 con 0x7f53cc100af0 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 -- 172.21.15.103:0/2108963111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 msgr2=0x7f53cc100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/2108963111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0098e0 tx=0x7f53bc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 -- 172.21.15.103:0/2108963111 shutdown_connections 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/2108963111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc100f00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 -- 172.21.15.103:0/2108963111 >> 172.21.15.103:0/2108963111 conn(0x7f53cc0fc570 msgr2=0x7f53cc0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 -- 172.21.15.103:0/2108963111 shutdown_connections 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.896+0000 7f53d2d9d700 1 -- 172.21.15.103:0/2108963111 wait complete. 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d2d9d700 1 Processor -- start 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d2d9d700 1 -- start start 2024-03-22T02:19:50.884 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d2d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d2d9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc111a30 con 0x7f53cc100af0 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54242/0 (socket says 172.21.15.103:54242) 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d1d9b700 1 -- 172.21.15.103:0/3544453525 learned_addr learned my addr 172.21.15.103:0/3544453525 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d1d9b700 1 -- 172.21.15.103:0/3544453525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0094f0 con 0x7f53cc100af0 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.897+0000 7f53d1d9b700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f53bc011680 tx=0x7f53bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc010890 con 0x7f53cc100af0 2024-03-22T02:19:50.885 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53bc010ec0 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc111c30 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc0243f0 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc1127a0 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.898+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f53bc0109f0 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.899+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc05f5f0 con 0x7f53cc100af0 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.899+0000 7f53c27fc700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 0x7f53b8038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:50.886 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.899+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f53bc04a1b0 con 0x7f53cc100af0 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.899+0000 7f53d159a700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 0x7f53b8038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.899+0000 7f53d159a700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 0x7f53b8038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009990 tx=0x7f53c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:49.904+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f53bc02b940 con 0x7f53cc100af0 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.134+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f53cc0fd890 con 0x7f53b8036710 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.209+0000 7f53c27fc700 1 -- 172.21.15.103:0/3544453525 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f53cc0fd890 con 0x7f53b8036710 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.211+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 msgr2=0x7f53b8038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.211+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 0x7f53b8038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009990 tx=0x7f53c8006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.211+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 msgr2=0x7f53cc1114f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.211+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f53bc011680 tx=0x7f53bc009250 comp rx=0 tx=0).stop 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.211+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 shutdown_connections 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.212+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f53b8036710 0x7f53b8038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.212+0000 7f53d2d9d700 1 --2- 172.21.15.103:0/3544453525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53cc100af0 0x7f53cc1114f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.212+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 >> 172.21.15.103:0/3544453525 conn(0x7f53cc0fc570 msgr2=0x7f53cc0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:50.888 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.212+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 shutdown_connections 2024-03-22T02:19:50.888 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:50.212+0000 7f53d2d9d700 1 -- 172.21.15.103:0/3544453525 wait complete. 2024-03-22T02:19:50.888 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial admin user... 2024-03-22T02:19:51.474 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:51 smithi103 ceph-mon[29258]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:51.474 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:51 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:51.475 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:51 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:51.475 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:51 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:51.475 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:51 smithi103 ceph-mon[29258]: mgrmap e12: smithi103.pfvmui(active, since 2s) 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$GpKkkSG.gOyqjcZPdGL0Y.NTlofLTBPyMsUGYjq6WLjZgi9F6XP/.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711073991, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.350+0000 7f4d2ad48700 1 Processor -- start 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.350+0000 7f4d2ad48700 1 -- start start 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d2ad48700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d2ad48700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24079080 con 0x7f4d24078c70 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d28ae4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d28ae4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54258/0 (socket says 172.21.15.103:54258) 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d28ae4700 1 -- 172.21.15.103:0/2955239218 learned_addr learned my addr 172.21.15.103:0/2955239218 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.351+0000 7f4d28ae4700 1 -- 172.21.15.103:0/2955239218 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240777b0 con 0x7f4d24078c70 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.352+0000 7f4d28ae4700 1 --2- 172.21.15.103:0/2955239218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d180098e0 tx=0x7f4d18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1a693e8f7fd12e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.352+0000 7f4d237fe700 1 -- 172.21.15.103:0/2955239218 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d1800b860 con 0x7f4d24078c70 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.352+0000 7f4d237fe700 1 -- 172.21.15.103:0/2955239218 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d1800be90 con 0x7f4d24078c70 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.352+0000 7f4d237fe700 1 -- 172.21.15.103:0/2955239218 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d180103c0 con 0x7f4d24078c70 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 -- 172.21.15.103:0/2955239218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 msgr2=0x7f4d24077270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/2955239218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d180098e0 tx=0x7f4d18009250 comp rx=0 tx=0).stop 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 -- 172.21.15.103:0/2955239218 shutdown_connections 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/2955239218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d24077270 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 -- 172.21.15.103:0/2955239218 >> 172.21.15.103:0/2955239218 conn(0x7f4d240fe040 msgr2=0x7f4d24100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 -- 172.21.15.103:0/2955239218 shutdown_connections 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 -- 172.21.15.103:0/2955239218 wait complete. 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.353+0000 7f4d2ad48700 1 Processor -- start 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d2ad48700 1 -- start start 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d2ad48700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d2ad48700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24113e30 con 0x7f4d24078c70 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d28ae4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d28ae4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54264/0 (socket says 172.21.15.103:54264) 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d28ae4700 1 -- 172.21.15.103:0/3130745009 learned_addr learned my addr 172.21.15.103:0/3130745009 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.354+0000 7f4d28ae4700 1 -- 172.21.15.103:0/3130745009 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180094f0 con 0x7f4d24078c70 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d28ae4700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d180094c0 tx=0x7f4d18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d18010890 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d24114030 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d24114ba0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d18010ec0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.355+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d180243f0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.356+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f4d1802d3f0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.356+0000 7f4d217fa700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 0x7f4d14038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.356+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4d1804b3d0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.356+0000 7f4d23fff700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 0x7f4d14038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.356+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d080052e0 con 0x7f4d24078c70 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.357+0000 7f4d23fff700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 0x7f4d14038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009990 tx=0x7f4d10006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.361+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4d1802b8e0 con 0x7f4d24078c70 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.569+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4d08002430 con 0x7f4d14036830 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.793+0000 7f4d217fa700 1 -- 172.21.15.103:0/3130745009 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f4d08002430 con 0x7f4d14036830 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 msgr2=0x7f4d14038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 0x7f4d14038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009990 tx=0x7f4d10006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 msgr2=0x7f4d241138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d180094c0 tx=0x7f4d18009250 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 shutdown_connections 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f4d14036830 0x7f4d14038ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 --2- 172.21.15.103:0/3130745009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d24078c70 0x7f4d241138f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 >> 172.21.15.103:0/3130745009 conn(0x7f4d240fe040 msgr2=0x7f4d240fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 shutdown_connections 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:51.795+0000 7f4d2ad48700 1 -- 172.21.15.103:0/3130745009 wait complete. 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi103.stdout:Fetching dashboard port number... 2024-03-22T02:19:53.107 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:52 smithi103 ceph-mon[29258]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:19:53.108 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:52 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:19:54.018 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:53 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1979101355' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-22T02:19:54.026 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 8443 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03de9f5700 1 Processor -- start 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03de9f5700 1 -- start start 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03de9f5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03de9f5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d81015d0 con 0x7f03d8100c80 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54268/0 (socket says 172.21.15.103:54268) 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03d7fff700 1 -- 172.21.15.103:0/89196969 learned_addr learned my addr 172.21.15.103:0/89196969 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.902+0000 7f03d7fff700 1 -- 172.21.15.103:0/89196969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8101710 con 0x7f03d8100c80 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03d7fff700 1 --2- 172.21.15.103:0/89196969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03c80098e0 tx=0x7f03c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6ca925bdf7b3be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03d6ffd700 1 -- 172.21.15.103:0/89196969 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c800b860 con 0x7f03d8100c80 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03d6ffd700 1 -- 172.21.15.103:0/89196969 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03c800be90 con 0x7f03d8100c80 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03d6ffd700 1 -- 172.21.15.103:0/89196969 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c80103c0 con 0x7f03d8100c80 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 -- 172.21.15.103:0/89196969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 msgr2=0x7f03d8101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 --2- 172.21.15.103:0/89196969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03c80098e0 tx=0x7f03c8009250 comp rx=0 tx=0).stop 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 -- 172.21.15.103:0/89196969 shutdown_connections 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 --2- 172.21.15.103:0/89196969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d8101090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 -- 172.21.15.103:0/89196969 >> 172.21.15.103:0/89196969 conn(0x7f03d80fc700 msgr2=0x7f03d80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 -- 172.21.15.103:0/89196969 shutdown_connections 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.903+0000 7f03de9f5700 1 -- 172.21.15.103:0/89196969 wait complete. 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03de9f5700 1 Processor -- start 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03de9f5700 1 -- start start 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03de9f5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03de9f5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8111c10 con 0x7f03d8100c80 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03d7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54278/0 (socket says 172.21.15.103:54278) 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.904+0000 7f03d7fff700 1 -- 172.21.15.103:0/1979101355 learned_addr learned my addr 172.21.15.103:0/1979101355 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03d7fff700 1 -- 172.21.15.103:0/1979101355 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80094f0 con 0x7f03d8100c80 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03d7fff700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f03c8011680 tx=0x7f03c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.029 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c8010890 con 0x7f03d8100c80 2024-03-22T02:19:54.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03c8010ec0 con 0x7f03d8100c80 2024-03-22T02:19:54.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d8111e10 con 0x7f03d8100c80 2024-03-22T02:19:54.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03c80243f0 con 0x7f03d8100c80 2024-03-22T02:19:54.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.905+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d8112980 con 0x7f03d8100c80 2024-03-22T02:19:54.030 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f03c802d3f0 con 0x7f03d8100c80 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d805f5f0 con 0x7f03d8100c80 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03d4ff9700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 0x7f03c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f03c804a4e0 con 0x7f03d8100c80 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03d77fe700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 0x7f03c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.906+0000 7f03d77fe700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 0x7f03c0038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009990 tx=0x7f03cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.031 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:52.911+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f03c802b8e0 con 0x7f03d8100c80 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.106+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f03d804bed0 con 0x7f03d8100c80 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.107+0000 7f03d4ff9700 1 -- 172.21.15.103:0/1979101355 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f03c802c350 con 0x7f03d8100c80 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 msgr2=0x7f03c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 0x7f03c0038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009990 tx=0x7f03cc006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 msgr2=0x7f03d81116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f03c8011680 tx=0x7f03c8009250 comp rx=0 tx=0).stop 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 shutdown_connections 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f03c00367e0 0x7f03c0038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.032 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 --2- 172.21.15.103:0/1979101355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03d8100c80 0x7f03d81116d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 >> 172.21.15.103:0/1979101355 conn(0x7f03d80fc700 msgr2=0x7f03d80fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 shutdown_connections 2024-03-22T02:19:54.033 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:53.109+0000 7f03de9f5700 1 -- 172.21.15.103:0/1979101355 wait complete. 2024-03-22T02:19:54.033 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:19:54.033 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-03-22T02:19:54.041 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:19:54.041 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-03-22T02:19:54.041 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-03-22T02:19:54.041 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph Dashboard is now available at: 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout: URL: https://smithi103.front.sepia.ceph.com:8443/ 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout: User: admin 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout: Password: nhh6yjuduq 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:54.043 INFO:teuthology.orchestra.run.smithi103.stdout:Saving cluster configuration to /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config directory 2024-03-22T02:19:54.044 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling autotune for osd_memory_target 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c858a700 1 Processor -- start 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c858a700 1 -- start start 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c858a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c858a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c01020f0 con 0x7fa8c01017a0 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c6326700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c6326700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40314/0 (socket says 172.21.15.103:40314) 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c6326700 1 -- 172.21.15.103:0/3769011259 learned_addr learned my addr 172.21.15.103:0/3769011259 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.476+0000 7fa8c6326700 1 -- 172.21.15.103:0/3769011259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0102230 con 0x7fa8c01017a0 2024-03-22T02:19:55.274 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c6326700 1 --2- 172.21.15.103:0/3769011259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3ac0c8cfa5b468 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c5324700 1 -- 172.21.15.103:0/3769011259 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b80092e0 con 0x7fa8c01017a0 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c5324700 1 -- 172.21.15.103:0/3769011259 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8b8012430 con 0x7fa8c01017a0 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c5324700 1 -- 172.21.15.103:0/3769011259 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b80075a0 con 0x7fa8c01017a0 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 -- 172.21.15.103:0/3769011259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 msgr2=0x7fa8c0101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 --2- 172.21.15.103:0/3769011259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e50 tx=0x7fa8b8009840 comp rx=0 tx=0).stop 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 -- 172.21.15.103:0/3769011259 shutdown_connections 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 --2- 172.21.15.103:0/3769011259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0101bb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 -- 172.21.15.103:0/3769011259 >> 172.21.15.103:0/3769011259 conn(0x7fa8c00fcf40 msgr2=0x7fa8c00ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 -- 172.21.15.103:0/3769011259 shutdown_connections 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.477+0000 7fa8c858a700 1 -- 172.21.15.103:0/3769011259 wait complete. 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c858a700 1 Processor -- start 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c858a700 1 -- start start 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c858a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c858a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c0107f50 con 0x7fa8c01017a0 2024-03-22T02:19:55.275 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c6326700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c6326700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40318/0 (socket says 172.21.15.103:40318) 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c6326700 1 -- 172.21.15.103:0/2533538709 learned_addr learned my addr 172.21.15.103:0/2533538709 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.478+0000 7fa8c6326700 1 -- 172.21.15.103:0/2533538709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009ae0 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8c6326700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80127d0 tx=0x7fa8b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b80092e0 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8b8007d30 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c0105fc0 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8025960 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.479+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c01064b0 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa8b8019050 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c004bed0 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8beffd700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 0x7fa8a8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa8b804a220 con 0x7fa8c01017a0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8c5b25700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 0x7fa8a8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:55.276 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.480+0000 7fa8c5b25700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 0x7fa8a8038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009990 tx=0x7fa8b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.485+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa8b8011570 con 0x7fa8c01017a0 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.669+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa8c0103540 con 0x7fa8c01017a0 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.669+0000 7fa8beffd700 1 -- 172.21.15.103:0/2533538709 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa8b8020070 con 0x7fa8c01017a0 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 msgr2=0x7fa8a8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 0x7fa8a8038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009990 tx=0x7fa8b4006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 msgr2=0x7fa8c0107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80127d0 tx=0x7fa8b8009840 comp rx=0 tx=0).stop 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 shutdown_connections 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fa8a80367e0 0x7fa8a8038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 --2- 172.21.15.103:0/2533538709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8c01017a0 0x7fa8c0107a10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 >> 172.21.15.103:0/2533538709 conn(0x7fa8c00fcf40 msgr2=0x7fa8c00fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 shutdown_connections 2024-03-22T02:19:55.277 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:54.671+0000 7fa8c858a700 1 -- 172.21.15.103:0/2533538709 wait complete. 2024-03-22T02:19:55.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:55 smithi103 ceph-mon[29258]: mgrmap e13: smithi103.pfvmui(active, since 6s) 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.692+0000 7f282d213700 1 Processor -- start 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.692+0000 7f282d213700 1 -- start start 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.692+0000 7f282d213700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.692+0000 7f282d213700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828105a40 con 0x7f28280712a0 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40334/0 (socket says 172.21.15.103:40334) 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2826d9d700 1 -- 172.21.15.103:0/1721818755 learned_addr learned my addr 172.21.15.103:0/1721818755 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2826d9d700 1 -- 172.21.15.103:0/1721818755 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828105b80 con 0x7f28280712a0 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2826d9d700 1 --2- 172.21.15.103:0/1721818755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f28180098e0 tx=0x7f2818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c0e11bd4f2b9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:56.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2825d9b700 1 -- 172.21.15.103:0/1721818755 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281800b860 con 0x7f28280712a0 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2825d9b700 1 -- 172.21.15.103:0/1721818755 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f281800be90 con 0x7f28280712a0 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.693+0000 7f2825d9b700 1 -- 172.21.15.103:0/1721818755 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28180103c0 con 0x7f28280712a0 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- 172.21.15.103:0/1721818755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 msgr2=0x7f2828105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 --2- 172.21.15.103:0/1721818755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f28180098e0 tx=0x7f2818009250 comp rx=0 tx=0).stop 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- 172.21.15.103:0/1721818755 shutdown_connections 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 --2- 172.21.15.103:0/1721818755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828105500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- 172.21.15.103:0/1721818755 >> 172.21.15.103:0/1721818755 conn(0x7f28280fcfb0 msgr2=0x7f28280ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- 172.21.15.103:0/1721818755 shutdown_connections 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- 172.21.15.103:0/1721818755 wait complete. 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 Processor -- start 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.694+0000 7f282d213700 1 -- start start 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f282d213700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:56.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f282d213700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828111cb0 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f2826d9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40350/0 (socket says 172.21.15.103:40350) 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f2826d9d700 1 -- 172.21.15.103:0/1764302644 learned_addr learned my addr 172.21.15.103:0/1764302644 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f2826d9d700 1 -- 172.21.15.103:0/1764302644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180094f0 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f2826d9d700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28180094c0 tx=0x7f2818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28180118f0 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2818010d60 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2828111eb0 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f281801a430 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.695+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2828112a20 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.696+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f281801a590 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.696+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282805f5f0 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.696+0000 7f280f7fe700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 0x7f2810038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.696+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f281804a910 con 0x7f28280712a0 2024-03-22T02:19:56.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.696+0000 7f282659c700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 0x7f2810038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.697+0000 7f282659c700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 0x7f2810038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f281c009990 tx=0x7f281c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.701+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f281802d3f0 con 0x7f28280712a0 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.990+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f282804bed0 con 0x7f28280712a0 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.995+0000 7f280f7fe700 1 -- 172.21.15.103:0/1764302644 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f2818018350 con 0x7f28280712a0 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 msgr2=0x7f2810038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 0x7f2810038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f281c009990 tx=0x7f281c006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 msgr2=0x7f2828111770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28180094c0 tx=0x7f2818009250 comp rx=0 tx=0).stop 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 shutdown_connections 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f2810036880 0x7f2810038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 --2- 172.21.15.103:0/1764302644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28280712a0 0x7f2828111770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 >> 172.21.15.103:0/1764302644 conn(0x7f28280fcfb0 msgr2=0x7f28280fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 shutdown_connections 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-03-22T02:19:55.997+0000 7f282d213700 1 -- 172.21.15.103:0/1764302644 wait complete. 2024-03-22T02:19:56.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout:Or, if you are only running a single cluster on this host: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: ceph telemetry on 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout:For more information see: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-22T02:19:56.577 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:19:56.578 INFO:teuthology.orchestra.run.smithi103.stdout:Bootstrap complete. 2024-03-22T02:19:56.610 INFO:tasks.cephadm:Fetching config... 2024-03-22T02:19:56.610 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:19:56.610 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-22T02:19:56.666 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-22T02:19:56.666 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:19:56.666 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-22T02:19:56.721 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-22T02:19:56.721 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:19:56.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/keyring of=/dev/stdout 2024-03-22T02:19:56.789 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-22T02:19:56.789 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:19:56.789 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-22T02:19:56.843 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-22T02:19:56.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVwv70uezTxr7yb7DSbnmiDoWU6urHN2UNp1+MpbYPnYKweUVn+/ssZz+Bh6gJS2b+ZcdWzTUcLJKEikNNU+3nvMIHToTsyoaUHTisiJPQZeiuzI0s0tEtwktsnT7xi1LN9/rJFZ+/uIJGtZVuzbWYRrBiPM44D7snnWk5nsllQo4dCrqs4Qen3zinTvrru897fysCLpF+t2UWV6de0EXmia57/TsIBmlyor000jsKyv/+nV6d7yQBXvp9xmSxD3Ni1F+a4jCBzBtr65y7A+6xYyt4R3QwBo3lCSxF3+TuAmp5Ywg7lwMSka3Ihq7hgqbaZCIcZLUMGH/wc5AQUTFqukTjkGGbIP16/LlRJBQSIHk1KWft3elBfSdC79X5V6cgUz4UsxF4GUT5sEkNRZ1ihM08aJrrQGNerkzWgTRkihVcIJYkw06e4+oZccD8PkxaEqmIkefwTYD/mJkqqudqSP390iPZZ1eyPkACxX7sg7WkKUSskMqkUUsMRgnRgNU= ceph-67659b4a-e7f2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T02:19:56.924 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVwv70uezTxr7yb7DSbnmiDoWU6urHN2UNp1+MpbYPnYKweUVn+/ssZz+Bh6gJS2b+ZcdWzTUcLJKEikNNU+3nvMIHToTsyoaUHTisiJPQZeiuzI0s0tEtwktsnT7xi1LN9/rJFZ+/uIJGtZVuzbWYRrBiPM44D7snnWk5nsllQo4dCrqs4Qen3zinTvrru897fysCLpF+t2UWV6de0EXmia57/TsIBmlyor000jsKyv/+nV6d7yQBXvp9xmSxD3Ni1F+a4jCBzBtr65y7A+6xYyt4R3QwBo3lCSxF3+TuAmp5Ywg7lwMSka3Ihq7hgqbaZCIcZLUMGH/wc5AQUTFqukTjkGGbIP16/LlRJBQSIHk1KWft3elBfSdC79X5V6cgUz4UsxF4GUT5sEkNRZ1ihM08aJrrQGNerkzWgTRkihVcIJYkw06e4+oZccD8PkxaEqmIkefwTYD/mJkqqudqSP390iPZZ1eyPkACxX7sg7WkKUSskMqkUUsMRgnRgNU= ceph-67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:19:56.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVwv70uezTxr7yb7DSbnmiDoWU6urHN2UNp1+MpbYPnYKweUVn+/ssZz+Bh6gJS2b+ZcdWzTUcLJKEikNNU+3nvMIHToTsyoaUHTisiJPQZeiuzI0s0tEtwktsnT7xi1LN9/rJFZ+/uIJGtZVuzbWYRrBiPM44D7snnWk5nsllQo4dCrqs4Qen3zinTvrru897fysCLpF+t2UWV6de0EXmia57/TsIBmlyor000jsKyv/+nV6d7yQBXvp9xmSxD3Ni1F+a4jCBzBtr65y7A+6xYyt4R3QwBo3lCSxF3+TuAmp5Ywg7lwMSka3Ihq7hgqbaZCIcZLUMGH/wc5AQUTFqukTjkGGbIP16/LlRJBQSIHk1KWft3elBfSdC79X5V6cgUz4UsxF4GUT5sEkNRZ1ihM08aJrrQGNerkzWgTRkihVcIJYkw06e4+oZccD8PkxaEqmIkefwTYD/mJkqqudqSP390iPZZ1eyPkACxX7sg7WkKUSskMqkUUsMRgnRgNU= ceph-67659b4a-e7f2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T02:19:56.987 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVwv70uezTxr7yb7DSbnmiDoWU6urHN2UNp1+MpbYPnYKweUVn+/ssZz+Bh6gJS2b+ZcdWzTUcLJKEikNNU+3nvMIHToTsyoaUHTisiJPQZeiuzI0s0tEtwktsnT7xi1LN9/rJFZ+/uIJGtZVuzbWYRrBiPM44D7snnWk5nsllQo4dCrqs4Qen3zinTvrru897fysCLpF+t2UWV6de0EXmia57/TsIBmlyor000jsKyv/+nV6d7yQBXvp9xmSxD3Ni1F+a4jCBzBtr65y7A+6xYyt4R3QwBo3lCSxF3+TuAmp5Ywg7lwMSka3Ihq7hgqbaZCIcZLUMGH/wc5AQUTFqukTjkGGbIP16/LlRJBQSIHk1KWft3elBfSdC79X5V6cgUz4UsxF4GUT5sEkNRZ1ihM08aJrrQGNerkzWgTRkihVcIJYkw06e4+oZccD8PkxaEqmIkefwTYD/mJkqqudqSP390iPZZ1eyPkACxX7sg7WkKUSskMqkUUsMRgnRgNU= ceph-67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:19:57.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-22T02:19:57.267 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:19:57.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:19:56 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1764302644' entity='client.admin' 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 -- 172.21.15.103:0/4291434360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 msgr2=0x7fec580fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/4291434360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec580fd460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0048c0 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 -- 172.21.15.103:0/4291434360 shutdown_connections 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/4291434360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec580fd460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 -- 172.21.15.103:0/4291434360 >> 172.21.15.103:0/4291434360 conn(0x7fec580f70d0 msgr2=0x7fec580f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 -- 172.21.15.103:0/4291434360 shutdown_connections 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 -- 172.21.15.103:0/4291434360 wait complete. 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.603+0000 7fec5dbf4700 1 Processor -- start 2024-03-22T02:19:59.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec5dbf4700 1 -- start start 2024-03-22T02:19:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec5dbf4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec5dbf4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5810c500 con 0x7fec580fd050 2024-03-22T02:19:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec577fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec577fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40358/0 (socket says 172.21.15.103:40358) 2024-03-22T02:19:59.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec577fe700 1 -- 172.21.15.103:0/2671931209 learned_addr learned my addr 172.21.15.103:0/2671931209 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:19:59.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec577fe700 1 -- 172.21.15.103:0/2671931209 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c005040 con 0x7fec580fd050 2024-03-22T02:19:59.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec577fe700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009680 tx=0x7fec4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:59.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec4c00b5c0 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5810c700 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.604+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec4c00be70 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.605+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec4c0135a0 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.605+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5810d270 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.605+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fec4c013810 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.605+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58101e70 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.606+0000 7fec3ffff700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 0x7fec40038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.606+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fec4c049370 con 0x7fec580fd050 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.606+0000 7fec56ffd700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 0x7fec40038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:59.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.606+0000 7fec56ffd700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 0x7fec40038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fec48009990 tx=0x7fec48006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:59.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.610+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fec4c011960 con 0x7fec580fd050 2024-03-22T02:19:59.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.801+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fec5810dc80 con 0x7fec580fd050 2024-03-22T02:19:59.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.804+0000 7fec3ffff700 1 -- 172.21.15.103:0/2671931209 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fec4c021070 con 0x7fec580fd050 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 msgr2=0x7fec40038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 0x7fec40038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fec48009990 tx=0x7fec48006e40 comp rx=0 tx=0).stop 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 msgr2=0x7fec5810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009680 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 shutdown_connections 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fec400368d0 0x7fec40038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 --2- 172.21.15.103:0/2671931209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec580fd050 0x7fec5810bfc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 >> 172.21.15.103:0/2671931209 conn(0x7fec580f70d0 msgr2=0x7fec58102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 shutdown_connections 2024-03-22T02:19:59.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:19:59.807+0000 7fec5dbf4700 1 -- 172.21.15.103:0/2671931209 wait complete. 2024-03-22T02:20:00.532 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-22T02:20:00.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-22T02:20:01.085 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:00 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2671931209' entity='client.admin' 2024-03-22T02:20:01.114 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:20:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:02 smithi103 ceph-mon[29258]: Deploying daemon crash.smithi103 on smithi103 2024-03-22T02:20:02.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.656+0000 7f009fbfb700 1 -- 172.21.15.103:0/4123789895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 msgr2=0x7f00980fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:02.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.656+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4123789895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f00980fbd70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f00940048c0 tx=0x7f0094009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:02.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.656+0000 7f009fbfb700 1 -- 172.21.15.103:0/4123789895 shutdown_connections 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.656+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4123789895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f00980fbd70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.656+0000 7f009fbfb700 1 -- 172.21.15.103:0/4123789895 >> 172.21.15.103:0/4123789895 conn(0x7f00980f70c0 msgr2=0x7f00980f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 -- 172.21.15.103:0/4123789895 shutdown_connections 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 -- 172.21.15.103:0/4123789895 wait complete. 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 Processor -- start 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 -- start start 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009fbfb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009810d610 con 0x7f00980fb960 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009d997700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009d997700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40384/0 (socket says 172.21.15.103:40384) 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.657+0000 7f009d997700 1 -- 172.21.15.103:0/4170450639 learned_addr learned my addr 172.21.15.103:0/4170450639 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:20:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f009d997700 1 -- 172.21.15.103:0/4170450639 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094005040 con 0x7f00980fb960 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f009d997700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f00940096b0 tx=0x7f0094009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f009400bed0 con 0x7f00980fb960 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009810d810 con 0x7f00980fb960 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f009400b5c0 con 0x7f00980fb960 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f009401c5b0 con 0x7f00980fb960 2024-03-22T02:20:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.658+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009810e380 con 0x7f00980fb960 2024-03-22T02:20:02.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f009401c820 con 0x7f00980fb960 2024-03-22T02:20:02.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0098101e70 con 0x7f00980fb960 2024-03-22T02:20:02.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f008e7fc700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 0x7f0084038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:02.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0094049a20 con 0x7f00980fb960 2024-03-22T02:20:02.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f009d196700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 0x7f0084038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.659+0000 7f009d196700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 0x7f0084038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0088009990 tx=0x7f0088006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.664+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0094023300 con 0x7f00980fb960 2024-03-22T02:20:02.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.855+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f009805f690 con 0x7f0084036880 2024-03-22T02:20:02.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.858+0000 7f008e7fc700 1 -- 172.21.15.103:0/4170450639 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f009805f690 con 0x7f0084036880 2024-03-22T02:20:02.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 msgr2=0x7f0084038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:02.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 0x7f0084038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0088009990 tx=0x7f0088006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:02.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 msgr2=0x7f009810d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:02.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f00940096b0 tx=0x7f0094009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:02.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 shutdown_connections 2024-03-22T02:20:02.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0084036880 0x7f0084038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:02.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 --2- 172.21.15.103:0/4170450639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00980fb960 0x7f009810d0d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:02.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 >> 172.21.15.103:0/4170450639 conn(0x7f00980f70c0 msgr2=0x7f0098065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:02.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 shutdown_connections 2024-03-22T02:20:02.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:02.861+0000 7f009fbfb700 1 -- 172.21.15.103:0/4170450639 wait complete. 2024-03-22T02:20:04.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:03 smithi103 ceph-mon[29258]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:20:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:03 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:04.389 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-03-22T02:20:04.389 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:20:04.389 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-03-22T02:20:04.408 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:20:04.409 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:20:04.466 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-03-22T02:20:04.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch host add smithi122 2024-03-22T02:20:04.960 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:20:06.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:05 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:05 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:05 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:05 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:05 smithi103 ceph-mon[29258]: Deploying daemon node-exporter.smithi103 on smithi103 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 -- 172.21.15.103:0/832603296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 msgr2=0x7f1da80fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/832603296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da80fbd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d980048c0 tx=0x7f1d98009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 -- 172.21.15.103:0/832603296 shutdown_connections 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/832603296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da80fbd60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 -- 172.21.15.103:0/832603296 >> 172.21.15.103:0/832603296 conn(0x7f1da80f7090 msgr2=0x7f1da80f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 -- 172.21.15.103:0/832603296 shutdown_connections 2024-03-22T02:20:06.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.788+0000 7f1daf8e5700 1 -- 172.21.15.103:0/832603296 wait complete. 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1daf8e5700 1 Processor -- start 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1daf8e5700 1 -- start start 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1daf8e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1daf8e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da810fd70 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1dad681700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1dad681700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47252/0 (socket says 172.21.15.103:47252) 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1dad681700 1 -- 172.21.15.103:0/2483032498 learned_addr learned my addr 172.21.15.103:0/2483032498 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1dad681700 1 -- 172.21.15.103:0/2483032498 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d98005040 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1dad681700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d980096b0 tx=0x7f1d98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9800bed0 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d9800b5c0 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.789+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da810ff10 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.790+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9801c5b0 con 0x7f1da80fb950 2024-03-22T02:20:06.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.790+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da81103a0 con 0x7f1da80fb950 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1d9801c820 con 0x7f1da80fb950 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da8105910 con 0x7f1da80fb950 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1d9dffb700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 0x7f1d94038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1d980499d0 con 0x7f1da80fb950 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1dace80700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 0x7f1d94038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:06.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.791+0000 7f1dace80700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 0x7f1d94038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009990 tx=0x7f1da4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:06.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.796+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1d98023300 con 0x7f1da80fb950 2024-03-22T02:20:06.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:06.987+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f1da805e400 con 0x7f1d94036880 2024-03-22T02:20:07.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:07 smithi103 ceph-mon[29258]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:20:09.102 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:08 smithi103 ceph-mon[29258]: Deploying cephadm binary to smithi122 2024-03-22T02:20:09.102 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:08 smithi103 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:09.102 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:08 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:09.175 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-03-22T02:20:09.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.173+0000 7f1d9dffb700 1 -- 172.21.15.103:0/2483032498 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1da805e400 con 0x7f1d94036880 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 msgr2=0x7f1d94038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 0x7f1d94038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009990 tx=0x7f1da4006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 msgr2=0x7f1da8066880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d980096b0 tx=0x7f1d98009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 shutdown_connections 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f1d94036880 0x7f1d94038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 --2- 172.21.15.103:0/2483032498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da80fb950 0x7f1da8066880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.175+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 >> 172.21.15.103:0/2483032498 conn(0x7f1da80f7090 msgr2=0x7f1da8102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.176+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 shutdown_connections 2024-03-22T02:20:09.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:09.176+0000 7f1daf8e5700 1 -- 172.21.15.103:0/2483032498 wait complete. 2024-03-22T02:20:09.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T02:20:10.079 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:20:10.492 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: Added host smithi122 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:10.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:10 smithi103 ceph-mon[29258]: Deploying daemon alertmanager.smithi103 on smithi103 2024-03-22T02:20:11.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:11 smithi103 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:12.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- 172.21.15.103:0/2637349881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 msgr2=0x7fc3c00fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:12.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 --2- 172.21.15.103:0/2637349881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c00fbd70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00048c0 tx=0x7fc3b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:12.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- 172.21.15.103:0/2637349881 shutdown_connections 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 --2- 172.21.15.103:0/2637349881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c00fbd70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- 172.21.15.103:0/2637349881 >> 172.21.15.103:0/2637349881 conn(0x7fc3c00f70c0 msgr2=0x7fc3c00f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- 172.21.15.103:0/2637349881 shutdown_connections 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- 172.21.15.103:0/2637349881 wait complete. 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 Processor -- start 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.071+0000 7fc3c6429700 1 -- start start 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3c6429700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3c6429700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c010c4d0 con 0x7fc3c00fb960 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47274/0 (socket says 172.21.15.103:47274) 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bffff700 1 -- 172.21.15.103:0/1487557547 learned_addr learned my addr 172.21.15.103:0/1487557547 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bffff700 1 -- 172.21.15.103:0/1487557547 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0005040 con 0x7fc3c00fb960 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bffff700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00096b0 tx=0x7fc3b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:12.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3b000bed0 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3b000b5c0 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3b001c5b0 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c010c6d0 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.072+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c010d240 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.073+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fc3b001c820 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.073+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c0101e70 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.073+0000 7fc3bcff9700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 0x7fc3a8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.073+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc3b0049ca0 con 0x7fc3c00fb960 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.074+0000 7fc3bf7fe700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 0x7fc3a8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.074+0000 7fc3bf7fe700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 0x7fc3a8038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009990 tx=0x7fc3b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.078+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc3b0023300 con 0x7fc3c00fb960 2024-03-22T02:20:12.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.275+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3c010d480 con 0x7fc3a8036880 2024-03-22T02:20:12.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.277+0000 7fc3bcff9700 1 -- 172.21.15.103:0/1487557547 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fc3c010d480 con 0x7fc3a8036880 2024-03-22T02:20:12.278 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:20:12.278 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 msgr2=0x7fc3a8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 0x7fc3a8038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009990 tx=0x7fc3b4006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 msgr2=0x7fc3c010bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00096b0 tx=0x7fc3b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 shutdown_connections 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc3a8036880 0x7fc3a8038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 --2- 172.21.15.103:0/1487557547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3c00fb960 0x7fc3c010bf90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 >> 172.21.15.103:0/1487557547 conn(0x7fc3c00f70c0 msgr2=0x7fc3c0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 shutdown_connections 2024-03-22T02:20:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:12.279+0000 7fc3c6429700 1 -- 172.21.15.103:0/1487557547 wait complete. 2024-03-22T02:20:13.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:13 smithi103 ceph-mon[29258]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:13.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:13 smithi103 ceph-mon[29258]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:20:13.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:13 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:13.837 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-22T02:20:13.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-22T02:20:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:20:15.613 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:15 smithi103 ceph-mon[29258]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:17.365 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:17 smithi103 ceph-mon[29258]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:18.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.648+0000 7f96b82c7700 1 -- 172.21.15.103:0/1466769870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 msgr2=0x7f96b00fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:18.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.648+0000 7f96b82c7700 1 --2- 172.21.15.103:0/1466769870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b00fe4b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0048c0 tx=0x7f96ac009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:18.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- 172.21.15.103:0/1466769870 shutdown_connections 2024-03-22T02:20:18.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 --2- 172.21.15.103:0/1466769870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b00fe4b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- 172.21.15.103:0/1466769870 >> 172.21.15.103:0/1466769870 conn(0x7f96b00f6840 msgr2=0x7f96b00f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- 172.21.15.103:0/1466769870 shutdown_connections 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- 172.21.15.103:0/1466769870 wait complete. 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 Processor -- start 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- start start 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b82c7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b010d600 con 0x7f96b00fe0a0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b6063700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b6063700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44014/0 (socket says 172.21.15.103:44014) 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.649+0000 7f96b6063700 1 -- 172.21.15.103:0/2926382757 learned_addr learned my addr 172.21.15.103:0/2926382757 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96b6063700 1 -- 172.21.15.103:0/2926382757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac005040 con 0x7f96b00fe0a0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96b6063700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0096b0 tx=0x7f96ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ac00bed0 con 0x7f96b00fe0a0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96ac00b5c0 con 0x7f96b00fe0a0 2024-03-22T02:20:18.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b010d800 con 0x7f96b00fe0a0 2024-03-22T02:20:18.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96ac01c5b0 con 0x7f96b00fe0a0 2024-03-22T02:20:18.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.650+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b010e370 con 0x7f96b00fe0a0 2024-03-22T02:20:18.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f96ac01c820 con 0x7f96b00fe0a0 2024-03-22T02:20:18.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b0102f70 con 0x7f96b00fe0a0 2024-03-22T02:20:18.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96a6ffd700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 0x7f969c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:18.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f96ac049a40 con 0x7f96b00fe0a0 2024-03-22T02:20:18.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96b5862700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 0x7f969c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:18.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.651+0000 7f96b5862700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 0x7f969c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009990 tx=0x7f96a0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:18.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.656+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f96ac023300 con 0x7f96b00fe0a0 2024-03-22T02:20:18.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:18.864+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f96b010ee40 con 0x7f96b00fe0a0 2024-03-22T02:20:19.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.221+0000 7f96a6ffd700 1 -- 172.21.15.103:0/2926382757 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f96ac011b60 con 0x7f96b00fe0a0 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 msgr2=0x7f969c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 0x7f969c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009990 tx=0x7f96a0006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 msgr2=0x7f96b010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0096b0 tx=0x7f96ac009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 shutdown_connections 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f969c036880 0x7f969c038d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.223+0000 7f96b82c7700 1 --2- 172.21.15.103:0/2926382757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96b00fe0a0 0x7f96b010d0c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.224+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 >> 172.21.15.103:0/2926382757 conn(0x7f96b00f6840 msgr2=0x7f96b0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.224+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 shutdown_connections 2024-03-22T02:20:19.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:20:19.224+0000 7f96b82c7700 1 -- 172.21.15.103:0/2926382757 wait complete. 2024-03-22T02:20:19.225 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-03-22T02:20:19.617 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:19 smithi103 ceph-mon[29258]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:19.617 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:19 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2926382757' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T02:20:20.378 INFO:tasks.cephadm:Adding mon.smithi103 on smithi103 2024-03-22T02:20:20.378 INFO:tasks.cephadm:Adding mon.smithi122 on smithi122 2024-03-22T02:20:20.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi103:172.21.15.103=smithi103;smithi122:172.21.15.122=smithi122' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2926382757' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T02:20:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:20 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:20.626 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:20.673 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:21.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:21 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T02:20:21.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:21 smithi103 ceph-mon[29258]: Deploying daemon grafana.smithi103 on smithi103 2024-03-22T02:20:21.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:21 smithi103 ceph-mon[29258]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.211+0000 7f16babb9700 1 -- 172.21.15.122:0/3286415623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 msgr2=0x7f16b40fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.211+0000 7f16babb9700 1 --2- 172.21.15.122:0/3286415623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b40fce70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f16a80048c0 tx=0x7f16a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- 172.21.15.122:0/3286415623 shutdown_connections 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 --2- 172.21.15.122:0/3286415623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b40fce70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- 172.21.15.122:0/3286415623 >> 172.21.15.122:0/3286415623 conn(0x7f16b40f81c0 msgr2=0x7f16b40fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- 172.21.15.122:0/3286415623 shutdown_connections 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- 172.21.15.122:0/3286415623 wait complete. 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 Processor -- start 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- start start 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.212+0000 7f16babb9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40722a0 con 0x7f16b40fca60 2024-03-22T02:20:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b8955700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:22.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b8955700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:49036/0 (socket says 172.21.15.122:49036) 2024-03-22T02:20:22.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b8955700 1 -- 172.21.15.122:0/504401310 learned_addr learned my addr 172.21.15.122:0/504401310 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:22.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b8955700 1 -- 172.21.15.122:0/504401310 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a8005040 con 0x7f16b40fca60 2024-03-22T02:20:22.214 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b8955700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f16a80096b0 tx=0x7f16a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a800bed0 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16a800b5c0 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a801c5b0 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b4070370 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.213+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b4070860 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.214+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f16a8013420 con 0x7f16b40fca60 2024-03-22T02:20:22.215 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.214+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b405f5f0 con 0x7f16b40fca60 2024-03-22T02:20:22.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.214+0000 7f16b17fa700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 0x7f16a4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:22.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.214+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f16a8049b10 con 0x7f16b40fca60 2024-03-22T02:20:22.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.214+0000 7f16b3fff700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 0x7f16a4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:22.216 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.215+0000 7f16b3fff700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 0x7f16a4038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009990 tx=0x7f16a0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:22.219 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.219+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f16a8025400 con 0x7f16b40fca60 2024-03-22T02:20:22.415 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.415+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 --> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi103:172.21.15.103=smithi103;smithi122:172.21.15.122=smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f16b4106560 con 0x7f16a4036880 2024-03-22T02:20:22.421 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.421+0000 7f16b17fa700 1 -- 172.21.15.122:0/504401310 <== mgr.14162 v2:172.21.15.103:6800/2790950462 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f16b4106560 con 0x7f16a4036880 2024-03-22T02:20:22.421 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-03-22T02:20:22.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 msgr2=0x7f16a4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:22.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 0x7f16a4038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009990 tx=0x7f16a0006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:22.423 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 msgr2=0x7f16b4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f16a80096b0 tx=0x7f16a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 shutdown_connections 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f16a4036880 0x7f16a4038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 --2- 172.21.15.122:0/504401310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16b40fca60 0x7f16b4071d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 >> 172.21.15.122:0/504401310 conn(0x7f16b40f81c0 msgr2=0x7f16b4103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 shutdown_connections 2024-03-22T02:20:22.424 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:22.423+0000 7f16babb9700 1 -- 172.21.15.122:0/504401310 wait complete. 2024-03-22T02:20:22.948 DEBUG:teuthology.orchestra.run.smithi122:mon.smithi122> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi122.service 2024-03-22T02:20:22.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:22.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:23.229 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:23.275 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:23.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:23 smithi103 ceph-mon[29258]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:23.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:23 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:23.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:23 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:24.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:24 smithi103 ceph-mon[29258]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi103:172.21.15.103=smithi103;smithi122:172.21.15.122=smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:20:24.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:24 smithi103 ceph-mon[29258]: Saving service mon spec with placement smithi103:172.21.15.103=smithi103;smithi122:172.21.15.122=smithi122;count:2 2024-03-22T02:20:24.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 -- 172.21.15.122:0/3144990568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 msgr2=0x7feb700fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.816 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 --2- 172.21.15.122:0/3144990568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb700fd460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb600048c0 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 -- 172.21.15.122:0/3144990568 shutdown_connections 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 --2- 172.21.15.122:0/3144990568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb700fd460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 -- 172.21.15.122:0/3144990568 >> 172.21.15.122:0/3144990568 conn(0x7feb700f70d0 msgr2=0x7feb700f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 -- 172.21.15.122:0/3144990568 shutdown_connections 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.803+0000 7feb748d6700 1 -- 172.21.15.122:0/3144990568 wait complete. 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb748d6700 1 Processor -- start 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb748d6700 1 -- start start 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb748d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb748d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7010d600 con 0x7feb700fd050 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb6e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb6e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:49044/0 (socket says 172.21.15.122:49044) 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb6e59c700 1 -- 172.21.15.122:0/4129116949 learned_addr learned my addr 172.21.15.122:0/4129116949 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.804+0000 7feb6e59c700 1 -- 172.21.15.122:0/4129116949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60005040 con 0x7feb700fd050 2024-03-22T02:20:24.817 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb6e59c700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb600096b0 tx=0x7feb60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6000bed0 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6000b5c0 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6001c5b0 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7010d800 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.805+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7010e370 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.806+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7feb60013a80 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.806+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb70101e70 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.806+0000 7feb5effd700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 0x7feb58038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.806+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7feb600499d0 con 0x7feb700fd050 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.806+0000 7feb6dd9b700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 0x7feb58038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.807+0000 7feb6dd9b700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 0x7feb58038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb64009990 tx=0x7feb64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.818 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:24.811+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7feb60011960 con 0x7feb700fd050 2024-03-22T02:20:25.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.068+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb7010e5b0 con 0x7feb700fd050 2024-03-22T02:20:25.069 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.068+0000 7feb5effd700 1 -- 172.21.15.122:0/4129116949 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7feb60011370 con 0x7feb700fd050 2024-03-22T02:20:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 msgr2=0x7feb58038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 0x7feb58038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb64009990 tx=0x7feb64006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 msgr2=0x7feb7010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feb600096b0 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 shutdown_connections 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7feb580368d0 0x7feb58038d80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 --2- 172.21.15.122:0/4129116949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb700fd050 0x7feb7010d0c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 >> 172.21.15.122:0/4129116949 conn(0x7feb700f70d0 msgr2=0x7feb70065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 shutdown_connections 2024-03-22T02:20:25.071 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:25.070+0000 7feb748d6700 1 -- 172.21.15.122:0/4129116949 wait complete. 2024-03-22T02:20:25.072 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:25.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:25 smithi103 ceph-mon[29258]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:25.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:25 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/4129116949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:26.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:26.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:26.889 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:26.935 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:27.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:27 smithi103 ceph-mon[29258]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 -- 172.21.15.122:0/3759075310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 msgr2=0x7ff6c40ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3759075310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c40ff6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00048c0 tx=0x7ff6c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 -- 172.21.15.122:0/3759075310 shutdown_connections 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3759075310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c40ff6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 -- 172.21.15.122:0/3759075310 >> 172.21.15.122:0/3759075310 conn(0x7ff6c40f70c0 msgr2=0x7ff6c40f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 -- 172.21.15.122:0/3759075310 shutdown_connections 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 -- 172.21.15.122:0/3759075310 wait complete. 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.431+0000 7ff6cc234700 1 Processor -- start 2024-03-22T02:20:28.433 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6cc234700 1 -- start start 2024-03-22T02:20:28.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6cc234700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:28.434 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6cc234700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c410d660 con 0x7ff6c4066980 2024-03-22T02:20:28.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6c9fd0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:28.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6c9fd0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:50384/0 (socket says 172.21.15.122:50384) 2024-03-22T02:20:28.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6c9fd0700 1 -- 172.21.15.122:0/3079200743 learned_addr learned my addr 172.21.15.122:0/3079200743 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:28.435 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6c9fd0700 1 -- 172.21.15.122:0/3079200743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c0005040 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.432+0000 7ff6c9fd0700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3b0 tx=0x7ff6c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.433+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6c0004e60 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.433+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c410d860 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.433+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6c0009650 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.433+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6c001c5b0 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.433+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c410e3d0 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7ff6c0013420 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c4102f60 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6ba7fc700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 0x7ff6b0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff6c0049960 con 0x7ff6c4066980 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6c97cf700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 0x7ff6b0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:28.436 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.434+0000 7ff6c97cf700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 0x7ff6b0038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009990 tx=0x7ff6b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:28.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.439+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff6c00139f0 con 0x7ff6c4066980 2024-03-22T02:20:28.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.701+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6c410ede0 con 0x7ff6c4066980 2024-03-22T02:20:28.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.701+0000 7ff6ba7fc700 1 -- 172.21.15.122:0/3079200743 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7ff6c0011370 con 0x7ff6c4066980 2024-03-22T02:20:28.703 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:28.703 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:28.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 msgr2=0x7ff6b0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:28.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 0x7ff6b0038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009990 tx=0x7ff6b4006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:28.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 msgr2=0x7ff6c410d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:28.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000b3b0 tx=0x7ff6c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 shutdown_connections 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7ff6b0036880 0x7ff6b0038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 --2- 172.21.15.122:0/3079200743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6c4066980 0x7ff6c410d120 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 >> 172.21.15.122:0/3079200743 conn(0x7ff6c40f70c0 msgr2=0x7ff6c4103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 shutdown_connections 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:28.703+0000 7ff6cc234700 1 -- 172.21.15.122:0/3079200743 wait complete. 2024-03-22T02:20:28.705 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:29.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:29 smithi103 ceph-mon[29258]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:29.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:29 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3079200743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:30.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:30.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:30.545 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:30.590 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:31.610 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:31 smithi103 ceph-mon[29258]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- 172.21.15.122:0/850880412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 msgr2=0x7f0eb80fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/850880412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb80fce70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0048c0 tx=0x7f0eac009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- 172.21.15.122:0/850880412 shutdown_connections 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/850880412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb80fce70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- 172.21.15.122:0/850880412 >> 172.21.15.122:0/850880412 conn(0x7f0eb80f81c0 msgr2=0x7f0eb80fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- 172.21.15.122:0/850880412 shutdown_connections 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- 172.21.15.122:0/850880412 wait complete. 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 Processor -- start 2024-03-22T02:20:32.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.191+0000 7f0ebedbe700 1 -- start start 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebedbe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebedbe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb810d5d0 con 0x7f0eb80fca60 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebcb5a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebcb5a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:50400/0 (socket says 172.21.15.122:50400) 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebcb5a700 1 -- 172.21.15.122:0/2224435363 learned_addr learned my addr 172.21.15.122:0/2224435363 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebcb5a700 1 -- 172.21.15.122:0/2224435363 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac005040 con 0x7f0eb80fca60 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebcb5a700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009680 tx=0x7f0eac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eac00b5c0 con 0x7f0eb80fca60 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0eac00be70 con 0x7f0eb80fca60 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eac0135a0 con 0x7f0eb80fca60 2024-03-22T02:20:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb810d7d0 con 0x7f0eb80fca60 2024-03-22T02:20:32.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.192+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb810e340 con 0x7f0eb80fca60 2024-03-22T02:20:32.196 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.193+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f0eac01c430 con 0x7f0eb80fca60 2024-03-22T02:20:32.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.193+0000 7f0eb57fa700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 0x7f0ea8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:32.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.193+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb8102f70 con 0x7f0eb80fca60 2024-03-22T02:20:32.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.193+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0eac0493e0 con 0x7f0eb80fca60 2024-03-22T02:20:32.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.193+0000 7f0eb7fff700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 0x7f0ea8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:32.197 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.194+0000 7f0eb7fff700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 0x7f0ea8038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009990 tx=0x7f0ea4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:32.199 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.198+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0eac011550 con 0x7f0eb80fca60 2024-03-22T02:20:32.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.457+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eb810ed50 con 0x7f0eb80fca60 2024-03-22T02:20:32.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.457+0000 7f0eb57fa700 1 -- 172.21.15.122:0/2224435363 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0eac021070 con 0x7f0eb80fca60 2024-03-22T02:20:32.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:32.459 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:32.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 msgr2=0x7f0ea8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:32.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 0x7f0ea8038d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009990 tx=0x7f0ea4006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:32.460 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 msgr2=0x7f0eb810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009680 tx=0x7f0eac009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 shutdown_connections 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f0ea80368d0 0x7f0ea8038d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 --2- 172.21.15.122:0/2224435363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb80fca60 0x7f0eb810d090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 >> 172.21.15.122:0/2224435363 conn(0x7f0eb80f81c0 msgr2=0x7f0eb8103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 shutdown_connections 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:32.459+0000 7f0ebedbe700 1 -- 172.21.15.122:0/2224435363 wait complete. 2024-03-22T02:20:32.461 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/2224435363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:33.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:33 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:34.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:34.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:34.312 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:34.358 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:34.532 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:34 smithi103 ceph-mon[29258]: Deploying daemon prometheus.smithi103 on smithi103 2024-03-22T02:20:35.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:35 smithi103 ceph-mon[29258]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:35.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- 172.21.15.122:0/765188471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 msgr2=0x7f8d240fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:35.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 --2- 172.21.15.122:0/765188471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d240fce70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8d140048c0 tx=0x7f8d14009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:35.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- 172.21.15.122:0/765188471 shutdown_connections 2024-03-22T02:20:35.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 --2- 172.21.15.122:0/765188471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d240fce70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- 172.21.15.122:0/765188471 >> 172.21.15.122:0/765188471 conn(0x7f8d240f81c0 msgr2=0x7f8d240fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- 172.21.15.122:0/765188471 shutdown_connections 2024-03-22T02:20:35.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- 172.21.15.122:0/765188471 wait complete. 2024-03-22T02:20:35.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 Processor -- start 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.841+0000 7f8d288ff700 1 -- start start 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d288ff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d288ff700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2410d5d0 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:50414/0 (socket says 172.21.15.122:50414) 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d2259c700 1 -- 172.21.15.122:0/745199704 learned_addr learned my addr 172.21.15.122:0/745199704 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d2259c700 1 -- 172.21.15.122:0/745199704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14005040 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d2259c700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d140096b0 tx=0x7f8d14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1400bed0 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d1400b5c0 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2410d7d0 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d1401c5b0 con 0x7f8d240fca60 2024-03-22T02:20:35.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.842+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2410e340 con 0x7f8d240fca60 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.843+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f8d1401c820 con 0x7f8d240fca60 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.843+0000 7f8d12ffd700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 0x7f8d0c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.843+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8d14049b40 con 0x7f8d240fca60 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.843+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24102f70 con 0x7f8d240fca60 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.844+0000 7f8d21d9b700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 0x7f8d0c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:35.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.844+0000 7f8d21d9b700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 0x7f8d0c038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009990 tx=0x7f8d18006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:35.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:35.852+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8d14023300 con 0x7f8d240fca60 2024-03-22T02:20:36.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.115+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d2405f5f0 con 0x7f8d240fca60 2024-03-22T02:20:36.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.116+0000 7f8d12ffd700 1 -- 172.21.15.122:0/745199704 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8d2405f5f0 con 0x7f8d240fca60 2024-03-22T02:20:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:36.117 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 msgr2=0x7f8d0c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 0x7f8d0c038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009990 tx=0x7f8d18006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 msgr2=0x7f8d2410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d140096b0 tx=0x7f8d14009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 shutdown_connections 2024-03-22T02:20:36.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f8d0c036880 0x7f8d0c038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:36.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 --2- 172.21.15.122:0/745199704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d240fca60 0x7f8d2410d090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:36.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 >> 172.21.15.122:0/745199704 conn(0x7f8d240f81c0 msgr2=0x7f8d24103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:36.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 shutdown_connections 2024-03-22T02:20:36.120 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:36.118+0000 7f8d288ff700 1 -- 172.21.15.122:0/745199704 wait complete. 2024-03-22T02:20:36.120 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:36.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:36 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/745199704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:37.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:37 smithi103 ceph-mon[29258]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:37.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:37.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:37.970 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:38.016 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:39.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- 172.21.15.122:0/3078603196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 msgr2=0x7fb874101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:39.515 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/3078603196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb874101ad0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb8700048c0 tx=0x7fb870009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- 172.21.15.122:0/3078603196 shutdown_connections 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/3078603196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb874101ad0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- 172.21.15.122:0/3078603196 >> 172.21.15.122:0/3078603196 conn(0x7fb8740f70c0 msgr2=0x7fb8740f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- 172.21.15.122:0/3078603196 shutdown_connections 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- 172.21.15.122:0/3078603196 wait complete. 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 Processor -- start 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- start start 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.513+0000 7fb87b4ca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87410f800 con 0x7fb874066960 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb879266700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb879266700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:60968/0 (socket says 172.21.15.122:60968) 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb879266700 1 -- 172.21.15.122:0/1341300062 learned_addr learned my addr 172.21.15.122:0/1341300062 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:39.516 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb879266700 1 -- 172.21.15.122:0/1341300062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb870005040 con 0x7fb874066960 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb879266700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb870009680 tx=0x7fb870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87000bed0 con 0x7fb874066960 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87410fa00 con 0x7fb874066960 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb87000b5c0 con 0x7fb874066960 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87001c5b0 con 0x7fb874066960 2024-03-22T02:20:39.517 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.514+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb874110570 con 0x7fb874066960 2024-03-22T02:20:39.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:39 smithi103 ceph-mon[29258]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:39.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.515+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fb870004d30 con 0x7fb874066960 2024-03-22T02:20:39.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.515+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb874105170 con 0x7fb874066960 2024-03-22T02:20:39.518 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.515+0000 7fb869ffb700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 0x7fb860038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:39.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.515+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb870049a50 con 0x7fb874066960 2024-03-22T02:20:39.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.515+0000 7fb878a65700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 0x7fb860038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:39.519 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.516+0000 7fb878a65700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 0x7fb860038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb864009990 tx=0x7fb864006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:39.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.520+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb870011550 con 0x7fb874066960 2024-03-22T02:20:39.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.783+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb874110f80 con 0x7fb874066960 2024-03-22T02:20:39.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.784+0000 7fb869ffb700 1 -- 172.21.15.122:0/1341300062 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb870011370 con 0x7fb874066960 2024-03-22T02:20:39.785 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:39.785 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 msgr2=0x7fb860038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 0x7fb860038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb864009990 tx=0x7fb864006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 msgr2=0x7fb87410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb870009680 tx=0x7fb870009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 shutdown_connections 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fb860036880 0x7fb860038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:39.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 --2- 172.21.15.122:0/1341300062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb874066960 0x7fb87410f2c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:39.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 >> 172.21.15.122:0/1341300062 conn(0x7fb8740f70c0 msgr2=0x7fb874105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:39.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 shutdown_connections 2024-03-22T02:20:39.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:39.786+0000 7fb87b4ca700 1 -- 172.21.15.122:0/1341300062 wait complete. 2024-03-22T02:20:39.788 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:40.611 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:40 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/1341300062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:41.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:41.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:41.583 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:41.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:41 smithi103 ceph-mon[29258]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:43.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 -- 172.21.15.122:0/1007476404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 msgr2=0x7f79b40fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:43.175 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 --2- 172.21.15.122:0/1007476404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b40fce70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f79b00048c0 tx=0x7f79b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 -- 172.21.15.122:0/1007476404 shutdown_connections 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 --2- 172.21.15.122:0/1007476404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b40fce70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 -- 172.21.15.122:0/1007476404 >> 172.21.15.122:0/1007476404 conn(0x7f79b40f81c0 msgr2=0x7f79b40fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 -- 172.21.15.122:0/1007476404 shutdown_connections 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.173+0000 7f79bbabe700 1 -- 172.21.15.122:0/1007476404 wait complete. 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79bbabe700 1 Processor -- start 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79bbabe700 1 -- start start 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79bbabe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79bbabe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b410d5d0 con 0x7f79b40fca60 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79b985a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79b985a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:60978/0 (socket says 172.21.15.122:60978) 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.174+0000 7f79b985a700 1 -- 172.21.15.122:0/579474745 learned_addr learned my addr 172.21.15.122:0/579474745 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:43.176 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79b985a700 1 -- 172.21.15.122:0/579474745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0005040 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79b985a700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f79b00096b0 tx=0x7f79b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b000bed0 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79b000b5c0 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b410d7d0 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b001c5b0 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.175+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b410e340 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f79b001c820 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b4102f70 con 0x7f79b40fca60 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79aa7fc700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 0x7f79a0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:43.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f79b0049b70 con 0x7f79b40fca60 2024-03-22T02:20:43.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79b9059700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 0x7f79a0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:43.179 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.176+0000 7f79b9059700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 0x7f79a0038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009990 tx=0x7f79a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:43.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.181+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f79b0023300 con 0x7f79b40fca60 2024-03-22T02:20:43.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.438+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79b410ed50 con 0x7f79b40fca60 2024-03-22T02:20:43.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.438+0000 7f79aa7fc700 1 -- 172.21.15.122:0/579474745 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f79b0011b60 con 0x7f79b40fca60 2024-03-22T02:20:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:43.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 msgr2=0x7f79a0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 0x7f79a0038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009990 tx=0x7f79a4006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 msgr2=0x7f79b410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f79b00096b0 tx=0x7f79b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 shutdown_connections 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7f79a0036880 0x7f79a0038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 --2- 172.21.15.122:0/579474745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79b40fca60 0x7f79b410d090 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 >> 172.21.15.122:0/579474745 conn(0x7f79b40f81c0 msgr2=0x7f79b4103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 shutdown_connections 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:43.440+0000 7f79bbabe700 1 -- 172.21.15.122:0/579474745 wait complete. 2024-03-22T02:20:43.442 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:43.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:20:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-22T02:20:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:43 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:44.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:44 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/579474745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:44.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:44 smithi103 ceph-mon[29258]: from='mgr.14162 172.21.15.103:0/4206351812' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-22T02:20:44.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:44 smithi103 ceph-mon[29258]: mgrmap e14: smithi103.pfvmui(active, since 55s) 2024-03-22T02:20:45.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:45.026 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:45.282 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:45.329 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:46.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- 172.21.15.122:0/3759353655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 msgr2=0x7fab680fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:46.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/3759353655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fe4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fab640048c0 tx=0x7fab64009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:46.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- 172.21.15.122:0/3759353655 shutdown_connections 2024-03-22T02:20:46.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/3759353655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fe4b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- 172.21.15.122:0/3759353655 >> 172.21.15.122:0/3759353655 conn(0x7fab680f6840 msgr2=0x7fab680f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- 172.21.15.122:0/3759353655 shutdown_connections 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- 172.21.15.122:0/3759353655 wait complete. 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 Processor -- start 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- start start 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.820+0000 7fab6f0c0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab680fdcf0 con 0x7fab680fe0a0 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6ce5c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6ce5c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:60988/0 (socket says 172.21.15.122:60988) 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6ce5c700 1 -- 172.21.15.122:0/1355898392 learned_addr learned my addr 172.21.15.122:0/1355898392 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:46.834 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6ce5c700 1 -- 172.21.15.122:0/1355898392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64005040 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6ce5c700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fab64009680 tx=0x7fab64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab6400b5c0 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab6400be70 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab640135a0 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab680fbdc0 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.821+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab680fc2b0 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7fab6401c430 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6805f5f0 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab5d7fa700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 0x7fab50038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fab64049400 con 0x7fab680fe0a0 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab5ffff700 1 -- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 msgr2=0x7fab50038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/2790950462 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.822+0000 7fab5ffff700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 0x7fab50038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:20:46.835 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:46.827+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fab64004d30 con 0x7fab680fe0a0 2024-03-22T02:20:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.023+0000 7fab5ffff700 1 -- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 msgr2=0x7fab50038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/2790950462 2024-03-22T02:20:47.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.023+0000 7fab5ffff700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 0x7fab50038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:20:47.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.088+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab68106440 con 0x7fab680fe0a0 2024-03-22T02:20:47.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.088+0000 7fab5d7fa700 1 -- 172.21.15.122:0/1355898392 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fab64021070 con 0x7fab680fe0a0 2024-03-22T02:20:47.090 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:47.090 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 msgr2=0x7fab50038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 0x7fab50038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 msgr2=0x7fab680fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fab64009680 tx=0x7fab64009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 shutdown_connections 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fab500368d0 0x7fab50038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 --2- 172.21.15.122:0/1355898392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab680fe0a0 0x7fab680fd7b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:47.092 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 >> 172.21.15.122:0/1355898392 conn(0x7fab680f6840 msgr2=0x7fab68103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:47.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 shutdown_connections 2024-03-22T02:20:47.093 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:47.091+0000 7fab6f0c0700 1 -- 172.21.15.122:0/1355898392 wait complete. 2024-03-22T02:20:47.093 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:47.461 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:47 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/1355898392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:48.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:48.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:48.837 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:48.883 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.326+0000 7fc89035f700 1 -- 172.21.15.122:0/4134474976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 msgr2=0x7fc8880fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.326+0000 7fc89035f700 1 --2- 172.21.15.122:0/4134474976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880fbcf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc8840048c0 tx=0x7fc884009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.326+0000 7fc89035f700 1 -- 172.21.15.122:0/4134474976 shutdown_connections 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.326+0000 7fc89035f700 1 --2- 172.21.15.122:0/4134474976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880fbcf0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.326+0000 7fc89035f700 1 -- 172.21.15.122:0/4134474976 >> 172.21.15.122:0/4134474976 conn(0x7fc8880f7080 msgr2=0x7fc8880f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 -- 172.21.15.122:0/4134474976 shutdown_connections 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 -- 172.21.15.122:0/4134474976 wait complete. 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 Processor -- start 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 -- start start 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc89035f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8880f9500 con 0x7fc8880fb8e0 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc88e0fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc88e0fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:33090/0 (socket says 172.21.15.122:33090) 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.327+0000 7fc88e0fb700 1 -- 172.21.15.122:0/3412327568 learned_addr learned my addr 172.21.15.122:0/3412327568 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:50.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc88e0fb700 1 -- 172.21.15.122:0/3412327568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884005040 con 0x7fc8880fb8e0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc88e0fb700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8840096b0 tx=0x7fc884009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc88400bed0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc88400b5c0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8880f75d0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc88401c5b0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.328+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8880f7ac0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7fc88401c820 con 0x7fc8880fb8e0 2024-03-22T02:20:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88805f5f0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc87effd700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 0x7fc874038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc884049c60 con 0x7fc8880fb8e0 2024-03-22T02:20:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc88d8fa700 1 -- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 msgr2=0x7fc874038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/2790950462 2024-03-22T02:20:50.333 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.329+0000 7fc88d8fa700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 0x7fc874038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:20:50.334 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.334+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc884011550 con 0x7fc8880fb8e0 2024-03-22T02:20:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.359+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7fc884013a80 con 0x7fc8880fb8e0 2024-03-22T02:20:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.359+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 msgr2=0x7fc874038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:20:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.359+0000 7fc87effd700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 0x7fc874038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.601+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc88804bed0 con 0x7fc8880fb8e0 2024-03-22T02:20:50.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.602+0000 7fc87effd700 1 -- 172.21.15.122:0/3412327568 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fc884011730 con 0x7fc8880fb8e0 2024-03-22T02:20:50.603 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:50.603 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.603+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 msgr2=0x7fc8880f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.603+0000 7fc89035f700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8840096b0 tx=0x7fc884009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 shutdown_connections 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:6800/2790950462,v1:172.21.15.103:6801/2790950462] conn(0x7fc8740368d0 0x7fc874038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 --2- 172.21.15.122:0/3412327568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8880fb8e0 0x7fc8880f8fc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 >> 172.21.15.122:0/3412327568 conn(0x7fc8880f7080 msgr2=0x7fc888065d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 shutdown_connections 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:50.604+0000 7fc89035f700 1 -- 172.21.15.122:0/3412327568 wait complete. 2024-03-22T02:20:50.605 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: Active manager daemon smithi103.pfvmui restarted 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: osdmap e5: 0 total, 0 up, 0 in 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: mgrmap e15: smithi103.pfvmui(active, starting, since 0.0029325s) 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:20:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:20:50.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:20:50.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:20:50.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:20:51.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3412327568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:51 smithi103 ceph-mon[29258]: mgrmap e16: smithi103.pfvmui(active, since 1.00555s) 2024-03-22T02:20:52.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:52.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:52.474 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:52.521 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:52 smithi103 ceph-mon[29258]: [22/Mar/2024:02:20:51] ENGINE Bus STARTING 2024-03-22T02:20:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:52 smithi103 ceph-mon[29258]: [22/Mar/2024:02:20:51] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:20:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:52 smithi103 ceph-mon[29258]: [22/Mar/2024:02:20:51] ENGINE Bus STARTED 2024-03-22T02:20:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:52 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:52 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:54.290 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 -- 172.21.15.122:0/416430595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 msgr2=0x7f6f900fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 --2- 172.21.15.122:0/416430595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f900fe4b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f800048c0 tx=0x7f6f80009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 -- 172.21.15.122:0/416430595 shutdown_connections 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 --2- 172.21.15.122:0/416430595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f900fe4b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 -- 172.21.15.122:0/416430595 >> 172.21.15.122:0/416430595 conn(0x7f6f900f6840 msgr2=0x7f6f900f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 -- 172.21.15.122:0/416430595 shutdown_connections 2024-03-22T02:20:54.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.290+0000 7f6f958cd700 1 -- 172.21.15.122:0/416430595 wait complete. 2024-03-22T02:20:54.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f958cd700 1 Processor -- start 2024-03-22T02:20:54.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f958cd700 1 -- start start 2024-03-22T02:20:54.292 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f958cd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f958cd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9010d5f0 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f8effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f8effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:33104/0 (socket says 172.21.15.122:33104) 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.291+0000 7f6f8effd700 1 -- 172.21.15.122:0/2948951307 learned_addr learned my addr 172.21.15.122:0/2948951307 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f8effd700 1 -- 172.21.15.122:0/2948951307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f80005040 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f8effd700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f800096b0 tx=0x7f6f80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f8000bed0 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9010d7f0 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f8000b5c0 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f8001c5b0 con 0x7f6f900fe0a0 2024-03-22T02:20:54.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.292+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9010e360 con 0x7f6f900fe0a0 2024-03-22T02:20:54.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.293+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f6f80013a80 con 0x7f6f900fe0a0 2024-03-22T02:20:54.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.293+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f90102f70 con 0x7f6f900fe0a0 2024-03-22T02:20:54.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.293+0000 7f6f777fe700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 0x7f6f78038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:54.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.293+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6f80049c20 con 0x7f6f900fe0a0 2024-03-22T02:20:54.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.293+0000 7f6f8e7fc700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 0x7f6f78038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:54.295 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.294+0000 7f6f8e7fc700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 0x7f6f78038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:54.298 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.298+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6f80011550 con 0x7f6f900fe0a0 2024-03-22T02:20:54.573 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.573+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f9010ed70 con 0x7f6f900fe0a0 2024-03-22T02:20:54.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.574+0000 7f6f777fe700 1 -- 172.21.15.122:0/2948951307 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6f80011370 con 0x7f6f900fe0a0 2024-03-22T02:20:54.574 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:54.574 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:54.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 msgr2=0x7f6f78038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:54.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 0x7f6f78038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:54.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 msgr2=0x7f6f9010d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:54.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f800096b0 tx=0x7f6f80009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 shutdown_connections 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6f78036970 0x7f6f78038e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 --2- 172.21.15.122:0/2948951307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f900fe0a0 0x7f6f9010d0b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 >> 172.21.15.122:0/2948951307 conn(0x7f6f900f6840 msgr2=0x7f6f90103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 shutdown_connections 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:54.576+0000 7f6f958cd700 1 -- 172.21.15.122:0/2948951307 wait complete. 2024-03-22T02:20:54.577 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:54.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:54 smithi103 ceph-mon[29258]: mgrmap e17: smithi103.pfvmui(active, since 2s) 2024-03-22T02:20:55.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:55 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/2948951307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:20:56.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:20:56.173 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:20:56.429 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:56.484 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:20:58.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:58 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:58 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:58 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:58 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:20:58.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:58 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:58.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 -- 172.21.15.122:0/2973408966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 msgr2=0x7f07340ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 --2- 172.21.15.122:0/2973408966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f07340ffa30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f07280048c0 tx=0x7f0728009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:58.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 -- 172.21.15.122:0/2973408966 shutdown_connections 2024-03-22T02:20:58.767 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 --2- 172.21.15.122:0/2973408966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f07340ffa30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 -- 172.21.15.122:0/2973408966 >> 172.21.15.122:0/2973408966 conn(0x7f07340f70d0 msgr2=0x7f07340f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 -- 172.21.15.122:0/2973408966 shutdown_connections 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.765+0000 7f073ac53700 1 -- 172.21.15.122:0/2973408966 wait complete. 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f073ac53700 1 Processor -- start 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f073ac53700 1 -- start start 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f073ac53700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f073ac53700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073410d5c0 con 0x7f07340fd650 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f07389ef700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f07389ef700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:35200/0 (socket says 172.21.15.122:35200) 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.766+0000 7f07389ef700 1 -- 172.21.15.122:0/3660958689 learned_addr learned my addr 172.21.15.122:0/3660958689 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f07389ef700 1 -- 172.21.15.122:0/3660958689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728005040 con 0x7f07340fd650 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f07389ef700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0728009680 tx=0x7f0728009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f072800bed0 con 0x7f07340fd650 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073410d7c0 con 0x7f07340fd650 2024-03-22T02:20:58.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f072800b5c0 con 0x7f07340fd650 2024-03-22T02:20:58.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f072801c5b0 con 0x7f07340fd650 2024-03-22T02:20:58.769 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.767+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073410e330 con 0x7f07340fd650 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.768+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f072801c710 con 0x7f07340fd650 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.768+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0734102f60 con 0x7f07340fd650 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.768+0000 7f07317fa700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 0x7f0724038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.768+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0728049c70 con 0x7f07340fd650 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.768+0000 7f0733fff700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 0x7f0724038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.770 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.769+0000 7f0733fff700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 0x7f0724038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0720009990 tx=0x7f0720006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:58.773+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0728025400 con 0x7f07340fd650 2024-03-22T02:20:59.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.044+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f073410ed40 con 0x7f07340fd650 2024-03-22T02:20:59.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.044+0000 7f07317fa700 1 -- 172.21.15.122:0/3660958689 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0728011370 con 0x7f07340fd650 2024-03-22T02:20:59.045 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:20:59.045 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 msgr2=0x7f0724038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 0x7f0724038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0720009990 tx=0x7f0720006e40 comp rx=0 tx=0).stop 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 msgr2=0x7f073410d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0728009680 tx=0x7f0728009e30 comp rx=0 tx=0).stop 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 shutdown_connections 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0724036970 0x7f0724038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 --2- 172.21.15.122:0/3660958689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07340fd650 0x7f073410d080 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 >> 172.21.15.122:0/3660958689 conn(0x7f07340f70d0 msgr2=0x7f0734103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 shutdown_connections 2024-03-22T02:20:59.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:20:59.046+0000 7f073ac53700 1 -- 172.21.15.122:0/3660958689 wait complete. 2024-03-22T02:20:59.048 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:20:59.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:20:59 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3660958689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:00.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:21:00.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:21:00.870 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:21:00.915 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-22T02:21:02.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:21:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:02 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:21:02.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- 172.21.15.122:0/4068359560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 msgr2=0x7f08cc101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:02.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 --2- 172.21.15.122:0/4068359560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc101da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0048c0 tx=0x7f08bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:02.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- 172.21.15.122:0/4068359560 shutdown_connections 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 --2- 172.21.15.122:0/4068359560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc101da0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- 172.21.15.122:0/4068359560 >> 172.21.15.122:0/4068359560 conn(0x7f08cc071800 msgr2=0x7f08cc071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- 172.21.15.122:0/4068359560 shutdown_connections 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- 172.21.15.122:0/4068359560 wait complete. 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 Processor -- start 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.666+0000 7f08d22e4700 1 -- start start 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08d22e4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08d22e4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc066020 con 0x7f08cc1019d0 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08cbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08cbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:35218/0 (socket says 172.21.15.122:35218) 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08cbfff700 1 -- 172.21.15.122:0/1776795911 learned_addr learned my addr 172.21.15.122:0/1776795911 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08cbfff700 1 -- 172.21.15.122:0/1776795911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc005040 con 0x7f08cc1019d0 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08cbfff700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00b820 tx=0x7f08bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:02.668 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08bc0093c0 con 0x7f08cc1019d0 2024-03-22T02:21:02.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc10e7f0 con 0x7f08cc1019d0 2024-03-22T02:21:02.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08bc013420 con 0x7f08cc1019d0 2024-03-22T02:21:02.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08bc01c5b0 con 0x7f08cc1019d0 2024-03-22T02:21:02.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.667+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc10ece0 con 0x7f08cc1019d0 2024-03-22T02:21:02.669 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.668+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f08bc01c7f0 con 0x7f08cc1019d0 2024-03-22T02:21:02.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.668+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc100c40 con 0x7f08cc1019d0 2024-03-22T02:21:02.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.668+0000 7f08c97fa700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 0x7f08b803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:02.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.669+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f08bc049d50 con 0x7f08cc1019d0 2024-03-22T02:21:02.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.669+0000 7f08c3fff700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 0x7f08b803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:02.670 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.669+0000 7f08c3fff700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 0x7f08b803d230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009990 tx=0x7f08b4006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:02.674 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.674+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f08bc023300 con 0x7f08cc1019d0 2024-03-22T02:21:02.942 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.942+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08cc102ad0 con 0x7f08cc1019d0 2024-03-22T02:21:02.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.943+0000 7f08c97fa700 1 -- 172.21.15.122:0/1776795911 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f08cc102ad0 con 0x7f08cc1019d0 2024-03-22T02:21:02.943 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:21:02.943 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 msgr2=0x7f08b803d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 0x7f08b803d230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009990 tx=0x7f08b4006e40 comp rx=0 tx=0).stop 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 msgr2=0x7f08cc10e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00b820 tx=0x7f08bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 shutdown_connections 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08b803ad80 0x7f08b803d230 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 --2- 172.21.15.122:0/1776795911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08cc1019d0 0x7f08cc10e250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 >> 172.21.15.122:0/1776795911 conn(0x7f08cc071800 msgr2=0x7f08cc1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:02.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 shutdown_connections 2024-03-22T02:21:02.946 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:02.945+0000 7f08d22e4700 1 -- 172.21.15.122:0/1776795911 wait complete. 2024-03-22T02:21:02.946 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: Deploying daemon crash.smithi122 on smithi122 2024-03-22T02:21:04.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:03 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/1776795911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:04.602 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:21:04.602 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:21:04.856 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:05.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:06.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 -- 172.21.15.122:0/1146873253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 msgr2=0x7fb030102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:06.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 --2- 172.21.15.122:0/1146873253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030102eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0200048c0 tx=0x7fb020009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:06.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 -- 172.21.15.122:0/1146873253 shutdown_connections 2024-03-22T02:21:06.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 --2- 172.21.15.122:0/1146873253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030102eb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:06.493 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 -- 172.21.15.122:0/1146873253 >> 172.21.15.122:0/1146873253 conn(0x7fb0300f85e0 msgr2=0x7fb0300fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 -- 172.21.15.122:0/1146873253 shutdown_connections 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.490+0000 7fb034d46700 1 -- 172.21.15.122:0/1146873253 wait complete. 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb034d46700 1 Processor -- start 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb034d46700 1 -- start start 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb034d46700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb034d46700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0301136a0 con 0x7fb030102ae0 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb02e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb02e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:35234/0 (socket says 172.21.15.122:35234) 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.491+0000 7fb02e59c700 1 -- 172.21.15.122:0/2730431715 learned_addr learned my addr 172.21.15.122:0/2730431715 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb02e59c700 1 -- 172.21.15.122:0/2730431715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020005040 con 0x7fb030102ae0 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb02e59c700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b7f0 tx=0x7fb020009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb020009370 con 0x7fb030102ae0 2024-03-22T02:21:06.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb020013420 con 0x7fb030102ae0 2024-03-22T02:21:06.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0301138a0 con 0x7fb030102ae0 2024-03-22T02:21:06.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb02001c590 con 0x7fb030102ae0 2024-03-22T02:21:06.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.492+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb030113d90 con 0x7fb030102ae0 2024-03-22T02:21:06.495 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.493+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0300fe1e0 con 0x7fb030102ae0 2024-03-22T02:21:06.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.493+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fb020013a70 con 0x7fb030102ae0 2024-03-22T02:21:06.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.493+0000 7fb01effd700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 0x7fb018038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:06.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.493+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb0200493f0 con 0x7fb030102ae0 2024-03-22T02:21:06.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.493+0000 7fb02dd9b700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 0x7fb018038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:06.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.494+0000 7fb02dd9b700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 0x7fb018038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb024009990 tx=0x7fb024006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:06.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.498+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb020024d30 con 0x7fb030102ae0 2024-03-22T02:21:06.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.767+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb03005f5f0 con 0x7fb030102ae0 2024-03-22T02:21:06.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.781+0000 7fb01effd700 1 -- 172.21.15.122:0/2730431715 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb020021070 con 0x7fb030102ae0 2024-03-22T02:21:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:21:06.783 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 msgr2=0x7fb018038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 0x7fb018038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb024009990 tx=0x7fb024006e40 comp rx=0 tx=0).stop 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 msgr2=0x7fb030113160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b7f0 tx=0x7fb020009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 shutdown_connections 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb018036970 0x7fb018038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 --2- 172.21.15.122:0/2730431715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb030102ae0 0x7fb030113160 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.783+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 >> 172.21.15.122:0/2730431715 conn(0x7fb0300f85e0 msgr2=0x7fb0300f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.784+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 shutdown_connections 2024-03-22T02:21:06.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:06.784+0000 7fb034d46700 1 -- 172.21.15.122:0/2730431715 wait complete. 2024-03-22T02:21:06.786 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:21:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:06 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/2730431715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:08.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:21:08.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:21:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:08 smithi103 ceph-mon[29258]: Deploying daemon node-exporter.smithi122 on smithi122 2024-03-22T02:21:08.787 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:11.530 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/435221626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 msgr2=0x7fb8b40fb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/435221626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b40fb6b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40048c0 tx=0x7fb8a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/435221626 shutdown_connections 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/435221626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b40fb6b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/435221626 >> 172.21.15.122:0/435221626 conn(0x7fb8b4073050 msgr2=0x7fb8b4073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/435221626 shutdown_connections 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.528+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/435221626 wait complete. 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b9b8d700 1 Processor -- start 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b9b8d700 1 -- start start 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b9b8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b9b8d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b41136f0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:11.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:43638/0 (socket says 172.21.15.122:43638) 2024-03-22T02:21:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.529+0000 7fb8b37fe700 1 -- 172.21.15.122:0/2749891143 learned_addr learned my addr 172.21.15.122:0/2749891143 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:21:11.532 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb8b37fe700 1 -- 172.21.15.122:0/2749891143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a4005040 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb8b37fe700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b7f0 tx=0x7fb8a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8a4009370 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b41138f0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8a4013420 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8a401c590 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.530+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b4113d60 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.531+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b40fe3b0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.531+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fb8a40094d0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.531+0000 7fb89bfff700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 0x7fb89c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.531+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb8a40493d0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.533 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.531+0000 7fb8b2ffd700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 0x7fb89c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:11.534 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.532+0000 7fb8b2ffd700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 0x7fb89c038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009990 tx=0x7fb8a8006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:11.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.536+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb8a4013590 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.817+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8b405f5f0 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.818+0000 7fb89bfff700 1 -- 172.21.15.122:0/2749891143 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb8a4021030 con 0x7fb8b40fb2e0 2024-03-22T02:21:11.820 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:21:11.820 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:21:11.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 msgr2=0x7fb89c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:11.821 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 0x7fb89c038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009990 tx=0x7fb8a8006e40 comp rx=0 tx=0).stop 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 msgr2=0x7fb8b41131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b7f0 tx=0x7fb8a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 shutdown_connections 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb89c036970 0x7fb89c038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 --2- 172.21.15.122:0/2749891143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8b40fb2e0 0x7fb8b41131b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 >> 172.21.15.122:0/2749891143 conn(0x7fb8b4073050 msgr2=0x7fb8b40f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 shutdown_connections 2024-03-22T02:21:11.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:11.820+0000 7fb8b9b8d700 1 -- 172.21.15.122:0/2749891143 wait complete. 2024-03-22T02:21:11.823 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:21:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:11 smithi103 ceph-mon[29258]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:11.885 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:21:13.070 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:13.071 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: Deploying daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:21:13.071 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:12 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/2749891143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:13.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:21:13.477 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:21:13.740 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:14.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:13 smithi103 ceph-mon[29258]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:15.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 -- 172.21.15.122:0/2870816475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 msgr2=0x7efc44101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:15.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 --2- 172.21.15.122:0/2870816475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc44101d80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efc380048c0 tx=0x7efc38009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:15.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 -- 172.21.15.122:0/2870816475 shutdown_connections 2024-03-22T02:21:15.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 --2- 172.21.15.122:0/2870816475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc44101d80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 -- 172.21.15.122:0/2870816475 >> 172.21.15.122:0/2870816475 conn(0x7efc440f7510 msgr2=0x7efc440f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 -- 172.21.15.122:0/2870816475 shutdown_connections 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.347+0000 7efc4ad81700 1 -- 172.21.15.122:0/2870816475 wait complete. 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.348+0000 7efc4ad81700 1 Processor -- start 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.348+0000 7efc4ad81700 1 -- start start 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.348+0000 7efc4ad81700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.348+0000 7efc4ad81700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4410e640 con 0x7efc441019b0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc48b1d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc48b1d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:43648/0 (socket says 172.21.15.122:43648) 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc48b1d700 1 -- 172.21.15.122:0/3195285351 learned_addr learned my addr 172.21.15.122:0/3195285351 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc48b1d700 1 -- 172.21.15.122:0/3195285351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc38005040 con 0x7efc441019b0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc48b1d700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc3800b820 tx=0x7efc38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc380093c0 con 0x7efc441019b0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc38013420 con 0x7efc441019b0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4410e840 con 0x7efc441019b0 2024-03-22T02:21:15.352 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3801c5b0 con 0x7efc441019b0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.349+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4410ed30 con 0x7efc441019b0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.350+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4410ef20 con 0x7efc441019b0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.350+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7efc38013a90 con 0x7efc441019b0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.350+0000 7efc417fa700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 0x7efc34038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.350+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7efc38049a90 con 0x7efc441019b0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.351+0000 7efc43fff700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 0x7efc34038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:15.353 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.351+0000 7efc43fff700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 0x7efc34038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc30009990 tx=0x7efc30006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:15.356 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.355+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efc380098a0 con 0x7efc441019b0 2024-03-22T02:21:15.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.638+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc4404bed0 con 0x7efc441019b0 2024-03-22T02:21:15.656 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.655+0000 7efc417fa700 1 -- 172.21.15.122:0/3195285351 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7efc38011370 con 0x7efc441019b0 2024-03-22T02:21:15.656 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:21:15.656 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:18:38.790476Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 msgr2=0x7efc34038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 0x7efc34038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc30009990 tx=0x7efc30006e40 comp rx=0 tx=0).stop 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 msgr2=0x7efc4410e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc3800b820 tx=0x7efc38009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 shutdown_connections 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efc34036970 0x7efc34038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:15.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 --2- 172.21.15.122:0/3195285351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc441019b0 0x7efc4410e100 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:15.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 >> 172.21.15.122:0/3195285351 conn(0x7efc440f7510 msgr2=0x7efc440f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:15.659 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 shutdown_connections 2024-03-22T02:21:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:15.657+0000 7efc4ad81700 1 -- 172.21.15.122:0/3195285351 wait complete. 2024-03-22T02:21:15.675 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-22T02:21:16.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:15 smithi103 ceph-mon[29258]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:15 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3195285351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: Deploying daemon mon.smithi122 on smithi122 2024-03-22T02:21:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:16 smithi103 ceph-mon[29258]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:17.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-22T02:21:17.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:21:18.945 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:21:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:19 smithi103 ceph-mon[29258]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:21.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 -- 172.21.15.122:0/1968496222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 msgr2=0x7fb1880706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:21.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 --2- 172.21.15.122:0/1968496222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1880706d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb1780048c0 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:21.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 -- 172.21.15.122:0/1968496222 shutdown_connections 2024-03-22T02:21:21.029 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 --2- 172.21.15.122:0/1968496222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1880706d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 -- 172.21.15.122:0/1968496222 >> 172.21.15.122:0/1968496222 conn(0x7fb1880f86f0 msgr2=0x7fb1880fab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 -- 172.21.15.122:0/1968496222 shutdown_connections 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.918+0000 7fb190567700 1 -- 172.21.15.122:0/1968496222 wait complete. 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb190567700 1 Processor -- start 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb190567700 1 -- start start 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb190567700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb190567700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881159f0 con 0x7fb188070300 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb18e303700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb18e303700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:56966/0 (socket says 172.21.15.122:56966) 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.919+0000 7fb18e303700 1 -- 172.21.15.122:0/3715008903 learned_addr learned my addr 172.21.15.122:0/3715008903 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb18e303700 1 -- 172.21.15.122:0/3715008903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178005040 con 0x7fb188070300 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb18e303700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b820 tx=0x7fb178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1780093c0 con 0x7fb188070300 2024-03-22T02:21:21.030 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb188115bf0 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb178013420 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb17801c5b0 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.920+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb188116060 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.921+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fb178009520 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.921+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188071a40 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.921+0000 7fb17effd700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 0x7fb174038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.921+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb178049b10 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.921+0000 7fb18db02700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 0x7fb174038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.922+0000 7fb18db02700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 0x7fb174038e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb184009990 tx=0x7fb184006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.926+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb178013a90 con 0x7fb188070300 2024-03-22T02:21:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:20.984+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb17804e020 con 0x7fb188070300 2024-03-22T02:21:21.201 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:21 smithi122 systemd[1]: Started Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:21:21.201 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:21.200+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb18804bed0 con 0x7fb188070300 2024-03-22T02:21:26.032 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.031+0000 7fb17effd700 1 -- 172.21.15.122:0/3715008903 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7fb178011370 con 0x7fb188070300 2024-03-22T02:21:26.033 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-22T02:21:26.033 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","modified":"2024-03-22T02:21:20.985271Z","created":"2024-03-22T02:18:38.790476Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi122","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 msgr2=0x7fb174038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 0x7fb174038e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb184009990 tx=0x7fb184006e40 comp rx=0 tx=0).stop 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 msgr2=0x7fb1881154b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b820 tx=0x7fb178009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 shutdown_connections 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb174036970 0x7fb174038e20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 --2- 172.21.15.122:0/3715008903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb188070300 0x7fb1881154b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 >> 172.21.15.122:0/3715008903 conn(0x7fb1880f86f0 msgr2=0x7fb188105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 shutdown_connections 2024-03-22T02:21:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:21:26.034+0000 7fb190567700 1 -- 172.21.15.122:0/3715008903 wait complete. 2024-03-22T02:21:26.036 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-03-22T02:21:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:21:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: mon.smithi103 calling monitor election 2024-03-22T02:21:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/3715008903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: mon.smithi122 calling monitor election 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:21:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: monmap e2: 2 mons at {smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],smithi122=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: fsmap 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: osdmap e5: 0 total, 0 up, 0 in 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: mgrmap e17: smithi103.pfvmui(active, since 35s) 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: overall HEALTH_OK 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:21:26.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:26 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: mon.smithi103 calling monitor election 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3715008903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:21:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: mon.smithi122 calling monitor election 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: monmap e2: 2 mons at {smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],smithi122=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: fsmap 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: osdmap e5: 0 total, 0 up, 0 in 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: mgrmap e17: smithi103.pfvmui(active, since 35s) 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: overall HEALTH_OK 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:21:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:26.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:21:26.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/3892846722' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:21:26.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:26 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:26.899 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-22T02:21:26.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-22T02:21:27.169 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: mgrmap e18: smithi103.pfvmui(active, since 35s), standbys: smithi122.prgtti 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:27.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:27.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: mgrmap e18: smithi103.pfvmui(active, since 35s), standbys: smithi122.prgtti 2024-03-22T02:21:27.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:21:27.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:27.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:21:27.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:27.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 -- 172.21.15.103:0/36884263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c101a20 msgr2=0x7f9d6c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 --2- 172.21.15.103:0/36884263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c101a20 0x7f9d6c101df0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9d680048c0 tx=0x7f9d68009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 -- 172.21.15.103:0/36884263 shutdown_connections 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 --2- 172.21.15.103:0/36884263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c101a20 0x7f9d6c101df0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 -- 172.21.15.103:0/36884263 >> 172.21.15.103:0/36884263 conn(0x7f9d6c0f7540 msgr2=0x7f9d6c0f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 -- 172.21.15.103:0/36884263 shutdown_connections 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 -- 172.21.15.103:0/36884263 wait complete. 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.738+0000 7f9d73aed700 1 Processor -- start 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d73aed700 1 -- start start 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d73aed700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d73aed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d73aed700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c1173b0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d73aed700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c1174f0 con 0x7f9d6c101a20 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56892/0 (socket says 172.21.15.103:56892) 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 -- 172.21.15.103:0/1444045360 learned_addr learned my addr 172.21.15.103:0/1444045360 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71889700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71889700 1 -- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 msgr2=0x7f9d6c10e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71889700 1 -- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 msgr2=0x7f9d6c10e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71889700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71889700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 -- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 msgr2=0x7f9d6c10e2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:28.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 -- 172.21.15.103:0/1444045360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68005040 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.739+0000 7f9d71088700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b790 tx=0x7f9d5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.740+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c010870 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.740+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d5c00f3e0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.740+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c1176f0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.740+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c010ea0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.740+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c117cb0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.741+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c100c90 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.741+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f9d5c00f550 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.742+0000 7f9d627fc700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 0x7f9d5806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.742+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9d5c086650 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.742+0000 7f9d71889700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 0x7f9d5806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.742+0000 7f9d71889700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 0x7f9d5806b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d680043d0 tx=0x7f9d68009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:28.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.747+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9d5c053a60 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.951+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9d6c0663e0 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.951+0000 7f9d627fc700 1 -- 172.21.15.103:0/1444045360 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f9d5c014100 con 0x7f9d6c10e7f0 2024-03-22T02:21:28.952 INFO:teuthology.orchestra.run.smithi103.stdout:# minimal ceph.conf for 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:21:28.952 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-03-22T02:21:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:21:28.952 INFO:teuthology.orchestra.run.smithi103.stdout: mon_host = [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-03-22T02:21:28.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 msgr2=0x7f9d5806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 0x7f9d5806b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d680043d0 tx=0x7f9d68009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 msgr2=0x7f9d6c10ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b790 tx=0x7f9d5c009250 comp rx=0 tx=0).stop 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 shutdown_connections 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9d58069370 0x7f9d5806b820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d6c101a20 0x7f9d6c10e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 --2- 172.21.15.103:0/1444045360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d6c10e7f0 0x7f9d6c10ec60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 >> 172.21.15.103:0/1444045360 conn(0x7f9d6c0f7540 msgr2=0x7f9d6c0f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 shutdown_connections 2024-03-22T02:21:28.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:28.953+0000 7f9d73aed700 1 -- 172.21.15.103:0/1444045360 wait complete. 2024-03-22T02:21:29.550 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:29.550 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.550 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.550 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1444045360' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:21:29.551 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.574 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-22T02:21:29.575 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:21:29.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T02:21:29.604 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:21:29.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:21:29.671 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:21:29.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T02:21:29.702 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:21:29.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1444045360' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:21:29.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:29.768 INFO:tasks.cephadm:Deploying OSDs... 2024-03-22T02:21:29.769 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:21:29.769 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T02:21:29.785 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T02:21:29.785 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-22 02:20:42.213979867 +0000 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-22 02:12:26.371490558 +0000 2024-03-22T02:21:29.840 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-22 02:12:26.371490558 +0000 2024-03-22T02:21:29.841 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-22 02:12:26.371490558 +0000 2024-03-22T02:21:29.841 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T02:21:29.907 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-22T02:21:29.908 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-22T02:21:29.908 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000259772 s, 2.0 MB/s 2024-03-22T02:21:29.910 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T02:21:29.965 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-03-22T02:21:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T02:21:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.022 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-22T02:21:30.022 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.022 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.023 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-22 02:20:42.214979847 +0000 2024-03-22T02:21:30.023 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-22 02:12:26.724483617 +0000 2024-03-22T02:21:30.023 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-22 02:12:26.724483617 +0000 2024-03-22T02:21:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-22 02:12:26.724483617 +0000 2024-03-22T02:21:30.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T02:21:30.088 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-22T02:21:30.088 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-22T02:21:30.088 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000201181 s, 2.5 MB/s 2024-03-22T02:21:30.089 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T02:21:30.145 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-03-22T02:21:30.199 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T02:21:30.199 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.199 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-22 02:20:42.214979847 +0000 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-22 02:12:27.069476866 +0000 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-22 02:12:27.069476866 +0000 2024-03-22T02:21:30.200 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-22 02:12:27.069476866 +0000 2024-03-22T02:21:30.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T02:21:30.266 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-22T02:21:30.266 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-22T02:21:30.266 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000183763 s, 2.8 MB/s 2024-03-22T02:21:30.267 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T02:21:30.324 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-03-22T02:21:30.379 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T02:21:30.379 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-22 02:20:42.215979827 +0000 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-22 02:12:27.406470271 +0000 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-22 02:12:27.406470271 +0000 2024-03-22T02:21:30.380 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-22 02:12:27.406470271 +0000 2024-03-22T02:21:30.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T02:21:30.445 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-22T02:21:30.445 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-22T02:21:30.445 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000176411 s, 2.9 MB/s 2024-03-22T02:21:30.446 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T02:21:30.503 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:21:30.503 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T02:21:30.521 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T02:21:30.521 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-03-22T02:21:30.578 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T02:21:30.578 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-22 02:21:19.259664217 +0000 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-22 02:13:34.517832635 +0000 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-22 02:13:34.517832635 +0000 2024-03-22T02:21:30.579 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-22 02:13:34.517832635 +0000 2024-03-22T02:21:30.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T02:21:30.647 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-22T02:21:30.647 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-22T02:21:30.647 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000301993 s, 1.7 MB/s 2024-03-22T02:21:30.648 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T02:21:30.706 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-22 02:21:19.259664217 +0000 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-22 02:13:34.871823820 +0000 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-22 02:13:34.871823820 +0000 2024-03-22T02:21:30.763 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-22 02:13:34.871823820 +0000 2024-03-22T02:21:30.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T02:21:30.775 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:30 smithi103 ceph-mon[29258]: Reconfiguring mon.smithi103 (unknown last config time)... 2024-03-22T02:21:30.775 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:30 smithi103 ceph-mon[29258]: Reconfiguring daemon mon.smithi103 on smithi103 2024-03-22T02:21:30.775 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:30 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:30.827 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:30 smithi122 ceph-mon[36403]: Reconfiguring mon.smithi103 (unknown last config time)... 2024-03-22T02:21:30.828 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:30 smithi122 ceph-mon[36403]: Reconfiguring daemon mon.smithi103 on smithi103 2024-03-22T02:21:30.828 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:30 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:30.830 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-22T02:21:30.831 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-22T02:21:30.831 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000581029 s, 881 kB/s 2024-03-22T02:21:30.832 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T02:21:30.888 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-22 02:21:19.260664191 +0000 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-22 02:13:35.229814905 +0000 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-22 02:13:35.229814905 +0000 2024-03-22T02:21:30.944 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-22 02:13:35.229814905 +0000 2024-03-22T02:21:30.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T02:21:31.011 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-22T02:21:31.011 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-22T02:21:31.011 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000196846 s, 2.6 MB/s 2024-03-22T02:21:31.012 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T02:21:31.068 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-22 02:21:19.260664191 +0000 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-22 02:13:35.570806414 +0000 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-22 02:13:35.570806414 +0000 2024-03-22T02:21:31.124 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-22 02:13:35.570806414 +0000 2024-03-22T02:21:31.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T02:21:31.189 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-22T02:21:31.190 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-22T02:21:31.190 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000179726 s, 2.8 MB/s 2024-03-22T02:21:31.191 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T02:21:31.246 INFO:tasks.cephadm:Deploying osd.0 on smithi103 with /dev/vg_nvme/lv_4... 2024-03-22T02:21:31.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: Reconfiguring mgr.smithi103.pfvmui (unknown last config time)... 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:21:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:21:32.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:32.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:31 smithi122 ceph-mon[36403]: Reconfiguring daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: Reconfiguring mgr.smithi103.pfvmui (unknown last config time)... 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:31 smithi103 ceph-mon[29258]: Reconfiguring daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:21:32.208 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:21:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: Reconfiguring crash.smithi103 (monmap changed)... 2024-03-22T02:21:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:21:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:33.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: Reconfiguring daemon crash.smithi103 on smithi103 2024-03-22T02:21:33.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:33 smithi103 ceph-mon[29258]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: Reconfiguring crash.smithi103 (monmap changed)... 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: Reconfiguring daemon crash.smithi103 on smithi103 2024-03-22T02:21:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:33 smithi122 ceph-mon[36403]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112947 s, 928 MB/s 2024-03-22T02:21:35.213 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: Reconfiguring alertmanager.smithi103 (dependencies changed)... 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: Reconfiguring daemon alertmanager.smithi103 on smithi103 2024-03-22T02:21:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: Reconfiguring alertmanager.smithi103 (dependencies changed)... 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: Reconfiguring daemon alertmanager.smithi103 on smithi103 2024-03-22T02:21:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:37 smithi122 ceph-mon[36403]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:37 smithi103 ceph-mon[29258]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:37.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-03-22T02:21:39.123 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:21:39.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:39 smithi103 ceph-mon[29258]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:39 smithi122 ceph-mon[36403]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- 172.21.15.103:0/1060857312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 msgr2=0x7fde700fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 --2- 172.21.15.103:0/1060857312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700fc0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde6c004940 tx=0x7fde6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- 172.21.15.103:0/1060857312 shutdown_connections 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 --2- 172.21.15.103:0/1060857312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 --2- 172.21.15.103:0/1060857312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700fc0e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:40.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- 172.21.15.103:0/1060857312 >> 172.21.15.103:0/1060857312 conn(0x7fde700f7450 msgr2=0x7fde700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- 172.21.15.103:0/1060857312 shutdown_connections 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- 172.21.15.103:0/1060857312 wait complete. 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 Processor -- start 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.744+0000 7fde7741c700 1 -- start start 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700fe800 con 0x7fde700fbcd0 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700fe970 con 0x7fde700fcf80 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47028/0 (socket says 172.21.15.103:47028) 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 -- 172.21.15.103:0/555533144 learned_addr learned my addr 172.21.15.103:0/555533144 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:21:40.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde749b7700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 -- 172.21.15.103:0/555533144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 msgr2=0x7fde700fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 -- 172.21.15.103:0/555533144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c005040 con 0x7fde700fbcd0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde749b7700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde751b8700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00b3b0 tx=0x7fde6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c00e040 con 0x7fde700fbcd0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde6c00bcc0 con 0x7fde700fbcd0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde700febf0 con 0x7fde700fbcd0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c01c630 con 0x7fde700fbcd0 2024-03-22T02:21:40.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.745+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70115a60 con 0x7fde700fbcd0 2024-03-22T02:21:40.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.746+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7004c710 con 0x7fde700fbcd0 2024-03-22T02:21:40.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.747+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fde6c013980 con 0x7fde700fbcd0 2024-03-22T02:21:40.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.747+0000 7fde65ffb700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:40.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.747+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fde6c086df0 con 0x7fde700fbcd0 2024-03-22T02:21:40.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.747+0000 7fde749b7700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:40.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.748+0000 7fde749b7700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 0x7fde5c06b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde60009b10 tx=0x7fde60008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:40.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.752+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fde6c057c90 con 0x7fde700fbcd0 2024-03-22T02:21:40.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:21:40.950+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fde7005e400 con 0x7fde5c069210 2024-03-22T02:21:41.699 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:41.699 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:21:41.700 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:21:41.700 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:41.700 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:41.700 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:42.765 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:42 smithi103 ceph-mon[29258]: from='client.14242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:21:42.765 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:42 smithi103 ceph-mon[29258]: Reconfiguring grafana.smithi103 (dependencies changed)... 2024-03-22T02:21:42.765 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:42 smithi103 ceph-mon[29258]: Reconfiguring daemon grafana.smithi103 on smithi103 2024-03-22T02:21:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:42 smithi122 ceph-mon[36403]: from='client.14242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:21:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:42 smithi122 ceph-mon[36403]: Reconfiguring grafana.smithi103 (dependencies changed)... 2024-03-22T02:21:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:42 smithi122 ceph-mon[36403]: Reconfiguring daemon grafana.smithi103 on smithi103 2024-03-22T02:21:43.752 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:43 smithi103 ceph-mon[29258]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:43.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:43 smithi122 ceph-mon[36403]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:45.562 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:45 smithi103 ceph-mon[29258]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:45 smithi122 ceph-mon[36403]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:47 smithi122 ceph-mon[36403]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:21:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:47 smithi122 ceph-mon[36403]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:21:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:47 smithi122 ceph-mon[36403]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:47.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:47.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:47 smithi103 ceph-mon[29258]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:21:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:47 smithi103 ceph-mon[29258]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:21:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:47 smithi103 ceph-mon[29258]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:48 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3506566117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1ef5a-9736-4d69-9026-3dc88ac0880d"}]: dispatch 2024-03-22T02:21:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:48 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3506566117' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1ef5a-9736-4d69-9026-3dc88ac0880d"}]': finished 2024-03-22T02:21:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:48 smithi122 ceph-mon[36403]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T02:21:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:48 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:21:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:48 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/364628904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:21:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3506566117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d1ef5a-9736-4d69-9026-3dc88ac0880d"}]: dispatch 2024-03-22T02:21:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3506566117' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d1ef5a-9736-4d69-9026-3dc88ac0880d"}]': finished 2024-03-22T02:21:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:48 smithi103 ceph-mon[29258]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T02:21:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:48 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:21:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/364628904' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:21:49.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:49 smithi122 ceph-mon[36403]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:49.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:49 smithi103 ceph-mon[29258]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:50.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:21:51.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:51 smithi103 ceph-mon[29258]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:51 smithi122 ceph-mon[36403]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:53.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:53 smithi103 ceph-mon[29258]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:53 smithi122 ceph-mon[36403]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:55 smithi103 ceph-mon[29258]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:55 smithi122 ceph-mon[36403]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:57.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:57 smithi122 ceph-mon[36403]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:57.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:57.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:57.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:21:57.757 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:57.799 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:57 smithi103 ceph-mon[29258]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:57.799 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:57.799 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:57.799 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:21:57.799 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:58 smithi103 ceph-mon[29258]: Reconfiguring crash.smithi122 (monmap changed)... 2024-03-22T02:21:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:58 smithi103 ceph-mon[29258]: Reconfiguring daemon crash.smithi122 on smithi122 2024-03-22T02:21:58.738 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:58 smithi122 ceph-mon[36403]: Reconfiguring crash.smithi122 (monmap changed)... 2024-03-22T02:21:58.738 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:58 smithi122 ceph-mon[36403]: Reconfiguring daemon crash.smithi122 on smithi122 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: Deploying daemon osd.0 on smithi103 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:59.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: Reconfiguring mgr.smithi122.prgtti (monmap changed)... 2024-03-22T02:21:59.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:21:59.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:21:59.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:59.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:21:59 smithi122 ceph-mon[36403]: Reconfiguring daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:21:59.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:21:59.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: Deploying daemon osd.0 on smithi103 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: Reconfiguring mgr.smithi122.prgtti (monmap changed)... 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:21:59 smithi103 ceph-mon[29258]: Reconfiguring daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:22:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: Reconfiguring mon.smithi122 (monmap changed)... 2024-03-22T02:22:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:22:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:22:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: Reconfiguring daemon mon.smithi122 on smithi122 2024-03-22T02:22:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:01 smithi122 ceph-mon[36403]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:01.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:01.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: Reconfiguring mon.smithi122 (monmap changed)... 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: Reconfiguring daemon mon.smithi122 on smithi122 2024-03-22T02:22:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:01 smithi103 ceph-mon[29258]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:03.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi103.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi103.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T02:22:03.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:02 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi103.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T02:22:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi103.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:02 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:03 smithi122 ceph-mon[36403]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:04.106 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:03 smithi103 ceph-mon[29258]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:04.106 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:04.106 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:05.329 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:05.329 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:05.330 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:05 smithi103 ceph-mon[29258]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:05.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:05.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:05.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:05 smithi122 ceph-mon[36403]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:06.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:06 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:06 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:07 smithi122 ceph-mon[36403]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:07 smithi103 ceph-mon[29258]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:08.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.011+0000 7fde65ffb700 1 -- 172.21.15.103:0/555533144 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde7005e400 con 0x7fde5c069210 2024-03-22T02:22:08.011 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 0 on host 'smithi103' 2024-03-22T02:22:08.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 msgr2=0x7fde5c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:08.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 0x7fde5c06b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde60009b10 tx=0x7fde60008040 comp rx=0 tx=0).stop 2024-03-22T02:22:08.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 msgr2=0x7fde700ffc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:08.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00b3b0 tx=0x7fde6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 shutdown_connections 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fde5c069210 0x7fde5c06b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde700fbcd0 0x7fde700ffc10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 --2- 172.21.15.103:0/555533144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde700fcf80 0x7fde700fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 >> 172.21.15.103:0/555533144 conn(0x7fde700f7450 msgr2=0x7fde70066860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 shutdown_connections 2024-03-22T02:22:08.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:08.013+0000 7fde7741c700 1 -- 172.21.15.103:0/555533144 wait complete. 2024-03-22T02:22:09.000 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:09.000 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:09.000 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:08 smithi103 ceph-mon[29258]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:22:09.000 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:08 smithi103 ceph-mon[29258]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:08 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:08 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:08 smithi122 ceph-mon[36403]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:22:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:08 smithi122 ceph-mon[36403]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:09.747 DEBUG:teuthology.orchestra.run.smithi103:osd.0> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.0.service 2024-03-22T02:22:09.749 INFO:tasks.cephadm:Deploying osd.1 on smithi103 with /dev/vg_nvme/lv_3... 2024-03-22T02:22:09.749 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T02:22:10.062 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:10.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:10.311 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[48258]: 2024-03-22T02:22:09.983+0000 7fd3710bf700 -1 osd.0 0 waiting for initial osdmap 2024-03-22T02:22:10.311 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:22:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[48258]: 2024-03-22T02:22:09.987+0000 7fd36c6bb700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:11.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:10 smithi103 ceph-mon[29258]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:22:11.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:10 smithi103 ceph-mon[29258]: osdmap e8: 1 total, 0 up, 1 in 2024-03-22T02:22:11.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:11.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:11.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:10 smithi103 ceph-mon[29258]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:10 smithi122 ceph-mon[36403]: from='osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:22:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:10 smithi122 ceph-mon[36403]: osdmap e8: 1 total, 0 up, 1 in 2024-03-22T02:22:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:10 smithi122 ceph-mon[36403]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:22:11.917 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126552 s, 829 MB/s 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 2024-03-22T02:22:11.918 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] boot 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T02:22:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:22:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:22:12.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:12.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: osd.0 [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] boot 2024-03-22T02:22:12.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T02:22:12.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:22:13.267 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:12 smithi103 ceph-mon[29258]: osdmap e10: 1 total, 1 up, 1 in 2024-03-22T02:22:13.268 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:12 smithi103 ceph-mon[29258]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:12 smithi122 ceph-mon[36403]: osdmap e10: 1 total, 1 up, 1 in 2024-03-22T02:22:13.378 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:12 smithi122 ceph-mon[36403]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:13.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-03-22T02:22:14.727 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi103 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi103 to 12864M 2024-03-22T02:22:15.949 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:15.950 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:15.950 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:15.950 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:15.950 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:15 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi103 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi103 to 12864M 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:16.068 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:15 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 -- 172.21.15.103:0/1680457391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fc2b0 msgr2=0x7fc4800fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 --2- 172.21.15.103:0/1680457391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fc2b0 0x7fc4800fc6c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc474004940 tx=0x7fc474009e30 comp rx=0 tx=0).stop 2024-03-22T02:22:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 -- 172.21.15.103:0/1680457391 shutdown_connections 2024-03-22T02:22:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 --2- 172.21.15.103:0/1680457391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fcc00 0x7fc4800fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:16.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 --2- 172.21.15.103:0/1680457391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fc2b0 0x7fc4800fc6c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 -- 172.21.15.103:0/1680457391 >> 172.21.15.103:0/1680457391 conn(0x7fc4800f6ba0 msgr2=0x7fc4800f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 -- 172.21.15.103:0/1680457391 shutdown_connections 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.306+0000 7fc485bee700 1 -- 172.21.15.103:0/1680457391 wait complete. 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 Processor -- start 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 -- start start 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 0x7fc4800ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800fe960 con 0x7fc4800fcc00 2024-03-22T02:22:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc485bee700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800fead0 con 0x7fc4800fc2b0 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39328/0 (socket says 172.21.15.103:39328) 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47effd700 1 -- 172.21.15.103:0/1729855442 learned_addr learned my addr 172.21.15.103:0/1729855442 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47f7fe700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 0x7fc4800ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47f7fe700 1 -- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 msgr2=0x7fc4800fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47f7fe700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47f7fe700 1 -- 172.21.15.103:0/1729855442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474005040 con 0x7fc4800fc2b0 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.307+0000 7fc47effd700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:16.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc47f7fe700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 0x7fc4800ffee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc474000c00 tx=0x7fc474009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc47400e030 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4800fed50 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc480115a20 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc474013de0 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.308+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc47401cdc0 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.309+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48005f5f0 con 0x7fc4800fc2b0 2024-03-22T02:22:16.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.310+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fc4740123c0 con 0x7fc4800fc2b0 2024-03-22T02:22:16.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.310+0000 7fc467fff700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 0x7fc46806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:16.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.310+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fc474087920 con 0x7fc4800fc2b0 2024-03-22T02:22:16.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.310+0000 7fc47effd700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 0x7fc46806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:16.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.311+0000 7fc47effd700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 0x7fc46806b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4700041d0 tx=0x7fc47000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:16.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.315+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc474054a50 con 0x7fc4800fc2b0 2024-03-22T02:22:16.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:16.517+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc48005e400 con 0x7fc468069210 2024-03-22T02:22:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:22:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:22:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:22:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:22:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:17.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:17 smithi103 ceph-mon[29258]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:17 smithi103 ceph-mon[29258]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:22:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:17 smithi122 ceph-mon[36403]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:17 smithi122 ceph-mon[36403]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:22:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:19 smithi103 ceph-mon[29258]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:19 smithi122 ceph-mon[36403]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:21.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2237778454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65887971-4f0f-4bda-a25a-621be3121b4d"}]: dispatch 2024-03-22T02:22:21.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2237778454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65887971-4f0f-4bda-a25a-621be3121b4d"}]': finished 2024-03-22T02:22:21.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T02:22:21.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:21.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:21.052 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1777495682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2237778454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65887971-4f0f-4bda-a25a-621be3121b4d"}]: dispatch 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2237778454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65887971-4f0f-4bda-a25a-621be3121b4d"}]': finished 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:20 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1777495682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:22.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:22.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:23 smithi103 ceph-mon[29258]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:23 smithi122 ceph-mon[36403]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:26.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:25 smithi103 ceph-mon[29258]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:25 smithi122 ceph-mon[36403]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:28.236 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:27 smithi103 ceph-mon[29258]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:28.236 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:22:28.237 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:27 smithi122 ceph-mon[36403]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:22:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:28 smithi103 ceph-mon[29258]: Deploying daemon osd.1 on smithi103 2024-03-22T02:22:29.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:28 smithi122 ceph-mon[36403]: Deploying daemon osd.1 on smithi103 2024-03-22T02:22:30.054 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:29 smithi103 ceph-mon[29258]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:30.341 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:29 smithi122 ceph-mon[36403]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:32.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:31 smithi103 ceph-mon[29258]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:31 smithi122 ceph-mon[36403]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:33.583 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:33.583 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:33.584 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:33 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:33.584 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:33 smithi122 ceph-mon[36403]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:33 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:33 smithi103 ceph-mon[29258]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:35 smithi122 ceph-mon[36403]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:35 smithi103 ceph-mon[29258]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:35.690 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 1 on host 'smithi103' 2024-03-22T02:22:35.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.688+0000 7fc467fff700 1 -- 172.21.15.103:0/1729855442 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc48005e400 con 0x7fc468069210 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.690+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 msgr2=0x7fc46806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.690+0000 7fc485bee700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 0x7fc46806b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc4700041d0 tx=0x7fc47000b040 comp rx=0 tx=0).stop 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.690+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 msgr2=0x7fc4800ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.690+0000 7fc485bee700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 0x7fc4800ffee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc474000c00 tx=0x7fc474009e30 comp rx=0 tx=0).stop 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 shutdown_connections 2024-03-22T02:22:35.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc468069210 0x7fc46806b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4800fc2b0 0x7fc4800ffee0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 --2- 172.21.15.103:0/1729855442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4800fcc00 0x7fc4800fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 >> 172.21.15.103:0/1729855442 conn(0x7fc4800f6ba0 msgr2=0x7fc4800f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 shutdown_connections 2024-03-22T02:22:35.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:35.691+0000 7fc485bee700 1 -- 172.21.15.103:0/1729855442 wait complete. 2024-03-22T02:22:36.314 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:36 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:36.314 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:36 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:36.314 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:36 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:36.338 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.1.service 2024-03-22T02:22:36.340 INFO:tasks.cephadm:Deploying osd.2 on smithi103 with /dev/vg_nvme/lv_2... 2024-03-22T02:22:36.341 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T02:22:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:36 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:36 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:36 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:37.460 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:22:37.460 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:37.460 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:22:37.460 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T02:22:37.460 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:37.461 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-mon[29258]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:37 smithi122 ceph-mon[36403]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:22:37.761 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:22:37.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[55397]: 2024-03-22T02:22:37.691+0000 7ffa1bec7700 -1 osd.1 0 waiting for initial osdmap 2024-03-22T02:22:37.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:22:37 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[55397]: 2024-03-22T02:22:37.695+0000 7ffa154bf700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:22:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:38 smithi103 ceph-mon[29258]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:22:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:38 smithi103 ceph-mon[29258]: osdmap e13: 2 total, 1 up, 2 in 2024-03-22T02:22:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:38 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:38 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:38 smithi122 ceph-mon[36403]: from='osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:22:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:38 smithi122 ceph-mon[36403]: osdmap e13: 2 total, 1 up, 2 in 2024-03-22T02:22:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:38 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:38 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] boot 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T02:22:39.955 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:39 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:22:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:22:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:22:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: osd.1 [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] boot 2024-03-22T02:22:40.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T02:22:40.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:39 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142853 s, 734 MB/s 2024-03-22T02:22:40.515 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-22T02:22:40.794 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:40 smithi103 ceph-mon[29258]: osdmap e15: 2 total, 2 up, 2 in 2024-03-22T02:22:41.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-03-22T02:22:41.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:40 smithi122 ceph-mon[36403]: osdmap e15: 2 total, 2 up, 2 in 2024-03-22T02:22:41.365 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:22:41.814 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:41 smithi103 ceph-mon[29258]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:41.814 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:41.814 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:41 smithi122 ceph-mon[36403]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:44.035 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:43 smithi103 ceph-mon[29258]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:43 smithi122 ceph-mon[36403]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:44.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 -- 172.21.15.103:0/3912099311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 msgr2=0x7fe58c0ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 --2- 172.21.15.103:0/3912099311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c0ffa00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe588004940 tx=0x7fe588009e30 comp rx=0 tx=0).stop 2024-03-22T02:22:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 -- 172.21.15.103:0/3912099311 shutdown_connections 2024-03-22T02:22:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 --2- 172.21.15.103:0/3912099311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 0x7fe58c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 --2- 172.21.15.103:0/3912099311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c0ffa00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:44.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 -- 172.21.15.103:0/3912099311 >> 172.21.15.103:0/3912099311 conn(0x7fe58c0f7430 msgr2=0x7fe58c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 -- 172.21.15.103:0/3912099311 shutdown_connections 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.132+0000 7fe594485700 1 -- 172.21.15.103:0/3912099311 wait complete. 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 Processor -- start 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 -- start start 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 0x7fe58c10df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c10e5b0 con 0x7fe58c066f30 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe594485700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c112f00 con 0x7fe58c0fff40 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe592221700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe592221700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59314/0 (socket says 172.21.15.103:59314) 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.133+0000 7fe592221700 1 -- 172.21.15.103:0/2271791264 learned_addr learned my addr 172.21.15.103:0/2271791264 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe591a20700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 0x7fe58c10df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe592221700 1 -- 172.21.15.103:0/2271791264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 msgr2=0x7fe58c10df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:44.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe592221700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 0x7fe58c10df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe592221700 1 -- 172.21.15.103:0/2271791264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588005040 con 0x7fe58c066f30 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe592221700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe58800b3b0 tx=0x7fe588009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58800e040 con 0x7fe58c066f30 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe58800bcc0 con 0x7fe58c066f30 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c113100 con 0x7fe58c066f30 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58801c630 con 0x7fe58c066f30 2024-03-22T02:22:44.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.134+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c113670 con 0x7fe58c066f30 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.135+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c04c710 con 0x7fe58c066f30 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.135+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fe5880139f0 con 0x7fe58c066f30 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.136+0000 7fe57effd700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 0x7fe57806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.136+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fe5880871a0 con 0x7fe58c066f30 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.136+0000 7fe591a20700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 0x7fe57806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:44.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.136+0000 7fe591a20700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 0x7fe57806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe580006fd0 tx=0x7fe580008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:44.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.140+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe588057cd0 con 0x7fe58c066f30 2024-03-22T02:22:44.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:22:44.338+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe58c0f8f70 con 0x7fe578069210 2024-03-22T02:22:44.995 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:44 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:22:44.995 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:44 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:22:44.995 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:44 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:44 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:22:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:44 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:22:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:44 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:46.036 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:45 smithi103 ceph-mon[29258]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:22:46.036 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:45 smithi103 ceph-mon[29258]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:45 smithi122 ceph-mon[36403]: from='client.14278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:22:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:45 smithi122 ceph-mon[36403]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:46.985 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi103 2024-03-22T02:22:46.985 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:46.985 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi103 to 6432M 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:46.986 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:46 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi103 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi103 to 6432M 2024-03-22T02:22:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:46 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:48.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:47 smithi103 ceph-mon[29258]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:48.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:47 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2464507053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d03ea66-4cf1-4f34-a046-709ed8f6f037"}]: dispatch 2024-03-22T02:22:48.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:47 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2464507053' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d03ea66-4cf1-4f34-a046-709ed8f6f037"}]': finished 2024-03-22T02:22:48.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:47 smithi103 ceph-mon[29258]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T02:22:48.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:22:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:47 smithi122 ceph-mon[36403]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:47 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2464507053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d03ea66-4cf1-4f34-a046-709ed8f6f037"}]: dispatch 2024-03-22T02:22:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:47 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2464507053' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d03ea66-4cf1-4f34-a046-709ed8f6f037"}]': finished 2024-03-22T02:22:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:47 smithi122 ceph-mon[36403]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T02:22:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:22:49.189 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/498763202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:22:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:48 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/498763202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:22:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:49 smithi122 ceph-mon[36403]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:50.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:49 smithi103 ceph-mon[29258]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:51.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:51.240 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:22:53.725 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:53 smithi103 ceph-mon[29258]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:53 smithi122 ceph-mon[36403]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:55 smithi103 ceph-mon[29258]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:22:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:55 smithi103 ceph-mon[29258]: Deploying daemon osd.2 on smithi103 2024-03-22T02:22:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:55 smithi122 ceph-mon[36403]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:22:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:55 smithi122 ceph-mon[36403]: Deploying daemon osd.2 on smithi103 2024-03-22T02:22:57.589 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:57 smithi103 ceph-mon[29258]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:57 smithi122 ceph-mon[36403]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:59.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:22:59 smithi122 ceph-mon[36403]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:22:59.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:22:59 smithi103 ceph-mon[29258]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:01 smithi122 ceph-mon[36403]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:01.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:01 smithi103 ceph-mon[29258]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:03.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.341+0000 7fe57effd700 1 -- 172.21.15.103:0/2271791264 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe58c0f8f70 con 0x7fe578069210 2024-03-22T02:23:03.343 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 2 on host 'smithi103' 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 msgr2=0x7fe57806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 0x7fe57806b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe580006fd0 tx=0x7fe580008040 comp rx=0 tx=0).stop 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 msgr2=0x7fe58c10d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe58800b3b0 tx=0x7fe588009e30 comp rx=0 tx=0).stop 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 shutdown_connections 2024-03-22T02:23:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe578069210 0x7fe57806b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:03.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe58c066f30 0x7fe58c10d9e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:03.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 --2- 172.21.15.103:0/2271791264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe58c0fff40 0x7fe58c10df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:03.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 >> 172.21.15.103:0/2271791264 conn(0x7fe58c0f7430 msgr2=0x7fe58c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:03.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 shutdown_connections 2024-03-22T02:23:03.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:23:03.344+0000 7fe594485700 1 -- 172.21.15.103:0/2271791264 wait complete. 2024-03-22T02:23:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:03 smithi103 ceph-mon[29258]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:03 smithi103 ceph-mon[29258]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:23:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:03 smithi122 ceph-mon[36403]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:03 smithi122 ceph-mon[36403]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:23:04.171 DEBUG:teuthology.orchestra.run.smithi103:osd.2> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.2.service 2024-03-22T02:23:04.173 INFO:tasks.cephadm:Deploying osd.3 on smithi122 with /dev/vg_nvme/lv_4... 2024-03-22T02:23:04.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T02:23:04.446 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:05.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:05 smithi122 ceph-mon[36403]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:23:05.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:05 smithi122 ceph-mon[36403]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T02:23:05.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:05.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:05 smithi122 ceph-mon[36403]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:23:05.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:05 smithi122 ceph-mon[36403]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-mon[29258]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:23:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-mon[29258]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T02:23:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-mon[29258]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:23:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-mon[29258]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:05.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[63196]: 2024-03-22T02:23:05.322+0000 7efd36da3700 -1 osd.2 0 waiting for initial osdmap 2024-03-22T02:23:05.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:05 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[63196]: 2024-03-22T02:23:05.329+0000 7efd31b9e700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110556 s, 948 MB/s 2024-03-22T02:23:06.317 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-22T02:23:06.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:06 smithi122 ceph-mon[36403]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:23:06.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:06 smithi122 ceph-mon[36403]: osdmap e18: 3 total, 2 up, 3 in 2024-03-22T02:23:06.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:06 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:06.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:06 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:06.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:06 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:06.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:06 smithi103 ceph-mon[29258]: from='osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-22T02:23:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:06 smithi103 ceph-mon[29258]: osdmap e18: 3 total, 2 up, 3 in 2024-03-22T02:23:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:06 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:06 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:06 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:06.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] boot 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:07.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:07.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: osd.2 [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] boot 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-22T02:23:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-22T02:23:08.570 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:08 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-22T02:23:08.571 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:08 smithi122 ceph-mon[36403]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T02:23:08.571 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:08 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:23:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-22T02:23:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:08 smithi103 ceph-mon[29258]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T02:23:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:08 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 -- 172.21.15.122:0/1002162933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 msgr2=0x7f4f900fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 --2- 172.21.15.122:0/1002162933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f900fd3f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4f84004940 tx=0x7f4f84009e30 comp rx=0 tx=0).stop 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 -- 172.21.15.122:0/1002162933 shutdown_connections 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 --2- 172.21.15.122:0/1002162933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f900fd3f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 --2- 172.21.15.122:0/1002162933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f900fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 -- 172.21.15.122:0/1002162933 >> 172.21.15.122:0/1002162933 conn(0x7f4f900f7430 msgr2=0x7f4f900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 -- 172.21.15.122:0/1002162933 shutdown_connections 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.767+0000 7f4f962c4700 1 -- 172.21.15.122:0/1002162933 wait complete. 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 Processor -- start 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 -- start start 2024-03-22T02:23:08.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f9010def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9010e4f0 con 0x7f4f900fbcd0 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f962c4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9010ed40 con 0x7f4f900fcf80 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f8f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f9010def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f8ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f8ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.122:53808/0 (socket says 172.21.15.122:53808) 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.768+0000 7f4f8ffff700 1 -- 172.21.15.122:0/125660678 learned_addr learned my addr 172.21.15.122:0/125660678 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8ffff700 1 -- 172.21.15.122:0/125660678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 msgr2=0x7f4f9010def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8ffff700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f9010def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8ffff700 1 -- 172.21.15.122:0/125660678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84005040 con 0x7f4f900fbcd0 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8f7fe700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f9010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8ffff700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f80002a80 tx=0x7f4f800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.772 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f80007cc0 con 0x7f4f900fbcd0 2024-03-22T02:23:08.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9010f020 con 0x7f4f900fbcd0 2024-03-22T02:23:08.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f8000f040 con 0x7f4f900fbcd0 2024-03-22T02:23:08.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9010f560 con 0x7f4f900fbcd0 2024-03-22T02:23:08.773 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.769+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8000da80 con 0x7f4f900fbcd0 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.770+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f90101b10 con 0x7f4f900fbcd0 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.771+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f4f8000dbe0 con 0x7f4f900fbcd0 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.771+0000 7f4f8cff9700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 0x7f4f7806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.771+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f4f800865d0 con 0x7f4f900fbcd0 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.771+0000 7f4f8f7fe700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 0x7f4f7806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.774 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.772+0000 7f4f8f7fe700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 0x7f4f7806b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b3b0 tx=0x7f4f84009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.776+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4f80056c50 con 0x7f4f900fbcd0 2024-03-22T02:23:08.968 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:08.967+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f9010f7d0 con 0x7f4f78069210 2024-03-22T02:23:09.111 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:08 smithi103 sudo[66430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:23:09.111 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:08 smithi103 sudo[66430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:09.111 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:08 smithi103 sudo[66430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:09.111 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:08 smithi103 sudo[66430]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:09.373 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:23:09.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:09.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67776]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:09.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67776]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:09.643 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67780]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:23:09.643 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:09.643 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67780]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:09.643 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67780]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:09.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:09.690 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:23:09.690 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T02:23:09.690 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:09.691 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:23:09.691 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:23:09.691 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:09.691 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:09.897 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67785]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:23:09.897 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67785]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:09.897 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67785]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:09.897 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67785]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:10.143 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:10.141+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4f80053590 con 0x7f4f900fbcd0 2024-03-22T02:23:10.171 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67820]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:23:10.171 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67820]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:10.171 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67820]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:10.171 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:09 smithi103 sudo[67820]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:10.434 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67910]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:23:10.434 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67910]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:10.434 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67910]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:10.434 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67910]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:10.697 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67978]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:23:10.698 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67978]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:10.698 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67978]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:10.698 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[67978]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi103 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:10.954 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi103 to 4288M 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: mgrmap e19: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 ceph-mon[36403]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:23:10.955 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 sudo[39689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[68000]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[68000]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[68000]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 sudo[68000]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi103 2024-03-22T02:23:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi103 to 4288M 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: mgrmap e19: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:23:11.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:10 smithi103 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:23:11.217 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 sudo[39689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:11.217 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 sudo[39689]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:11.217 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:10 smithi122 sudo[39689]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:11.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 sudo[39711]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:23:11.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 sudo[39711]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:23:11.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 sudo[39711]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:23:11.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 sudo[39711]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:23:11.992 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/567585732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]: dispatch 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]: dispatch 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]': finished 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T02:23:11.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:11 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:23:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/567585732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]: dispatch 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]: dispatch 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12aeb69e-863c-492d-b243-d1696e37848f"}]': finished 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T02:23:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:11 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:12 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/2274523661' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:23:13.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:12 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/2274523661' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:23:14.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:13 smithi122 ceph-mon[36403]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:14.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:13 smithi103 ceph-mon[29258]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:14 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:14 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:14 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:14 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:14 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:15.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:14 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:15.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:14 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:15.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:14 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:14 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:14 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:15 smithi122 ceph-mon[36403]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:15 smithi103 ceph-mon[29258]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:18.097 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:17 smithi122 ceph-mon[36403]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-22T02:23:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:17 smithi103 ceph-mon[29258]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-22T02:23:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:23:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:19.297 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:23:19.297 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:19 smithi122 ceph-mon[36403]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-22T02:23:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:19 smithi122 ceph-mon[36403]: Deploying daemon osd.3 on smithi122 2024-03-22T02:23:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:19 smithi103 ceph-mon[29258]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-22T02:23:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:19 smithi103 ceph-mon[29258]: Deploying daemon osd.3 on smithi122 2024-03-22T02:23:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:21.173 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:22.086 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:21 smithi122 ceph-mon[36403]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-22T02:23:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:21 smithi103 ceph-mon[29258]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-22T02:23:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:23 smithi122 ceph-mon[36403]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-03-22T02:23:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:23 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:23 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:23 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:23 smithi103 ceph-mon[29258]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-03-22T02:23:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:23 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:23 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:23 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:26.211 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:25 smithi122 ceph-mon[36403]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-22T02:23:26.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:25 smithi103 ceph-mon[29258]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-22T02:23:27.135 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.134+0000 7f4f8cff9700 1 -- 172.21.15.122:0/125660678 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4f9010f7d0 con 0x7f4f78069210 2024-03-22T02:23:27.136 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 3 on host 'smithi122' 2024-03-22T02:23:27.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 msgr2=0x7f4f7806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:27.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 0x7f4f7806b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b3b0 tx=0x7f4f84009e30 comp rx=0 tx=0).stop 2024-03-22T02:23:27.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 msgr2=0x7f4f9010d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:27.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f80002a80 tx=0x7f4f800046b0 comp rx=0 tx=0).stop 2024-03-22T02:23:27.137 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 shutdown_connections 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f78069210 0x7f4f7806b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f900fbcd0 0x7f4f9010d9b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 --2- 172.21.15.122:0/125660678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f900fcf80 0x7f4f9010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 >> 172.21.15.122:0/125660678 conn(0x7f4f900f7430 msgr2=0x7f4f900f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 shutdown_connections 2024-03-22T02:23:27.138 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:27.136+0000 7f4f962c4700 1 -- 172.21.15.122:0/125660678 wait complete. 2024-03-22T02:23:27.919 DEBUG:teuthology.orchestra.run.smithi122:osd.3> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.3.service 2024-03-22T02:23:27.922 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_3... 2024-03-22T02:23:27.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T02:23:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:27 smithi122 ceph-mon[36403]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-22T02:23:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:27 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:27 smithi122 ceph-mon[36403]: from='osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:23:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:27 smithi122 ceph-mon[36403]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:23:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:27 smithi103 ceph-mon[29258]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-22T02:23:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:27 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:27 smithi103 ceph-mon[29258]: from='osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:23:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:27 smithi103 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:23:28.584 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:23:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T02:23:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:29.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: from='osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:29.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:29.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-mon[36403]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:29.349 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[41797]: 2024-03-22T02:23:29.104+0000 7fdcb907c700 -1 osd.3 0 waiting for initial osdmap 2024-03-22T02:23:29.349 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:23:29 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[41797]: 2024-03-22T02:23:29.109+0000 7fdcb1e73700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: from='osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:29 smithi103 ceph-mon[29258]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:30.333 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:23:30.333 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T02:23:30.334 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:30.334 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:30.334 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:30.334 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:30 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:30.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:23:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T02:23:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:30 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:30.514 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T02:23:30.515 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T02:23:30.515 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-22T02:23:30.515 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-22T02:23:30.515 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122777 s, 854 MB/s 2024-03-22T02:23:30.515 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] boot 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:31.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:31 smithi122 ceph-mon[36403]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: osd.3 [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] boot 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:23:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:31 smithi103 ceph-mon[29258]: pgmap v94: 1 pgs: 1 peering; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-22T02:23:32.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:32 smithi122 ceph-mon[36403]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T02:23:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:32 smithi103 ceph-mon[29258]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T02:23:32.451 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-03-22T02:23:33.223 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:33 smithi122 ceph-mon[36403]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T02:23:33.223 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:33 smithi122 ceph-mon[36403]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:33.344 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:33.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:33 smithi103 ceph-mon[29258]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T02:23:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:33 smithi103 ceph-mon[29258]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi122 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi122 to 15936M 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:34 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.405 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi122 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi122 to 15936M 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:34.406 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:34.407 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:34 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:34.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 -- 172.21.15.122:0/835423217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 msgr2=0x7f55000fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:34.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 --2- 172.21.15.122:0/835423217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f55000fe840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f54fc004940 tx=0x7f54fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:23:34.919 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 -- 172.21.15.122:0/835423217 shutdown_connections 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 --2- 172.21.15.122:0/835423217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 0x7f55000faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 --2- 172.21.15.122:0/835423217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f55000fe840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.917+0000 7f55079c6700 1 -- 172.21.15.122:0/835423217 >> 172.21.15.122:0/835423217 conn(0x7f55000f6bd0 msgr2=0x7f55000f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 -- 172.21.15.122:0/835423217 shutdown_connections 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 -- 172.21.15.122:0/835423217 wait complete. 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 Processor -- start 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 -- start start 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 0x7f550010de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550010e440 con 0x7f55000fed80 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f55079c6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550010e5b0 con 0x7f55000fe430 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f5505762700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f5505762700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:52846/0 (socket says 172.21.15.122:52846) 2024-03-22T02:23:34.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f5505762700 1 -- 172.21.15.122:0/2166502529 learned_addr learned my addr 172.21.15.122:0/2166502529 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.918+0000 7f5504f61700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 0x7f550010de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f5505762700 1 -- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 msgr2=0x7f550010de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f5505762700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 0x7f550010de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f5505762700 1 -- 172.21.15.122:0/2166502529 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc005040 con 0x7f55000fe430 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f5505762700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b3b0 tx=0x7f54fc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54fc00e040 con 0x7f55000fe430 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5500112fb0 con 0x7f55000fe430 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54fc00bbf0 con 0x7f55000fe430 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54fc01c630 con 0x7f55000fe430 2024-03-22T02:23:34.921 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.919+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001134e0 con 0x7f55000fe430 2024-03-22T02:23:34.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.920+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550004c710 con 0x7f55000fe430 2024-03-22T02:23:34.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.920+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f54fc013420 con 0x7f55000fe430 2024-03-22T02:23:34.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.921+0000 7f54f5ffb700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 0x7f54ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:34.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.921+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f54fc0877c0 con 0x7f55000fe430 2024-03-22T02:23:34.922 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.921+0000 7f5504f61700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 0x7f54ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:34.923 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.921+0000 7f5504f61700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 0x7f54ec06b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54f00041d0 tx=0x7f54f00043c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:34.926 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:34.926+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f54fc057c20 con 0x7f55000fe430 2024-03-22T02:23:35.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:35.121+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5500113700 con 0x7f54ec0692a0 2024-03-22T02:23:35.532 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:35 smithi122 ceph-mon[36403]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T02:23:35.532 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:35 smithi122 ceph-mon[36403]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:35.533 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:23:35.533 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:23:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:35 smithi103 ceph-mon[29258]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T02:23:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:35 smithi103 ceph-mon[29258]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:23:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:23:36.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:36 smithi122 ceph-mon[36403]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:23:36.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:36 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:36.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:36 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:36.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:36 smithi103 ceph-mon[29258]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:23:36.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:36 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:36.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:36 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:37 smithi122 ceph-mon[36403]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-22T02:23:37.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:37 smithi103 ceph-mon[29258]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-22T02:23:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/204530780' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]: dispatch 2024-03-22T02:23:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]: dispatch 2024-03-22T02:23:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]': finished 2024-03-22T02:23:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T02:23:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:38 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/1716250822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/204530780' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]: dispatch 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]: dispatch 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ede66c5-d812-4f2c-b642-ac6595a340d8"}]': finished 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:38 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/1716250822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:23:39.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:39 smithi122 ceph-mon[36403]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T02:23:39.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:39 smithi103 ceph-mon[29258]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T02:23:41.766 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:41 smithi122 ceph-mon[36403]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-22T02:23:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:41 smithi103 ceph-mon[29258]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-22T02:23:43.713 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:43 smithi122 ceph-mon[36403]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-22T02:23:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:43 smithi103 ceph-mon[29258]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-22T02:23:45.570 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:45 smithi122 ceph-mon[36403]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:23:45.570 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:45 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:23:45.570 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:45 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:45.570 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:45 smithi122 ceph-mon[36403]: Deploying daemon osd.4 on smithi122 2024-03-22T02:23:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:45 smithi103 ceph-mon[29258]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:23:45.890 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:45 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:23:45.890 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:45 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:45.891 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:45 smithi103 ceph-mon[29258]: Deploying daemon osd.4 on smithi122 2024-03-22T02:23:47.540 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:47 smithi122 ceph-mon[36403]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:47.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:47 smithi103 ceph-mon[29258]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:49 smithi103 ceph-mon[29258]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:49.723 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:49 smithi122 ceph-mon[36403]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:51.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:51.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:51.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:50 smithi122 ceph-mon[36403]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:51.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:51.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:50 smithi103 ceph-mon[29258]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:23:53.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.182+0000 7f54f5ffb700 1 -- 172.21.15.122:0/2166502529 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5500113700 con 0x7f54ec0692a0 2024-03-22T02:23:53.184 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2024-03-22T02:23:53.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 msgr2=0x7f54ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:53.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 0x7f54ec06b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54f00041d0 tx=0x7f54f00043c0 comp rx=0 tx=0).stop 2024-03-22T02:23:53.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 msgr2=0x7f550010d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:53.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b3b0 tx=0x7f54fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 shutdown_connections 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54ec0692a0 0x7f54ec06b750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55000fe430 0x7f550010d900 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 --2- 172.21.15.122:0/2166502529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55000fed80 0x7f550010de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 >> 172.21.15.122:0/2166502529 conn(0x7f55000f6bd0 msgr2=0x7f55000f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 shutdown_connections 2024-03-22T02:23:53.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:23:53.184+0000 7f55079c6700 1 -- 172.21.15.122:0/2166502529 wait complete. 2024-03-22T02:23:53.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:53 smithi122 ceph-mon[36403]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:53.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:53.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:53.800 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.4.service 2024-03-22T02:23:53.802 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_2... 2024-03-22T02:23:53.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T02:23:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:53 smithi103 ceph-mon[29258]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:54.098 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:23:53 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:23:53.837+0000 7f0370640540 -1 osd.4 0 log_to_monitors true 2024-03-22T02:23:54.505 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:54.792 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:54 smithi122 ceph-mon[36403]: from='osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:23:54.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:54 smithi122 ceph-mon[36403]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:23:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:54 smithi103 ceph-mon[29258]: from='osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:23:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:54 smithi103 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:23:55.795 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:23:55.462+0000 7f0366e12700 -1 osd.4 0 waiting for initial osdmap 2024-03-22T02:23:55.796 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:23:55.469+0000 7f0363c11700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:55.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:23:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:55.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.017886 s, 586 MB/s 2024-03-22T02:23:56.504 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-22T02:23:56.572 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:56 smithi122 ceph-mon[36403]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:23:56.572 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:56 smithi122 ceph-mon[36403]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T02:23:56.572 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:56 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:56 smithi103 ceph-mon[29258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:23:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:56 smithi103 ceph-mon[29258]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T02:23:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:56 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:23:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:23:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] boot 2024-03-22T02:23:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T02:23:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: osd.4 [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] boot 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T02:23:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:23:57.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-03-22T02:23:58.527 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:23:58.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:58 smithi122 ceph-mon[36403]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T02:23:58.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:58 smithi103 ceph-mon[29258]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T02:23:59.626 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:23:59 smithi103 ceph-mon[29258]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:23:59.768 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:23:59 smithi122 ceph-mon[36403]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:00.785 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.781+0000 7f23d89d9700 1 -- 172.21.15.122:0/2401163349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 msgr2=0x7f23d40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.781+0000 7f23d89d9700 1 --2- 172.21.15.122:0/2401163349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d40fc0e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f23c4004940 tx=0x7f23c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- 172.21.15.122:0/2401163349 shutdown_connections 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 --2- 172.21.15.122:0/2401163349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 0x7f23d40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 --2- 172.21.15.122:0/2401163349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d40fc0e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- 172.21.15.122:0/2401163349 >> 172.21.15.122:0/2401163349 conn(0x7f23d40f7450 msgr2=0x7f23d40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- 172.21.15.122:0/2401163349 shutdown_connections 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- 172.21.15.122:0/2401163349 wait complete. 2024-03-22T02:24:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 Processor -- start 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- start start 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 0x7f23d4112750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4112c90 con 0x7f23d40fcf80 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d89d9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d4112e00 con 0x7f23d40fbcd0 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:54140/0 (socket says 172.21.15.122:54140) 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.782+0000 7f23d259c700 1 -- 172.21.15.122:0/1352850666 learned_addr learned my addr 172.21.15.122:0/1352850666 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:24:00.787 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d1d9b700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 0x7f23d4112750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d259c700 1 -- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 msgr2=0x7f23d4112750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d259c700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 0x7f23d4112750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d259c700 1 -- 172.21.15.122:0/1352850666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4005040 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d259c700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f23c400b3b0 tx=0x7f23c4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23c400e040 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23c400bcc0 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d4113080 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23c401c630 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.783+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d4113640 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.784+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d404c710 con 0x7f23d40fbcd0 2024-03-22T02:24:00.788 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.784+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f23c4013980 con 0x7f23d40fbcd0 2024-03-22T02:24:00.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.785+0000 7f23c2ffd700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 0x7f23bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:00.789 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.785+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f23c4087ad0 con 0x7f23d40fbcd0 2024-03-22T02:24:00.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.785+0000 7f23d1d9b700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 0x7f23bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:00.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.786+0000 7f23d1d9b700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 0x7f23bc06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006fd0 tx=0x7f23c8008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:00.790 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.789+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f23c4057d00 con 0x7f23d40fbcd0 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi122 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi122 to 7968M 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:00.984 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:00.983+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f23d4101250 con 0x7f23bc0692a0 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi122 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi122 to 7968M 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:01.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:02.089 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:01 smithi122 ceph-mon[36403]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:02.090 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:24:02.090 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:24:02.090 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:01 smithi103 ceph-mon[29258]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:24:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:24:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:02.902 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:02 smithi122 ceph-mon[36403]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:24:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:02 smithi103 ceph-mon[29258]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:24:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/3129630735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]: dispatch 2024-03-22T02:24:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]: dispatch 2024-03-22T02:24:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]': finished 2024-03-22T02:24:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T02:24:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3129630735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]: dispatch 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]: dispatch 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9918a976-1e8c-4fb8-93ec-6600de78f79a"}]': finished 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T02:24:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:04 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/1601896249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:24:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:04 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/1601896249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:24:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:05 smithi122 ceph-mon[36403]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:05 smithi103 ceph-mon[29258]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:07.958 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:07 smithi122 ceph-mon[36403]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:07 smithi103 ceph-mon[29258]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:10.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:09 smithi122 ceph-mon[36403]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:10.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:09 smithi103 ceph-mon[29258]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:11.072 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:24:11.072 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:24:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:11 smithi122 ceph-mon[36403]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:11 smithi122 ceph-mon[36403]: Deploying daemon osd.5 on smithi122 2024-03-22T02:24:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:11 smithi103 ceph-mon[29258]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:11 smithi103 ceph-mon[29258]: Deploying daemon osd.5 on smithi122 2024-03-22T02:24:14.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:13 smithi122 ceph-mon[36403]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:13 smithi103 ceph-mon[29258]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:16.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:15 smithi122 ceph-mon[36403]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:16.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:15 smithi103 ceph-mon[29258]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:17.234 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:17.235 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:17.235 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:16 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:16 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:18.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:17 smithi122 ceph-mon[36403]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:17 smithi103 ceph-mon[29258]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:19.450 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.450+0000 7f23c2ffd700 1 -- 172.21.15.122:0/1352850666 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f23d4101250 con 0x7f23bc0692a0 2024-03-22T02:24:19.450 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2024-03-22T02:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 msgr2=0x7f23bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 0x7f23bc06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006fd0 tx=0x7f23c8008040 comp rx=0 tx=0).stop 2024-03-22T02:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 msgr2=0x7f23d4066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:19.452 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f23c400b3b0 tx=0x7f23c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 shutdown_connections 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f23bc0692a0 0x7f23bc06b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23d40fbcd0 0x7f23d4066cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 --2- 172.21.15.122:0/1352850666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23d40fcf80 0x7f23d4112750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 >> 172.21.15.122:0/1352850666 conn(0x7f23d40f7450 msgr2=0x7f23d40f8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 shutdown_connections 2024-03-22T02:24:19.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:19.452+0000 7f23d89d9700 1 -- 172.21.15.122:0/1352850666 wait complete. 2024-03-22T02:24:20.073 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:19 smithi122 ceph-mon[36403]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:20.073 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:19 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:20.073 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:19 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:20.073 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:19 smithi122 ceph-mon[36403]: from='osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:24:20.073 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:19 smithi122 ceph-mon[36403]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:24:20.166 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.5.service 2024-03-22T02:24:20.168 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-22T02:24:20.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T02:24:20.198 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:19 smithi103 ceph-mon[29258]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:20.199 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:19 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:20.199 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:19 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:20.199 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:19 smithi103 ceph-mon[29258]: from='osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:24:20.199 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:19 smithi103 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:24:20.438 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:21.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:21.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:24:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:21.766 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[54950]: 2024-03-22T02:24:21.450+0000 7f981727a700 -1 osd.5 0 waiting for initial osdmap 2024-03-22T02:24:21.766 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[54950]: 2024-03-22T02:24:21.458+0000 7f9814079700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:24:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-mon[36403]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-mon[36403]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:24:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-mon[36403]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T02:24:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:21 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:22.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 -- 172.21.15.103:0/1044378629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fcf80 msgr2=0x7f70cc0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:22.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1044378629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fcf80 0x7f70cc0fd3f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70b4004940 tx=0x7f70b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 -- 172.21.15.103:0/1044378629 shutdown_connections 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1044378629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fcf80 0x7f70cc0fd3f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1044378629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 -- 172.21.15.103:0/1044378629 >> 172.21.15.103:0/1044378629 conn(0x7f70cc0f7410 msgr2=0x7f70cc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 -- 172.21.15.103:0/1044378629 shutdown_connections 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.174+0000 7f70d14bb700 1 -- 172.21.15.103:0/1044378629 wait complete. 2024-03-22T02:24:22.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 Processor -- start 2024-03-22T02:24:22.177 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:21 smithi103 ceph-mon[29258]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:22.177 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:21 smithi103 ceph-mon[29258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-22T02:24:22.177 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:21 smithi103 ceph-mon[29258]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T02:24:22.178 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:22.178 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:21 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 -- start start 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 0x7f70cc110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc110700 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70d14bb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc110f00 con 0x7f70cc0fcf80 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70caffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70caffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50106/0 (socket says 172.21.15.103:50106) 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70caffd700 1 -- 172.21.15.103:0/1462705799 learned_addr learned my addr 172.21.15.103:0/1462705799 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:22.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.175+0000 7f70ca7fc700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 0x7f70cc110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70caffd700 1 -- 172.21.15.103:0/1462705799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 msgr2=0x7f70cc110100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70caffd700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 0x7f70cc110100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70caffd700 1 -- 172.21.15.103:0/1462705799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b4005040 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70ca7fc700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 0x7f70cc110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70caffd700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70c400eb40 tx=0x7f70c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c4007cc0 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc111180 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70c400f040 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c400ca10 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.176+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc1116c0 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.178+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc101910 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.178+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f70c400cb70 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.178+0000 7f70c37fe700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 0x7f70b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:22.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.178+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f70c4086880 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.178+0000 7f70ca7fc700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 0x7f70b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:22.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.179+0000 7f70ca7fc700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 0x7f70b806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009b30 tx=0x7f70b4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:22.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.183+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f70c404e190 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.373+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f70cc05f5f0 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.374+0000 7f70c37fe700 1 -- 172.21.15.103:0/1462705799 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f70c4056b40 con 0x7f70cc0fbcd0 2024-03-22T02:24:22.376 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:22.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 msgr2=0x7f70b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:22.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 0x7f70b806b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009b30 tx=0x7f70b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:22.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 msgr2=0x7f70cc10fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:22.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70c400eb40 tx=0x7f70c400b040 comp rx=0 tx=0).stop 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 shutdown_connections 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f70b80692a0 0x7f70b806b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70cc0fbcd0 0x7f70cc10fbc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 --2- 172.21.15.103:0/1462705799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70cc0fcf80 0x7f70cc110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 >> 172.21.15.103:0/1462705799 conn(0x7f70cc0f7410 msgr2=0x7f70cc0663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 shutdown_connections 2024-03-22T02:24:22.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:22.376+0000 7f70d14bb700 1 -- 172.21.15.103:0/1462705799 wait complete. 2024-03-22T02:24:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: purged_snaps scrub starts 2024-03-22T02:24:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: purged_snaps scrub ok 2024-03-22T02:24:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1462705799' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:24:23.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] boot 2024-03-22T02:24:23.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T02:24:23.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:22 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: purged_snaps scrub starts 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: purged_snaps scrub ok 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1462705799' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: osd.5 [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] boot 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T02:24:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:22 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:24:23.159 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1711074236,"num_in_osds":6,"osd_in_since":1711074243,"num_remapped_pgs":0} 2024-03-22T02:24:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:23 smithi122 ceph-mon[36403]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:23 smithi122 ceph-mon[36403]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T02:24:24.160 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T02:24:24.190 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:23 smithi103 ceph-mon[29258]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-22T02:24:24.191 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:23 smithi103 ceph-mon[29258]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T02:24:24.440 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: Detected new or changed devices on smithi122 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi122 to 5312M 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:25.542 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:25.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:25.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:25.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.543 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:25 smithi103 ceph-mon[29258]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi122 2024-03-22T02:24:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi122 to 5312M 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:24:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:25 smithi122 ceph-mon[36403]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:26.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 -- 172.21.15.103:0/2121416666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 msgr2=0x7f26940fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 --2- 172.21.15.103:0/2121416666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940fc0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2684004940 tx=0x7f2684009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 -- 172.21.15.103:0/2121416666 shutdown_connections 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 --2- 172.21.15.103:0/2121416666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 --2- 172.21.15.103:0/2121416666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940fc0e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.033+0000 7f269b6a7700 1 -- 172.21.15.103:0/2121416666 >> 172.21.15.103:0/2121416666 conn(0x7f26940f7430 msgr2=0x7f26940f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 -- 172.21.15.103:0/2121416666 shutdown_connections 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 -- 172.21.15.103:0/2121416666 wait complete. 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 Processor -- start 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 -- start start 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694100250 con 0x7f26940fbcd0 2024-03-22T02:24:26.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f269b6a7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26940fe830 con 0x7f26940fcf80 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f2699443700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f2699443700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50626/0 (socket says 172.21.15.103:50626) 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f2699443700 1 -- 172.21.15.103:0/885919761 learned_addr learned my addr 172.21.15.103:0/885919761 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.034+0000 7f2698c42700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2699443700 1 -- 172.21.15.103:0/885919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 msgr2=0x7f26940fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2699443700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2699443700 1 -- 172.21.15.103:0/885919761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684005040 con 0x7f26940fbcd0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2698c42700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2699443700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f268400b3b0 tx=0x7f2684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268400e040 con 0x7f26940fbcd0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26940feab0 con 0x7f26940fbcd0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f268400bcc0 con 0x7f26940fbcd0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268401c630 con 0x7f26940fbcd0 2024-03-22T02:24:26.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.035+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26940ff070 con 0x7f26940fbcd0 2024-03-22T02:24:26.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.036+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269404c710 con 0x7f26940fbcd0 2024-03-22T02:24:26.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.036+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2684013980 con 0x7f26940fbcd0 2024-03-22T02:24:26.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.037+0000 7f2689ffb700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 0x7f268006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:26.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.037+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2684087c70 con 0x7f26940fbcd0 2024-03-22T02:24:26.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.037+0000 7f2698c42700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 0x7f268006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:26.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.037+0000 7f2698c42700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 0x7f268006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2690009b10 tx=0x7f2690008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:26.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.041+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2684057cf0 con 0x7f26940fbcd0 2024-03-22T02:24:26.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.230+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2694101af0 con 0x7f26940fbcd0 2024-03-22T02:24:26.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.230+0000 7f2689ffb700 1 -- 172.21.15.103:0/885919761 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2684057880 con 0x7f26940fbcd0 2024-03-22T02:24:26.231 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:26.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 msgr2=0x7f268006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 0x7f268006b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2690009b10 tx=0x7f2690008040 comp rx=0 tx=0).stop 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 msgr2=0x7f26940ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f268400b3b0 tx=0x7f2684009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 shutdown_connections 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26800692a0 0x7f268006b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26940fbcd0 0x7f26940ffbc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 --2- 172.21.15.103:0/885919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26940fcf80 0x7f26940fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 >> 172.21.15.103:0/885919761 conn(0x7f26940f7430 msgr2=0x7f26940667a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 shutdown_connections 2024-03-22T02:24:26.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:26.232+0000 7f269b6a7700 1 -- 172.21.15.103:0/885919761 wait complete. 2024-03-22T02:24:26.588 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:26 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/885919761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:24:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:26 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/885919761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:24:26.916 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1711074262,"num_in_osds":6,"osd_in_since":1711074243,"num_remapped_pgs":0} 2024-03-22T02:24:26.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:24:27.212 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:27.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:27 smithi103 ceph-mon[29258]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:27 smithi122 ceph-mon[36403]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:28.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- 172.21.15.103:0/814662853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fded0 msgr2=0x7f87440fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 --2- 172.21.15.103:0/814662853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fded0 0x7f87440fe2a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8738004940 tx=0x7f8738009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- 172.21.15.103:0/814662853 shutdown_connections 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 --2- 172.21.15.103:0/814662853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fe870 0x7f8744106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 --2- 172.21.15.103:0/814662853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fded0 0x7f87440fe2a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- 172.21.15.103:0/814662853 >> 172.21.15.103:0/814662853 conn(0x7f8744072bc0 msgr2=0x7f8744072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- 172.21.15.103:0/814662853 shutdown_connections 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- 172.21.15.103:0/814662853 wait complete. 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 Processor -- start 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.774+0000 7f874ae8a700 1 -- start start 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f874ae8a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fded0 0x7f8744115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f874ae8a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:28.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f874ae8a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441168c0 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f874ae8a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874410fce0 con 0x7f87440fded0 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50634/0 (socket says 172.21.15.103:50634) 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 -- 172.21.15.103:0/3046254294 learned_addr learned my addr 172.21.15.103:0/3046254294 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8748c26700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fded0 0x7f8744115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 -- 172.21.15.103:0/3046254294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fded0 msgr2=0x7f8744115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fded0 0x7f8744115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 -- 172.21.15.103:0/3046254294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738005040 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f8743fff700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8730002a50 tx=0x7f87300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.775+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8730007cc0 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.776+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874410ffc0 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.776+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f873000f040 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.776+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873000da30 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.776+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744110500 con 0x7f87440fe870 2024-03-22T02:24:28.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.777+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8744106a00 con 0x7f87440fe870 2024-03-22T02:24:28.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.777+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f873000db90 con 0x7f87440fe870 2024-03-22T02:24:28.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.777+0000 7f87417fa700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 0x7f873406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:28.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.778+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8730086c00 con 0x7f87440fe870 2024-03-22T02:24:28.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.778+0000 7f8748c26700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 0x7f873406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:28.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.778+0000 7f8748c26700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 0x7f873406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8738004f90 tx=0x7f8738009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:28.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.782+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f873008c020 con 0x7f87440fe870 2024-03-22T02:24:28.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.969+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8744065df0 con 0x7f87440fe870 2024-03-22T02:24:28.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.970+0000 7f87417fa700 1 -- 172.21.15.103:0/3046254294 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11133 (secure 0 0 0) 0x7f87300569f0 con 0x7f87440fe870 2024-03-22T02:24:28.971 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:28.971 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":40,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:24:23.447675+0000","last_up_change":"2024-03-22T02:24:22.446345+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6803","nonce":1382751588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6805","nonce":1382751588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6809","nonce":1382751588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6807","nonce":1382751588}]},"public_addr":"172.21.15.103:6803/1382751588","cluster_addr":"172.21.15.103:6805/1382751588","heartbeat_back_addr":"172.21.15.103:6809/1382751588","heartbeat_front_addr":"172.21.15.103:6807/1382751588","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6811","nonce":3229649513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6813","nonce":3229649513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6817","nonce":3229649513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6815","nonce":3229649513}]},"public_addr":"172.21.15.103:6811/3229649513","cluster_addr":"172.21.15.103:6813/3229649513","heartbeat_back_addr":"172.21.15.103:6817/3229649513","heartbeat_front_addr":"172.21.15.103:6815/3229649513","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6819","nonce":2311429465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6821","nonce":2311429465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6825","nonce":2311429465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6823","nonce":2311429465}]},"public_addr":"172.21.15.103:6819/2311429465","cluster_addr":"172.21.15.103:6821/2311429465","heartbeat_back_addr":"172.21.15.103:6825/2311429465","heartbeat_front_addr":"172.21.15.103:6823/2311429465","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6801","nonce":599760037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6803","nonce":599760037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6807","nonce":599760037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6805","nonce":599760037}]},"public_addr":"172.21.15.122:6801/599760037","cluster_addr":"172.21.15.122:6803/599760037","heartbeat_back_addr":"172.21.15.122:6807/599760037","heartbeat_front_addr":"172.21.15.122:6805/599760037","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6809","nonce":1379244401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6811","nonce":1379244401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6815","nonce":1379244401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6813","nonce":1379244401}]},"public_addr":"172.21.15.122:6809/1379244401","cluster_addr":"172.21.15.122:6811/1379244401","heartbeat_back_addr":"172.21.15.122:6815/1379244401","heartbeat_front_addr":"172.21.15.122:6813/1379244401","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6817","nonce":2680393488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6819","nonce":2680393488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6823","nonce":2680393488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6821","nonce":2680393488}]},"public_addr":"172.21.15.122:6817/2680393488","cluster_addr":"172.21.15.122:6819/2680393488","heartbeat_back_addr":"172.21.15.122:6823/2680393488","heartbeat_front_addr":"172.21.15.122:6821/2680393488","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:24:28.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 msgr2=0x7f873406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:28.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 0x7f873406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8738004f90 tx=0x7f8738009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 msgr2=0x7f8744116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8730002a50 tx=0x7f87300046b0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 shutdown_connections 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f87340692a0 0x7f873406b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87440fded0 0x7f8744115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 --2- 172.21.15.103:0/3046254294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87440fe870 0x7f8744116270 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 >> 172.21.15.103:0/3046254294 conn(0x7f8744072bc0 msgr2=0x7f87440fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 shutdown_connections 2024-03-22T02:24:28.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:28.972+0000 7f874ae8a700 1 -- 172.21.15.103:0/3046254294 wait complete. 2024-03-22T02:24:29.708 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:29 smithi103 ceph-mon[29258]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:29.708 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:29 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3046254294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:29.709 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:23:06.693135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-22T02:24:29.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-22T02:24:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:29 smithi122 ceph-mon[36403]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:29 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3046254294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:29.970 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- 172.21.15.103:0/361498542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d4071f90 msgr2=0x7ff8d4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 --2- 172.21.15.103:0/361498542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d4071f90 0x7ff8d4070360 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0004970 tx=0x7ff8d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- 172.21.15.103:0/361498542 shutdown_connections 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 --2- 172.21.15.103:0/361498542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d40708a0 0x7ff8d4070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 --2- 172.21.15.103:0/361498542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d4071f90 0x7ff8d4070360 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- 172.21.15.103:0/361498542 >> 172.21.15.103:0/361498542 conn(0x7ff8d40f8950 msgr2=0x7ff8d40fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:31.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- 172.21.15.103:0/361498542 shutdown_connections 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- 172.21.15.103:0/361498542 wait complete. 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 Processor -- start 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 -- start start 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.570+0000 7ff8dc299700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8dc299700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 0x7ff8d411a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8dc299700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d411abf0 con 0x7ff8d40708a0 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8dc299700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d4114140 con 0x7ff8d4071f90 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50640/0 (socket says 172.21.15.103:50640) 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 -- 172.21.15.103:0/1040096248 learned_addr learned my addr 172.21.15.103:0/1040096248 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8d9834700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 0x7ff8d411a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 -- 172.21.15.103:0/1040096248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 msgr2=0x7ff8d411a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 0x7ff8d411a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 -- 172.21.15.103:0/1040096248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0005040 con 0x7ff8d40708a0 2024-03-22T02:24:31.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8d9834700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 0x7ff8d411a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8da035700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0000c00 tx=0x7ff8d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8d000e040 con 0x7ff8d40708a0 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8d000bce0 con 0x7ff8d40708a0 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d4114360 con 0x7ff8d40708a0 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8d001c630 con 0x7ff8d40708a0 2024-03-22T02:24:31.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.571+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d41148a0 con 0x7ff8d40708a0 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.572+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d4071aa0 con 0x7ff8d40708a0 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.573+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff8d0013980 con 0x7ff8d40708a0 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.573+0000 7ff8caffd700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 0x7ff8c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.573+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff8d00881b0 con 0x7ff8d40708a0 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.573+0000 7ff8d9834700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 0x7ff8c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:31.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.574+0000 7ff8d9834700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 0x7ff8c006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4115870 tx=0x7ff8c4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:31.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.577+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8d00582b0 con 0x7ff8d40708a0 2024-03-22T02:24:31.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:31 smithi103 ceph-mon[29258]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:31.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.765+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8d404c710 con 0x7ff8d40708a0 2024-03-22T02:24:31.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.765+0000 7ff8caffd700 1 -- 172.21.15.103:0/1040096248 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8d404c710 con 0x7ff8d40708a0 2024-03-22T02:24:31.766 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 msgr2=0x7ff8c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 0x7ff8c006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4115870 tx=0x7ff8c4008040 comp rx=0 tx=0).stop 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 msgr2=0x7ff8d4119fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0000c00 tx=0x7ff8d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 shutdown_connections 2024-03-22T02:24:31.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8c00692a0 0x7ff8c006b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8d40708a0 0x7ff8d4119fd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 --2- 172.21.15.103:0/1040096248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8d4071f90 0x7ff8d411a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:31.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.767+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 >> 172.21.15.103:0/1040096248 conn(0x7ff8d40f8950 msgr2=0x7ff8d40f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:31.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.768+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 shutdown_connections 2024-03-22T02:24:31.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:31.768+0000 7ff8dc299700 1 -- 172.21.15.103:0/1040096248 wait complete. 2024-03-22T02:24:31.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:31 smithi122 ceph-mon[36403]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:32.532 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:32 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1040096248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:24:32.557 INFO:tasks.cephadm:Setting up client nodes... 2024-03-22T02:24:32.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T02:24:32.811 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:32 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1040096248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:24:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:33 smithi122 ceph-mon[36403]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:33 smithi103 ceph-mon[29258]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 -- 172.21.15.103:0/1549574700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 msgr2=0x7f7a080fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 --2- 172.21.15.103:0/1549574700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a080fd400 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f79f8004940 tx=0x7f79f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 -- 172.21.15.103:0/1549574700 shutdown_connections 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 --2- 172.21.15.103:0/1549574700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a080fd400 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 --2- 172.21.15.103:0/1549574700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a080fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 -- 172.21.15.103:0/1549574700 >> 172.21.15.103:0/1549574700 conn(0x7f7a080f7440 msgr2=0x7f7a080f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 -- 172.21.15.103:0/1549574700 shutdown_connections 2024-03-22T02:24:34.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.484+0000 7f7a0f992700 1 -- 172.21.15.103:0/1549574700 wait complete. 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 Processor -- start 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 -- start start 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a08066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0810e570 con 0x7f7a080fcf90 2024-03-22T02:24:34.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0f992700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a081150d0 con 0x7f7a080fbce0 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53740/0 (socket says 172.21.15.103:53740) 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 -- 172.21.15.103:0/3561223800 learned_addr learned my addr 172.21.15.103:0/3561223800 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0d72e700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a08066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 -- 172.21.15.103:0/3561223800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 msgr2=0x7f7a08066d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a08066d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0cf2d700 1 -- 172.21.15.103:0/3561223800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8005040 con 0x7f7a080fcf90 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.485+0000 7f7a0d72e700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a08066d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f7a0cf2d700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009100 tx=0x7f79f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f800e040 con 0x7f7a080fcf90 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79f800bbf0 con 0x7f7a080fcf90 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a08115270 con 0x7f7a080fcf90 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f801c3f0 con 0x7f7a080fcf90 2024-03-22T02:24:34.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.486+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081157e0 con 0x7f7a080fcf90 2024-03-22T02:24:34.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.487+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0804c710 con 0x7f7a080fcf90 2024-03-22T02:24:34.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.487+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f79f801c660 con 0x7f7a080fcf90 2024-03-22T02:24:34.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.487+0000 7f79fdffb700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 0x7f79f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:34.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.488+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f79f8088110 con 0x7f7a080fcf90 2024-03-22T02:24:34.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.488+0000 7f7a0d72e700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 0x7f79f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:34.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.488+0000 7f7a0d72e700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 0x7f79f406b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a04007fd0 tx=0x7f7a04009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:34.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.492+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f79f8058320 con 0x7f7a080fcf90 2024-03-22T02:24:34.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.755+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7a081047f0 con 0x7f7a080fcf90 2024-03-22T02:24:34.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.758+0000 7f79fdffb700 1 -- 172.21.15.103:0/3561223800 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f79f8057eb0 con 0x7f7a080fcf90 2024-03-22T02:24:34.759 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-03-22T02:24:34.759 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDi6/xlcO4OLRAA9KxdoUXDlT3oyd9GOgWJuQ== 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 msgr2=0x7f79f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 0x7f79f406b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7a04007fd0 tx=0x7f7a04009040 comp rx=0 tx=0).stop 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 msgr2=0x7f7a08114b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009100 tx=0x7f79f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 shutdown_connections 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.760+0000 7f7a0f992700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f79f40692b0 0x7f79f406b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.761+0000 7f7a0f992700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a080fbce0 0x7f7a08066d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.761+0000 7f7a0f992700 1 --2- 172.21.15.103:0/3561223800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a080fcf90 0x7f7a08114b90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:34.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.761+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 >> 172.21.15.103:0/3561223800 conn(0x7f7a080f7440 msgr2=0x7f7a080f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:34.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.761+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 shutdown_connections 2024-03-22T02:24:34.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:34.761+0000 7f7a0f992700 1 -- 172.21.15.103:0/3561223800 wait complete. 2024-03-22T02:24:35.483 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:24:35.483 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-22T02:24:35.484 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-22T02:24:35.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T02:24:35.795 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi122/config 2024-03-22T02:24:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:35 smithi122 ceph-mon[36403]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:35 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3561223800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:35 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3561223800' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:24:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:35 smithi103 ceph-mon[29258]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:35 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3561223800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:35 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3561223800' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:24:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:37.364 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 -- 172.21.15.122:0/4070900376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 msgr2=0x7f7a200fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 --2- 172.21.15.122:0/4070900376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a200fe4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a08004940 tx=0x7f7a08009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 -- 172.21.15.122:0/4070900376 shutdown_connections 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 --2- 172.21.15.122:0/4070900376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a200fe4d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 --2- 172.21.15.122:0/4070900376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 0x7f7a200fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 -- 172.21.15.122:0/4070900376 >> 172.21.15.122:0/4070900376 conn(0x7f7a200f8570 msgr2=0x7f7a200fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 -- 172.21.15.122:0/4070900376 shutdown_connections 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 -- 172.21.15.122:0/4070900376 wait complete. 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.361+0000 7f7a258ac700 1 Processor -- start 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 -- start start 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 0x7f7a2010d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:37.365 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2010e590 con 0x7f7a200fce60 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20112e70 con 0x7f7a200fe060 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:51644/0 (socket says 172.21.15.122:51644) 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 -- 172.21.15.122:0/3387451768 learned_addr learned my addr 172.21.15.122:0/3387451768 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a1effd700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 0x7f7a2010d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 -- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 msgr2=0x7f7a2010d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 0x7f7a2010d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 -- 172.21.15.122:0/3387451768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08005040 con 0x7f7a200fe060 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a167fc700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b3b0 tx=0x7f7a08009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a0800e040 con 0x7f7a200fe060 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a0800bbf0 con 0x7f7a200fe060 2024-03-22T02:24:37.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a0801c630 con 0x7f7a200fe060 2024-03-22T02:24:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.362+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a20113070 con 0x7f7a200fe060 2024-03-22T02:24:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.363+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a201135e0 con 0x7f7a200fe060 2024-03-22T02:24:37.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.363+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2004c710 con 0x7f7a200fe060 2024-03-22T02:24:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.364+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7a0800bd60 con 0x7f7a200fe060 2024-03-22T02:24:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.364+0000 7f7a17fff700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 0x7f7a0c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.364+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7a08087a30 con 0x7f7a200fe060 2024-03-22T02:24:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.364+0000 7f7a1effd700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 0x7f7a0c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:37.368 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.365+0000 7f7a1effd700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 0x7f7a0c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a200fdbd0 tx=0x7f7a100091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:37.370 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.369+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7a08057bd0 con 0x7f7a200fe060 2024-03-22T02:24:37.629 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.628+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7a20113a10 con 0x7f7a200fe060 2024-03-22T02:24:37.635 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.634+0000 7f7a17fff700 1 -- 172.21.15.122:0/3387451768 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7a08057760 con 0x7f7a200fe060 2024-03-22T02:24:37.636 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-03-22T02:24:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDl6/xlq72WJRAANrnxgTRQBEy/9lQiF5utmw== 2024-03-22T02:24:37.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 msgr2=0x7f7a0c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 0x7f7a0c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a200fdbd0 tx=0x7f7a100091a0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 msgr2=0x7f7a2010df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b3b0 tx=0x7f7a08009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 shutdown_connections 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7a0c0692a0 0x7f7a0c06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a200fce60 0x7f7a2010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 --2- 172.21.15.122:0/3387451768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a200fe060 0x7f7a2010df00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.636+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 >> 172.21.15.122:0/3387451768 conn(0x7f7a200f8570 msgr2=0x7f7a200f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:37.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.637+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 shutdown_connections 2024-03-22T02:24:37.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:24:37.637+0000 7f7a258ac700 1 -- 172.21.15.122:0/3387451768 wait complete. 2024-03-22T02:24:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:37 smithi122 ceph-mon[36403]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:37 smithi103 ceph-mon[29258]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:38.251 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:24:38.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-22T02:24:38.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-22T02:24:38.294 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-22T02:24:38.294 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-22T02:24:38.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-22T02:24:38.562 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:38 smithi122 ceph-mon[36403]: from='client.? 172.21.15.122:0/3387451768' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:38 smithi122 ceph-mon[36403]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:38 smithi122 ceph-mon[36403]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:24:38.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:38 smithi103 ceph-mon[29258]: from='client.? 172.21.15.122:0/3387451768' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:38 smithi103 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:24:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:38 smithi103 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:24:39.894 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:39 smithi103 ceph-mon[29258]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:39 smithi122 ceph-mon[36403]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2553653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f08102fd0 msgr2=0x7f9f081033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2553653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f08102fd0 0x7f9f081033a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9efc004940 tx=0x7f9efc009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2553653102 shutdown_connections 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2553653102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f080fb540 0x7f9f080fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2553653102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f08102fd0 0x7f9f081033a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2553653102 >> 172.21.15.103:0/2553653102 conn(0x7f9f08071530 msgr2=0x7f9f08071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2553653102 shutdown_connections 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2553653102 wait complete. 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 Processor -- start 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.241+0000 7f9f0ecfd700 1 -- start start 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f08102fd0 0x7f9f08113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f081145c0 con 0x7f9f080fb540 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0810dba0 con 0x7f9f08102fd0 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53764/0 (socket says 172.21.15.103:53764) 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 -- 172.21.15.103:0/2909846424 learned_addr learned my addr 172.21.15.103:0/2909846424 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f07fff700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f08102fd0 0x7f9f08113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:40.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 -- 172.21.15.103:0/2909846424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f08102fd0 msgr2=0x7f9f08113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f08102fd0 0x7f9f08113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 -- 172.21.15.103:0/2909846424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc005040 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ca99700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009150 tx=0x7f9efc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9efc00e040 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9efc013420 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0810de20 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9efc01c630 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.242+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0810e360 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.244+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f08100270 con 0x7f9f080fb540 2024-03-22T02:24:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.244+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9efc013590 con 0x7f9f080fb540 2024-03-22T02:24:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.244+0000 7f9f057fa700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 0x7f9ef806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.245+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9efc088120 con 0x7f9f080fb540 2024-03-22T02:24:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.245+0000 7f9f07fff700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 0x7f9ef806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.245+0000 7f9f07fff700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 0x7f9ef806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4006fd0 tx=0x7f9ef4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:40.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.249+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9efc054730 con 0x7f9f080fb540 2024-03-22T02:24:40.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.490+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9f0810a090 con 0x7f9f080fb540 2024-03-22T02:24:40.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.493+0000 7f9f057fa700 1 -- 172.21.15.103:0/2909846424 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167177 (secure 0 0 0) 0x7f9efc057d50 con 0x7f9f080fb540 2024-03-22T02:24:40.496 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.495+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 msgr2=0x7f9ef806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.495+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 0x7f9ef806b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4006fd0 tx=0x7f9ef4008040 comp rx=0 tx=0).stop 2024-03-22T02:24:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.495+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 msgr2=0x7f9f081139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.495+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009150 tx=0x7f9efc009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 shutdown_connections 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ef80692a0 0x7f9ef806b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f080fb540 0x7f9f081139a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 --2- 172.21.15.103:0/2909846424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f08102fd0 0x7f9f08113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 >> 172.21.15.103:0/2909846424 conn(0x7f9f08071530 msgr2=0x7f9f080fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 shutdown_connections 2024-03-22T02:24:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:40.496+0000 7f9f0ecfd700 1 -- 172.21.15.103:0/2909846424 wait complete. 2024-03-22T02:24:40.781 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:40 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2909846424' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T02:24:40.923 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:40 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2909846424' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T02:24:41.135 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":19,"active_gid":14208,"active_name":"smithi103.pfvmui","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3344821418},{"type":"v1","addr":"172.21.15.103:6801","nonce":3344821418}]},"active_addr":"172.21.15.103:6801/3344821418","active_change":"2024-03-22T02:20:50.358593+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14232,"name":"smithi122.prgtti","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.103:8443/","prometheus":"http://172.21.15.103:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2509522170}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3955856929}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1309215164}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":385301047}]}]} 2024-03-22T02:24:41.139 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-22T02:24:41.139 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-22T02:24:41.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:24:41.405 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:41 smithi122 ceph-mon[36403]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:42.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:41 smithi103 ceph-mon[29258]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:43.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 -- 172.21.15.103:0/294073774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fe7b0 msgr2=0x7f0d54106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:43.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 --2- 172.21.15.103:0/294073774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fe7b0 0x7f0d54106df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0d44004940 tx=0x7f0d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:43.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 -- 172.21.15.103:0/294073774 shutdown_connections 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 --2- 172.21.15.103:0/294073774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fe7b0 0x7f0d54106df0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 --2- 172.21.15.103:0/294073774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fde10 0x7f0d540fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 -- 172.21.15.103:0/294073774 >> 172.21.15.103:0/294073774 conn(0x7f0d540f7860 msgr2=0x7f0d540f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 -- 172.21.15.103:0/294073774 shutdown_connections 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 -- 172.21.15.103:0/294073774 wait complete. 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.178+0000 7f0d5a879700 1 Processor -- start 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d5a879700 1 -- start start 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d5a879700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fde10 0x7f0d54113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d5a879700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d5a879700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d541147d0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d5a879700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5410db30 con 0x7f0d540fde10 2024-03-22T02:24:43.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:43.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53774/0 (socket says 172.21.15.103:53774) 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 -- 172.21.15.103:0/469756577 learned_addr learned my addr 172.21.15.103:0/469756577 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d53fff700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fde10 0x7f0d54113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 -- 172.21.15.103:0/469756577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fde10 msgr2=0x7f0d54113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fde10 0x7f0d54113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.179+0000 7f0d537fe700 1 -- 172.21.15.103:0/469756577 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d540fe7b0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d537fe700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009380 tx=0x7f0d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d4400e040 con 0x7f0d540fe7b0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d44009af0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5410ddb0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d4401c630 con 0x7f0d540fe7b0 2024-03-22T02:24:43.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.180+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5410e2f0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.181+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54066770 con 0x7f0d540fe7b0 2024-03-22T02:24:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.181+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0d4400bbc0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.182+0000 7f0d50ff9700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 0x7f0d3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.182+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d44088160 con 0x7f0d540fe7b0 2024-03-22T02:24:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.182+0000 7f0d53fff700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 0x7f0d3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:43.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.182+0000 7f0d53fff700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 0x7f0d3c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d48006fd0 tx=0x7f0d48008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:43.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.186+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0d44058260 con 0x7f0d540fe7b0 2024-03-22T02:24:43.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.384+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d54109f20 con 0x7f0d540fe7b0 2024-03-22T02:24:43.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.385+0000 7f0d50ff9700 1 -- 172.21.15.103:0/469756577 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11133 (secure 0 0 0) 0x7f0d44057df0 con 0x7f0d540fe7b0 2024-03-22T02:24:43.386 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:43.386 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":40,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:24:23.447675+0000","last_up_change":"2024-03-22T02:24:22.446345+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6803","nonce":1382751588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6805","nonce":1382751588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6809","nonce":1382751588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6807","nonce":1382751588}]},"public_addr":"172.21.15.103:6803/1382751588","cluster_addr":"172.21.15.103:6805/1382751588","heartbeat_back_addr":"172.21.15.103:6809/1382751588","heartbeat_front_addr":"172.21.15.103:6807/1382751588","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6811","nonce":3229649513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6813","nonce":3229649513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6817","nonce":3229649513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6815","nonce":3229649513}]},"public_addr":"172.21.15.103:6811/3229649513","cluster_addr":"172.21.15.103:6813/3229649513","heartbeat_back_addr":"172.21.15.103:6817/3229649513","heartbeat_front_addr":"172.21.15.103:6815/3229649513","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6819","nonce":2311429465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6821","nonce":2311429465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6825","nonce":2311429465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6823","nonce":2311429465}]},"public_addr":"172.21.15.103:6819/2311429465","cluster_addr":"172.21.15.103:6821/2311429465","heartbeat_back_addr":"172.21.15.103:6825/2311429465","heartbeat_front_addr":"172.21.15.103:6823/2311429465","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6801","nonce":599760037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6803","nonce":599760037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6807","nonce":599760037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6805","nonce":599760037}]},"public_addr":"172.21.15.122:6801/599760037","cluster_addr":"172.21.15.122:6803/599760037","heartbeat_back_addr":"172.21.15.122:6807/599760037","heartbeat_front_addr":"172.21.15.122:6805/599760037","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6809","nonce":1379244401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6811","nonce":1379244401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6815","nonce":1379244401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6813","nonce":1379244401}]},"public_addr":"172.21.15.122:6809/1379244401","cluster_addr":"172.21.15.122:6811/1379244401","heartbeat_back_addr":"172.21.15.122:6815/1379244401","heartbeat_front_addr":"172.21.15.122:6813/1379244401","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6817","nonce":2680393488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6819","nonce":2680393488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6823","nonce":2680393488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6821","nonce":2680393488}]},"public_addr":"172.21.15.122:6817/2680393488","cluster_addr":"172.21.15.122:6819/2680393488","heartbeat_back_addr":"172.21.15.122:6823/2680393488","heartbeat_front_addr":"172.21.15.122:6821/2680393488","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:24:43.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.387+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 msgr2=0x7f0d3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:43.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.387+0000 7f0d5a879700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 0x7f0d3c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d48006fd0 tx=0x7f0d48008040 comp rx=0 tx=0).stop 2024-03-22T02:24:43.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 msgr2=0x7f0d541140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:43.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009380 tx=0x7f0d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:43.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 shutdown_connections 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f0d3c0692a0 0x7f0d3c06b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d540fde10 0x7f0d54113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 --2- 172.21.15.103:0/469756577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d540fe7b0 0x7f0d541140f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 >> 172.21.15.103:0/469756577 conn(0x7f0d540f7860 msgr2=0x7f0d541036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 shutdown_connections 2024-03-22T02:24:43.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:43.388+0000 7f0d5a879700 1 -- 172.21.15.103:0/469756577 wait complete. 2024-03-22T02:24:43.989 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:43 smithi103 ceph-mon[29258]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:43.989 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:43 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/469756577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:44.014 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-22T02:24:44.014 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:24:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:43 smithi122 ceph-mon[36403]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:43 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/469756577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:44.273 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:45.857 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:45 smithi103 ceph-mon[29258]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 -- 172.21.15.103:0/3533599008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d70 msgr2=0x7fc280102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 --2- 172.21.15.103:0/3533599008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d70 0x7fc280102140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc270004940 tx=0x7fc270009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 -- 172.21.15.103:0/3533599008 shutdown_connections 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 --2- 172.21.15.103:0/3533599008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280102710 0x7fc2800fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 --2- 172.21.15.103:0/3533599008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280101d70 0x7fc280102140 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 -- 172.21.15.103:0/3533599008 >> 172.21.15.103:0/3533599008 conn(0x7fc280072bc0 msgr2=0x7fc280072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.856+0000 7fc285c94700 1 -- 172.21.15.103:0/3533599008 shutdown_connections 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.857+0000 7fc285c94700 1 -- 172.21.15.103:0/3533599008 wait complete. 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.857+0000 7fc285c94700 1 Processor -- start 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.857+0000 7fc285c94700 1 -- start start 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc285c94700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280101d70 0x7fc28010eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc285c94700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc285c94700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28010f6d0 con 0x7fc280102710 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc285c94700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28010f810 con 0x7fc280101d70 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57114/0 (socket says 172.21.15.103:57114) 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 -- 172.21.15.103:0/616047070 learned_addr learned my addr 172.21.15.103:0/616047070 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27f7fe700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280101d70 0x7fc28010eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 -- 172.21.15.103:0/616047070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280101d70 msgr2=0x7fc28010eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280101d70 0x7fc28010eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 -- 172.21.15.103:0/616047070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270005040 con 0x7fc280102710 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc27effd700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc27400b790 tx=0x7fc274009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc27400f6f0 con 0x7fc280102710 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc280117b80 con 0x7fc280102710 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc27400fd20 con 0x7fc280102710 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.858+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2740103a0 con 0x7fc280102710 2024-03-22T02:24:45.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.859+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280118110 con 0x7fc280102710 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.860+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2800ff080 con 0x7fc280102710 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.860+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc274011400 con 0x7fc280102710 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.860+0000 7fc267fff700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 0x7fc26806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.861+0000 7fc27f7fe700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 0x7fc26806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.861+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc274086be0 con 0x7fc280102710 2024-03-22T02:24:45.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.861+0000 7fc27f7fe700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 0x7fc26806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc270004f90 tx=0x7fc270009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:45.865+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc274053250 con 0x7fc280102710 2024-03-22T02:24:46.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.053+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc280108ee0 con 0x7fc280102710 2024-03-22T02:24:46.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.054+0000 7fc267fff700 1 -- 172.21.15.103:0/616047070 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11133 (secure 0 0 0) 0x7fc274056870 con 0x7fc280102710 2024-03-22T02:24:46.059 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:24:46.059 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":40,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:24:23.447675+0000","last_up_change":"2024-03-22T02:24:22.446345+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6803","nonce":1382751588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6805","nonce":1382751588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6809","nonce":1382751588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6807","nonce":1382751588}]},"public_addr":"172.21.15.103:6803/1382751588","cluster_addr":"172.21.15.103:6805/1382751588","heartbeat_back_addr":"172.21.15.103:6809/1382751588","heartbeat_front_addr":"172.21.15.103:6807/1382751588","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6811","nonce":3229649513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6813","nonce":3229649513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6817","nonce":3229649513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6815","nonce":3229649513}]},"public_addr":"172.21.15.103:6811/3229649513","cluster_addr":"172.21.15.103:6813/3229649513","heartbeat_back_addr":"172.21.15.103:6817/3229649513","heartbeat_front_addr":"172.21.15.103:6815/3229649513","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6819","nonce":2311429465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6821","nonce":2311429465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6825","nonce":2311429465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6823","nonce":2311429465}]},"public_addr":"172.21.15.103:6819/2311429465","cluster_addr":"172.21.15.103:6821/2311429465","heartbeat_back_addr":"172.21.15.103:6825/2311429465","heartbeat_front_addr":"172.21.15.103:6823/2311429465","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6801","nonce":599760037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6803","nonce":599760037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6807","nonce":599760037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6805","nonce":599760037}]},"public_addr":"172.21.15.122:6801/599760037","cluster_addr":"172.21.15.122:6803/599760037","heartbeat_back_addr":"172.21.15.122:6807/599760037","heartbeat_front_addr":"172.21.15.122:6805/599760037","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6809","nonce":1379244401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6811","nonce":1379244401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6815","nonce":1379244401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6813","nonce":1379244401}]},"public_addr":"172.21.15.122:6809/1379244401","cluster_addr":"172.21.15.122:6811/1379244401","heartbeat_back_addr":"172.21.15.122:6815/1379244401","heartbeat_front_addr":"172.21.15.122:6813/1379244401","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6817","nonce":2680393488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6819","nonce":2680393488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6823","nonce":2680393488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6821","nonce":2680393488}]},"public_addr":"172.21.15.122:6817/2680393488","cluster_addr":"172.21.15.122:6819/2680393488","heartbeat_back_addr":"172.21.15.122:6823/2680393488","heartbeat_front_addr":"172.21.15.122:6821/2680393488","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 msgr2=0x7fc26806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 0x7fc26806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc270004f90 tx=0x7fc270009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 msgr2=0x7fc28010eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc27400b790 tx=0x7fc274009220 comp rx=0 tx=0).stop 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 shutdown_connections 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc2680692a0 0x7fc26806b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc280101d70 0x7fc28010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 --2- 172.21.15.103:0/616047070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc280102710 0x7fc28010eff0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.056+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 >> 172.21.15.103:0/616047070 conn(0x7fc280072bc0 msgr2=0x7fc2800fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.057+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 shutdown_connections 2024-03-22T02:24:46.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:46.057+0000 7fc285c94700 1 -- 172.21.15.103:0/616047070 wait complete. 2024-03-22T02:24:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:45 smithi122 ceph-mon[36403]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:46.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-22T02:24:46.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-22T02:24:46.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-22T02:24:46.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-22T02:24:46.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-22T02:24:46.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-22T02:24:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:46 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/616047070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:46.906 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:46.907 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:47.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:46 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/616047070' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:24:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:47 smithi122 ceph-mon[36403]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:48.122 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:47 smithi103 ceph-mon[29258]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:49.002 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:49.002 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:49.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 -- 172.21.15.103:0/1073319773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 msgr2=0x7f437c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:49.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 --2- 172.21.15.103:0/1073319773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c0fbd30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4370004940 tx=0x7f4370009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 -- 172.21.15.103:0/1073319773 shutdown_connections 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 --2- 172.21.15.103:0/1073319773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c0fbd30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 --2- 172.21.15.103:0/1073319773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 -- 172.21.15.103:0/1073319773 >> 172.21.15.103:0/1073319773 conn(0x7f437c0f78d0 msgr2=0x7f437c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 -- 172.21.15.103:0/1073319773 shutdown_connections 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.010+0000 7f43825bc700 1 -- 172.21.15.103:0/1073319773 wait complete. 2024-03-22T02:24:49.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.011+0000 7f43825bc700 1 Processor -- start 2024-03-22T02:24:49.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f43825bc700 1 -- start start 2024-03-22T02:24:49.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f43825bc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:49.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f43825bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f43825bc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c114560 con 0x7f437c1026c0 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f43825bc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c10d950 con 0x7f437c101d20 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f437bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f437b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f437bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:34516/0 (socket says 172.21.15.103:34516) 2024-03-22T02:24:49.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.012+0000 7f437bfff700 1 -- 172.21.15.103:0/253270610 learned_addr learned my addr 172.21.15.103:0/253270610 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f437bfff700 1 -- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 msgr2=0x7f437c113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f437bfff700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f437bfff700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370005040 con 0x7f437c101d20 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f437bfff700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f436c00eb40 tx=0x7f436c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436c007cc0 con 0x7f437c101d20 2024-03-22T02:24:49.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f436c00f040 con 0x7f437c101d20 2024-03-22T02:24:49.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.013+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c10dbd0 con 0x7f437c101d20 2024-03-22T02:24:49.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.014+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436c00ca10 con 0x7f437c101d20 2024-03-22T02:24:49.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.014+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c10e0e0 con 0x7f437c101d20 2024-03-22T02:24:49.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.015+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f437c10e5e0 con 0x7f437c101d20 2024-03-22T02:24:49.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.015+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f436c00cb70 con 0x7f437c101d20 2024-03-22T02:24:49.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4378ff9700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 0x7f436406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:49.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f436c086a00 con 0x7f437c101d20 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f437b7fe700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 0x7f436406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4378ff9700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 0x7f4364071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4364071a40 con 0x7f436406ef80 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f436c08b220 con 0x7f437c101d20 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.016+0000 7f4380b59700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 0x7f4364071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.017+0000 7f437b7fe700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 0x7f436406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4370009b30 tx=0x7f4370009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:49.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.017+0000 7f4380b59700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 0x7f4364071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:49.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.018+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== osd.2 v2:172.21.15.103:6818/2311429465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4364071a40 con 0x7f436406ef80 2024-03-22T02:24:49.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.041+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 --> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f437c1056b0 con 0x7f436406ef80 2024-03-22T02:24:49.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.041+0000 7f4378ff9700 1 -- 172.21.15.103:0/253270610 <== osd.2 v2:172.21.15.103:6818/2311429465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f437c1056b0 con 0x7f436406ef80 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 msgr2=0x7f4364071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 0x7f4364071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 msgr2=0x7f436406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 0x7f436406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4370009b30 tx=0x7f4370009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 msgr2=0x7f437c1139d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f436c00eb40 tx=0x7f436c00b040 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 shutdown_connections 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6818/2311429465,v1:172.21.15.103:6819/2311429465] conn(0x7f436406ef80 0x7f4364071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f43640692b0 0x7f436406b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f437c101d20 0x7f437c1139d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 --2- 172.21.15.103:0/253270610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f437c1026c0 0x7f437c113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:49.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 >> 172.21.15.103:0/253270610 conn(0x7f437c0f78d0 msgr2=0x7f437c104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:49.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 shutdown_connections 2024-03-22T02:24:49.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:49.042+0000 7f43825bc700 1 -- 172.21.15.103:0/253270610 wait complete. 2024-03-22T02:24:49.920 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:49 smithi103 ceph-mon[29258]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:50.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:49 smithi122 ceph-mon[36403]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:50.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- 172.21.15.103:0/329882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 msgr2=0x7fafc00fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 --2- 172.21.15.103:0/329882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc00fe1e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fafb0004940 tx=0x7fafb0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- 172.21.15.103:0/329882 shutdown_connections 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 --2- 172.21.15.103:0/329882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 --2- 172.21.15.103:0/329882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc00fe1e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- 172.21.15.103:0/329882 >> 172.21.15.103:0/329882 conn(0x7fafc00f7870 msgr2=0x7fafc00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- 172.21.15.103:0/329882 shutdown_connections 2024-03-22T02:24:50.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- 172.21.15.103:0/329882 wait complete. 2024-03-22T02:24:50.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 Processor -- start 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.405+0000 7fafc517f700 1 -- start start 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafc517f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafc517f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafc517f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc01145a0 con 0x7fafc00fde10 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafc517f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc010db10 con 0x7fafc00fe7b0 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57160/0 (socket says 172.21.15.103:57160) 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 -- 172.21.15.103:0/268883384 learned_addr learned my addr 172.21.15.103:0/268883384 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbe59c700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 -- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 msgr2=0x7fafc0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0005040 con 0x7fafc00fde10 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbe59c700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafbed9d700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009380 tx=0x7fafb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb000e040 con 0x7fafc00fde10 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc010dd90 con 0x7fafc00fde10 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafb0009af0 con 0x7fafc00fde10 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.406+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb001c630 con 0x7fafc00fde10 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.407+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc010e2d0 con 0x7fafc00fde10 2024-03-22T02:24:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.407+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafc010e7f0 con 0x7fafc00fde10 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fafb000bbc0 con 0x7fafc00fde10 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 0x7fafa806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fafb0088160 con 0x7fafc00fde10 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafbe59c700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 0x7fafa806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 0x7fafa8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafa8071a30 con 0x7fafa806ef70 2024-03-22T02:24:50.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.408+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb008a750 con 0x7fafc00fde10 2024-03-22T02:24:50.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.409+0000 7fafbe59c700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 0x7fafa806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafc010ecd0 tx=0x7fafb4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:50.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.409+0000 7fafbf59e700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 0x7fafa8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:50.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.409+0000 7fafbf59e700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 0x7fafa8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:50.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.410+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== osd.5 v2:172.21.15.122:6816/2680393488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fafa8071a30 con 0x7fafa806ef70 2024-03-22T02:24:50.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.430+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 --> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafc010e9b0 con 0x7fafa806ef70 2024-03-22T02:24:50.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafa77fe700 1 -- 172.21.15.103:0/268883384 <== osd.5 v2:172.21.15.122:6816/2680393488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fafc010e9b0 con 0x7fafa806ef70 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 msgr2=0x7fafa8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 0x7fafa8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 msgr2=0x7fafa806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 0x7fafa806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafc010ecd0 tx=0x7fafb4008040 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 msgr2=0x7fafc0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009380 tx=0x7fafb0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 shutdown_connections 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:6816/2680393488,v1:172.21.15.122:6817/2680393488] conn(0x7fafa806ef70 0x7fafa8071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fafa80692a0 0x7fafa806b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafc00fde10 0x7fafc0113980 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 --2- 172.21.15.103:0/268883384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafc00fe7b0 0x7fafc0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:50.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 >> 172.21.15.103:0/268883384 conn(0x7fafc00f7870 msgr2=0x7fafc00ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:50.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 shutdown_connections 2024-03-22T02:24:50.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:50.431+0000 7fafc517f700 1 -- 172.21.15.103:0/268883384 wait complete. 2024-03-22T02:24:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:24:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:51.867 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:51 smithi103 ceph-mon[29258]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:51 smithi122 ceph-mon[36403]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:52.363 INFO:teuthology.orchestra.run.smithi103.stdout:81604378648 2024-03-22T02:24:52.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-22T02:24:53.435 INFO:teuthology.orchestra.run.smithi103.stdout:167503724551 2024-03-22T02:24:53.436 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-22T02:24:53.973 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:53 smithi103 ceph-mon[29258]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:53 smithi122 ceph-mon[36403]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:54.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 -- 172.21.15.103:0/1670920757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 msgr2=0x7fbab41033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:54.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 --2- 172.21.15.103:0/1670920757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab41033a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004940 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 -- 172.21.15.103:0/1670920757 shutdown_connections 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 --2- 172.21.15.103:0/1670920757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 0x7fbab40fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 --2- 172.21.15.103:0/1670920757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab41033a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 -- 172.21.15.103:0/1670920757 >> 172.21.15.103:0/1670920757 conn(0x7fbab4071530 msgr2=0x7fbab4071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 -- 172.21.15.103:0/1670920757 shutdown_connections 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.875+0000 7fbab99dd700 1 -- 172.21.15.103:0/1670920757 wait complete. 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 Processor -- start 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 -- start start 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 0x7fbab4113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41144b0 con 0x7fbab4102fd0 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab99dd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab410dab0 con 0x7fbab40fb540 2024-03-22T02:24:54.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47048/0 (socket says 172.21.15.103:47048) 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab27fc700 1 -- 172.21.15.103:0/3101273867 learned_addr learned my addr 172.21.15.103:0/3101273867 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.876+0000 7fbab2ffd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 0x7fbab4113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab27fc700 1 -- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 msgr2=0x7fbab4113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab27fc700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 0x7fbab4113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab27fc700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4005040 con 0x7fbab4102fd0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab27fc700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8002a50 tx=0x7fbaa80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa8007cc0 con 0x7fbab4102fd0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaa800f040 con 0x7fbab4102fd0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab410dd90 con 0x7fbab4102fd0 2024-03-22T02:24:54.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbaa800da30 con 0x7fbab4102fd0 2024-03-22T02:24:54.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.877+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab410e350 con 0x7fbab4102fd0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.878+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbab410e880 con 0x7fbab4102fd0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.878+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbaa800db90 con 0x7fbab4102fd0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fba9b7fe700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 0x7fba9c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbaa8086b20 con 0x7fbab4102fd0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fbab2ffd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 0x7fba9c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fba9b7fe700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 0x7fba9c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba9c071a30 con 0x7fba9c06ef70 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaa808b220 con 0x7fbab4102fd0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fbab37fe700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 0x7fba9c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.879+0000 7fbab2ffd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 0x7fba9c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004f90 tx=0x7fbaa4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.880+0000 7fbab37fe700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 0x7fba9c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:54.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.880+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== osd.4 v2:172.21.15.122:6808/1379244401 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fba9c071a30 con 0x7fba9c06ef70 2024-03-22T02:24:54.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.895+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 --> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbab405e670 con 0x7fba9c06ef70 2024-03-22T02:24:54.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.895+0000 7fba9b7fe700 1 -- 172.21.15.103:0/3101273867 <== osd.4 v2:172.21.15.122:6808/1379244401 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbab405e670 con 0x7fba9c06ef70 2024-03-22T02:24:54.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 msgr2=0x7fba9c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 0x7fba9c071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 msgr2=0x7fba9c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 0x7fba9c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004f90 tx=0x7fbaa4009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 msgr2=0x7fbab4113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8002a50 tx=0x7fbaa80046b0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 shutdown_connections 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fba9c0692a0 0x7fba9c06b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:6808/1379244401,v1:172.21.15.122:6809/1379244401] conn(0x7fba9c06ef70 0x7fba9c071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbab40fb540 0x7fbab4113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 --2- 172.21.15.103:0/3101273867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbab4102fd0 0x7fbab4113e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:54.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 >> 172.21.15.103:0/3101273867 conn(0x7fbab4071530 msgr2=0x7fbab40fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 shutdown_connections 2024-03-22T02:24:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:54.896+0000 7fbab99dd700 1 -- 172.21.15.103:0/3101273867 wait complete. 2024-03-22T02:24:55.801 INFO:teuthology.orchestra.run.smithi103.stdout:146028888078 2024-03-22T02:24:55.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-22T02:24:55.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:55 smithi103 ceph-mon[29258]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 -- 172.21.15.103:0/4081451980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 msgr2=0x7f2c28106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/4081451980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c28106f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004940 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 -- 172.21.15.103:0/4081451980 shutdown_connections 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/4081451980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c28106f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/4081451980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 0x7f2c280fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 -- 172.21.15.103:0/4081451980 >> 172.21.15.103:0/4081451980 conn(0x7f2c28072bc0 msgr2=0x7f2c28072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:56.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 -- 172.21.15.103:0/4081451980 shutdown_connections 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.016+0000 7f2c2e92b700 1 -- 172.21.15.103:0/4081451980 wait complete. 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 Processor -- start 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 -- start start 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 0x7f2c28113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28114730 con 0x7f2c280fe830 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.017+0000 7f2c2e92b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2810db20 con 0x7f2c280fde90 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47064/0 (socket says 172.21.15.103:47064) 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 -- 172.21.15.103:0/2227749037 learned_addr learned my addr 172.21.15.103:0/2227749037 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c27fff700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 0x7f2c28113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 -- 172.21.15.103:0/2227749037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 msgr2=0x7f2c28113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 0x7f2c28113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18005040 con 0x7f2c280fe830 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c277fe700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b3b0 tx=0x7f2c18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1800e040 con 0x7f2c280fe830 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2810dda0 con 0x7f2c280fe830 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2810e2e0 con 0x7f2c280fe830 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c18013420 con 0x7f2c280fe830 2024-03-22T02:24:56.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.018+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1801c5f0 con 0x7f2c280fe830 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.019+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c2810e850 con 0x7f2c280fe830 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2c1801c860 con 0x7f2c280fe830 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 0x7f2c1006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2c18088000 con 0x7f2c280fe830 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c27fff700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 0x7f2c1006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 0x7f2c10071450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c10071b00 con 0x7f2c1006f040 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.020+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c1808a540 con 0x7f2c280fe830 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.021+0000 7f2c2cec8700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 0x7f2c10071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.021+0000 7f2c27fff700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 0x7f2c1006b820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c004f40 tx=0x7f2c1c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.021+0000 7f2c2cec8700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 0x7f2c10071450 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:56.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.022+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== osd.1 v2:172.21.15.103:6810/3229649513 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2c10071b00 con 0x7f2c1006f040 2024-03-22T02:24:56.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.050+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 --> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c28102400 con 0x7f2c1006f040 2024-03-22T02:24:56.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c24ff9700 1 -- 172.21.15.103:0/2227749037 <== osd.1 v2:172.21.15.103:6810/3229649513 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2c28102400 con 0x7f2c1006f040 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 msgr2=0x7f2c10071450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 0x7f2c10071450 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 msgr2=0x7f2c1006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 0x7f2c1006b820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c004f40 tx=0x7f2c1c00b040 comp rx=0 tx=0).stop 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 msgr2=0x7f2c281140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b3b0 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 shutdown_connections 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2c10069370 0x7f2c1006b820 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:6810/3229649513,v1:172.21.15.103:6811/3229649513] conn(0x7f2c1006f040 0x7f2c10071450 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c280fde90 0x7f2c28113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 --2- 172.21.15.103:0/2227749037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c280fe830 0x7f2c281140e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:56.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 >> 172.21.15.103:0/2227749037 conn(0x7f2c28072bc0 msgr2=0x7f2c281037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:56.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 shutdown_connections 2024-03-22T02:24:56.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:56.051+0000 7f2c2e92b700 1 -- 172.21.15.103:0/2227749037 wait complete. 2024-03-22T02:24:56.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:55 smithi122 ceph-mon[36403]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:57.354 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:57.393 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:24:57.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:57 smithi103 ceph-mon[29258]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:57 smithi122 ceph-mon[36403]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 -- 172.21.15.103:0/2288146374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c41026d0 msgr2=0x7fa7c40fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 --2- 172.21.15.103:0/2288146374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c41026d0 0x7fa7c40fbd40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0004940 tx=0x7fa7b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 -- 172.21.15.103:0/2288146374 shutdown_connections 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 --2- 172.21.15.103:0/2288146374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c41026d0 0x7fa7c40fbd40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 --2- 172.21.15.103:0/2288146374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c4101d30 0x7fa7c4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 -- 172.21.15.103:0/2288146374 >> 172.21.15.103:0/2288146374 conn(0x7fa7c40f78c0 msgr2=0x7fa7c40f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 -- 172.21.15.103:0/2288146374 shutdown_connections 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.134+0000 7fa7caab3700 1 -- 172.21.15.103:0/2288146374 wait complete. 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 Processor -- start 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 -- start start 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c41026d0 0x7fa7c4113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4114590 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7caab3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c410db20 con 0x7fa7c41026d0 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7c884f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7c884f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47078/0 (socket says 172.21.15.103:47078) 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.135+0000 7fa7c884f700 1 -- 172.21.15.103:0/1812494371 learned_addr learned my addr 172.21.15.103:0/1812494371 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c3fff700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c41026d0 0x7fa7c4113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c884f700 1 -- 172.21.15.103:0/1812494371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c41026d0 msgr2=0x7fa7c4113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c884f700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c41026d0 0x7fa7c4113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c884f700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0005040 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c884f700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800b790 tx=0x7fa7b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b800f6f0 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7b800fd20 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c410de00 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b80103a0 con 0x7fa7c4101d30 2024-03-22T02:24:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.136+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c410e390 con 0x7fa7c4101d30 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.137+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7c410e910 con 0x7fa7c4101d30 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa7b8011400 con 0x7fa7c4101d30 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 0x7fa7b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa7b80876b0 con 0x7fa7c4101d30 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c3fff700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 0x7fa7b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 0x7fa7b4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7b4071a30 con 0x7fa7b406ef70 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7b8087a90 con 0x7fa7c4101d30 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c9050700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 0x7fa7b4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.138+0000 7fa7c3fff700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 0x7fa7b406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0004890 tx=0x7fa7b0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.139+0000 7fa7c9050700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 0x7fa7b4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:59.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.139+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== osd.0 v2:172.21.15.103:6802/1382751588 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa7b4071a30 con 0x7fa7b406ef70 2024-03-22T02:24:59.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.154+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 --> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa7c40fd480 con 0x7fa7b406ef70 2024-03-22T02:24:59.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.154+0000 7fa7c17fa700 1 -- 172.21.15.103:0/1812494371 <== osd.0 v2:172.21.15.103:6802/1382751588 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa7c40fd480 con 0x7fa7b406ef70 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 msgr2=0x7fa7b4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 0x7fa7b4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 msgr2=0x7fa7b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 0x7fa7b406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0004890 tx=0x7fa7b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 msgr2=0x7fa7c4113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800b790 tx=0x7fa7b8009250 comp rx=0 tx=0).stop 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 shutdown_connections 2024-03-22T02:24:59.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6802/1382751588,v1:172.21.15.103:6803/1382751588] conn(0x7fa7b406ef70 0x7fa7b4071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa7b40692a0 0x7fa7b406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7c4101d30 0x7fa7c4113970 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 --2- 172.21.15.103:0/1812494371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7c41026d0 0x7fa7c4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 >> 172.21.15.103:0/1812494371 conn(0x7fa7c40f78c0 msgr2=0x7fa7c40f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 shutdown_connections 2024-03-22T02:24:59.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.155+0000 7fa7caab3700 1 -- 172.21.15.103:0/1812494371 wait complete. 2024-03-22T02:24:59.613 INFO:teuthology.orchestra.run.smithi103.stdout:60129542175 2024-03-22T02:24:59.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-22T02:24:59.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 -- 172.21.15.103:0/1102219813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a00fb4a0 msgr2=0x7fb0a00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 --2- 172.21.15.103:0/1102219813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a00fb910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb088004940 tx=0x7fb088009e30 comp rx=0 tx=0).stop 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 -- 172.21.15.103:0/1102219813 shutdown_connections 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 --2- 172.21.15.103:0/1102219813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a00fb910 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 --2- 172.21.15.103:0/1102219813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0102e70 0x7fb0a0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 -- 172.21.15.103:0/1102219813 >> 172.21.15.103:0/1102219813 conn(0x7fb0a00714c0 msgr2=0x7fb0a00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 -- 172.21.15.103:0/1102219813 shutdown_connections 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.792+0000 7fb0a7161700 1 -- 172.21.15.103:0/1102219813 wait complete. 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 Processor -- start 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 -- start start 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a0113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0114700 con 0x7fb0a0102e70 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.793+0000 7fb0a7161700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a010daf0 con 0x7fb0a00fb4a0 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47102/0 (socket says 172.21.15.103:47102) 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb0a4efd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a0113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 -- 172.21.15.103:0/329827007 learned_addr learned my addr 172.21.15.103:0/329827007 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 -- 172.21.15.103:0/329827007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a00fb4a0 msgr2=0x7fb0a0113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a0113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 -- 172.21.15.103:0/329827007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088005040 con 0x7fb0a0102e70 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09ffff700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb08800b3b0 tx=0x7fb088009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb08800e040 con 0x7fb0a0102e70 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb088013420 con 0x7fb0a0102e70 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a010dd70 con 0x7fb0a0102e70 2024-03-22T02:24:59.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb08801c630 con 0x7fb0a0102e70 2024-03-22T02:24:59.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.794+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a010e330 con 0x7fb0a0102e70 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.795+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a0100110 con 0x7fb0a0102e70 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.796+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb08800baf0 con 0x7fb0a0102e70 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.796+0000 7fb09d7fa700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 0x7fb08c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.796+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb088087fa0 con 0x7fb0a0102e70 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.796+0000 7fb0a4efd700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 0x7fb08c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:59.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.797+0000 7fb0a4efd700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 0x7fb08c06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb090004f40 tx=0x7fb09000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:59.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.801+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb088058140 con 0x7fb0a0102e70 2024-03-22T02:24:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:24:59 smithi103 ceph-mon[29258]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:24:59.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.997+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb0a010f1c0 con 0x7fb0a0102e70 2024-03-22T02:24:59.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:24:59.998+0000 7fb09d7fa700 1 -- 172.21.15.103:0/329827007 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb088057cd0 con 0x7fb0a0102e70 2024-03-22T02:24:59.999 INFO:teuthology.orchestra.run.smithi103.stdout:167503724553 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 msgr2=0x7fb08c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 0x7fb08c06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb090004f40 tx=0x7fb09000b040 comp rx=0 tx=0).stop 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 msgr2=0x7fb0a01140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb08800b3b0 tx=0x7fb088009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 shutdown_connections 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb08c0692a0 0x7fb08c06b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a00fb4a0 0x7fb0a0113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 --2- 172.21.15.103:0/329827007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0a0102e70 0x7fb0a01140b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 >> 172.21.15.103:0/329827007 conn(0x7fb0a00714c0 msgr2=0x7fb0a00fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 shutdown_connections 2024-03-22T02:25:00.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:00.000+0000 7fb0a7161700 1 -- 172.21.15.103:0/329827007 wait complete. 2024-03-22T02:25:00.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:24:59 smithi122 ceph-mon[36403]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:00 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/329827007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T02:25:00.919 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:00 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/329827007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T02:25:00.950 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724551 got 167503724553 for osd.5 2024-03-22T02:25:00.950 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:01.870 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:01 smithi103 ceph-mon[29258]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:02.096 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:01 smithi122 ceph-mon[36403]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:02.850 INFO:teuthology.orchestra.run.smithi103.stdout:38654705701 2024-03-22T02:25:02.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-22T02:25:03.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- 172.21.15.103:0/1209130692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 msgr2=0x7f5418106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 --2- 172.21.15.103:0/1209130692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418106dc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5408004940 tx=0x7f5408009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:03.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- 172.21.15.103:0/1209130692 shutdown_connections 2024-03-22T02:25:03.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 --2- 172.21.15.103:0/1209130692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418106dc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 --2- 172.21.15.103:0/1209130692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 0x7f54180fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- 172.21.15.103:0/1209130692 >> 172.21.15.103:0/1209130692 conn(0x7f54180f7890 msgr2=0x7f54180f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- 172.21.15.103:0/1209130692 shutdown_connections 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- 172.21.15.103:0/1209130692 wait complete. 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 Processor -- start 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.522+0000 7f5420268700 1 -- start start 2024-03-22T02:25:03.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 0x7f5418113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54181144b0 con 0x7f54180fe780 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541810dab0 con 0x7f54180fdde0 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47126/0 (socket says 172.21.15.103:47126) 2024-03-22T02:25:03.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 -- 172.21.15.103:0/3373549820 learned_addr learned my addr 172.21.15.103:0/3373549820 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541e004700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 0x7f5418113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 -- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 msgr2=0x7f5418113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 0x7f5418113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408005040 con 0x7f54180fe780 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f541d803700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5408009380 tx=0x7f5408009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:03.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f540800e040 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541810dd30 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5408009af0 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f540801c3f0 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.523+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541810e270 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.524+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f541810e850 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.525+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5408013420 con 0x7f54180fe780 2024-03-22T02:25:03.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.525+0000 7f540effd700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 0x7f540406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.525+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5408088c20 con 0x7f54180fe780 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.525+0000 7f541e004700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 0x7f540406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f540effd700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 0x7f5404071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5404071a30 con 0x7f540406ef70 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5408089000 con 0x7f54180fe780 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f541e004700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 0x7f540406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5414004f40 tx=0x7f541400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:03.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f541e805700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 0x7f5404071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.526+0000 7f541e805700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 0x7f5404071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:03.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.527+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== osd.3 v2:172.21.15.122:6800/599760037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5404071a30 con 0x7f540406ef70 2024-03-22T02:25:03.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.542+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 --> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54180f97e0 con 0x7f540406ef70 2024-03-22T02:25:03.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.542+0000 7f540effd700 1 -- 172.21.15.103:0/3373549820 <== osd.3 v2:172.21.15.122:6800/599760037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f54180f97e0 con 0x7f540406ef70 2024-03-22T02:25:03.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 msgr2=0x7f5404071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 0x7f5404071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 msgr2=0x7f540406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 0x7f540406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5414004f40 tx=0x7f541400b040 comp rx=0 tx=0).stop 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 msgr2=0x7f5418113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5408009380 tx=0x7f5408009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 shutdown_connections 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54040692a0 0x7f540406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:6800/599760037,v1:172.21.15.122:6801/599760037] conn(0x7f540406ef70 0x7f5404071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54180fdde0 0x7f5418113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 --2- 172.21.15.103:0/3373549820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54180fe780 0x7f5418113e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 >> 172.21.15.103:0/3373549820 conn(0x7f54180f7890 msgr2=0x7f54180f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:03.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 shutdown_connections 2024-03-22T02:25:03.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.543+0000 7f5420268700 1 -- 172.21.15.103:0/3373549820 wait complete. 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 -- 172.21.15.103:0/4002671961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 msgr2=0x7f617c1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 --2- 172.21.15.103:0/4002671961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c1033a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f616c004940 tx=0x7f616c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 -- 172.21.15.103:0/4002671961 shutdown_connections 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 --2- 172.21.15.103:0/4002671961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 0x7f617c0fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 --2- 172.21.15.103:0/4002671961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c1033a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 -- 172.21.15.103:0/4002671961 >> 172.21.15.103:0/4002671961 conn(0x7f617c071530 msgr2=0x7f617c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:03.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 -- 172.21.15.103:0/4002671961 shutdown_connections 2024-03-22T02:25:03.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.721+0000 7f61833c2700 1 -- 172.21.15.103:0/4002671961 wait complete. 2024-03-22T02:25:03.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.722+0000 7f61833c2700 1 Processor -- start 2024-03-22T02:25:03.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.722+0000 7f61833c2700 1 -- start start 2024-03-22T02:25:03.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.722+0000 7f61833c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 0x7f617c113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f61833c2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f61833c2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c1144b0 con 0x7f617c102fd0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f61833c2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10dab0 con 0x7f617c0fb540 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55518/0 (socket says 172.21.15.103:55518) 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 -- 172.21.15.103:0/3099233835 learned_addr learned my addr 172.21.15.103:0/3099233835 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618115e700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 0x7f617c113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 -- 172.21.15.103:0/3099233835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 msgr2=0x7f617c113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 0x7f617c113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 -- 172.21.15.103:0/3099233835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c005040 con 0x7f617c102fd0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f618095d700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6178002a50 tx=0x7f61780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6178007cc0 con 0x7f617c102fd0 2024-03-22T02:25:03.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f617800f040 con 0x7f617c102fd0 2024-03-22T02:25:03.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c10dd90 con 0x7f617c102fd0 2024-03-22T02:25:03.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617800da30 con 0x7f617c102fd0 2024-03-22T02:25:03.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.723+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c10e350 con 0x7f617c102fd0 2024-03-22T02:25:03.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.724+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c100270 con 0x7f617c102fd0 2024-03-22T02:25:03.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.725+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f617800db90 con 0x7f617c102fd0 2024-03-22T02:25:03.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.725+0000 7f6171ffb700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 0x7f616806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:03.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.725+0000 7f618115e700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 0x7f616806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:03.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.725+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6178086b20 con 0x7f617c102fd0 2024-03-22T02:25:03.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.726+0000 7f618115e700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 0x7f616806b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f616c004f90 tx=0x7f616c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:03.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.730+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6178056c20 con 0x7f617c102fd0 2024-03-22T02:25:03.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.919+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f617c10ef70 con 0x7f617c102fd0 2024-03-22T02:25:03.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.919+0000 7f6171ffb700 1 -- 172.21.15.103:0/3099233835 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61780567b0 con 0x7f617c102fd0 2024-03-22T02:25:03.921 INFO:teuthology.orchestra.run.smithi103.stdout:81604378651 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 msgr2=0x7f616806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 0x7f616806b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f616c004f90 tx=0x7f616c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 msgr2=0x7f617c113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6178002a50 tx=0x7f61780046b0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 shutdown_connections 2024-03-22T02:25:03.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61680692a0 0x7f616806b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f617c0fb540 0x7f617c113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 --2- 172.21.15.103:0/3099233835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f617c102fd0 0x7f617c113e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 >> 172.21.15.103:0/3099233835 conn(0x7f617c071530 msgr2=0x7f617c0fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 shutdown_connections 2024-03-22T02:25:03.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:03.922+0000 7f61833c2700 1 -- 172.21.15.103:0/3099233835 wait complete. 2024-03-22T02:25:03.934 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:03 smithi103 ceph-mon[29258]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:03 smithi122 ceph-mon[36403]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:04.419 INFO:teuthology.orchestra.run.smithi103.stdout:111669149716 2024-03-22T02:25:04.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-22T02:25:04.463 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:04.463 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:04.952 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:04 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3099233835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T02:25:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:04 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3099233835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T02:25:05.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-03-22T02:25:05.752 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:05 smithi122 ceph-mon[36403]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:06.099 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:05 smithi103 ceph-mon[29258]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:06.099 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:06.353 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:07.549 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 -- 172.21.15.103:0/3845829904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fe830 msgr2=0x7efea8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 --2- 172.21.15.103:0/3845829904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fe830 0x7efea8106f00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efe98004940 tx=0x7efe98009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 -- 172.21.15.103:0/3845829904 shutdown_connections 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 --2- 172.21.15.103:0/3845829904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fe830 0x7efea8106f00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 --2- 172.21.15.103:0/3845829904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fde90 0x7efea80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 -- 172.21.15.103:0/3845829904 >> 172.21.15.103:0/3845829904 conn(0x7efea8072bc0 msgr2=0x7efea8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:07.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 -- 172.21.15.103:0/3845829904 shutdown_connections 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.580+0000 7efeb066e700 1 -- 172.21.15.103:0/3845829904 wait complete. 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 Processor -- start 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 -- start start 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fe830 0x7efea80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea80fc820 con 0x7efea80fde90 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeb066e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea80fc960 con 0x7efea80fe830 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55546/0 (socket says 172.21.15.103:55546) 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 -- 172.21.15.103:0/2974552351 learned_addr learned my addr 172.21.15.103:0/2974552351 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeadc09700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fe830 0x7efea80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 -- 172.21.15.103:0/2974552351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fe830 msgr2=0x7efea80fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fe830 0x7efea80fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 -- 172.21.15.103:0/2974552351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98005040 con 0x7efea80fde90 2024-03-22T02:25:07.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.581+0000 7efeae40a700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efea4002a50 tx=0x7efea40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:07.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.582+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea4007cc0 con 0x7efea80fde90 2024-03-22T02:25:07.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.582+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efea400f040 con 0x7efea80fde90 2024-03-22T02:25:07.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.582+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea8119ea0 con 0x7efea80fde90 2024-03-22T02:25:07.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.582+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea400da30 con 0x7efea80fde90 2024-03-22T02:25:07.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.582+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea811a2a0 con 0x7efea80fde90 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.583+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea8065e30 con 0x7efea80fde90 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.583+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efea400db90 con 0x7efea80fde90 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.583+0000 7efe9effd700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 0x7efe9406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.584+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efea4086b20 con 0x7efea80fde90 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.584+0000 7efeadc09700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 0x7efe9406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:07.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.584+0000 7efeadc09700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 0x7efe9406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efea80fd690 tx=0x7efe98009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:07.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.588+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efea4053190 con 0x7efea80fde90 2024-03-22T02:25:07.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.779+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efea805f5f0 con 0x7efea80fde90 2024-03-22T02:25:07.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.779+0000 7efe9effd700 1 -- 172.21.15.103:0/2974552351 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efea40567b0 con 0x7efea80fde90 2024-03-22T02:25:07.781 INFO:teuthology.orchestra.run.smithi103.stdout:60129542177 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 msgr2=0x7efe9406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 0x7efe9406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efea80fd690 tx=0x7efe98009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 msgr2=0x7efea80fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efea4002a50 tx=0x7efea40046b0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 shutdown_connections 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7efe940692a0 0x7efe9406b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efea80fde90 0x7efea80fbd50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 --2- 172.21.15.103:0/2974552351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efea80fe830 0x7efea80fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.781+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 >> 172.21.15.103:0/2974552351 conn(0x7efea8072bc0 msgr2=0x7efea81037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.782+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 shutdown_connections 2024-03-22T02:25:07.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:07.782+0000 7efeb066e700 1 -- 172.21.15.103:0/2974552351 wait complete. 2024-03-22T02:25:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:07 smithi103 ceph-mon[29258]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:07 smithi122 ceph-mon[36403]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:08.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/465014633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba01037a0 msgr2=0x7f2ba00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/465014633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba01037a0 0x7f2ba00fce10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2b90004940 tx=0x7f2b90009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/465014633 shutdown_connections 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/465014633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba01037a0 0x7f2ba00fce10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/465014633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba0102e00 0x7f2ba01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/465014633 >> 172.21.15.103:0/465014633 conn(0x7f2ba00f89b0 msgr2=0x7f2ba00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/465014633 shutdown_connections 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.822+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/465014633 wait complete. 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 Processor -- start 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 -- start start 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba01037a0 0x7f2ba0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:08.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba0114510 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2ba4a0c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010db10 con 0x7f2ba01037a0 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2b9e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2b9e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55556/0 (socket says 172.21.15.103:55556) 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2b9e59c700 1 -- 172.21.15.103:0/80137461 learned_addr learned my addr 172.21.15.103:0/80137461 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.823+0000 7f2b9dd9b700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba01037a0 0x7f2ba0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b9e59c700 1 -- 172.21.15.103:0/80137461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba01037a0 msgr2=0x7f2ba0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b9e59c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba01037a0 0x7f2ba0113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b9e59c700 1 -- 172.21.15.103:0/80137461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90005040 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b9e59c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b94002a50 tx=0x7f2b940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b94007cc0 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba010ddf0 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b9400f040 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b9400da30 con 0x7f2ba0102e00 2024-03-22T02:25:08.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.824+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba010e380 con 0x7f2ba0102e00 2024-03-22T02:25:08.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.825+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba01052d0 con 0x7f2ba0102e00 2024-03-22T02:25:08.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.825+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2b9400dbb0 con 0x7f2ba0102e00 2024-03-22T02:25:08.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.826+0000 7f2b8effd700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 0x7f2b8806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:08.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.826+0000 7f2b9dd9b700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 0x7f2b8806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:08.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.826+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2b94086c00 con 0x7f2ba0102e00 2024-03-22T02:25:08.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.826+0000 7f2b9dd9b700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 0x7f2b8806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b90004890 tx=0x7f2b90009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:08.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:08.830+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2b9408c020 con 0x7f2ba0102e00 2024-03-22T02:25:09.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.023+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2ba010efd0 con 0x7f2ba0102e00 2024-03-22T02:25:09.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:08 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2974552351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T02:25:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.023+0000 7f2b8effd700 1 -- 172.21.15.103:0/80137461 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2b94056c80 con 0x7f2ba0102e00 2024-03-22T02:25:09.025 INFO:teuthology.orchestra.run.smithi103.stdout:146028888081 2024-03-22T02:25:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 msgr2=0x7f2b8806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 0x7f2b8806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b90004890 tx=0x7f2b90009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 msgr2=0x7f2ba0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2b94002a50 tx=0x7f2b940046b0 comp rx=0 tx=0).stop 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 shutdown_connections 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2b880692a0 0x7f2b8806b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ba0102e00 0x7f2ba0113980 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 --2- 172.21.15.103:0/80137461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ba01037a0 0x7f2ba0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 >> 172.21.15.103:0/80137461 conn(0x7f2ba00f89b0 msgr2=0x7f2ba01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 shutdown_connections 2024-03-22T02:25:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:09.026+0000 7f2ba4a0c700 1 -- 172.21.15.103:0/80137461 wait complete. 2024-03-22T02:25:09.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:08 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2974552351' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T02:25:09.181 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-03-22T02:25:09.181 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:09.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888081 for osd.4 2024-03-22T02:25:09.849 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:09.994 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:09 smithi103 ceph-mon[29258]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:09.995 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:09 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/80137461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T02:25:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:09 smithi122 ceph-mon[36403]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:09 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/80137461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T02:25:12.005 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:11 smithi103 ceph-mon[29258]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:12.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 -- 172.21.15.103:0/313993259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037b0 msgr2=0x7f4c4c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 --2- 172.21.15.103:0/313993259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c0fce20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4c38004940 tx=0x7f4c38009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 -- 172.21.15.103:0/313993259 shutdown_connections 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 --2- 172.21.15.103:0/313993259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c0fce20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 --2- 172.21.15.103:0/313993259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e10 0x7f4c4c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 -- 172.21.15.103:0/313993259 >> 172.21.15.103:0/313993259 conn(0x7f4c4c0f8980 msgr2=0x7f4c4c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 -- 172.21.15.103:0/313993259 shutdown_connections 2024-03-22T02:25:12.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.022+0000 7f4c531d8700 1 -- 172.21.15.103:0/313993259 wait complete. 2024-03-22T02:25:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 Processor -- start 2024-03-22T02:25:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 -- start start 2024-03-22T02:25:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c1140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:12.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c1147d0 con 0x7f4c4c102e10 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c531d8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c10db30 con 0x7f4c4c1037b0 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55578/0 (socket says 172.21.15.103:55578) 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 -- 172.21.15.103:0/1789531663 learned_addr learned my addr 172.21.15.103:0/1789531663 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c4bfff700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c1140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 -- 172.21.15.103:0/1789531663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037b0 msgr2=0x7f4c4c1140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c1140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.023+0000 7f4c50f74700 1 -- 172.21.15.103:0/1789531663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38005040 con 0x7f4c4c102e10 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c50f74700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000b790 tx=0x7f4c40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4000f770 con 0x7f4c4c102e10 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c10de10 con 0x7f4c4c102e10 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c4000fda0 con 0x7f4c4c102e10 2024-03-22T02:25:12.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4000d310 con 0x7f4c4c102e10 2024-03-22T02:25:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.024+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c10e3d0 con 0x7f4c4c102e10 2024-03-22T02:25:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.025+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c0fe560 con 0x7f4c4c102e10 2024-03-22T02:25:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.025+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4c40010420 con 0x7f4c4c102e10 2024-03-22T02:25:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.026+0000 7f4c497fa700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 0x7f4c3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.026+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4c40086ed0 con 0x7f4c4c102e10 2024-03-22T02:25:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.026+0000 7f4c4bfff700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 0x7f4c3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:12.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.026+0000 7f4c4bfff700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 0x7f4c3c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009fd0 tx=0x7f4c38009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:12.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.031+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4c40056f70 con 0x7f4c4c102e10 2024-03-22T02:25:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:11 smithi122 ceph-mon[36403]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:12.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.219+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4c4c04bed0 con 0x7f4c4c102e10 2024-03-22T02:25:12.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.220+0000 7f4c497fa700 1 -- 172.21.15.103:0/1789531663 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c40056b00 con 0x7f4c4c102e10 2024-03-22T02:25:12.222 INFO:teuthology.orchestra.run.smithi103.stdout:38654705703 2024-03-22T02:25:12.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 msgr2=0x7f4c3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 0x7f4c3c06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009fd0 tx=0x7f4c38009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 msgr2=0x7f4c4c113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000b790 tx=0x7f4c40009250 comp rx=0 tx=0).stop 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 shutdown_connections 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4c3c0692a0 0x7f4c3c06b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e10 0x7f4c4c113bb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 --2- 172.21.15.103:0/1789531663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037b0 0x7f4c4c1140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:12.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 >> 172.21.15.103:0/1789531663 conn(0x7f4c4c0f8980 msgr2=0x7f4c4c0f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:12.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 shutdown_connections 2024-03-22T02:25:12.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:12.222+0000 7f4c531d8700 1 -- 172.21.15.103:0/1789531663 wait complete. 2024-03-22T02:25:12.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:12 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1789531663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T02:25:12.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-03-22T02:25:12.953 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:12 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1789531663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T02:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 -- 172.21.15.103:0/469646189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 msgr2=0x7f18840fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 --2- 172.21.15.103:0/469646189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fe250 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1874004940 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 -- 172.21.15.103:0/469646189 shutdown_connections 2024-03-22T02:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 --2- 172.21.15.103:0/469646189 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f1884106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 --2- 172.21.15.103:0/469646189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fe250 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 -- 172.21.15.103:0/469646189 >> 172.21.15.103:0/469646189 conn(0x7f18840f7870 msgr2=0x7f18840f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 -- 172.21.15.103:0/469646189 shutdown_connections 2024-03-22T02:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.179+0000 7f18897a6700 1 -- 172.21.15.103:0/469646189 wait complete. 2024-03-22T02:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 Processor -- start 2024-03-22T02:25:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 -- start start 2024-03-22T02:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f18840fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840fdc90 con 0x7f18840fde80 2024-03-22T02:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18897a6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840fc730 con 0x7f18840fe820 2024-03-22T02:25:13.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55588/0 (socket says 172.21.15.103:55588) 2024-03-22T02:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 -- 172.21.15.103:0/3008541381 learned_addr learned my addr 172.21.15.103:0/3008541381 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18827fc700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f18840fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 -- 172.21.15.103:0/3008541381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 msgr2=0x7f18840fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:13.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f18840fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f1882ffd700 1 -- 172.21.15.103:0/3008541381 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874005040 con 0x7f18840fde80 2024-03-22T02:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.180+0000 7f18827fc700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f18840fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f1882ffd700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1874009380 tx=0x7f1874009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187400e040 con 0x7f18840fde80 2024-03-22T02:25:13.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1874009af0 con 0x7f18840fde80 2024-03-22T02:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18840fca10 con 0x7f18840fde80 2024-03-22T02:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f187401c630 con 0x7f18840fde80 2024-03-22T02:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.181+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188411a110 con 0x7f18840fde80 2024-03-22T02:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.182+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1884066b10 con 0x7f18840fde80 2024-03-22T02:25:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.182+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f187400bbc0 con 0x7f18840fde80 2024-03-22T02:25:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.183+0000 7f186b7fe700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 0x7f186c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.183+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1874088010 con 0x7f18840fde80 2024-03-22T02:25:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.183+0000 7f18827fc700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 0x7f186c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.183+0000 7f18827fc700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 0x7f186c06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.187+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1874054720 con 0x7f18840fde80 2024-03-22T02:25:13.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.381+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f18840fd190 con 0x7f18840fde80 2024-03-22T02:25:13.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.381+0000 7f186b7fe700 1 -- 172.21.15.103:0/3008541381 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1874057d40 con 0x7f18840fde80 2024-03-22T02:25:13.382 INFO:teuthology.orchestra.run.smithi103.stdout:111669149718 2024-03-22T02:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 msgr2=0x7f186c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 0x7f186c06b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878008040 comp rx=0 tx=0).stop 2024-03-22T02:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 msgr2=0x7f18840fbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1874009380 tx=0x7f1874009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:13.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 shutdown_connections 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f186c0692a0 0x7f186c06b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18840fde80 0x7f18840fbcb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 --2- 172.21.15.103:0/3008541381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18840fe820 0x7f18840fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 >> 172.21.15.103:0/3008541381 conn(0x7f18840f7870 msgr2=0x7f18840f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 shutdown_connections 2024-03-22T02:25:13.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:13.383+0000 7f18897a6700 1 -- 172.21.15.103:0/3008541381 wait complete. 2024-03-22T02:25:14.008 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:13 smithi103 ceph-mon[29258]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:14.009 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:13 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3008541381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T02:25:14.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-03-22T02:25:14.034 DEBUG:teuthology.parallel:result is None 2024-03-22T02:25:14.034 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-22T02:25:14.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T02:25:14.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:13 smithi122 ceph-mon[36403]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:14.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:13 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3008541381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T02:25:14.292 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:15.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.935+0000 7f08b098b700 1 -- 172.21.15.103:0/296458034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac102e00 msgr2=0x7f08ac1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:15.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.935+0000 7f08b098b700 1 --2- 172.21.15.103:0/296458034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac102e00 0x7f08ac1031d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f089c004940 tx=0x7f089c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:15.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- 172.21.15.103:0/296458034 shutdown_connections 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 --2- 172.21.15.103:0/296458034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac1037a0 0x7f08ac0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 --2- 172.21.15.103:0/296458034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac102e00 0x7f08ac1031d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- 172.21.15.103:0/296458034 >> 172.21.15.103:0/296458034 conn(0x7f08ac0f89b0 msgr2=0x7f08ac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- 172.21.15.103:0/296458034 shutdown_connections 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- 172.21.15.103:0/296458034 wait complete. 2024-03-22T02:25:15.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 Processor -- start 2024-03-22T02:25:15.938 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:15 smithi103 ceph-mon[29258]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- start start 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac102e00 0x7f08ac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac0722a0 con 0x7f08ac1037a0 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08b098b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac070d40 con 0x7f08ac102e00 2024-03-22T02:25:15.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08a9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08a9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36242/0 (socket says 172.21.15.103:36242) 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.936+0000 7f08a9d9b700 1 -- 172.21.15.103:0/1258896815 learned_addr learned my addr 172.21.15.103:0/1258896815 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08aa59c700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac102e00 0x7f08ac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08a9d9b700 1 -- 172.21.15.103:0/1258896815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac102e00 msgr2=0x7f08ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08a9d9b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac102e00 0x7f08ac0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08a9d9b700 1 -- 172.21.15.103:0/1258896815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c005040 con 0x7f08ac1037a0 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08a9d9b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08a000b790 tx=0x7f08a0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08a000f6f0 con 0x7f08ac1037a0 2024-03-22T02:25:15.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac071020 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08a000fd20 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08a00103a0 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.937+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac11a190 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.938+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac0fe550 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.939+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f08a0011400 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.939+0000 7f089affd700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 0x7f089406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.939+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f08a0086c10 con 0x7f08ac1037a0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.939+0000 7f08aa59c700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 0x7f089406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:15.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.939+0000 7f08aa59c700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 0x7f089406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f089c004f90 tx=0x7f089c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:15.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:15.943+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f08a0053280 con 0x7f08ac1037a0 2024-03-22T02:25:16.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:15 smithi122 ceph-mon[36403]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.123+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08ac109f70 con 0x7f08940692a0 2024-03-22T02:25:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.125+0000 7f089affd700 1 -- 172.21.15.103:0/1258896815 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28138 (secure 0 0 0) 0x7f08ac109f70 con 0x7f08940692a0 2024-03-22T02:25:16.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 msgr2=0x7f089406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 0x7f089406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f089c004f90 tx=0x7f089c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 msgr2=0x7f08ac070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f08a000b790 tx=0x7f08a0009220 comp rx=0 tx=0).stop 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 shutdown_connections 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f08940692a0 0x7f089406b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08ac102e00 0x7f08ac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 --2- 172.21.15.103:0/1258896815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08ac1037a0 0x7f08ac070800 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 >> 172.21.15.103:0/1258896815 conn(0x7f08ac0f89b0 msgr2=0x7f08ac0f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 shutdown_connections 2024-03-22T02:25:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:16.127+0000 7f08b098b700 1 -- 172.21.15.103:0/1258896815 wait complete. 2024-03-22T02:25:16.129 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-03-22T02:25:16.800 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-22T02:25:14.408746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002156"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-22T02:24:22.449495+0000","last_change":"2024-03-22T02:23:56.464047+0000","last_active":"2024-03-22T02:24:22.449495+0000","last_peered":"2024-03-22T02:24:22.449495+0000","last_clean":"2024-03-22T02:24:22.449495+0000","last_became_active":"2024-03-22T02:23:56.463699+0000","last_became_peered":"2024-03-22T02:23:56.463699+0000","last_unstale":"2024-03-22T02:24:22.449495+0000","last_undegraded":"2024-03-22T02:24:22.449495+0000","last_fullsized":"2024-03-22T02:24:22.449495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:23:07.366487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:23:07.366487+0000","last_clean_scrub_stamp":"2024-03-22T02:23:07.366487+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:52:44.264957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.501}]},{"osd":3,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.438},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Mar 22 02:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.32700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.376,"15min":0.375},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.27600000000000002}]},{"osd":1,"last update":"Fri Mar 22 02:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.33300000000000002}]},{"osd":3,"last update":"Fri Mar 22 02:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Mar 22 02:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 22 02:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.316,"15min":0.309},"min":{"1min":0.25,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.29299999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Mar 22 02:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.378,"15min":0.38100000000000001},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.38100000000000001,"15min":0.38500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29299999999999998}]},{"osd":3,"last update":"Fri Mar 22 02:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.68300000000000005}]},{"osd":4,"last update":"Fri Mar 22 02:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.628},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.219,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.21199999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.27300000000000002}]},{"osd":2,"last update":"Fri Mar 22 02:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45400000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.317},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.374,"15min":0.38},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46500000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Mar 22 02:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.67900000000000005}]},{"osd":4,"last update":"Fri Mar 22 02:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.61299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:25:16.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T02:25:17.058 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:17 smithi122 ceph-mon[36403]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:25:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:17 smithi122 ceph-mon[36403]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:18.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:17 smithi103 ceph-mon[29258]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:25:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:17 smithi103 ceph-mon[29258]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.731+0000 7f59e166d700 1 -- 172.21.15.103:0/1270093649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 msgr2=0x7f59dc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.731+0000 7f59e166d700 1 --2- 172.21.15.103:0/1270093649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc0fce10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004940 tx=0x7f59d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- 172.21.15.103:0/1270093649 shutdown_connections 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 --2- 172.21.15.103:0/1270093649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc0fce10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 --2- 172.21.15.103:0/1270093649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- 172.21.15.103:0/1270093649 >> 172.21.15.103:0/1270093649 conn(0x7f59dc0f89b0 msgr2=0x7f59dc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- 172.21.15.103:0/1270093649 shutdown_connections 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- 172.21.15.103:0/1270093649 wait complete. 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 Processor -- start 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- start start 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc1145b0 con 0x7f59dc1037a0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.732+0000 7f59e166d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc10db40 con 0x7f59dc102e00 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36254/0 (socket says 172.21.15.103:36254) 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 -- 172.21.15.103:0/675291750 learned_addr learned my addr 172.21.15.103:0/675291750 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59daffd700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 -- 172.21.15.103:0/675291750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 msgr2=0x7f59dc113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 -- 172.21.15.103:0/675291750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0005040 con 0x7f59dc1037a0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59daffd700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59da7fc700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009100 tx=0x7f59d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d000e040 con 0x7f59dc1037a0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59dc10ddc0 con 0x7f59dc1037a0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d0013420 con 0x7f59dc1037a0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d001c630 con 0x7f59dc1037a0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.733+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59dc10e380 con 0x7f59dc1037a0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.734+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59dc100110 con 0x7f59dc1037a0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.735+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f59d0013590 con 0x7f59dc1037a0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.735+0000 7f59c37fe700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 0x7f59c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.735+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f59d0088110 con 0x7f59dc1037a0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.735+0000 7f59daffd700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 0x7f59c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.735+0000 7f59daffd700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 0x7f59c406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009b10 tx=0x7f59cc008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.740+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f59d0054780 con 0x7f59dc1037a0 2024-03-22T02:25:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.920+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59dc109f50 con 0x7f59c40692a0 2024-03-22T02:25:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.922+0000 7f59c37fe700 1 -- 172.21.15.103:0/675291750 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28167 (secure 0 0 0) 0x7f59dc109f50 con 0x7f59c40692a0 2024-03-22T02:25:18.923 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 msgr2=0x7f59c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 0x7f59c406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009b10 tx=0x7f59cc008040 comp rx=0 tx=0).stop 2024-03-22T02:25:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 msgr2=0x7f59dc113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009100 tx=0x7f59d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 shutdown_connections 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f59c40692a0 0x7f59c406b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59dc102e00 0x7f59dc113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 --2- 172.21.15.103:0/675291750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1037a0 0x7f59dc113ed0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 >> 172.21.15.103:0/675291750 conn(0x7f59dc0f89b0 msgr2=0x7f59dc0fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 shutdown_connections 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:18.924+0000 7f59e166d700 1 -- 172.21.15.103:0/675291750 wait complete. 2024-03-22T02:25:18.926 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-03-22T02:25:19.658 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-03-22T02:25:18.409343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002069"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-22T02:24:22.449495+0000","last_change":"2024-03-22T02:23:56.464047+0000","last_active":"2024-03-22T02:24:22.449495+0000","last_peered":"2024-03-22T02:24:22.449495+0000","last_clean":"2024-03-22T02:24:22.449495+0000","last_became_active":"2024-03-22T02:23:56.463699+0000","last_became_peered":"2024-03-22T02:23:56.463699+0000","last_unstale":"2024-03-22T02:24:22.449495+0000","last_undegraded":"2024-03-22T02:24:22.449495+0000","last_fullsized":"2024-03-22T02:24:22.449495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:23:07.366487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:23:07.366487+0000","last_clean_scrub_stamp":"2024-03-22T02:23:07.366487+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:52:44.264957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42499999999999999}]},{"osd":1,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.497},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.442}]},{"osd":2,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Mar 22 02:24:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.45100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.747},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63}]},{"osd":1,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.78900000000000003}]},{"osd":2,"last update":"Fri Mar 22 02:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Mar 22 02:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38800000000000001,"15min":0.39400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.376,"15min":0.375},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.308}]},{"osd":1,"last update":"Fri Mar 22 02:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.27200000000000002}]},{"osd":3,"last update":"Fri Mar 22 02:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.61399999999999999}]},{"osd":4,"last update":"Fri Mar 22 02:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 22 02:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.316,"15min":0.309},"min":{"1min":0.25,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.29299999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Mar 22 02:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.378,"15min":0.38100000000000001},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.38100000000000001,"15min":0.38500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28399999999999997}]},{"osd":3,"last update":"Fri Mar 22 02:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Mar 22 02:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 02:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.219,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.21199999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Mar 22 02:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.38400000000000001,"15min":0.39200000000000002},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45400000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.374,"15min":0.38},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.46500000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Mar 22 02:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.495}]},{"osd":4,"last update":"Fri Mar 22 02:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:25:19.659 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-22T02:25:19.659 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-22T02:25:19.659 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-22T02:25:19.659 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-22T02:25:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:20.044 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:19 smithi103 ceph-mon[29258]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:20.044 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:19 smithi103 ceph-mon[29258]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:25:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:19 smithi122 ceph-mon[36403]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:19 smithi122 ceph-mon[36403]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:25:21.028 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:21.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 -- 172.21.15.103:0/2629574359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 msgr2=0x7f96900fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:21.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 --2- 172.21.15.103:0/2629574359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f96900fb990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9680004940 tx=0x7f9680009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:21.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 -- 172.21.15.103:0/2629574359 shutdown_connections 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 --2- 172.21.15.103:0/2629574359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f96900fb990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 --2- 172.21.15.103:0/2629574359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f96901033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 -- 172.21.15.103:0/2629574359 >> 172.21.15.103:0/2629574359 conn(0x7f9690071530 msgr2=0x7f9690071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 -- 172.21.15.103:0/2629574359 shutdown_connections 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.552+0000 7f9698596700 1 -- 172.21.15.103:0/2629574359 wait complete. 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 Processor -- start 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 -- start start 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f9690113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96901145b0 con 0x7f96900fb540 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9698596700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969010db40 con 0x7f9690102fd0 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36262/0 (socket says 172.21.15.103:36262) 2024-03-22T02:25:21.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 -- 172.21.15.103:0/503286504 learned_addr learned my addr 172.21.15.103:0/503286504 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9695b31700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f9690113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 -- 172.21.15.103:0/503286504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 msgr2=0x7f9690113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f9690113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 -- 172.21.15.103:0/503286504 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680005040 con 0x7f96900fb540 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9695b31700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f9690113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.553+0000 7f9696332700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f968c00b790 tx=0x7f968c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.554+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f968c00f770 con 0x7f96900fb540 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.554+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f968c00fda0 con 0x7f96900fb540 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.554+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969010ddf0 con 0x7f96900fb540 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.554+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f968c00d310 con 0x7f96900fb540 2024-03-22T02:25:21.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.554+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969010e330 con 0x7f96900fb540 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.555+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690100270 con 0x7f96900fb540 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.555+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f968c00f8d0 con 0x7f96900fb540 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.555+0000 7f9686ffd700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 0x7f967c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.555+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f968c086ec0 con 0x7f96900fb540 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.555+0000 7f9695b31700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 0x7f967c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:21.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.556+0000 7f9695b31700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 0x7f967c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:21.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.560+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f968c053530 con 0x7f96900fb540 2024-03-22T02:25:21.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.785+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f969010efe0 con 0x7f96900fb540 2024-03-22T02:25:21.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.785+0000 7f9686ffd700 1 -- 172.21.15.103:0/503286504 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f968c020250 con 0x7f96900fb540 2024-03-22T02:25:21.786 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:21.786 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-22T02:25:21.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 msgr2=0x7f967c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:21.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 0x7f967c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:21.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 msgr2=0x7f9690113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:21.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f968c00b790 tx=0x7f968c009250 comp rx=0 tx=0).stop 2024-03-22T02:25:21.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 shutdown_connections 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f967c0692a0 0x7f967c06b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96900fb540 0x7f9690113990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 --2- 172.21.15.103:0/503286504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9690102fd0 0x7f9690113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 >> 172.21.15.103:0/503286504 conn(0x7f9690071530 msgr2=0x7f96900fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 shutdown_connections 2024-03-22T02:25:21.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:21.787+0000 7f9698596700 1 -- 172.21.15.103:0/503286504 wait complete. 2024-03-22T02:25:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:21 smithi103 ceph-mon[29258]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:21 smithi122 ceph-mon[36403]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:22.596 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-22T02:25:22.596 INFO:tasks.cephadm:Setup complete, yielding 2024-03-22T02:25:22.597 INFO:teuthology.run_tasks:Running task print... 2024-03-22T02:25:22.606 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-22T02:25:22.606 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:22.626 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:22.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-22T02:25:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:22 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/503286504' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T02:25:22.888 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:22 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/503286504' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T02:25:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:23 smithi103 ceph-mon[29258]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:23 smithi122 ceph-mon[36403]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- 172.21.15.103:0/2814036741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 msgr2=0x7f293c0fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 --2- 172.21.15.103:0/2814036741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c0fc110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2934009ce0 tx=0x7f2934004d10 comp rx=0 tx=0).stop 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- 172.21.15.103:0/2814036741 shutdown_connections 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 --2- 172.21.15.103:0/2814036741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 0x7f293c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 --2- 172.21.15.103:0/2814036741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c0fc110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- 172.21.15.103:0/2814036741 >> 172.21.15.103:0/2814036741 conn(0x7f293c0f7440 msgr2=0x7f293c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- 172.21.15.103:0/2814036741 shutdown_connections 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- 172.21.15.103:0/2814036741 wait complete. 2024-03-22T02:25:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 Processor -- start 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.577+0000 7f2942bca700 1 -- start start 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 0x7f293c1128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c112de0 con 0x7f293c0fbd00 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c112f20 con 0x7f293c0fcfb0 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42772/0 (socket says 172.21.15.103:42772) 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 -- 172.21.15.103:0/2330342544 learned_addr learned my addr 172.21.15.103:0/2330342544 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f293bfff700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 0x7f293c1128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 -- 172.21.15.103:0/2330342544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 msgr2=0x7f293c1128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 0x7f293c1128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 -- 172.21.15.103:0/2330342544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934009940 con 0x7f293c0fbd00 2024-03-22T02:25:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2940966700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2934009380 tx=0x7f2934004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293400e040 con 0x7f293c0fbd00 2024-03-22T02:25:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c113120 con 0x7f293c0fbd00 2024-03-22T02:25:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29340128e0 con 0x7f293c0fbd00 2024-03-22T02:25:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293401c930 con 0x7f293c0fbd00 2024-03-22T02:25:24.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.578+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c1136e0 con 0x7f293c0fbd00 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.579+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c04c710 con 0x7f293c0fbd00 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.580+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2934012a50 con 0x7f293c0fbd00 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.580+0000 7f29397fa700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 0x7f292806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.581+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29340889e0 con 0x7f293c0fbd00 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.581+0000 7f293bfff700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 0x7f292806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:24.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.581+0000 7f293bfff700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 0x7f292806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2924004f40 tx=0x7f292400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:24.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.585+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2934058a60 con 0x7f293c0fbd00 2024-03-22T02:25:24.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.783+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f293c1047f0 con 0x7f293c0fbd00 2024-03-22T02:25:24.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.787+0000 7f29397fa700 1 -- 172.21.15.103:0/2330342544 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2934012d00 con 0x7f293c0fbd00 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.789+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 msgr2=0x7f292806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.789+0000 7f2942bca700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 0x7f292806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2924004f40 tx=0x7f292400b040 comp rx=0 tx=0).stop 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.789+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 msgr2=0x7f293c066c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.789+0000 7f2942bca700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2934009380 tx=0x7f2934004d10 comp rx=0 tx=0).stop 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 shutdown_connections 2024-03-22T02:25:24.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f29280692a0 0x7f292806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f293c0fbd00 0x7f293c066c70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 --2- 172.21.15.103:0/2330342544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c0fcfb0 0x7f293c1128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 >> 172.21.15.103:0/2330342544 conn(0x7f293c0f7440 msgr2=0x7f293c0f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 shutdown_connections 2024-03-22T02:25:24.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:24.790+0000 7f2942bca700 1 -- 172.21.15.103:0/2330342544 wait complete. 2024-03-22T02:25:24.905 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:24 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:25.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:24 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:25.569 INFO:teuthology.run_tasks:Running task print... 2024-03-22T02:25:25.579 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-22T02:25:25.579 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:25.599 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:25.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-22T02:25:25.871 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:25.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2330342544' entity='client.admin' 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:25 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2330342544' entity='client.admin' 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:25 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 -- 172.21.15.103:0/903921623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 msgr2=0x7f49340fb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 --2- 172.21.15.103:0/903921623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f49340fb890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4924004a50 tx=0x7f492400af40 comp rx=0 tx=0).stop 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 -- 172.21.15.103:0/903921623 shutdown_connections 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 --2- 172.21.15.103:0/903921623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 0x7f49340fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 --2- 172.21.15.103:0/903921623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f49340fb890 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 -- 172.21.15.103:0/903921623 >> 172.21.15.103:0/903921623 conn(0x7f49340f6be0 msgr2=0x7f49340f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:27.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 -- 172.21.15.103:0/903921623 shutdown_connections 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 -- 172.21.15.103:0/903921623 wait complete. 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.520+0000 7f493a9cd700 1 Processor -- start 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 -- start start 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 0x7f493410dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493410e4c0 con 0x7f49340fb480 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493410e630 con 0x7f49340fc730 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42798/0 (socket says 172.21.15.103:42798) 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 -- 172.21.15.103:0/3425584200 learned_addr learned my addr 172.21.15.103:0/3425584200 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f49337fe700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 0x7f493410dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 -- 172.21.15.103:0/3425584200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 msgr2=0x7f493410dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:27.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 0x7f493410dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 -- 172.21.15.103:0/3425584200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924005040 con 0x7f49340fb480 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4933fff700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f492400c3b0 tx=0x7f492400af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f492400f050 con 0x7f49340fb480 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f492401d950 con 0x7f49340fb480 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4934113030 con 0x7f49340fb480 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.521+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49240148e0 con 0x7f49340fb480 2024-03-22T02:25:27.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.522+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49341134b0 con 0x7f49340fb480 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.522+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493404c710 con 0x7f49340fb480 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.523+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f492401dac0 con 0x7f49340fb480 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.523+0000 7f4930ff9700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 0x7f491c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.523+0000 7f49337fe700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 0x7f491c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.523+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4924088b60 con 0x7f49340fb480 2024-03-22T02:25:27.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.524+0000 7f49337fe700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 0x7f491c06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:27.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.528+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4924058c60 con 0x7f49340fb480 2024-03-22T02:25:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.726+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4934100cf0 con 0x7f491c0692a0 2024-03-22T02:25:27.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.727+0000 7f4930ff9700 1 -- 172.21.15.103:0/3425584200 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4934100cf0 con 0x7f491c0692a0 2024-03-22T02:25:27.727 INFO:teuthology.orchestra.run.smithi103.stdout:Backend: cephadm 2024-03-22T02:25:27.727 INFO:teuthology.orchestra.run.smithi103.stdout:Available: Yes 2024-03-22T02:25:27.727 INFO:teuthology.orchestra.run.smithi103.stdout:Paused: No 2024-03-22T02:25:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 msgr2=0x7f491c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 0x7f491c06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4928007fd0 tx=0x7f4928009040 comp rx=0 tx=0).stop 2024-03-22T02:25:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 msgr2=0x7f493410d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f492400c3b0 tx=0x7f492400af40 comp rx=0 tx=0).stop 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 shutdown_connections 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f491c0692a0 0x7f491c06b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49340fb480 0x7f493410d980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 --2- 172.21.15.103:0/3425584200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49340fc730 0x7f493410dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 >> 172.21.15.103:0/3425584200 conn(0x7f49340f6be0 msgr2=0x7f49340f81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 shutdown_connections 2024-03-22T02:25:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:27.729+0000 7f493a9cd700 1 -- 172.21.15.103:0/3425584200 wait complete. 2024-03-22T02:25:28.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:27 smithi103 ceph-mon[29258]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:27 smithi122 ceph-mon[36403]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:28.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-22T02:25:28.596 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:28.969 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:28 smithi103 ceph-mon[29258]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:29.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:28 smithi122 ceph-mon[36403]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:29.984 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:29 smithi103 ceph-mon[29258]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:29 smithi122 ceph-mon[36403]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 -- 172.21.15.103:0/1745240985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 msgr2=0x7f3d500fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 --2- 172.21.15.103:0/1745240985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d500fd280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c004940 tx=0x7f3d4c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 -- 172.21.15.103:0/1745240985 shutdown_connections 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 --2- 172.21.15.103:0/1745240985 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d500fdfe0 0x7f3d500fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 --2- 172.21.15.103:0/1745240985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d500fd280 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 -- 172.21.15.103:0/1745240985 >> 172.21.15.103:0/1745240985 conn(0x7f3d500f8550 msgr2=0x7f3d500fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 -- 172.21.15.103:0/1745240985 shutdown_connections 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.206+0000 7f3d57540700 1 -- 172.21.15.103:0/1745240985 wait complete. 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 Processor -- start 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 -- start start 2024-03-22T02:25:30.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d500fdfe0 0x7f3d5010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5010e490 con 0x7f3d500fce70 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d57540700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5010e600 con 0x7f3d500fdfe0 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d552dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d552dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42808/0 (socket says 172.21.15.103:42808) 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.207+0000 7f3d552dc700 1 -- 172.21.15.103:0/2626048090 learned_addr learned my addr 172.21.15.103:0/2626048090 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d552dc700 1 -- 172.21.15.103:0/2626048090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d500fdfe0 msgr2=0x7f3d5010de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d552dc700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d500fdfe0 0x7f3d5010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d552dc700 1 -- 172.21.15.103:0/2626048090 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c005040 con 0x7f3d500fce70 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d552dc700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00b3b0 tx=0x7f3d4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c00e040 con 0x7f3d500fce70 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d4c00bcc0 con 0x7f3d500fce70 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d50113000 con 0x7f3d500fce70 2024-03-22T02:25:30.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c01c630 con 0x7f3d500fce70 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.208+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d501135c0 con 0x7f3d500fce70 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.209+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5004c710 con 0x7f3d500fce70 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.210+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3d4c013980 con 0x7f3d500fce70 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.210+0000 7f3d45ffb700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 0x7f3d3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.210+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3d4c087bc0 con 0x7f3d500fce70 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.210+0000 7f3d54adb700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 0x7f3d3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:30.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.210+0000 7f3d54adb700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 0x7f3d3c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d40004d00 tx=0x7f3d40009ce0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:30.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.214+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3d4c057cc0 con 0x7f3d500fce70 2024-03-22T02:25:30.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.409+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d50102340 con 0x7f3d3c0692a0 2024-03-22T02:25:30.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.417+0000 7f3d45ffb700 1 -- 172.21.15.103:0/2626048090 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7f3d50102340 con 0x7f3d3c0692a0 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 31b45ef21cbd 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (5m) 2m ago 5m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (4m) 69s ago 4m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (3m) 2m ago 4m 78.4M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (6m) 2m ago 6m 473M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (4m) 69s ago 4m 432M - 17.2.7-726-g9c4e0c34 837124da804f 29f8988d41c7 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (6m) 2m ago 6m 45.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (4m) 69s ago 4m 38.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:25:30.418 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (5m) 2m ago 5m 15.9M - 1.5.0 0da6a335fe13 136d30f21984 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 69s ago 4m 16.0M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (3m) 2m ago 3m 38.6M 4288M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (2m) 2m ago 2m 38.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (2m) 2m ago 2m 41.2M 4288M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (2m) 69s ago 2m 39.1M 5312M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (100s) 69s ago 100s 39.3M 5312M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (74s) 69s ago 74s 10.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:25:30.419 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (3m) 2m ago 4m 37.6M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 msgr2=0x7f3d3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 0x7f3d3c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d40004d00 tx=0x7f3d40009ce0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 msgr2=0x7f3d5010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00b3b0 tx=0x7f3d4c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 shutdown_connections 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f3d3c0692a0 0x7f3d3c06b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d500fce70 0x7f3d5010d950 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 --2- 172.21.15.103:0/2626048090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d500fdfe0 0x7f3d5010de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 >> 172.21.15.103:0/2626048090 conn(0x7f3d500f8550 msgr2=0x7f3d500f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 shutdown_connections 2024-03-22T02:25:30.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:30.419+0000 7f3d57540700 1 -- 172.21.15.103:0/2626048090 wait complete. 2024-03-22T02:25:31.172 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-22T02:25:31.428 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:31.809 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:31 smithi103 ceph-mon[29258]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:31.809 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:31 smithi103 ceph-mon[29258]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:31 smithi122 ceph-mon[36403]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:31 smithi122 ceph-mon[36403]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:33.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/1923337434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0071e20 msgr2=0x7f5ef0070330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/1923337434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0071e20 0x7f5ef0070330 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0004940 tx=0x7f5ee0009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:33.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/1923337434 shutdown_connections 2024-03-22T02:25:33.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/1923337434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0070870 0x7f5ef0070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/1923337434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0071e20 0x7f5ef0070330 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/1923337434 >> 172.21.15.103:0/1923337434 conn(0x7f5ef00f8550 msgr2=0x7f5ef00fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/1923337434 shutdown_connections 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/1923337434 wait complete. 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 Processor -- start 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.166+0000 7f5ef4e3a700 1 -- start start 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 0x7f5ef01100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01106d0 con 0x7f5ef0070870 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0110840 con 0x7f5ef0071e20 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42822/0 (socket says 172.21.15.103:42822) 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 -- 172.21.15.103:0/372070058 learned_addr learned my addr 172.21.15.103:0/372070058 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eedd9b700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 0x7f5ef01100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.169 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 -- 172.21.15.103:0/372070058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 msgr2=0x7f5ef01100d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 0x7f5ef01100d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 -- 172.21.15.103:0/372070058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0005040 con 0x7f5ef0070870 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eedd9b700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 0x7f5ef01100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5eee59c700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000b970 tx=0x7f5ee0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee000e040 con 0x7f5ef0070870 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef0115240 con 0x7f5ef0070870 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef0115800 con 0x7f5ef0070870 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ee000bbf0 con 0x7f5ef0070870 2024-03-22T02:25:33.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.167+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee001c630 con 0x7f5ef0070870 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.168+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef004c710 con 0x7f5ef0070870 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.169+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5ee000bd60 con 0x7f5ef0070870 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.169+0000 7f5edeffd700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 0x7f5ed806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.169+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5ee0088140 con 0x7f5ef0070870 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.169+0000 7f5eedd9b700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 0x7f5ed806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.169+0000 7f5eedd9b700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 0x7f5ed806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40041d0 tx=0x7f5ee4009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.174+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5ee0058390 con 0x7f5ef0070870 2024-03-22T02:25:33.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.367+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef0103f10 con 0x7f5ed80692a0 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.371+0000 7f5edeffd700 1 -- 172.21.15.103:0/372070058 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f5ef0103f10 con 0x7f5ed80692a0 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:crash 2/2 2m ago 6m * 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:mgr 2/2 2m ago 6m count:2 2024-03-22T02:25:33.372 INFO:teuthology.orchestra.run.smithi103.stdout:mon 2/2 2m ago 5m smithi103:172.21.15.103=smithi103;smithi122:172.21.15.122=smithi122;count:2 2024-03-22T02:25:33.373 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-03-22T02:25:33.373 INFO:teuthology.orchestra.run.smithi103.stdout:osd 6 2m ago - 2024-03-22T02:25:33.373 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-22T02:25:33.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.373+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 msgr2=0x7f5ed806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.373+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 0x7f5ed806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40041d0 tx=0x7f5ee4009040 comp rx=0 tx=0).stop 2024-03-22T02:25:33.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.373+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 msgr2=0x7f5ef010fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.373+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000b970 tx=0x7f5ee0009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 shutdown_connections 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f5ed80692a0 0x7f5ed806b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5ef0070870 0x7f5ef010fb90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 --2- 172.21.15.103:0/372070058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ef0071e20 0x7f5ef01100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 >> 172.21.15.103:0/372070058 conn(0x7f5ef00f8550 msgr2=0x7f5ef00f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 shutdown_connections 2024-03-22T02:25:33.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:33.374+0000 7f5ef4e3a700 1 -- 172.21.15.103:0/372070058 wait complete. 2024-03-22T02:25:34.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-22T02:25:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:33 smithi122 ceph-mon[36403]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:34.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:33 smithi103 ceph-mon[29258]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:34.276 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:35.061 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:34 smithi103 ceph-mon[29258]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:35.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:34 smithi122 ceph-mon[36403]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 -- 172.21.15.103:0/4061218378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 msgr2=0x7f660c0fbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 --2- 172.21.15.103:0/4061218378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c0fbf60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6600004940 tx=0x7f6600009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 -- 172.21.15.103:0/4061218378 shutdown_connections 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 --2- 172.21.15.103:0/4061218378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 0x7f660c0fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 --2- 172.21.15.103:0/4061218378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c0fbf60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.891+0000 7f66119a1700 1 -- 172.21.15.103:0/4061218378 >> 172.21.15.103:0/4061218378 conn(0x7f660c0f72d0 msgr2=0x7f660c0f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 -- 172.21.15.103:0/4061218378 shutdown_connections 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 -- 172.21.15.103:0/4061218378 wait complete. 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 Processor -- start 2024-03-22T02:25:35.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 -- start start 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 0x7f660c112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c066e30 con 0x7f660c0fbb50 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f66119a1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c066f70 con 0x7f660c0fce00 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f661099f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f661099f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46800/0 (socket says 172.21.15.103:46800) 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.892+0000 7f661099f700 1 -- 172.21.15.103:0/819015811 learned_addr learned my addr 172.21.15.103:0/819015811 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f661099f700 1 -- 172.21.15.103:0/819015811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 msgr2=0x7f660c112ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f660bfff700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 0x7f660c112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f661099f700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 0x7f660c112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f661099f700 1 -- 172.21.15.103:0/819015811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600005040 con 0x7f660c0fbb50 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f661099f700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6600009150 tx=0x7f6600009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660000e040 con 0x7f660c0fbb50 2024-03-22T02:25:35.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f660000bbf0 con 0x7f660c0fbb50 2024-03-22T02:25:35.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660001c630 con 0x7f660c0fbb50 2024-03-22T02:25:35.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660c1130b0 con 0x7f660c0fbb50 2024-03-22T02:25:35.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.893+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660c113640 con 0x7f660c0fbb50 2024-03-22T02:25:35.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.894+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660c0faf50 con 0x7f660c0fbb50 2024-03-22T02:25:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.894+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f660000bd60 con 0x7f660c0fbb50 2024-03-22T02:25:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.895+0000 7f66097fa700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 0x7f65fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.895+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6600088180 con 0x7f660c0fbb50 2024-03-22T02:25:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.895+0000 7f660bfff700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 0x7f65fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:35.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.895+0000 7f660bfff700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 0x7f65fc06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f65f8006fd0 tx=0x7f65f8008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:35.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:35.899+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6600054890 con 0x7f660c0fbb50 2024-03-22T02:25:36.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.089+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f660c1010a0 con 0x7f65fc0692a0 2024-03-22T02:25:36.090 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:35 smithi103 ceph-mon[29258]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:36.090 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:36.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.091+0000 7f66097fa700 1 -- 172.21.15.103:0/819015811 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f660c1010a0 con 0x7f65fc0692a0 2024-03-22T02:25:36.091 INFO:teuthology.orchestra.run.smithi103.stdout:HOST ADDR LABELS STATUS 2024-03-22T02:25:36.091 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 172.21.15.103 2024-03-22T02:25:36.091 INFO:teuthology.orchestra.run.smithi103.stdout:smithi122 172.21.15.122 2024-03-22T02:25:36.091 INFO:teuthology.orchestra.run.smithi103.stdout:2 hosts in cluster 2024-03-22T02:25:36.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 msgr2=0x7f65fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:36.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 0x7f65fc06b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f65f8006fd0 tx=0x7f65f8008040 comp rx=0 tx=0).stop 2024-03-22T02:25:36.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 msgr2=0x7f660c110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:36.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6600009150 tx=0x7f6600009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:36.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 shutdown_connections 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f65fc0692a0 0x7f65fc06b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fbb50 0x7f660c110560 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 --2- 172.21.15.103:0/819015811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fce00 0x7f660c112ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 >> 172.21.15.103:0/819015811 conn(0x7f660c0f72d0 msgr2=0x7f660c0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 shutdown_connections 2024-03-22T02:25:36.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:36.094+0000 7f66119a1700 1 -- 172.21.15.103:0/819015811 wait complete. 2024-03-22T02:25:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:35 smithi122 ceph-mon[36403]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:36.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-22T02:25:36.993 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:38.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:37 smithi103 ceph-mon[29258]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:38.093 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:37 smithi103 ceph-mon[29258]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:38.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:37 smithi122 ceph-mon[36403]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:38.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:37 smithi122 ceph-mon[36403]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:38.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 -- 172.21.15.103:0/1600669829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fbcd0 msgr2=0x7fa4840fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 --2- 172.21.15.103:0/1600669829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fbcd0 0x7fa4840fc0e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa474004940 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 -- 172.21.15.103:0/1600669829 shutdown_connections 2024-03-22T02:25:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 --2- 172.21.15.103:0/1600669829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fcf80 0x7fa4840fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 --2- 172.21.15.103:0/1600669829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fbcd0 0x7fa4840fc0e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 -- 172.21.15.103:0/1600669829 >> 172.21.15.103:0/1600669829 conn(0x7fa4840f7430 msgr2=0x7fa4840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 -- 172.21.15.103:0/1600669829 shutdown_connections 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.577+0000 7fa48935c700 1 -- 172.21.15.103:0/1600669829 wait complete. 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 Processor -- start 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 -- start start 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fbcd0 0x7fa4840ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:38.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa484100250 con 0x7fa4840fcf80 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa48935c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4840fe830 con 0x7fa4840fbcd0 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa4827fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa4827fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46816/0 (socket says 172.21.15.103:46816) 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.578+0000 7fa4827fc700 1 -- 172.21.15.103:0/1149620823 learned_addr learned my addr 172.21.15.103:0/1149620823 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa482ffd700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fbcd0 0x7fa4840ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa4827fc700 1 -- 172.21.15.103:0/1149620823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fbcd0 msgr2=0x7fa4840ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:38.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa4827fc700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fbcd0 0x7fa4840ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa4827fc700 1 -- 172.21.15.103:0/1149620823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474005040 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa4827fc700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa47800b790 tx=0x7fa478009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa47800f6f0 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa47800fd20 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4840feb10 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4780103a0 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.579+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4840ff0d0 con 0x7fa4840fcf80 2024-03-22T02:25:38.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.580+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48404c710 con 0x7fa4840fcf80 2024-03-22T02:25:38.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.580+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa478011400 con 0x7fa4840fcf80 2024-03-22T02:25:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.581+0000 7fa46b7fe700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 0x7fa46c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.581+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa478086c30 con 0x7fa4840fcf80 2024-03-22T02:25:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.581+0000 7fa482ffd700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 0x7fa46c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:38.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.581+0000 7fa482ffd700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 0x7fa46c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3b0 tx=0x7fa474009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:38.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.585+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa478056d30 con 0x7fa4840fcf80 2024-03-22T02:25:38.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.781+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa48402b310 con 0x7fa46c0692a0 2024-03-22T02:25:38.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.782+0000 7fa46b7fe700 1 -- 172.21.15.103:0/1149620823 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fa48402b310 con 0x7fa46c0692a0 2024-03-22T02:25:38.783 INFO:teuthology.orchestra.run.smithi103.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-22T02:25:38.783 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-22T02:25:38.783 INFO:teuthology.orchestra.run.smithi103.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 372G No 74s ago Has a FileSystem, LVM detected 2024-03-22T02:25:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.784+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 msgr2=0x7fa46c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.784+0000 7fa48935c700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 0x7fa46c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3b0 tx=0x7fa474009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.784+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 msgr2=0x7fa4840fe2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.784+0000 7fa48935c700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa47800b790 tx=0x7fa478009250 comp rx=0 tx=0).stop 2024-03-22T02:25:38.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 shutdown_connections 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa46c0692a0 0x7fa46c06b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4840fbcd0 0x7fa4840ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 --2- 172.21.15.103:0/1149620823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4840fcf80 0x7fa4840fe2c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 >> 172.21.15.103:0/1149620823 conn(0x7fa4840f7430 msgr2=0x7fa4840667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 shutdown_connections 2024-03-22T02:25:38.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:38.785+0000 7fa48935c700 1 -- 172.21.15.103:0/1149620823 wait complete. 2024-03-22T02:25:39.399 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:39.409 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:39.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-22T02:25:39.664 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:39 smithi122 ceph-mon[36403]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:39 smithi122 ceph-mon[36403]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:39 smithi103 ceph-mon[29258]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:39 smithi103 ceph-mon[29258]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 -- 172.21.15.103:0/1368233879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3238066220 msgr2=0x7f32380ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 --2- 172.21.15.103:0/1368233879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3238066220 0x7f32380ffa80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3228004940 tx=0x7f3228009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 -- 172.21.15.103:0/1368233879 shutdown_connections 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 --2- 172.21.15.103:0/1368233879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32380fffc0 0x7f32381023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 --2- 172.21.15.103:0/1368233879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3238066220 0x7f32380ffa80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:41.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 -- 172.21.15.103:0/1368233879 >> 172.21.15.103:0/1368233879 conn(0x7f32380f7430 msgr2=0x7f32380f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 -- 172.21.15.103:0/1368233879 shutdown_connections 2024-03-22T02:25:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.302+0000 7f323e0d2700 1 -- 172.21.15.103:0/1368233879 wait complete. 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.303+0000 7f323e0d2700 1 Processor -- start 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.303+0000 7f323e0d2700 1 -- start start 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.303+0000 7f323e0d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3238066220 0x7f323810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f323e0d2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f323e0d2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323810e650 con 0x7f32380fffc0 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f323e0d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323810ee40 con 0x7f3238066220 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46838/0 (socket says 172.21.15.103:46838) 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 -- 172.21.15.103:0/2532972453 learned_addr learned my addr 172.21.15.103:0/2532972453 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f32377fe700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3238066220 0x7f323810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 -- 172.21.15.103:0/2532972453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3238066220 msgr2=0x7f323810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3238066220 0x7f323810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:41.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 -- 172.21.15.103:0/2532972453 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228005040 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f3236ffd700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f322c00b790 tx=0x7f322c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c00f6f0 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f322c00fd20 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323810f120 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c0103a0 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.304+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323810f6e0 con 0x7f32380fffc0 2024-03-22T02:25:41.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.305+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323805f5f0 con 0x7f32380fffc0 2024-03-22T02:25:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.306+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f322c011400 con 0x7f32380fffc0 2024-03-22T02:25:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.306+0000 7f321ffff700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 0x7f322006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.306+0000 7f32377fe700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 0x7f322006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.306+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f322c086c20 con 0x7f32380fffc0 2024-03-22T02:25:41.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.307+0000 7f32377fe700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 0x7f322006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3228004890 tx=0x7f3228009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:41.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.311+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f322c056d20 con 0x7f32380fffc0 2024-03-22T02:25:41.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:41.528+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f323810f9d0 con 0x7f32200692a0 2024-03-22T02:25:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:41 smithi122 ceph-mon[36403]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:41 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-22T02:25:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:41 smithi103 ceph-mon[29258]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:41 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-22T02:25:42.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.859+0000 7f321ffff700 1 -- 172.21.15.103:0/2532972453 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f323810f9d0 con 0x7f32200692a0 2024-03-22T02:25:42.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 msgr2=0x7f322006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:42.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 0x7f322006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3228004890 tx=0x7f3228009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:42.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 msgr2=0x7f323810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f322c00b790 tx=0x7f322c009220 comp rx=0 tx=0).stop 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 shutdown_connections 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f32200692a0 0x7f322006b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3238066220 0x7f323810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 --2- 172.21.15.103:0/2532972453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32380fffc0 0x7f323810dfc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 >> 172.21.15.103:0/2532972453 conn(0x7f32380f7430 msgr2=0x7f32380f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 shutdown_connections 2024-03-22T02:25:42.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:42.862+0000 7f323e0d2700 1 -- 172.21.15.103:0/2532972453 wait complete. 2024-03-22T02:25:43.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:42 smithi122 ceph-mon[36403]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:43.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:42 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-22T02:25:43.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:42 smithi122 ceph-mon[36403]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T02:25:43.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:42 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-22T02:25:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:42 smithi103 ceph-mon[29258]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:25:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:42 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-22T02:25:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:42 smithi103 ceph-mon[29258]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T02:25:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:42 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-22T02:25:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:42 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[29254]: 2024-03-22T02:25:42.836+0000 7f0572604700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:25:43.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: pgmap v175: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-22T02:25:43.868 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: fsmap cephfs:0 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: Saving service mds.cephfs spec with placement count:4 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:43.869 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:43 smithi103 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:25:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: pgmap v175: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: fsmap cephfs:0 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: Saving service mds.cephfs spec with placement count:4 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:44.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:44.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:44.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:44.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:44.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:43 smithi122 ceph-mon[36403]: Deploying daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:25:44.827 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:44 smithi103 ceph-mon[29258]: osdmap e44: 6 total, 6 up, 6 in 2024-03-22T02:25:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:44 smithi122 ceph-mon[36403]: osdmap e44: 6 total, 6 up, 6 in 2024-03-22T02:25:46.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:45 smithi103 ceph-mon[29258]: pgmap v179: 65 pgs: 13 creating+peering, 1 active+clean, 51 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:46.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:45 smithi103 ceph-mon[29258]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:25:46.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:45 smithi103 ceph-mon[29258]: osdmap e45: 6 total, 6 up, 6 in 2024-03-22T02:25:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:45 smithi122 ceph-mon[36403]: pgmap v179: 65 pgs: 13 creating+peering, 1 active+clean, 51 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:45 smithi122 ceph-mon[36403]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:25:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:45 smithi122 ceph-mon[36403]: osdmap e45: 6 total, 6 up, 6 in 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 -- 172.21.15.103:0/2738523640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 msgr2=0x7fb4d80fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 --2- 172.21.15.103:0/2738523640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d80fd3f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc004940 tx=0x7fb4cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 -- 172.21.15.103:0/2738523640 shutdown_connections 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 --2- 172.21.15.103:0/2738523640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d80fd3f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 --2- 172.21.15.103:0/2738523640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 0x7fb4d80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 -- 172.21.15.103:0/2738523640 >> 172.21.15.103:0/2738523640 conn(0x7fb4d80f7430 msgr2=0x7fb4d80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 -- 172.21.15.103:0/2738523640 shutdown_connections 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.646+0000 7fb4de411700 1 -- 172.21.15.103:0/2738523640 wait complete. 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 Processor -- start 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 -- start start 2024-03-22T02:25:46.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 0x7fb4d810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d810e470 con 0x7fb4d80fcf80 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4de411700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d810e5e0 con 0x7fb4d80fbcd0 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52110/0 (socket says 172.21.15.103:52110) 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 -- 172.21.15.103:0/62828160 learned_addr learned my addr 172.21.15.103:0/62828160 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d7fff700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 0x7fb4d810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 -- 172.21.15.103:0/62828160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 msgr2=0x7fb4d810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 0x7fb4d810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.647+0000 7fb4d77fe700 1 -- 172.21.15.103:0/62828160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc005040 con 0x7fb4d80fcf80 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4d77fe700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00b3b0 tx=0x7fb4cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4cc00e040 con 0x7fb4d80fcf80 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4cc00bbf0 con 0x7fb4d80fcf80 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d8112fe0 con 0x7fb4d80fcf80 2024-03-22T02:25:46.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4cc01c630 con 0x7fb4d80fcf80 2024-03-22T02:25:46.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.648+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d8113570 con 0x7fb4d80fcf80 2024-03-22T02:25:46.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.649+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d8101b10 con 0x7fb4d80fcf80 2024-03-22T02:25:46.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.649+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb4cc013420 con 0x7fb4d80fcf80 2024-03-22T02:25:46.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.650+0000 7fb4d4ff9700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 0x7fb4c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.650+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb4cc087e60 con 0x7fb4d80fcf80 2024-03-22T02:25:46.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.650+0000 7fb4d7fff700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 0x7fb4c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.650+0000 7fb4d7fff700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 0x7fb4c006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8004f40 tx=0x7fb4c800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.654+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb4cc057c20 con 0x7fb4d80fcf80 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.908+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb4d804c710 con 0x7fb4d80fcf80 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.908+0000 7fb4d4ff9700 1 -- 172.21.15.103:0/62828160 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1249 (secure 0 0 0) 0x7fb4cc0577b0 con 0x7fb4d80fcf80 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:e4 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:46.910 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 4 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:25:46.863629+0000 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:25:46.911 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 0 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 1 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:in 0 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14460} 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 0 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{0:14460} state up:creating seq 1 addr [v2:172.21.15.103:6826/3786434277,v1:172.21.15.103:6827/3786434277] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:25:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 msgr2=0x7fb4c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 0x7fb4c006b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8004f40 tx=0x7fb4c800b040 comp rx=0 tx=0).stop 2024-03-22T02:25:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 msgr2=0x7fb4d810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00b3b0 tx=0x7fb4cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 shutdown_connections 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fb4c00692a0 0x7fb4c006b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4d80fbcd0 0x7fb4d810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 --2- 172.21.15.103:0/62828160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4d80fcf80 0x7fb4d810de70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.910+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 >> 172.21.15.103:0/62828160 conn(0x7fb4d80f7430 msgr2=0x7fb4d80667f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.911+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 shutdown_connections 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:46.911+0000 7fb4de411700 1 -- 172.21.15.103:0/62828160 wait complete. 2024-03-22T02:25:46.914 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 4 2024-03-22T02:25:47.553 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:47.563 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:47.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-22T02:25:47.822 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: pgmap v181: 65 pgs: 13 creating+peering, 28 active+clean, 24 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6826/3786434277,v1:172.21.15.103:6827/3786434277] up:boot 2024-03-22T02:25:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: daemon mds.cephfs.smithi103.gtuatp assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-22T02:25:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:25:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: Cluster is now healthy 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: fsmap cephfs:0 1 up:standby 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:creating} 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: daemon mds.cephfs.smithi103.gtuatp is now active in filesystem cephfs as rank 0 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/62828160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:25:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:47 smithi122 ceph-mon[36403]: Deploying daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:25:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: pgmap v181: 65 pgs: 13 creating+peering, 28 active+clean, 24 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:25:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6826/3786434277,v1:172.21.15.103:6827/3786434277] up:boot 2024-03-22T02:25:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: daemon mds.cephfs.smithi103.gtuatp assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: Cluster is now healthy 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: fsmap cephfs:0 1 up:standby 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:creating} 2024-03-22T02:25:48.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: daemon mds.cephfs.smithi103.gtuatp is now active in filesystem cephfs as rank 0 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/62828160' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:47 smithi103 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:25:48.902 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:48 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6826/3786434277,v1:172.21.15.103:6827/3786434277] up:active 2024-03-22T02:25:48.903 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:48 smithi103 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:active} 2024-03-22T02:25:49.214 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:48 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6826/3786434277,v1:172.21.15.103:6827/3786434277] up:active 2024-03-22T02:25:49.215 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:48 smithi122 ceph-mon[36403]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:active} 2024-03-22T02:25:49.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 -- 172.21.15.103:0/3704328819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 msgr2=0x7f73d80fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 --2- 172.21.15.103:0/3704328819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d80fe450 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f73cc004940 tx=0x7f73cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 -- 172.21.15.103:0/3704328819 shutdown_connections 2024-03-22T02:25:49.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 --2- 172.21.15.103:0/3704328819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d80fe450 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 --2- 172.21.15.103:0/3704328819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d80fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 -- 172.21.15.103:0/3704328819 >> 172.21.15.103:0/3704328819 conn(0x7f73d80f8550 msgr2=0x7f73d80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 -- 172.21.15.103:0/3704328819 shutdown_connections 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.390+0000 7f73de848700 1 -- 172.21.15.103:0/3704328819 wait complete. 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 Processor -- start 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 -- start start 2024-03-22T02:25:49.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d8110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d8110600 con 0x7f73d80fdfe0 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73de848700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d8110770 con 0x7f73d80fce70 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73d7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73d7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:50712/0 (socket says 172.21.15.103:50712) 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73d77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d8110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.391+0000 7f73d7fff700 1 -- 172.21.15.103:0/3810539308 learned_addr learned my addr 172.21.15.103:0/3810539308 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:49.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d7fff700 1 -- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 msgr2=0x7f73d8110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d7fff700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d8110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d7fff700 1 -- 172.21.15.103:0/3810539308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc005040 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d7fff700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f73c800eb40 tx=0x7f73c800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c8007cc0 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73c800f040 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d8115170 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c800ca10 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.392+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d81156b0 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.393+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d804c710 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.393+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f73c800cb70 con 0x7f73d80fce70 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.394+0000 7f73d4ff9700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 0x7f73c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.394+0000 7f73d77fe700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 0x7f73c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.394+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f73c8086d40 con 0x7f73d80fce70 2024-03-22T02:25:49.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.394+0000 7f73d77fe700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 0x7f73c006b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009b30 tx=0x7f73cc009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.398+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f73c804e190 con 0x7f73d80fce70 2024-03-22T02:25:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.638+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f73d8115b40 con 0x7f73d80fce70 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.873+0000 7f73d4ff9700 1 -- 172.21.15.103:0/3810539308 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f73c8056b10 con 0x7f73d80fce70 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 msgr2=0x7f73c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 0x7f73c006b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009b30 tx=0x7f73cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 msgr2=0x7f73d810fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f73c800eb40 tx=0x7f73c800b040 comp rx=0 tx=0).stop 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 shutdown_connections 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f73c00692a0 0x7f73c006b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73d80fce70 0x7f73d810fac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 --2- 172.21.15.103:0/3810539308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73d80fdfe0 0x7f73d8110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.875+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 >> 172.21.15.103:0/3810539308 conn(0x7f73d80f8550 msgr2=0x7f73d80f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.876+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 shutdown_connections 2024-03-22T02:25:49.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:49.876+0000 7f73de848700 1 -- 172.21.15.103:0/3810539308 wait complete. 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3810539308' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-22T02:25:49.892 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:49 smithi103 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: pgmap v182: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:50.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:50.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3810539308' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-22T02:25:50.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:49 smithi122 ceph-mon[36403]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-22T02:25:51.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:25:51.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:25:51.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:boot 2024-03-22T02:25:51.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-22T02:25:51.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: daemon mds.cephfs.smithi122.ztrkgp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:active} 1 up:standby 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:creating} 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: daemon mds.cephfs.smithi122.ztrkgp is now active in filesystem cephfs as rank 1 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:51.016 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:51.039 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:51.050 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:51.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-03-22T02:25:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: Deploying daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:25:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:25:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:boot 2024-03-22T02:25:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: daemon mds.cephfs.smithi122.ztrkgp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: fsmap cephfs:1 {0=cephfs.smithi103.gtuatp=up:active} 1 up:standby 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:creating} 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: daemon mds.cephfs.smithi122.ztrkgp is now active in filesystem cephfs as rank 1 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:51.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:25:51.583 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:51.997 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:51 smithi103 ceph-mon[29258]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-22T02:25:51.997 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:51 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:active 2024-03-22T02:25:51.998 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:51 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2024-03-22T02:25:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:51 smithi122 ceph-mon[36403]: pgmap v183: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-22T02:25:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:51 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:active 2024-03-22T02:25:52.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:51 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 7 op/s 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:53.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 -- 172.21.15.103:0/1958667572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 msgr2=0x7f616c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 --2- 172.21.15.103:0/1958667572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c0ffa40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f615c004940 tx=0x7f615c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:53.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 -- 172.21.15.103:0/1958667572 shutdown_connections 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 --2- 172.21.15.103:0/1958667572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 --2- 172.21.15.103:0/1958667572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c0ffa40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 -- 172.21.15.103:0/1958667572 >> 172.21.15.103:0/1958667572 conn(0x7f616c0f7430 msgr2=0x7f616c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 -- 172.21.15.103:0/1958667572 shutdown_connections 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.938+0000 7f6170b7c700 1 -- 172.21.15.103:0/1958667572 wait complete. 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 Processor -- start 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 -- start start 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c10dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c10e5b0 con 0x7f616c066260 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6170b7c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c10ee00 con 0x7f616c0fff80 2024-03-22T02:25:53.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58452/0 (socket says 172.21.15.103:58452) 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 -- 172.21.15.103:0/650285543 learned_addr learned my addr 172.21.15.103:0/650285543 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6161d9b700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c10dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 -- 172.21.15.103:0/650285543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 msgr2=0x7f616c10dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c10dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 -- 172.21.15.103:0/650285543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c005040 con 0x7f616c066260 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f6161d9b700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c10dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f616a59c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f615c00b3b0 tx=0x7f615c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.939+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615c00e040 con 0x7f616c066260 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.940+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f615c00bcc0 con 0x7f616c066260 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.940+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c10f080 con 0x7f616c066260 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.940+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f615c01c630 con 0x7f616c066260 2024-03-22T02:25:53.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.940+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c10f5c0 con 0x7f616c066260 2024-03-22T02:25:53.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.941+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c05f5f0 con 0x7f616c066260 2024-03-22T02:25:53.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.941+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f615c0139f0 con 0x7f616c066260 2024-03-22T02:25:53.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.942+0000 7f61637fe700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 0x7f615806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:53.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.942+0000 7f6161d9b700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 0x7f615806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:53.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.942+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f615c087f90 con 0x7f616c066260 2024-03-22T02:25:53.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.942+0000 7f6161d9b700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 0x7f615806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fd0 tx=0x7f6154009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:53.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:53.946+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f615c057d50 con 0x7f616c066260 2024-03-22T02:25:54.176 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 7 op/s 2024-03-22T02:25:54.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:54.338 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:54.338 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:54.338 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:25:54.338 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:25:54.338 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:54.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.175+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f616c1058b0 con 0x7f616c066260 2024-03-22T02:25:54.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.895+0000 7f61637fe700 1 -- 172.21.15.103:0/650285543 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f615c0578e0 con 0x7f616c066260 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 msgr2=0x7f615806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 0x7f615806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6154009fd0 tx=0x7f6154009ee0 comp rx=0 tx=0).stop 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 msgr2=0x7f616c10da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f615c00b3b0 tx=0x7f615c009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 shutdown_connections 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f61580692a0 0x7f615806b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f616c066260 0x7f616c10da70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 --2- 172.21.15.103:0/650285543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f616c0fff80 0x7f616c10dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:54.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 >> 172.21.15.103:0/650285543 conn(0x7f616c0f7430 msgr2=0x7f616c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:54.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 shutdown_connections 2024-03-22T02:25:54.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:54.897+0000 7f6170b7c700 1 -- 172.21.15.103:0/650285543 wait complete. 2024-03-22T02:25:55.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: Deploying daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:25:55.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-22T02:25:55.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: Cluster is now healthy 2024-03-22T02:25:55.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:boot 2024-03-22T02:25:55.215 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:55.215 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:25:55.215 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:54 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/650285543' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: Deploying daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: Cluster is now healthy 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:boot 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:25:55.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:54 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/650285543' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-03-22T02:25:55.517 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:55.532 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:55.533 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-22T02:25:55.796 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:25:56.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:55 smithi122 ceph-mon[36403]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-03-22T02:25:56.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:55 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/650285543' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-22T02:25:56.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:55 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:56.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:55 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:56.167 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:55 smithi103 ceph-mon[29258]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-03-22T02:25:56.167 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:55 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/650285543' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-03-22T02:25:56.167 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:55 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:56.167 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:55 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 -- 172.21.15.103:0/812483195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fdfe0 msgr2=0x7f06100fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:57.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 --2- 172.21.15.103:0/812483195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fdfe0 0x7f06100fe450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0604004940 tx=0x7f0604009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:57.435 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 7 op/s 2024-03-22T02:25:57.436 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] up:boot 2024-03-22T02:25:57.436 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:25:57.436 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:57 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 -- 172.21.15.103:0/812483195 shutdown_connections 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 --2- 172.21.15.103:0/812483195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fdfe0 0x7f06100fe450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 --2- 172.21.15.103:0/812483195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fce70 0x7f06100fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 -- 172.21.15.103:0/812483195 >> 172.21.15.103:0/812483195 conn(0x7f06100f8550 msgr2=0x7f06100fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 -- 172.21.15.103:0/812483195 shutdown_connections 2024-03-22T02:25:57.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.433+0000 7f0614d39700 1 -- 172.21.15.103:0/812483195 wait complete. 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 Processor -- start 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 -- start start 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fdfe0 0x7f061010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061010e460 con 0x7f06100fce70 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f0614d39700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061010e5d0 con 0x7f06100fdfe0 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58474/0 (socket says 172.21.15.103:58474) 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 -- 172.21.15.103:0/3637594301 learned_addr learned my addr 172.21.15.103:0/3637594301 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060dd9b700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fdfe0 0x7f061010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 -- 172.21.15.103:0/3637594301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fdfe0 msgr2=0x7f061010de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:57.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fdfe0 0x7f061010de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 -- 172.21.15.103:0/3637594301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0604005040 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.434+0000 7f060e59c700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0600002a50 tx=0x7f06000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.435+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0600007cc0 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.435+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f060000f040 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.435+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0610112fd0 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.435+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f060000da30 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.435+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0610113560 con 0x7f06100fce70 2024-03-22T02:25:57.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.436+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061004c710 con 0x7f06100fce70 2024-03-22T02:25:57.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.436+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f060000db90 con 0x7f06100fce70 2024-03-22T02:25:57.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.437+0000 7f05feffd700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 0x7f05f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:57.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.437+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0600086f40 con 0x7f06100fce70 2024-03-22T02:25:57.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.437+0000 7f060dd9b700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 0x7f05f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:57.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.437+0000 7f060dd9b700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 0x7f05f806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f060400b970 tx=0x7f0604009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:57.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.441+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0600056c80 con 0x7f06100fce70 2024-03-22T02:25:57.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 7 op/s 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] up:boot 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:25:57.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:57 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:25:57.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:57.700+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f06101058d0 con 0x7f06100fce70 2024-03-22T02:25:58.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.326+0000 7f05feffd700 1 -- 172.21.15.103:0/3637594301 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0600056810 con 0x7f06100fce70 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 msgr2=0x7f05f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 0x7f05f806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f060400b970 tx=0x7f0604009e30 comp rx=0 tx=0).stop 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 msgr2=0x7f061010d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0600002a50 tx=0x7f06000046b0 comp rx=0 tx=0).stop 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 shutdown_connections 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f05f80692a0 0x7f05f806b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06100fce70 0x7f061010d920 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0600002a50 tx=0x7f06000046b0 comp rx=0 tx=0).stop 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 --2- 172.21.15.103:0/3637594301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06100fdfe0 0x7f061010de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 >> 172.21.15.103:0/3637594301 conn(0x7f06100f8550 msgr2=0x7f06100f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 shutdown_connections 2024-03-22T02:25:58.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:25:58.329+0000 7f0614d39700 1 -- 172.21.15.103:0/3637594301 wait complete. 2024-03-22T02:25:58.332 INFO:teuthology.orchestra.run.smithi103.stderr:inline data disabled 2024-03-22T02:25:58.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:58 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3637594301' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-22T02:25:58.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:58 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3637594301' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-22T02:25:59.235 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:25:59.245 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:25:59.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:standby 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3637594301' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:active 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: Dropping low affinity active daemon mds.cephfs.smithi103.gtuatp in favor of higher affinity standby. 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: Replacing daemon mds.cephfs.smithi103.gtuatp as rank 0 with standby daemon mds.cephfs.smithi103.xhmfjp 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: osdmap e46: 6 total, 6 up, 6 in 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:replay,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2024-03-22T02:25:59.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:25:59 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:standby 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3637594301' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] up:active 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: Dropping low affinity active daemon mds.cephfs.smithi103.gtuatp in favor of higher affinity standby. 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: Replacing daemon mds.cephfs.smithi103.gtuatp as rank 0 with standby daemon mds.cephfs.smithi103.xhmfjp 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:25:59.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:25:59.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: osdmap e46: 6 total, 6 up, 6 in 2024-03-22T02:25:59.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:replay,1=cephfs.smithi122.ztrkgp=up:active} 1 up:standby 2024-03-22T02:25:59.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2024-03-22T02:25:59.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:59.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:25:59 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:25:59.651 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:00.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:00 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:resolve 2024-03-22T02:26:00.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:00 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] up:boot 2024-03-22T02:26:00.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:00 smithi122 ceph-mon[36403]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:resolve,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:00.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:00 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:26:00.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:00 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:resolve 2024-03-22T02:26:00.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:00 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] up:boot 2024-03-22T02:26:00.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:00 smithi103 ceph-mon[29258]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:resolve,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:00.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:00 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:26:01.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:01 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:reconnect 2024-03-22T02:26:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:01 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] up:standby 2024-03-22T02:26:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:01 smithi122 ceph-mon[36403]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:reconnect,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:01 smithi122 ceph-mon[36403]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 409 B/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-22T02:26:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:01 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:01 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:reconnect 2024-03-22T02:26:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:01 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] up:standby 2024-03-22T02:26:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:01 smithi103 ceph-mon[29258]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:reconnect,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:01 smithi103 ceph-mon[29258]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 409 B/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-22T02:26:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:01 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 -- 172.21.15.103:0/1525935008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fcf80 msgr2=0x7f679c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 --2- 172.21.15.103:0/1525935008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fcf80 0x7f679c0fd3f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6784004940 tx=0x7f6784009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 -- 172.21.15.103:0/1525935008 shutdown_connections 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 --2- 172.21.15.103:0/1525935008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fcf80 0x7f679c0fd3f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 --2- 172.21.15.103:0/1525935008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fbcd0 0x7f679c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 -- 172.21.15.103:0/1525935008 >> 172.21.15.103:0/1525935008 conn(0x7f679c0f7430 msgr2=0x7f679c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 -- 172.21.15.103:0/1525935008 shutdown_connections 2024-03-22T02:26:02.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 -- 172.21.15.103:0/1525935008 wait complete. 2024-03-22T02:26:02.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.330+0000 7f67a1a75700 1 Processor -- start 2024-03-22T02:26:02.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f67a1a75700 1 -- start start 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f67a1a75700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f67a1a75700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fcf80 0x7f679c112c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f67a1a75700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0663e0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f67a1a75700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c066520 con 0x7f679c0fcf80 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:02.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58510/0 (socket says 172.21.15.103:58510) 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 -- 172.21.15.103:0/3844094726 learned_addr learned my addr 172.21.15.103:0/3844094726 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679affd700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fcf80 0x7f679c112c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 -- 172.21.15.103:0/3844094726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fcf80 msgr2=0x7f679c112c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fcf80 0x7f679c112c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 -- 172.21.15.103:0/3844094726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784005040 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.331+0000 7f679b7fe700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f679400bbb0 tx=0x7f6794009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.332+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67940092e0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.332+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f679400cb20 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.332+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1132e0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.332+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6794015d70 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.332+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c113870 con 0x7f679c0fbcd0 2024-03-22T02:26:02.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.333+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c05f5f0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.333+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6794012070 con 0x7f679c0fbcd0 2024-03-22T02:26:02.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.334+0000 7f6793fff700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 0x7f678806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:02.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.334+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6794086cb0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.334+0000 7f679affd700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 0x7f678806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:02.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.334+0000 7f679affd700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 0x7f678806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f678400b3b0 tx=0x7f6784009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:02.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.339+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6794052f90 con 0x7f679c0fbcd0 2024-03-22T02:26:02.585 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:02 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:rejoin 2024-03-22T02:26:02.585 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:02 smithi103 ceph-mon[29258]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:rejoin,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:02.585 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:02 smithi103 ceph-mon[29258]: daemon mds.cephfs.smithi103.xhmfjp is now active in filesystem cephfs as rank 0 2024-03-22T02:26:02.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.586+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f679c113b10 con 0x7f679c0fbcd0 2024-03-22T02:26:02.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.587+0000 7f6793fff700 1 -- 172.21.15.103:0/3844094726 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 17 v17) v1 ==== 76+0+1824 (secure 0 0 0) 0x7f67940565b0 con 0x7f679c0fbcd0 2024-03-22T02:26:02.588 INFO:teuthology.orchestra.run.smithi103.stdout:e17 2024-03-22T02:26:02.588 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 17 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:26:02.350646+0000 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:26:02.589 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:26:02.590 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:26:02.591 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:26:02.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 msgr2=0x7f678806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:02.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 0x7f678806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f678400b3b0 tx=0x7f6784009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:02.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 msgr2=0x7f679c110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:02.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f679400bbb0 tx=0x7f6794009840 comp rx=0 tx=0).stop 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 shutdown_connections 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f67880692a0 0x7f678806b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f679c0fbcd0 0x7f679c110730 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 --2- 172.21.15.103:0/3844094726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f679c0fcf80 0x7f679c112c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 >> 172.21.15.103:0/3844094726 conn(0x7f679c0f7430 msgr2=0x7f679c0f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 shutdown_connections 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:02.590+0000 7f67a1a75700 1 -- 172.21.15.103:0/3844094726 wait complete. 2024-03-22T02:26:02.593 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 17 2024-03-22T02:26:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:02 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:rejoin 2024-03-22T02:26:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:02 smithi122 ceph-mon[36403]: fsmap cephfs:2/2 {0=cephfs.smithi103.xhmfjp=up:rejoin,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:02 smithi122 ceph-mon[36403]: daemon mds.cephfs.smithi103.xhmfjp is now active in filesystem cephfs as rank 0 2024-03-22T02:26:03.357 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: Cluster is now healthy 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:active 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-22T02:26:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:03 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3844094726' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:26:03.776 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:26:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: Cluster is now healthy 2024-03-22T02:26:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: mds.? [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] up:active 2024-03-22T02:26:03.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:26:03.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-22T02:26:03.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:03 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3844094726' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:26:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 -- 172.21.15.103:0/3862175750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fdfe0 msgr2=0x7fc3140fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3862175750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fdfe0 0x7fc3140fe450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc304004940 tx=0x7fc304009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 -- 172.21.15.103:0/3862175750 shutdown_connections 2024-03-22T02:26:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3862175750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fdfe0 0x7fc3140fe450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3862175750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fce70 0x7fc3140fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.493+0000 7fc31b71d700 1 -- 172.21.15.103:0/3862175750 >> 172.21.15.103:0/3862175750 conn(0x7fc3140f8550 msgr2=0x7fc3140fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 -- 172.21.15.103:0/3862175750 shutdown_connections 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 -- 172.21.15.103:0/3862175750 wait complete. 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 Processor -- start 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 -- start start 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fdfe0 0x7fc314110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc314110680 con 0x7fc3140fce70 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc31b71d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3141107f0 con 0x7fc3140fdfe0 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc3194b9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc3194b9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35692/0 (socket says 172.21.15.103:35692) 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc3194b9700 1 -- 172.21.15.103:0/3535687490 learned_addr learned my addr 172.21.15.103:0/3535687490 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.494+0000 7fc318cb8700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fdfe0 0x7fc314110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc3194b9700 1 -- 172.21.15.103:0/3535687490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fdfe0 msgr2=0x7fc314110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc3194b9700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fdfe0 0x7fc314110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc3194b9700 1 -- 172.21.15.103:0/3535687490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc304005040 con 0x7fc3140fce70 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc3194b9700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc310002a50 tx=0x7fc3100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc310007cc0 con 0x7fc3140fce70 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc31000f040 con 0x7fc3140fce70 2024-03-22T02:26:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc314115250 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31000da30 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.495+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc314115810 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.496+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc314101970 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.496+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc31000db90 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.497+0000 7fc309ffb700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 0x7fc30006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.497+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc310086f90 con 0x7fc3140fce70 2024-03-22T02:26:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.497+0000 7fc318cb8700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 0x7fc30006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:05.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.497+0000 7fc318cb8700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 0x7fc30006b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc30400b3b0 tx=0x7fc304009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:05.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.501+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc310056c80 con 0x7fc3140fce70 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 409 B/s wr, 2 op/s 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:26:05.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:05.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.756+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc31404c710 con 0x7fc3140fce70 2024-03-22T02:26:05.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.757+0000 7fc309ffb700 1 -- 172.21.15.103:0/3535687490 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4715 (secure 0 0 0) 0x7fc310056810 con 0x7fc3140fce70 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 msgr2=0x7fc30006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 0x7fc30006b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc30400b3b0 tx=0x7fc304009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 msgr2=0x7fc31410fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc310002a50 tx=0x7fc3100046b0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 shutdown_connections 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc3000692a0 0x7fc30006b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3140fce70 0x7fc31410fb40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 --2- 172.21.15.103:0/3535687490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3140fdfe0 0x7fc314110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:05.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 >> 172.21.15.103:0/3535687490 conn(0x7fc3140f8550 msgr2=0x7fc3140f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:05.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 shutdown_connections 2024-03-22T02:26:05.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:05.759+0000 7fc31b71d700 1 -- 172.21.15.103:0/3535687490 wait complete. 2024-03-22T02:26:05.761 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 17 2024-03-22T02:26:05.770 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:26:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 409 B/s wr, 2 op/s 2024-03-22T02:26:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:05.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:06.717 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:06 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3535687490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:06 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3535687490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:07.025 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-22T02:26:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:07 smithi103 ceph-mon[29258]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-22T02:26:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:07.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:07 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:07.845 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:07 smithi122 ceph-mon[36403]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-22T02:26:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:07 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:09.541 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:09 smithi103 ceph-mon[29258]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-22T02:26:09.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:09 smithi122 ceph-mon[36403]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-22T02:26:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.007+0000 7f464009f700 1 -- 172.21.15.103:0/4274761868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 msgr2=0x7f46380fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.007+0000 7f464009f700 1 --2- 172.21.15.103:0/4274761868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f46380fd280 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4634004940 tx=0x7f4634009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 -- 172.21.15.103:0/4274761868 shutdown_connections 2024-03-22T02:26:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 --2- 172.21.15.103:0/4274761868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f46380fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 --2- 172.21.15.103:0/4274761868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f46380fd280 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 -- 172.21.15.103:0/4274761868 >> 172.21.15.103:0/4274761868 conn(0x7f46380f8550 msgr2=0x7f46380fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 -- 172.21.15.103:0/4274761868 shutdown_connections 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 -- 172.21.15.103:0/4274761868 wait complete. 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.008+0000 7f464009f700 1 Processor -- start 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f464009f700 1 -- start start 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f464009f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f464009f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f4638110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f464009f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4638110700 con 0x7f46380fce70 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f464009f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381150a0 con 0x7f46380fdfe0 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35708/0 (socket says 172.21.15.103:35708) 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 -- 172.21.15.103:0/1143190429 learned_addr learned my addr 172.21.15.103:0/1143190429 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463d63a700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f4638110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 -- 172.21.15.103:0/1143190429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 msgr2=0x7f4638110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f4638110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 -- 172.21.15.103:0/1143190429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634005040 con 0x7f46380fce70 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463d63a700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f4638110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.009+0000 7f463de3b700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4634009150 tx=0x7f4634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.010+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463400e040 con 0x7f46380fce70 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.010+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46381152a0 con 0x7f46380fce70 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.010+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f463400bcc0 con 0x7f46380fce70 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.010+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f463401c630 con 0x7f46380fce70 2024-03-22T02:26:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.010+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4638115810 con 0x7f46380fce70 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.011+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4638101970 con 0x7f46380fce70 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.012+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4634013420 con 0x7f46380fce70 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.012+0000 7f462e7fc700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 0x7f462406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.012+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4634087fe0 con 0x7f46380fce70 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.012+0000 7f463d63a700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 0x7f462406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:11.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.013+0000 7f463d63a700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 0x7f462406b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4628009b10 tx=0x7f4628008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:11.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.017+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4634057d50 con 0x7f46380fce70 2024-03-22T02:26:11.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.294+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f463802b360 con 0x7f46380fce70 2024-03-22T02:26:11.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.294+0000 7f462e7fc700 1 -- 172.21.15.103:0/1143190429 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v17) v1 ==== 78+0+99 (secure 0 0 0) 0x7f46340578e0 con 0x7f46380fce70 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 msgr2=0x7f462406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 0x7f462406b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4628009b10 tx=0x7f4628008040 comp rx=0 tx=0).stop 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 msgr2=0x7f463810fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4634009150 tx=0x7f4634009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 shutdown_connections 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f46240692a0 0x7f462406b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46380fce70 0x7f463810fb30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 --2- 172.21.15.103:0/1143190429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46380fdfe0 0x7f4638110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 >> 172.21.15.103:0/1143190429 conn(0x7f46380f8550 msgr2=0x7f46380f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:11.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 shutdown_connections 2024-03-22T02:26:11.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:11.296+0000 7f464009f700 1 -- 172.21.15.103:0/1143190429 wait complete. 2024-03-22T02:26:11.308 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:26:11.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:11 smithi103 ceph-mon[29258]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-22T02:26:11.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:11 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1143190429' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-22T02:26:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:11 smithi122 ceph-mon[36403]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-22T02:26:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:11 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1143190429' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-22T02:26:11.927 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-22T02:26:11.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:26:12.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 -- 172.21.15.103:0/2923637961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 msgr2=0x7f6dd80fc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2923637961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd80fc7e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc0048d0 tx=0x7f6dcc009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 -- 172.21.15.103:0/2923637961 shutdown_connections 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2923637961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd80fc7e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2923637961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 0x7f6dd80fb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 -- 172.21.15.103:0/2923637961 >> 172.21.15.103:0/2923637961 conn(0x7f6dd80f6840 msgr2=0x7f6dd80f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 -- 172.21.15.103:0/2923637961 shutdown_connections 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 -- 172.21.15.103:0/2923637961 wait complete. 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.063+0000 7f6dde35d640 1 Processor -- start 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 -- start start 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 0x7f6dd810ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd810d870 con 0x7f6dd80fc370 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd8112230 con 0x7f6dd80fb0a0 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35732/0 (socket says 172.21.15.103:35732) 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 -- 172.21.15.103:0/2558461439 learned_addr learned my addr 172.21.15.103:0/2558461439 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd7fff640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 0x7f6dd810ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 -- 172.21.15.103:0/2558461439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 msgr2=0x7f6dd810ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 0x7f6dd810ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 -- 172.21.15.103:0/2558461439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc005040 con 0x7f6dd80fc370 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd77fe640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007dd0 tx=0x7f6dcc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dcc0094c0 con 0x7f6dd80fc370 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd8112370 con 0x7f6dd80fc370 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dcc009620 con 0x7f6dd80fc370 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.064+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dcc03b640 con 0x7f6dd80fc370 2024-03-22T02:26:12.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.065+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd8112860 con 0x7f6dd80fc370 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.066+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd804bcc0 con 0x7f6dd80fc370 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.066+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6dcc02f070 con 0x7f6dd80fc370 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.066+0000 7f6dd4ff9640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 0x7f6dc006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.066+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6dcc0a96b0 con 0x7f6dd80fc370 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.066+0000 7f6dd7fff640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 0x7f6dc006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.067+0000 7f6dd7fff640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 0x7f6dc006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80fa420 tx=0x7f6dc800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.071+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6dcc075910 con 0x7f6dd80fc370 2024-03-22T02:26:12.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.245+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dd8103bf0 con 0x7f6dd80fc370 2024-03-22T02:26:12.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.246+0000 7f6dd4ff9640 1 -- 172.21.15.103:0/2558461439 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14450 (secure 0 0 0) 0x7f6dcc078f30 con 0x7f6dd80fc370 2024-03-22T02:26:12.247 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:12.247 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":46,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:25:58.328465+0000","last_up_change":"2024-03-22T02:24:22.446345+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-22T02:25:41.764005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-22T02:25:42.064180+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6803","nonce":1382751588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6805","nonce":1382751588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6809","nonce":1382751588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1382751588},{"type":"v1","addr":"172.21.15.103:6807","nonce":1382751588}]},"public_addr":"172.21.15.103:6803/1382751588","cluster_addr":"172.21.15.103:6805/1382751588","heartbeat_back_addr":"172.21.15.103:6809/1382751588","heartbeat_front_addr":"172.21.15.103:6807/1382751588","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6811","nonce":3229649513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6813","nonce":3229649513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6817","nonce":3229649513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3229649513},{"type":"v1","addr":"172.21.15.103:6815","nonce":3229649513}]},"public_addr":"172.21.15.103:6811/3229649513","cluster_addr":"172.21.15.103:6813/3229649513","heartbeat_back_addr":"172.21.15.103:6817/3229649513","heartbeat_front_addr":"172.21.15.103:6815/3229649513","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6819","nonce":2311429465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6821","nonce":2311429465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6825","nonce":2311429465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":2311429465},{"type":"v1","addr":"172.21.15.103:6823","nonce":2311429465}]},"public_addr":"172.21.15.103:6819/2311429465","cluster_addr":"172.21.15.103:6821/2311429465","heartbeat_back_addr":"172.21.15.103:6825/2311429465","heartbeat_front_addr":"172.21.15.103:6823/2311429465","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6801","nonce":599760037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6803","nonce":599760037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6807","nonce":599760037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":599760037},{"type":"v1","addr":"172.21.15.122:6805","nonce":599760037}]},"public_addr":"172.21.15.122:6801/599760037","cluster_addr":"172.21.15.122:6803/599760037","heartbeat_back_addr":"172.21.15.122:6807/599760037","heartbeat_front_addr":"172.21.15.122:6805/599760037","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6809","nonce":1379244401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6811","nonce":1379244401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6815","nonce":1379244401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1379244401},{"type":"v1","addr":"172.21.15.122:6813","nonce":1379244401}]},"public_addr":"172.21.15.122:6809/1379244401","cluster_addr":"172.21.15.122:6811/1379244401","heartbeat_back_addr":"172.21.15.122:6815/1379244401","heartbeat_front_addr":"172.21.15.122:6813/1379244401","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6817","nonce":2680393488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6819","nonce":2680393488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6823","nonce":2680393488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2680393488},{"type":"v1","addr":"172.21.15.122:6821","nonce":2680393488}]},"public_addr":"172.21.15.122:6817/2680393488","cluster_addr":"172.21.15.122:6819/2680393488","heartbeat_back_addr":"172.21.15.122:6823/2680393488","heartbeat_front_addr":"172.21.15.122:6821/2680393488","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6826/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6827/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 msgr2=0x7f6dc006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 0x7f6dc006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80fa420 tx=0x7f6dc800b040 comp rx=0 tx=0).stop 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 msgr2=0x7f6dd810d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007dd0 tx=0x7f6dcc009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 shutdown_connections 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f6dc0069210 0x7f6dc006b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd80fc370 0x7f6dd810d1f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 --2- 172.21.15.103:0/2558461439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd80fb0a0 0x7f6dd810ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 >> 172.21.15.103:0/2558461439 conn(0x7f6dd80f6840 msgr2=0x7f6dd8064370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 shutdown_connections 2024-03-22T02:26:12.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.248+0000 7f6dde35d640 1 -- 172.21.15.103:0/2558461439 wait complete. 2024-03-22T02:26:12.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:23:06.693135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-22T02:25:41.764005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-22T02:25:42.064180+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:26:12.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- 172.21.15.103:0/3963898147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 msgr2=0x7f4f500fb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3963898147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f500fb4a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009c50 tx=0x7f4f48004d10 comp rx=0 tx=0).stop 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- 172.21.15.103:0/3963898147 shutdown_connections 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3963898147 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 0x7f4f500fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3963898147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f500fb4a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- 172.21.15.103:0/3963898147 >> 172.21.15.103:0/3963898147 conn(0x7f4f500f6870 msgr2=0x7f4f500f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- 172.21.15.103:0/3963898147 shutdown_connections 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- 172.21.15.103:0/3963898147 wait complete. 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 Processor -- start 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- start start 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 0x7f4f5010d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5010d8a0 con 0x7f4f500fb090 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.362+0000 7f4f5859d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50112260 con 0x7f4f500fc360 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35748/0 (socket says 172.21.15.103:35748) 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 -- 172.21.15.103:0/3103041541 learned_addr learned my addr 172.21.15.103:0/3103041541 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f55b38640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 0x7f4f5010d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 -- 172.21.15.103:0/3103041541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 msgr2=0x7f4f5010d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 0x7f4f5010d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 -- 172.21.15.103:0/3103041541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f480098b0 con 0x7f4f500fb090 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f56339640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009c20 tx=0x7f4f48004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f4803a070 con 0x7f4f500fb090 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f48035b20 con 0x7f4f500fb090 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f501123a0 con 0x7f4f500fb090 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f4803eb20 con 0x7f4f500fb090 2024-03-22T02:26:12.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.363+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501128f0 con 0x7f4f500fb090 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.364+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5004bcc0 con 0x7f4f500fb090 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.364+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4f48032070 con 0x7f4f500fb090 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.365+0000 7f4f46ffd640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 0x7f4f3c06b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.365+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4f480a99c0 con 0x7f4f500fb090 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.365+0000 7f4f55b38640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 0x7f4f3c06b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.365+0000 7f4f55b38640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 0x7f4f3c06b6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4f40007fd0 tx=0x7f4f40009e90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.369+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4f48075ca0 con 0x7f4f500fb090 2024-03-22T02:26:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.534+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f50112dc0 con 0x7f4f500fb090 2024-03-22T02:26:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.534+0000 7f4f46ffd640 1 -- 172.21.15.103:0/3103041541 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f480792c0 con 0x7f4f500fb090 2024-03-22T02:26:12.535 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 msgr2=0x7f4f3c06b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 0x7f4f3c06b6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4f40007fd0 tx=0x7f4f40009e90 comp rx=0 tx=0).stop 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 msgr2=0x7f4f5010cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009c20 tx=0x7f4f48004d10 comp rx=0 tx=0).stop 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 shutdown_connections 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f4f3c069220 0x7f4f3c06b6d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f500fc360 0x7f4f5010d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 --2- 172.21.15.103:0/3103041541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f500fb090 0x7f4f5010cd10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 >> 172.21.15.103:0/3103041541 conn(0x7f4f500f6870 msgr2=0x7f4f500f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 shutdown_connections 2024-03-22T02:26:12.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.536+0000 7f4f5859d640 1 -- 172.21.15.103:0/3103041541 wait complete. 2024-03-22T02:26:12.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-22T02:26:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 -- 172.21.15.103:0/1635709239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 msgr2=0x7f3020101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 --2- 172.21.15.103:0/1635709239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f3020101740 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f30100048d0 tx=0x7f3010009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 -- 172.21.15.103:0/1635709239 shutdown_connections 2024-03-22T02:26:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 --2- 172.21.15.103:0/1635709239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f3020101740 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 --2- 172.21.15.103:0/1635709239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f30200fee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 -- 172.21.15.103:0/1635709239 >> 172.21.15.103:0/1635709239 conn(0x7f30200f6850 msgr2=0x7f30200f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 -- 172.21.15.103:0/1635709239 shutdown_connections 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.647+0000 7f302742e640 1 -- 172.21.15.103:0/1635709239 wait complete. 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 Processor -- start 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 -- start start 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f302010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302010d920 con 0x7f30200ff360 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f302742e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3020112290 con 0x7f30200661b0 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35760/0 (socket says 172.21.15.103:35760) 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 -- 172.21.15.103:0/2186902165 learned_addr learned my addr 172.21.15.103:0/2186902165 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30251ca640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f302010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 -- 172.21.15.103:0/2186902165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 msgr2=0x7f302010cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f302010cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30249c9640 1 -- 172.21.15.103:0/2186902165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010005040 con 0x7f30200ff360 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.648+0000 7f30251ca640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f302010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f30249c9640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f301002bda0 tx=0x7f3010009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30100094c0 con 0x7f30200ff360 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3010009620 con 0x7f30200ff360 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30201123d0 con 0x7f30200ff360 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f301003b640 con 0x7f30200ff360 2024-03-22T02:26:12.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.649+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30201128a0 con 0x7f30200ff360 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.650+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302004bcc0 con 0x7f30200ff360 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.650+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f301002f070 con 0x7f30200ff360 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.651+0000 7f3015ffb640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 0x7f300c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.651+0000 7f30251ca640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 0x7f300c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.651+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f30100a97f0 con 0x7f30200ff360 2024-03-22T02:26:12.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.651+0000 7f30251ca640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 0x7f300c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f302010e200 tx=0x7f301c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.655+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3010075a50 con 0x7f30200ff360 2024-03-22T02:26:12.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.823+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f3020112a30 con 0x7f30200ff360 2024-03-22T02:26:12.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.824+0000 7f3015ffb640 1 -- 172.21.15.103:0/2186902165 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3010079070 con 0x7f30200ff360 2024-03-22T02:26:12.824 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:26:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 msgr2=0x7f300c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 0x7f300c06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f302010e200 tx=0x7f301c00b040 comp rx=0 tx=0).stop 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 msgr2=0x7f302010d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f301002bda0 tx=0x7f3010009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 shutdown_connections 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f300c069210 0x7f300c06b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.825+0000 7f302742e640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30200ff360 0x7f302010d2d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.826+0000 7f302742e640 1 --2- 172.21.15.103:0/2186902165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30200661b0 0x7f302010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.826+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 >> 172.21.15.103:0/2186902165 conn(0x7f30200f6850 msgr2=0x7f30200f6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.826+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 shutdown_connections 2024-03-22T02:26:12.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.826+0000 7f302742e640 1 -- 172.21.15.103:0/2186902165 wait complete. 2024-03-22T02:26:12.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-22T02:26:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- 172.21.15.103:0/3562352379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fb0b0 msgr2=0x7f9d100fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/3562352379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fb0b0 0x7f9d100fb4c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9d000048d0 tx=0x7f9d00009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- 172.21.15.103:0/3562352379 shutdown_connections 2024-03-22T02:26:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/3562352379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fc380 0x7f9d100fc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/3562352379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fb0b0 0x7f9d100fb4c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- 172.21.15.103:0/3562352379 >> 172.21.15.103:0/3562352379 conn(0x7f9d100f6890 msgr2=0x7f9d100f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- 172.21.15.103:0/3562352379 shutdown_connections 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- 172.21.15.103:0/3562352379 wait complete. 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 Processor -- start 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- start start 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 0x7f9d10065f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10112020 con 0x7f9d100fc380 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d14a6e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10112160 con 0x7f9d100fb0b0 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35780/0 (socket says 172.21.15.103:35780) 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d07fff640 1 -- 172.21.15.103:0/2183675121 learned_addr learned my addr 172.21.15.103:0/2183675121 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.937+0000 7f9d0e59c640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 0x7f9d10065f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d07fff640 1 -- 172.21.15.103:0/2183675121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 msgr2=0x7f9d10065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d07fff640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 0x7f9d10065f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d07fff640 1 -- 172.21.15.103:0/2183675121 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00005040 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d0e59c640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 0x7f9d10065f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d07fff640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800eb70 tx=0x7f9cf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cf8007c30 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d10112360 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cf800f040 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cf80095b0 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.938+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d10112850 con 0x7f9d100fc380 2024-03-22T02:26:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.939+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d101055b0 con 0x7f9d100fc380 2024-03-22T02:26:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.939+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9cf8009730 con 0x7f9d100fc380 2024-03-22T02:26:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.940+0000 7f9d06ffd640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 0x7f9cfc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.940+0000 7f9d0e59c640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 0x7f9cfc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.940+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9cf8086f60 con 0x7f9d100fc380 2024-03-22T02:26:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.940+0000 7f9d0e59c640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 0x7f9cfc06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d000317e0 tx=0x7f9d00009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:12.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:12.944+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9cf80531c0 con 0x7f9d100fc380 2024-03-22T02:26:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.113+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f9d10112aa0 con 0x7f9d100fc380 2024-03-22T02:26:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.114+0000 7f9d06ffd640 1 -- 172.21.15.103:0/2183675121 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9cf80567e0 con 0x7f9d100fc380 2024-03-22T02:26:13.114 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:26:13.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 msgr2=0x7f9cfc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 0x7f9cfc06b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d000317e0 tx=0x7f9d00009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 msgr2=0x7f9d10111ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800eb70 tx=0x7f9cf800b040 comp rx=0 tx=0).stop 2024-03-22T02:26:13.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 shutdown_connections 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9cfc069210 0x7f9cfc06b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d100fc380 0x7f9d10111ae0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 --2- 172.21.15.103:0/2183675121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d100fb0b0 0x7f9d10065f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.115+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 >> 172.21.15.103:0/2183675121 conn(0x7f9d100f6890 msgr2=0x7f9d100ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.116+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 shutdown_connections 2024-03-22T02:26:13.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.116+0000 7f9d14a6e640 1 -- 172.21.15.103:0/2183675121 wait complete. 2024-03-22T02:26:13.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2558461439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3103041541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:26:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:13 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2186902165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:26:13.127 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2384613309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 msgr2=0x7fdf040fc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2384613309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf040fc7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdef40048d0 tx=0x7fdef4009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2384613309 shutdown_connections 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2384613309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf040fc7e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2384613309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 0x7fdf040fb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.223+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2384613309 >> 172.21.15.103:0/2384613309 conn(0x7fdf040f6840 msgr2=0x7fdf040f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2384613309 shutdown_connections 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2384613309 wait complete. 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 Processor -- start 2024-03-22T02:26:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 -- start start 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 0x7fdf0410cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410d8f0 con 0x7fdf040fc370 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf0a99d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041122b0 con 0x7fdf040fb0a0 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf037fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35792/0 (socket says 172.21.15.103:35792) 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf037fe640 1 -- 172.21.15.103:0/2986491252 learned_addr learned my addr 172.21.15.103:0/2986491252 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.224+0000 7fdf03fff640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 0x7fdf0410cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf037fe640 1 -- 172.21.15.103:0/2986491252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 msgr2=0x7fdf0410cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf037fe640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 0x7fdf0410cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf037fe640 1 -- 172.21.15.103:0/2986491252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4005040 con 0x7fdf040fc370 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf037fe640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdef4031720 tx=0x7fdef4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef40092e0 con 0x7fdf040fc370 2024-03-22T02:26:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdef403d450 con 0x7fdf040fc370 2024-03-22T02:26:13.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf041123f0 con 0x7fdf040fc370 2024-03-22T02:26:13.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef403b640 con 0x7fdf040fc370 2024-03-22T02:26:13.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.225+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf041128c0 con 0x7fdf040fc370 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.226+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0404bcc0 con 0x7fdf040fc370 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.227+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdef402f070 con 0x7fdf040fc370 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.227+0000 7fdf00ff9640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 0x7fdeec06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.227+0000 7fdf03fff640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 0x7fdeec06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.227+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdef40a9630 con 0x7fdf040fc370 2024-03-22T02:26:13.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.227+0000 7fdf03fff640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 0x7fdeec06b6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf0410e220 tx=0x7fdef800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.231+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdef4075890 con 0x7fdf040fc370 2024-03-22T02:26:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2558461439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3103041541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:26:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:13 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2186902165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:26:13.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.434+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf04103c10 con 0x7fdf040fc370 2024-03-22T02:26:13.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.435+0000 7fdf00ff9640 1 -- 172.21.15.103:0/2986491252 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4715 (secure 0 0 0) 0x7fdef4078eb0 con 0x7fdf040fc370 2024-03-22T02:26:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:26:13.436 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:26:02.350646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14472,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi103.xhmfjp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6829/2193581075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2193581075},{"type":"v1","addr":"172.21.15.103:6829","nonce":2193581075}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi122.ztrkgp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6825/1237228570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1237228570},{"type":"v1","addr":"172.21.15.122:6825","nonce":1237228570}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-22T02:26:13.436 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 17 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 msgr2=0x7fdeec06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 0x7fdeec06b6c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf0410e220 tx=0x7fdef800b040 comp rx=0 tx=0).stop 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 msgr2=0x7fdf0410d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdef4031720 tx=0x7fdef4009da0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 shutdown_connections 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fdeec069210 0x7fdeec06b6c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf040fc370 0x7fdf0410d2a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 --2- 172.21.15.103:0/2986491252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf040fb0a0 0x7fdf0410cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 >> 172.21.15.103:0/2986491252 conn(0x7fdf040f6840 msgr2=0x7fdf040f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 shutdown_connections 2024-03-22T02:26:13.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:13.437+0000 7fdf0a99d640 1 -- 172.21.15.103:0/2986491252 wait complete. 2024-03-22T02:26:13.449 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 17, 'max_mds': 2, 'flags': 18} 2024-03-22T02:26:13.449 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-22T02:26:13.474 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-22T02:26:13.474 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-22T02:26:13.474 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-22T02:26:13.474 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T02:26:13.475 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:13.475 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-22T02:26:13.475 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T02:26:13.475 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:13.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:13.475 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-03-22T02:26:13.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:13.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-03-22T02:26:13.560 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-03-22T02:26:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:26:13.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:13.561 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-22T02:26:13.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:13.581 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2024-03-22T02:26:13.658 INFO:teuthology.orchestra.run.smithi122.stderr:Cannot find device "ceph-brx" 2024-03-22T02:26:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:26:13.660 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-22T02:26:13.660 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T02:26:13.917 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:14 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2183675121' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:26:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:14 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2986491252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:14 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2183675121' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:26:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:14 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2986491252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:15.216 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:15 smithi103 ceph-mon[29258]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:15 smithi122 ceph-mon[36403]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:15.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/313421184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4103740 msgr2=0x7fa4c40fcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:15.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/313421184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4103740 0x7fa4c40fcdb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0004940 tx=0x7fa4c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:15.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/313421184 shutdown_connections 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/313421184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4103740 0x7fa4c40fcdb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/313421184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4102da0 0x7fa4c4103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/313421184 >> 172.21.15.103:0/313421184 conn(0x7fa4c40f8970 msgr2=0x7fa4c40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/313421184 shutdown_connections 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.432+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/313421184 wait complete. 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 Processor -- start 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 -- start start 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4103740 0x7fa4c4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4114690 con 0x7fa4c4102da0 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4cb8ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c410da80 con 0x7fa4c4103740 2024-03-22T02:26:15.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38424/0 (socket says 172.21.15.103:38424) 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 -- 172.21.15.103:0/301923932 learned_addr learned my addr 172.21.15.103:0/301923932 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c8e49700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4103740 0x7fa4c4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 -- 172.21.15.103:0/301923932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4103740 msgr2=0x7fa4c4114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4103740 0x7fa4c4114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 -- 172.21.15.103:0/301923932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0005040 con 0x7fa4c4102da0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.433+0000 7fa4c964a700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c830 tx=0x7fa4b4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.434+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4b40106f0 con 0x7fa4c4102da0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.434+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c410dd60 con 0x7fa4c4102da0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.434+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4b4010d20 con 0x7fa4c4102da0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.434+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4b40113a0 con 0x7fa4c4102da0 2024-03-22T02:26:15.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.434+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c410e2f0 con 0x7fa4c4102da0 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.435+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c40fe4f0 con 0x7fa4c4102da0 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.435+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa4b4012400 con 0x7fa4c4102da0 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.435+0000 7fa4b9ffb700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 0x7fa4b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.436+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa4b40871a0 con 0x7fa4c4102da0 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.436+0000 7fa4c8e49700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 0x7fa4b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.436+0000 7fa4c8e49700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 0x7fa4b006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0004890 tx=0x7fa4c0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.440+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa4b4056f10 con 0x7fa4c4102da0 2024-03-22T02:26:15.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.668+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa4c4109ed0 con 0x7fa4c4102da0 2024-03-22T02:26:15.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.668+0000 7fa4b9ffb700 1 -- 172.21.15.103:0/301923932 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+83 (secure 0 0 0) 0x7fa4b4056aa0 con 0x7fa4c4102da0 2024-03-22T02:26:15.669 INFO:teuthology.orchestra.run.smithi103.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 msgr2=0x7fa4b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 0x7fa4b006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0004890 tx=0x7fa4c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 msgr2=0x7fa4c4113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c830 tx=0x7fa4b4004470 comp rx=0 tx=0).stop 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 shutdown_connections 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa4b00692a0 0x7fa4b006b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4c4102da0 0x7fa4c4113b00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 --2- 172.21.15.103:0/301923932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4c4103740 0x7fa4c4114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:15.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 >> 172.21.15.103:0/301923932 conn(0x7fa4c40f8970 msgr2=0x7fa4c40f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 shutdown_connections 2024-03-22T02:26:15.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:15.670+0000 7fa4cb8ae700 1 -- 172.21.15.103:0/301923932 wait complete. 2024-03-22T02:26:16.296 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:16 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/301923932' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T02:26:16.297 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T02:26:16.298 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-22T02:26:16.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:16.298 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:16.314 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42106sec preferred_lft 42106sec 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-03-22T02:26:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:16.316 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T02:26:16.316 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-03-22T02:26:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:16 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/301923932' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:26:16.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:16.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:16.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:16.512 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T02:26:16.539 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-03-22T02:26:16.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:16.541 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-03-22T02:26:16.611 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-03-22T02:26:16.611 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T02:26:16.612 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-22T02:26:16.612 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-22T02:26:16.612 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-22T02:26:16.612 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-03-22T02:26:16.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:16.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:16.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:16.935 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-03-22T02:26:16.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:16.950 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-03-22T02:26:17.005 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:17.005 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-03-22T02:26:17.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:26:17.005 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-22T02:26:17.005 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-03-22T02:26:17.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:17.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:17.129 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-22T02:26:17.129 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-03-22T02:26:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:17.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:17.263 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:17.263 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-03-22T02:26:17.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-03-22T02:26:17.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T02:26:17.263 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-03-22T02:26:17.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:17.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:17.346 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-22T02:26:17.346 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:26:17.346 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:17.361 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-22T02:26:17.361 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:26:17.361 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:17.373 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:17 smithi103 ceph-mon[29258]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:17.390 DEBUG:teuthology.orchestra.run.smithi103:> sudo modprobe fuse 2024-03-22T02:26:17.459 DEBUG:teuthology.orchestra.run.smithi103:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:26:17.515 INFO:teuthology.orchestra.run.smithi103.stdout:/proc 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/dev 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/security 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/shm 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/pts 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/run 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/cgroup 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/pstore 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/bpf 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/ 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/selinux 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/mqueue 2024-03-22T02:26:17.516 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/hugepages 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/debug 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/tracing 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/fuse/connections 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/config 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/1000 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/ceph 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/faf27891b11290c644d564a49f2c4ce4130ef7ad3704f3b15f7cd6841fd2f887/merged 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/4f4cc8ead2b528402fd1087ecef744400e5056579404533a740f99e851ee4ad3/merged 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/0 2024-03-22T02:26:17.517 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/655b25d3bc080c3b4dacc6b7e9ee0799a239dffa7b11a22139587efb5bb1558e/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/81093cf9cfff2e9b8b8e9ac4b6092cbc624955fffbeefdaf94bebf1d56b45d41/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/5e432866abdccfe0a59bfe27fad17d6318e188e53510918cd048bf6001ef5dde/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/62691c06087c222b6d29fc502eea58a9fec231073fe6ce532cab1519422baedc/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/750434f674e24fda0277ced0a32e36e3da639ec2bbe38d6847563f8de423144a/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/b4c0def3c372ca54757024921b0f84614d7059a9b13c436f3e1fca42107f27e6/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/3f973385b13f95f2870a0bf6a9b27305f003a8d4a0d8931e62ef5a66676aa84c/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/82cd958c7889e3e834712bd68bdab858bbe0084e8f98126a8c8e0d9fe06f8135/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/25a16816ee03ef0a0433b5b6bfa2f52d2b9f55a92f4f7dc9e6e0cdf086b0fd7a/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/efa865d544974af9f5140b26841a3c0469632133f25fb443f6c7219fc988e171/merged 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns 2024-03-22T02:26:17.518 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:26:17.519 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:26:17.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:17.519 DEBUG:teuthology.orchestra.run.smithi103:> ls /sys/fs/fuse/connections 2024-03-22T02:26:17.571 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-22T02:26:17.571 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-22T02:26:17.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:17 smithi122 ceph-mon[36403]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-22T02:26:17.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo modprobe fuse 2024-03-22T02:26:17.640 DEBUG:teuthology.orchestra.run.smithi103:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:26:17.696 INFO:teuthology.orchestra.run.smithi103.stdout:/proc 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/dev 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/security 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/shm 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/pts 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/run 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/cgroup 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/pstore 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/bpf 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/ 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/selinux 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/mqueue 2024-03-22T02:26:17.697 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/hugepages 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/debug 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/tracing 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/fuse/connections 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/config 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/1000 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/ceph 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/faf27891b11290c644d564a49f2c4ce4130ef7ad3704f3b15f7cd6841fd2f887/merged 2024-03-22T02:26:17.698 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/4f4cc8ead2b528402fd1087ecef744400e5056579404533a740f99e851ee4ad3/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/0 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/655b25d3bc080c3b4dacc6b7e9ee0799a239dffa7b11a22139587efb5bb1558e/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/81093cf9cfff2e9b8b8e9ac4b6092cbc624955fffbeefdaf94bebf1d56b45d41/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/5e432866abdccfe0a59bfe27fad17d6318e188e53510918cd048bf6001ef5dde/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/62691c06087c222b6d29fc502eea58a9fec231073fe6ce532cab1519422baedc/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/750434f674e24fda0277ced0a32e36e3da639ec2bbe38d6847563f8de423144a/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/b4c0def3c372ca54757024921b0f84614d7059a9b13c436f3e1fca42107f27e6/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/3f973385b13f95f2870a0bf6a9b27305f003a8d4a0d8931e62ef5a66676aa84c/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/82cd958c7889e3e834712bd68bdab858bbe0084e8f98126a8c8e0d9fe06f8135/merged 2024-03-22T02:26:17.699 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/25a16816ee03ef0a0433b5b6bfa2f52d2b9f55a92f4f7dc9e6e0cdf086b0fd7a/merged 2024-03-22T02:26:17.700 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/efa865d544974af9f5140b26841a3c0469632133f25fb443f6c7219fc988e171/merged 2024-03-22T02:26:17.700 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns 2024-03-22T02:26:17.700 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:26:17.700 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:26:17.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:17.701 DEBUG:teuthology.orchestra.run.smithi103:> ls /sys/fs/fuse/connections 2024-03-22T02:26:17.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi103.stderr:2024-03-22T02:26:17.708+0000 7fefe85d8300 -1 init, newargv = 0x559420635870 newargc=15 2024-03-22T02:26:17.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi103.stderr:ceph-fuse[77676]: starting ceph client 2024-03-22T02:26:17.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi103.stderr:ceph-fuse[77676]: starting fuse 2024-03-22T02:26:17.725 INFO:teuthology.orchestra.run.smithi103.stdout:85 2024-03-22T02:26:17.725 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [85] 2024-03-22T02:26:17.725 DEBUG:teuthology.orchestra.run.smithi103:> sudo stdin-killer -- python3 -c ' 2024-03-22T02:26:17.725 DEBUG:teuthology.orchestra.run.smithi103:> import glob 2024-03-22T02:26:17.725 DEBUG:teuthology.orchestra.run.smithi103:> import re 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> import os 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> import subprocess 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> def _find_admin_socket(client_name): 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> files = glob.glob(asok_path) 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-22T02:26:17.726 DEBUG:teuthology.orchestra.run.smithi103:> 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> # Given a non-glob path, it better be there 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> if "*" not in asok_path: 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> assert(len(files) == 1) 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> return files[0] 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> for f in files: 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-22T02:26:17.727 DEBUG:teuthology.orchestra.run.smithi103:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> contents = proc_f.read() 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> if mountpoint in contents: 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> return f 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> print(_find_admin_socket("client.0")) 2024-03-22T02:26:17.728 DEBUG:teuthology.orchestra.run.smithi103:> ' 2024-03-22T02:26:17.846 INFO:teuthology.orchestra.run.smithi103.stdout:/var/run/ceph/ceph-client.0.77676.asok 2024-03-22T02:26:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:17.857 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77676.asok 2024-03-22T02:26:17.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:17.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77676.asok status 2024-03-22T02:26:17.989 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "metadata": { 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_sha1": "9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "entity_id": "0", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "hostname": "smithi103", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "pid": "77676", 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "root": "/" 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "dentry_count": 0, 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "dentry_pinned_count": 0, 2024-03-22T02:26:17.990 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 14526, 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "inst": { 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "name": { 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client", 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "num": 14526 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": { 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4037363279 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": { 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-03-22T02:26:17.991 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 4037363279 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "inst_str": "client.14526 192.168.0.1:0/4037363279", 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "addr_str": "192.168.0.1:0/4037363279", 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "inode_count": 1, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_epoch": 17, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_epoch": 46, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_epoch_barrier": 0, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklisted": false, 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs" 2024-03-22T02:26:17.992 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:26:17.998 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T02:26:18.255 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:19.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:19 smithi103 ceph-mon[29258]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1 op/s 2024-03-22T02:26:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:19 smithi122 ceph-mon[36403]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1 op/s 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 -- 172.21.15.103:0/3568577007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 msgr2=0x7f54781031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 --2- 172.21.15.103:0/3568577007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f54781031f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f546c009ce0 tx=0x7f546c004d10 comp rx=0 tx=0).stop 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 -- 172.21.15.103:0/3568577007 shutdown_connections 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 --2- 172.21.15.103:0/3568577007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f54780fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 --2- 172.21.15.103:0/3568577007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f54781031f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 -- 172.21.15.103:0/3568577007 >> 172.21.15.103:0/3568577007 conn(0x7f54780f89d0 msgr2=0x7f54780fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 -- 172.21.15.103:0/3568577007 shutdown_connections 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.943+0000 7f547fd9a700 1 -- 172.21.15.103:0/3568577007 wait complete. 2024-03-22T02:26:19.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 Processor -- start 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 -- start start 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f5478114220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478114870 con 0x7f5478102e20 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547fd9a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547810dc40 con 0x7f54781037c0 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38448/0 (socket says 172.21.15.103:38448) 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 -- 172.21.15.103:0/1231684603 learned_addr learned my addr 172.21.15.103:0/1231684603 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547d335700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f5478114220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 -- 172.21.15.103:0/1231684603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 msgr2=0x7f5478114220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f5478114220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547db36700 1 -- 172.21.15.103:0/1231684603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c009940 con 0x7f5478102e20 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.944+0000 7f547d335700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f5478114220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:26:19.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f547db36700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f546c003400 tx=0x7f546c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:19.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f546c00e040 con 0x7f5478102e20 2024-03-22T02:26:19.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547810dec0 con 0x7f5478102e20 2024-03-22T02:26:19.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f546c004610 con 0x7f5478102e20 2024-03-22T02:26:19.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f546c01c6d0 con 0x7f5478102e20 2024-03-22T02:26:19.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.945+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547810e480 con 0x7f5478102e20 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.946+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5478100130 con 0x7f5478102e20 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.946+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f546c012410 con 0x7f5478102e20 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.946+0000 7f54767fc700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 0x7f546806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.947+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f546c088c80 con 0x7f5478102e20 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.947+0000 7f547d335700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 0x7f546806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:19.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.947+0000 7f547d335700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 0x7f546806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5464009fd0 tx=0x7f546400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:19.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:19.951+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f546c058970 con 0x7f5478102e20 2024-03-22T02:26:20.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.179+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f547804bed0 con 0x7f5478102e20 2024-03-22T02:26:20.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.179+0000 7f54767fc700 1 -- 172.21.15.103:0/1231684603 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+83 (secure 0 0 0) 0x7f546c004e10 con 0x7f5478102e20 2024-03-22T02:26:20.181 INFO:teuthology.orchestra.run.smithi103.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-22T02:26:20.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 msgr2=0x7f546806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:20.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 0x7f546806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5464009fd0 tx=0x7f546400b040 comp rx=0 tx=0).stop 2024-03-22T02:26:20.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 msgr2=0x7f5478113ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:20.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f546c003400 tx=0x7f546c004d10 comp rx=0 tx=0).stop 2024-03-22T02:26:20.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 shutdown_connections 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f54680692a0 0x7f546806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5478102e20 0x7f5478113ce0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 --2- 172.21.15.103:0/1231684603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54781037c0 0x7f5478114220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 >> 172.21.15.103:0/1231684603 conn(0x7f54780f89d0 msgr2=0x7f5478107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 shutdown_connections 2024-03-22T02:26:20.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:20.181+0000 7f547fd9a700 1 -- 172.21.15.103:0/1231684603 wait complete. 2024-03-22T02:26:20.816 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1231684603' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:26:20.816 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:20.839 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-22T02:26:20.839 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T02:26:20.839 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-03-22T02:26:20.839 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-22T02:26:20.839 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:20.840 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-22T02:26:20.840 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T02:26:20.840 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T02:26:20.840 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-22T02:26:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:20.840 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-03-22T02:26:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:20 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1231684603' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:26:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:20.859 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T02:26:20.859 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T02:26:20.859 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T02:26:20.860 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:26:20.861 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-22T02:26:20.861 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42125sec preferred_lft 42125sec 2024-03-22T02:26:20.861 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-03-22T02:26:20.861 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:26:20.861 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add name ceph-brx type bridge 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr flush dev ceph-brx 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set ceph-brx up 2024-03-22T02:26:20.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T02:26:20.862 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-22T02:26:20.954 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:21.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:21.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:21.075 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T02:26:21.101 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-22T02:26:21.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:21.103 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-03-22T02:26:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-03-22T02:26:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T02:26:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-22T02:26:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-22T02:26:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T02:26:21.178 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:21.179 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-22T02:26:21.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-22T02:26:21.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-22T02:26:21.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-22T02:26:21.179 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-22T02:26:21.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:21.486 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:21.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:21.493 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-03-22T02:26:21.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:21.508 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2024-03-22T02:26:21.552 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:21 smithi122 ceph-mon[36403]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-22T02:26:21.563 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:21.563 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-22T02:26:21.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-22T02:26:21.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-22T02:26:21.563 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-22T02:26:21.651 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:21.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:21.688 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-22T02:26:21.688 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-22T02:26:21.738 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:21.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:21.833 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:26:21.833 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-03-22T02:26:21.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-03-22T02:26:21.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T02:26:21.834 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-03-22T02:26:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:21 smithi103 ceph-mon[29258]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-03-22T02:26:21.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:26:21.909 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:21.915 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-22T02:26:21.915 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:26:21.915 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:21.929 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-22T02:26:21.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:26:21.930 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:21.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-22T02:26:22.053 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-22T02:26:22.109 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-22T02:26:22.110 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/0 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/b30fb43add43d30d1a7f715ef095708f329990958b3ce4809bb97b16ca7c6848/merged 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a155d0d2d4ac2970ffbd026e965bf5c567502c11c54f1d7a007c056078ea989d/merged 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a3a66de9f3e1e152a97c824fcb6c662c8d2610433461da976aec3edb924a8062/merged 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/76908ddba0d0db5909298816fb5bb485785db3483098299ece1a638460246e39/merged 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/86a19f57eb20cde6292dff6132934b0ec32206af81f18dfb19cbbef08548404d/merged 2024-03-22T02:26:22.111 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/cfc7717057ba8490fed1080bf028a9a1f39d27f7d79dfd075b06a3f0edd24c3b/merged 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/867afdeae381ad657508737e2c5862e2dfd19089f7070bf6721c3fe611bbdd17/merged 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/414a546b3682aa6dace0e40975221d001cd4c796f7d355ad0fab5217548f9d28/merged 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/b88177434c9da63d5fda1cc9a2d028fac4847b3f7ea786386ff1e98c67744a49/merged 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-22T02:26:22.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.112 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-22T02:26:22.164 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-22T02:26:22.165 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-22T02:26:22.208 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-03-22T02:26:22.236 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-03-22T02:26:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/0 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:26:22.292 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/b30fb43add43d30d1a7f715ef095708f329990958b3ce4809bb97b16ca7c6848/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a155d0d2d4ac2970ffbd026e965bf5c567502c11c54f1d7a007c056078ea989d/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a3a66de9f3e1e152a97c824fcb6c662c8d2610433461da976aec3edb924a8062/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/76908ddba0d0db5909298816fb5bb485785db3483098299ece1a638460246e39/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/86a19f57eb20cde6292dff6132934b0ec32206af81f18dfb19cbbef08548404d/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/cfc7717057ba8490fed1080bf028a9a1f39d27f7d79dfd075b06a3f0edd24c3b/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/867afdeae381ad657508737e2c5862e2dfd19089f7070bf6721c3fe611bbdd17/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/414a546b3682aa6dace0e40975221d001cd4c796f7d355ad0fab5217548f9d28/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/b88177434c9da63d5fda1cc9a2d028fac4847b3f7ea786386ff1e98c67744a49/merged 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-22T02:26:22.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.294 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-03-22T02:26:22.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:2024-03-22T02:26:22.334+0000 7fdef022a300 -1 init, newargv = 0x5597cea2b970 newargc=15 2024-03-22T02:26:22.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[61255]: starting ceph client 2024-03-22T02:26:22.342 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[61255]: starting fuse 2024-03-22T02:26:22.347 INFO:teuthology.orchestra.run.smithi122.stdout:115 2024-03-22T02:26:22.348 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-03-22T02:26:22.348 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> 2024-03-22T02:26:22.349 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.1")) 2024-03-22T02:26:22.350 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-03-22T02:26:22.468 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.1.61255.asok 2024-03-22T02:26:22.471 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-22T02:26:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:26:22.478 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61255.asok 2024-03-22T02:26:22.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61255.asok status 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45", 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "1", 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-22T02:26:22.610 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "61255", 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 24329, 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 24329 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-22T02:26:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1177476251 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 1177476251 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.24329 192.168.0.1:0/1177476251", 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/1177476251", 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 17, 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 46, 2024-03-22T02:26:22.612 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-03-22T02:26:22.613 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-03-22T02:26:22.613 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-03-22T02:26:22.613 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-22T02:26:22.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.618 DEBUG:teuthology.orchestra.run.smithi103:> stat --file-system '--printf=%T 2024-03-22T02:26:22.618 DEBUG:teuthology.orchestra.run.smithi103:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:22.638 INFO:teuthology.orchestra.run.smithi103.stdout:fuseblk 2024-03-22T02:26:22.638 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:22.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.638 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:22.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.713 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-03-22T02:26:22.713 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:22.728 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-03-22T02:26:22.728 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:22.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:22.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:23.556 INFO:teuthology.run_tasks:Running task print... 2024-03-22T02:26:23.566 INFO:teuthology.task.print:**** done client 2024-03-22T02:26:23.566 INFO:teuthology.run_tasks:Running task parallel... 2024-03-22T02:26:23.592 INFO:teuthology.task.parallel:starting parallel... 2024-03-22T02:26:23.593 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-22T02:26:23.593 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-22T02:26:23.594 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:26:23.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-22T02:26:23.594 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-22T02:26:23.595 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-22T02:26:23.598 INFO:tasks.workunit:Pulling workunits from ref c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:26:23.598 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-22T02:26:23.599 INFO:tasks.workunit:timeout=3h 2024-03-22T02:26:23.599 INFO:tasks.workunit:cleanup=True 2024-03-22T02:26:23.599 DEBUG:teuthology.orchestra.run.smithi103:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 55h/85d Inode: 1 Links: 2 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-22 02:25:46.867358022 +0000 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-22 02:26:22.709622959 +0000 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-03-22T02:26:23.619 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-22T02:26:23.620 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-22T02:26:23.694 DEBUG:teuthology.orchestra.run.smithi122:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:23.713 INFO:teuthology.orchestra.run.smithi122.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:23.713 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-22T02:26:23.713 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-03-22T02:26:23.713 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:26:23.713 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-22T02:26:23.714 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-22T02:26:23.714 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-22 02:26:23.687892182 +0000 2024-03-22T02:26:23.714 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-22 02:26:23.687892182 +0000 2024-03-22T02:26:23.714 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-22T02:26:23.714 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-22T02:26:23.714 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-22T02:26:23.778 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:23 smithi122 ceph-mon[36403]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-22T02:26:23.788 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:26:23.788 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:26:23.804 INFO:tasks.workunit.client.0.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-22T02:26:23.844 INFO:tasks.workunit.client.1.smithi122.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-22T02:26:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:23 smithi103 ceph-mon[29258]: pgmap v200: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-03-22T02:26:23.877 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- 172.21.15.103:0/3236635137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 msgr2=0x7f03400fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 --2- 172.21.15.103:0/3236635137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400fc0f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0330004940 tx=0x7f0330009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- 172.21.15.103:0/3236635137 shutdown_connections 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 --2- 172.21.15.103:0/3236635137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 --2- 172.21.15.103:0/3236635137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400fc0f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- 172.21.15.103:0/3236635137 >> 172.21.15.103:0/3236635137 conn(0x7f03400f7480 msgr2=0x7f03400f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:25.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- 172.21.15.103:0/3236635137 shutdown_connections 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- 172.21.15.103:0/3236635137 wait complete. 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 Processor -- start 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 -- start start 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.470+0000 7f034505b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f034505b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f034505b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340100210 con 0x7f03400fbce0 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f034505b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03400fe810 con 0x7f03400fcf90 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53258/0 (socket says 172.21.15.103:53258) 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 -- 172.21.15.103:0/2167755606 learned_addr learned my addr 172.21.15.103:0/2167755606 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033e59c700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 -- 172.21.15.103:0/2167755606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 msgr2=0x7f03400fe2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:25.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fe2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 -- 172.21.15.103:0/2167755606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330005040 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033e59c700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f033ed9d700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f033000b3b0 tx=0x7f0330009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033000e040 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03400fea90 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f033000bcc0 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033001c630 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.471+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03400ff050 con 0x7f03400fbce0 2024-03-22T02:26:25.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.472+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034004c710 con 0x7f03400fbce0 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.473+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0330026440 con 0x7f03400fbce0 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.473+0000 7f03277fe700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 0x7f032806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.473+0000 7f033e59c700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 0x7f032806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.473+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0330088000 con 0x7f03400fbce0 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.474+0000 7f033e59c700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 0x7f032806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0334006fd0 tx=0x7f0334008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:25.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.478+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0330057cf0 con 0x7f03400fbce0 2024-03-22T02:26:25.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.663+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f0340101ae0 con 0x7f03400fbce0 2024-03-22T02:26:25.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.664+0000 7f03277fe700 1 -- 172.21.15.103:0/2167755606 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f0330057880 con 0x7f03400fbce0 2024-03-22T02:26:25.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 msgr2=0x7f032806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 0x7f032806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0334006fd0 tx=0x7f0334008040 comp rx=0 tx=0).stop 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 msgr2=0x7f03400ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f033000b3b0 tx=0x7f0330009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 shutdown_connections 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f03280692a0 0x7f032806b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03400fbce0 0x7f03400ffb80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 --2- 172.21.15.103:0/2167755606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03400fcf90 0x7f03400fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 >> 172.21.15.103:0/2167755606 conn(0x7f03400f7480 msgr2=0x7f03400668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 shutdown_connections 2024-03-22T02:26:25.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:25.666+0000 7f034505b700 1 -- 172.21.15.103:0/2167755606 wait complete. 2024-03-22T02:26:25.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:25 smithi122 ceph-mon[36403]: pgmap v201: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-22T02:26:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:25 smithi103 ceph-mon[29258]: pgmap v201: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-03-22T02:26:26.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-22T02:26:26.583 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:27.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:27 smithi103 ceph-mon[29258]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:27 smithi122 ceph-mon[36403]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/1972844354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 msgr2=0x7f9eb80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/1972844354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb80fc0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8004940 tx=0x7f9ea8009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/1972844354 shutdown_connections 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/1972844354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 0x7f9eb80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/1972844354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb80fc0e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/1972844354 >> 172.21.15.103:0/1972844354 conn(0x7f9eb80f7430 msgr2=0x7f9eb80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:28.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/1972844354 shutdown_connections 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/1972844354 wait complete. 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.270+0000 7f9ebe9f1700 1 Processor -- start 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9ebe9f1700 1 -- start start 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9ebe9f1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9ebe9f1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 0x7f9eb810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9ebe9f1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb810e5c0 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9ebe9f1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb810ed50 con 0x7f9eb80fcf80 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53278/0 (socket says 172.21.15.103:53278) 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 -- 172.21.15.103:0/3077695506 learned_addr learned my addr 172.21.15.103:0/3077695506 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb77fe700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 0x7f9eb810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 -- 172.21.15.103:0/3077695506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 msgr2=0x7f9eb810dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:28.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 0x7f9eb810dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 -- 172.21.15.103:0/3077695506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8005040 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.271+0000 7f9eb7fff700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009150 tx=0x7f9ea8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.272+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea800e040 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.272+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb810efd0 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.272+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ea800bbf0 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.272+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea801c630 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.272+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb810f510 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.273+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8101b10 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.273+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9ea800bd60 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.273+0000 7f9eb4ff9700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 0x7f9ea006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.274+0000 7f9eb77fe700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 0x7f9ea006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.274+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9ea8088540 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.274+0000 7f9eb77fe700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 0x7f9ea006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9eac004f40 tx=0x7f9eac00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:28.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.278+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9ea8058360 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.464+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f9eb805f5f0 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.465+0000 7f9eb4ff9700 1 -- 172.21.15.103:0/3077695506 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f9ea8013e60 con 0x7f9eb80fbcd0 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 msgr2=0x7f9ea006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 0x7f9ea006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9eac004f40 tx=0x7f9eac00b040 comp rx=0 tx=0).stop 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 msgr2=0x7f9eb810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009150 tx=0x7f9ea8009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 shutdown_connections 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f9ea00692b0 0x7f9ea006b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9eb80fbcd0 0x7f9eb810da80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 --2- 172.21.15.103:0/3077695506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9eb80fcf80 0x7f9eb810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 >> 172.21.15.103:0/3077695506 conn(0x7f9eb80f7430 msgr2=0x7f9eb8065c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:28.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 shutdown_connections 2024-03-22T02:26:28.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:28.467+0000 7f9ebe9f1700 1 -- 172.21.15.103:0/3077695506 wait complete. 2024-03-22T02:26:29.164 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-22T02:26:29.439 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:29.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:29 smithi103 ceph-mon[29258]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:29 smithi122 ceph-mon[36403]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 -- 172.21.15.103:0/1521308511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fbce0 msgr2=0x7f269c0fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/1521308511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fbce0 0x7f269c0fc0f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f268c004940 tx=0x7f268c009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 -- 172.21.15.103:0/1521308511 shutdown_connections 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/1521308511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fcf90 0x7f269c0fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/1521308511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fbce0 0x7f269c0fc0f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 -- 172.21.15.103:0/1521308511 >> 172.21.15.103:0/1521308511 conn(0x7f269c0f7440 msgr2=0x7f269c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:31.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 -- 172.21.15.103:0/1521308511 shutdown_connections 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.152+0000 7f26a0f0f700 1 -- 172.21.15.103:0/1521308511 wait complete. 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 Processor -- start 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 -- start start 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 0x7f269c0ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c100210 con 0x7f269c0fcf90 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f26a0f0f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0fe810 con 0x7f269c0fbce0 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f2699d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f2699d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53294/0 (socket says 172.21.15.103:53294) 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f2699d9b700 1 -- 172.21.15.103:0/2364359329 learned_addr learned my addr 172.21.15.103:0/2364359329 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.153+0000 7f269a59c700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 0x7f269c0ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f2699d9b700 1 -- 172.21.15.103:0/2364359329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 msgr2=0x7f269c0ffb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f2699d9b700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 0x7f269c0ffb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f2699d9b700 1 -- 172.21.15.103:0/2364359329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c005040 con 0x7f269c0fcf90 2024-03-22T02:26:31.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f269a59c700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 0x7f269c0ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:31.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f2699d9b700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f269000b790 tx=0x7f2690009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269000f770 con 0x7f269c0fcf90 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c0feaf0 con 0x7f269c0fcf90 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c0ff0b0 con 0x7f269c0fcf90 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.154+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f269000fda0 con 0x7f269c0fcf90 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.155+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f269000d310 con 0x7f269c0fcf90 2024-03-22T02:26:31.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.155+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c101b20 con 0x7f269c0fcf90 2024-03-22T02:26:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.156+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f269000f8d0 con 0x7f269c0fcf90 2024-03-22T02:26:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.156+0000 7f268affd700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 0x7f268406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.156+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2690087200 con 0x7f269c0fcf90 2024-03-22T02:26:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.156+0000 7f269a59c700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 0x7f268406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:31.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.156+0000 7f269a59c700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 0x7f268406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b3b0 tx=0x7f268c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:31.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.160+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2690056f70 con 0x7f269c0fcf90 2024-03-22T02:26:31.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.346+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f269c04c710 con 0x7f269c0fcf90 2024-03-22T02:26:31.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.347+0000 7f268affd700 1 -- 172.21.15.103:0/2364359329 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2690020250 con 0x7f269c0fcf90 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 msgr2=0x7f268406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 0x7f268406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b3b0 tx=0x7f268c009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 msgr2=0x7f269c0fe2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f269000b790 tx=0x7f2690009250 comp rx=0 tx=0).stop 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 shutdown_connections 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f26840692a0 0x7f268406b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f269c0fbce0 0x7f269c0ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 --2- 172.21.15.103:0/2364359329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f269c0fcf90 0x7f269c0fe2d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 >> 172.21.15.103:0/2364359329 conn(0x7f269c0f7440 msgr2=0x7f269c066880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:31.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 shutdown_connections 2024-03-22T02:26:31.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:31.349+0000 7f26a0f0f700 1 -- 172.21.15.103:0/2364359329 wait complete. 2024-03-22T02:26:31.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:31 smithi122 ceph-mon[36403]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-22T02:26:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:31 smithi103 ceph-mon[29258]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-22T02:26:32.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-22T02:26:32.318 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:26:33.720 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:33 smithi103 ceph-mon[29258]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:33 smithi122 ceph-mon[36403]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-03-22T02:26:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 -- 172.21.15.103:0/929388012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 msgr2=0x7f50500fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 --2- 172.21.15.103:0/929388012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f50500fd400 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5040004940 tx=0x7f5040009e30 comp rx=0 tx=0).stop 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 -- 172.21.15.103:0/929388012 shutdown_connections 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 --2- 172.21.15.103:0/929388012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f50500fd400 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 --2- 172.21.15.103:0/929388012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 0x7f50500fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 -- 172.21.15.103:0/929388012 >> 172.21.15.103:0/929388012 conn(0x7f50500f7440 msgr2=0x7f50500f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 -- 172.21.15.103:0/929388012 shutdown_connections 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.926+0000 7f5056e47700 1 -- 172.21.15.103:0/929388012 wait complete. 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 Processor -- start 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 -- start start 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 0x7f5050066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050112ca0 con 0x7f50500fcf90 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5056e47700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050112de0 con 0x7f50500fbce0 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5047fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5047fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49538/0 (socket says 172.21.15.103:49538) 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5047fff700 1 -- 172.21.15.103:0/4261993640 learned_addr learned my addr 172.21.15.103:0/4261993640 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.927+0000 7f5054be3700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 0x7f5050066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5047fff700 1 -- 172.21.15.103:0/4261993640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 msgr2=0x7f5050066d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5047fff700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 0x7f5050066d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5047fff700 1 -- 172.21.15.103:0/4261993640 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040005040 con 0x7f50500fcf90 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5047fff700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f504000b3b0 tx=0x7f5040009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504000e040 con 0x7f50500fcf90 2024-03-22T02:26:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5050112fe0 con 0x7f50500fcf90 2024-03-22T02:26:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f504000bbf0 con 0x7f50500fcf90 2024-03-22T02:26:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504001c630 con 0x7f50500fcf90 2024-03-22T02:26:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.928+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5050113570 con 0x7f50500fcf90 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.929+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505004c710 con 0x7f50500fcf90 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.929+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f504000bd60 con 0x7f50500fcf90 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.930+0000 7f50457fa700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 0x7f503806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.930+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5040087ed0 con 0x7f50500fcf90 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.930+0000 7f5054be3700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 0x7f503806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:33.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.930+0000 7f5054be3700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 0x7f503806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f504c004f40 tx=0x7f504c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:33.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:33.934+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5040057c40 con 0x7f50500fcf90 2024-03-22T02:26:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:26:34.137+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f5050101350 con 0x7f50380692a0 2024-03-22T02:26:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:35 smithi122 ceph-mon[36403]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:26:35.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:35 smithi122 ceph-mon[36403]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-22T02:26:35.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:35 smithi103 ceph-mon[29258]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:26:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:35 smithi103 ceph-mon[29258]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-03-22T02:26:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:37 smithi122 ceph-mon[36403]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:26:37.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:37 smithi103 ceph-mon[29258]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:26:39.780 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:39 smithi103 ceph-mon[29258]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-22T02:26:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:39 smithi122 ceph-mon[36403]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-22T02:26:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:41 smithi122 ceph-mon[36403]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-22T02:26:41.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:41 smithi103 ceph-mon[29258]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-03-22T02:26:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:43 smithi122 ceph-mon[36403]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:43.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:43 smithi103 ceph-mon[29258]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:45 smithi122 ceph-mon[36403]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:45.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:45 smithi103 ceph-mon[29258]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:47 smithi122 ceph-mon[36403]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:47 smithi103 ceph-mon[29258]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:26:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:49 smithi103 ceph-mon[29258]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:49 smithi122 ceph-mon[36403]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:50.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:26:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:51 smithi122 ceph-mon[36403]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:51.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:51 smithi103 ceph-mon[29258]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:53 smithi122 ceph-mon[36403]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:53 smithi103 ceph-mon[29258]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:55 smithi122 ceph-mon[36403]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:55 smithi103 ceph-mon[29258]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:57 smithi122 ceph-mon[36403]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:57 smithi103 ceph-mon[29258]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:59.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:26:59 smithi103 ceph-mon[29258]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:26:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:26:59 smithi122 ceph-mon[36403]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:01.604 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:01 smithi103 ceph-mon[29258]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:01 smithi122 ceph-mon[36403]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:02.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.644+0000 7f50457fa700 1 -- 172.21.15.103:0/4261993640 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5050101350 con 0x7f50380692a0 2024-03-22T02:27:02.645 INFO:teuthology.orchestra.run.smithi103.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:27:02.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 msgr2=0x7f503806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:02.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 0x7f503806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f504c004f40 tx=0x7f504c00b040 comp rx=0 tx=0).stop 2024-03-22T02:27:02.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 msgr2=0x7f5050112760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:02.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f504000b3b0 tx=0x7f5040009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:02.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 shutdown_connections 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f50380692a0 0x7f503806b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50500fbce0 0x7f5050066d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 --2- 172.21.15.103:0/4261993640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50500fcf90 0x7f5050112760 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 >> 172.21.15.103:0/4261993640 conn(0x7f50500f7440 msgr2=0x7f50500f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 shutdown_connections 2024-03-22T02:27:02.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:02.646+0000 7f5056e47700 1 -- 172.21.15.103:0/4261993640 wait complete. 2024-03-22T02:27:03.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T02:27:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:27:03.898 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:03 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:27:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:27:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:03 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:04.956 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:04 smithi103 ceph-mon[29258]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:27:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:04 smithi122 ceph-mon[36403]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:27:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:05 smithi103 ceph-mon[29258]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:05 smithi122 ceph-mon[36403]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:07.740 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:07 smithi103 ceph-mon[29258]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:07 smithi122 ceph-mon[36403]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:09.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 -- 172.21.15.103:0/1743589225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 msgr2=0x7f21b40fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/1743589225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b40fd290 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f21b0004940 tx=0x7f21b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 -- 172.21.15.103:0/1743589225 shutdown_connections 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/1743589225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 0x7f21b40fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/1743589225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b40fd290 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 -- 172.21.15.103:0/1743589225 >> 172.21.15.103:0/1743589225 conn(0x7f21b40f8540 msgr2=0x7f21b40fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.550+0000 7f21bc2f5700 1 -- 172.21.15.103:0/1743589225 shutdown_connections 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.551+0000 7f21bc2f5700 1 -- 172.21.15.103:0/1743589225 wait complete. 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.551+0000 7f21bc2f5700 1 Processor -- start 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21bc2f5700 1 -- start start 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21bc2f5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21bc2f5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 0x7f21b410df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21bc2f5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410e520 con 0x7f21b40fce80 2024-03-22T02:27:09.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21bc2f5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410ed70 con 0x7f21b40fdff0 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41886/0 (socket says 172.21.15.103:41886) 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 -- 172.21.15.103:0/833807167 learned_addr learned my addr 172.21.15.103:0/833807167 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21b9890700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 0x7f21b410df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 -- 172.21.15.103:0/833807167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 msgr2=0x7f21b410df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 0x7f21b410df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 -- 172.21.15.103:0/833807167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b0005040 con 0x7f21b40fce80 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.552+0000 7f21ba091700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f21b0013870 tx=0x7f21b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.553+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b000e070 con 0x7f21b40fce80 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.553+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b410eff0 con 0x7f21b40fce80 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.553+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21b0013de0 con 0x7f21b40fce80 2024-03-22T02:27:09.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.553+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21b001cea0 con 0x7f21b40fce80 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.553+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b410f4d0 con 0x7f21b40fce80 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.554+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b405f5f0 con 0x7f21b40fce80 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.554+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f21b0013910 con 0x7f21b40fce80 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.555+0000 7f21aaffd700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 0x7f21a006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.555+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f21b00888e0 con 0x7f21b40fce80 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.555+0000 7f21b9890700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 0x7f21a006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.555+0000 7f21b9890700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 0x7f21a006b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009b80 tx=0x7f21a400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:09.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.559+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f21b0058650 con 0x7f21b40fce80 2024-03-22T02:27:09.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.792+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21b410f800 con 0x7f21a00692b0 2024-03-22T02:27:09.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.793+0000 7f21aaffd700 1 -- 172.21.15.103:0/833807167 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f21b410f800 con 0x7f21a00692b0 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 msgr2=0x7f21a006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 0x7f21a006b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009b80 tx=0x7f21a400b040 comp rx=0 tx=0).stop 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 msgr2=0x7f21b410d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f21b0013870 tx=0x7f21b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 shutdown_connections 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f21a00692b0 0x7f21a006b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b40fce80 0x7f21b410d9e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 --2- 172.21.15.103:0/833807167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b40fdff0 0x7f21b410df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.795+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 >> 172.21.15.103:0/833807167 conn(0x7f21b40f8540 msgr2=0x7f21b40f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:09.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.796+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 shutdown_connections 2024-03-22T02:27:09.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.796+0000 7f21bc2f5700 1 -- 172.21.15.103:0/833807167 wait complete. 2024-03-22T02:27:09.808 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:27:09.808 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:09 smithi103 ceph-mon[29258]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:09.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.902+0000 7f1b84b76700 1 -- 172.21.15.103:0/501317195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fbcd0 msgr2=0x7f1b800fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.902+0000 7f1b84b76700 1 --2- 172.21.15.103:0/501317195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fbcd0 0x7f1b800fc0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1b70004940 tx=0x7f1b70009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:09.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- 172.21.15.103:0/501317195 shutdown_connections 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 --2- 172.21.15.103:0/501317195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fcf80 0x7f1b800fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 --2- 172.21.15.103:0/501317195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fbcd0 0x7f1b800fc0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- 172.21.15.103:0/501317195 >> 172.21.15.103:0/501317195 conn(0x7f1b800f7430 msgr2=0x7f1b800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- 172.21.15.103:0/501317195 shutdown_connections 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- 172.21.15.103:0/501317195 wait complete. 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 Processor -- start 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- start start 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 0x7f1b80112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80066350 con 0x7f1b800fcf80 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.903+0000 7f1b84b76700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80066490 con 0x7f1b800fbcd0 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:45384/0 (socket says 172.21.15.103:45384) 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 -- 172.21.15.103:0/119956672 learned_addr learned my addr 172.21.15.103:0/119956672 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:09.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7dd9b700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 0x7f1b80112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 -- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 msgr2=0x7f1b80112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:09.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 0x7f1b80112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:09.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 -- 172.21.15.103:0/119956672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70005040 con 0x7f1b800fbcd0 2024-03-22T02:27:09.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7dd9b700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 0x7f1b80112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b7e59c700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b0d0 tx=0x7f1b70009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7000e070 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b80113120 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b7000b410 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b70012600 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.904+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b801136e0 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.906+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8005f5f0 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.906+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1b7000ba80 con 0x7f1b800fbcd0 2024-03-22T02:27:09.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.906+0000 7f1b6effd700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 0x7f1b6806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:09.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.906+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1b70087d90 con 0x7f1b800fbcd0 2024-03-22T02:27:09.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.906+0000 7f1b7dd9b700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 0x7f1b6806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:09.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.907+0000 7f1b7dd9b700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 0x7f1b6806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009b80 tx=0x7f1b7400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:09.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:09.911+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1b70054100 con 0x7f1b800fbcd0 2024-03-22T02:27:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:09 smithi122 ceph-mon[36403]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:10.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.165+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b801012f0 con 0x7f1b680692b0 2024-03-22T02:27:10.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.167+0000 7f1b6effd700 1 -- 172.21.15.103:0/119956672 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f1b801012f0 con 0x7f1b680692b0 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 msgr2=0x7f1b6806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 0x7f1b6806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009b80 tx=0x7f1b7400b040 comp rx=0 tx=0).stop 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 msgr2=0x7f1b80110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b0d0 tx=0x7f1b70009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 shutdown_connections 2024-03-22T02:27:10.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f1b680692b0 0x7f1b6806b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800fbcd0 0x7f1b80110690 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 --2- 172.21.15.103:0/119956672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b800fcf80 0x7f1b80112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 >> 172.21.15.103:0/119956672 conn(0x7f1b800f7430 msgr2=0x7f1b800f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:10.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 shutdown_connections 2024-03-22T02:27:10.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.169+0000 7f1b84b76700 1 -- 172.21.15.103:0/119956672 wait complete. 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 -- 172.21.15.103:0/2888759327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fcef0 msgr2=0x7f35a40fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 --2- 172.21.15.103:0/2888759327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fcef0 0x7f35a40fd300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3594004940 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 -- 172.21.15.103:0/2888759327 shutdown_connections 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 --2- 172.21.15.103:0/2888759327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fe060 0x7f35a40fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 --2- 172.21.15.103:0/2888759327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fcef0 0x7f35a40fd300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 -- 172.21.15.103:0/2888759327 >> 172.21.15.103:0/2888759327 conn(0x7f35a40f85b0 msgr2=0x7f35a40fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:10.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 -- 172.21.15.103:0/2888759327 shutdown_connections 2024-03-22T02:27:10.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.273+0000 7f35aa31a700 1 -- 172.21.15.103:0/2888759327 wait complete. 2024-03-22T02:27:10.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 Processor -- start 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 -- start start 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a410e4f0 con 0x7f35a40fe060 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35aa31a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a410ed40 con 0x7f35a40fcef0 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35a37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35a3fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35a3fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:45398/0 (socket says 172.21.15.103:45398) 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35a3fff700 1 -- 172.21.15.103:0/3147543442 learned_addr learned my addr 172.21.15.103:0/3147543442 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.274+0000 7f35a37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41920/0 (socket says 172.21.15.103:41920) 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a3fff700 1 -- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 msgr2=0x7f35a410def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a3fff700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a3fff700 1 -- 172.21.15.103:0/3147543442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594005040 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a37fe700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a3fff700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f359400b0d0 tx=0x7f3594009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f359400e030 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a410efc0 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f359401c400 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f359400b630 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.275+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a410f500 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.276+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a405f5f0 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.276+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f359400b8f0 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.277+0000 7f35a0ff9700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 0x7f358c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.277+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3594088c30 con 0x7f35a40fcef0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.277+0000 7f35a37fe700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 0x7f358c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.277+0000 7f35a37fe700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 0x7f358c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3598009b80 tx=0x7f359800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:10.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.282+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f359408f080 con 0x7f35a40fcef0 2024-03-22T02:27:10.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.499+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f35a410f7e0 con 0x7f358c0692a0 2024-03-22T02:27:10.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.511+0000 7f35a0ff9700 1 -- 172.21.15.103:0/3147543442 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f35a410f7e0 con 0x7f358c0692a0 2024-03-22T02:27:10.511 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:27:10.511 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (5m) 58s ago 6m 24.8M - 0.25.0 c8568f914cd2 31b45ef21cbd 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (7m) 58s ago 7m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (6m) 63s ago 6m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (5m) 58s ago 6m 84.7M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (84s) 58s ago 83s 10.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (77s) 58s ago 76s 14.1M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (74s) 63s ago 74s 11.9M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (81s) 63s ago 81s 16.8M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (8m) 58s ago 8m 481M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (5m) 63s ago 5m 432M - 17.2.7-726-g9c4e0c34 837124da804f 29f8988d41c7 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (8m) 58s ago 8m 48.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (5m) 63s ago 5m 36.4M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (7m) 58s ago 7m 16.2M - 1.5.0 0da6a335fe13 136d30f21984 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 63s ago 5m 16.4M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (5m) 58s ago 5m 45.4M 4288M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (4m) 58s ago 4m 45.3M 4288M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:27:10.512 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (4m) 58s ago 4m 47.6M 4288M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (3m) 63s ago 3m 45.6M 5312M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (3m) 63s ago 3m 44.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (2m) 63s ago 2m 41.3M 5312M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (5m) 58s ago 6m 54.3M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 msgr2=0x7f358c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 0x7f358c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3598009b80 tx=0x7f359800b040 comp rx=0 tx=0).stop 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 msgr2=0x7f35a410d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f359400b0d0 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 shutdown_connections 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f358c0692a0 0x7f358c06b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35a40fcef0 0x7f35a410d9b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 --2- 172.21.15.103:0/3147543442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35a40fe060 0x7f35a410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 >> 172.21.15.103:0/3147543442 conn(0x7f35a40f85b0 msgr2=0x7f35a40f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 shutdown_connections 2024-03-22T02:27:10.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.513+0000 7f35aa31a700 1 -- 172.21.15.103:0/3147543442 wait complete. 2024-03-22T02:27:10.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.615+0000 7ffadf617700 1 -- 172.21.15.103:0/2307561334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fbcd0 msgr2=0x7ffad80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.615+0000 7ffadf617700 1 --2- 172.21.15.103:0/2307561334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fbcd0 0x7ffad80fc0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffad4004940 tx=0x7ffad4009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:10.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.615+0000 7ffadf617700 1 -- 172.21.15.103:0/2307561334 shutdown_connections 2024-03-22T02:27:10.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.615+0000 7ffadf617700 1 --2- 172.21.15.103:0/2307561334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fcf80 0x7ffad80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 --2- 172.21.15.103:0/2307561334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fbcd0 0x7ffad80fc0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- 172.21.15.103:0/2307561334 >> 172.21.15.103:0/2307561334 conn(0x7ffad80f7430 msgr2=0x7ffad80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- 172.21.15.103:0/2307561334 shutdown_connections 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- 172.21.15.103:0/2307561334 wait complete. 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 Processor -- start 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- start start 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 0x7ffad8110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad80663b0 con 0x7ffad80fcf80 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.616+0000 7ffadf617700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad80664f0 con 0x7ffad80fbcd0 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41940/0 (socket says 172.21.15.103:41940) 2024-03-22T02:27:10.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadd3b3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 0x7ffad8110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 -- 172.21.15.103:0/1612030158 learned_addr learned my addr 172.21.15.103:0/1612030158 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 -- 172.21.15.103:0/1612030158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 msgr2=0x7ffad8110630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 0x7ffad8110630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 -- 172.21.15.103:0/1612030158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4005040 con 0x7ffad80fcf80 2024-03-22T02:27:10.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadd3b3700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 0x7ffad8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:10.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadcbb2700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffac800eb40 tx=0x7ffac800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:10.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffac8007cc0 con 0x7ffad80fcf80 2024-03-22T02:27:10.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffac800f040 con 0x7ffad80fcf80 2024-03-22T02:27:10.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad8113120 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffac800ca10 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.617+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad8113660 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.618+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad805f5f0 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.619+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ffac800cc80 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.619+0000 7ffacdffb700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 0x7ffac406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.619+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffac8086dd0 con 0x7ffad80fcf80 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.619+0000 7ffadd3b3700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 0x7ffac406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:10.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.620+0000 7ffadd3b3700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 0x7ffac406b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffad4004890 tx=0x7ffad4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:10.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.624+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ffac804e190 con 0x7ffad80fcf80 2024-03-22T02:27:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:10 smithi103 ceph-mon[29258]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:10 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:10.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.924+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffad8113980 con 0x7ffad80fcf80 2024-03-22T02:27:10.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.925+0000 7ffacdffb700 1 -- 172.21.15.103:0/1612030158 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7ffac80530e0 con 0x7ffad80fcf80 2024-03-22T02:27:10.925 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:27:10.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:10.927 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:27:10.927 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-22T02:27:10.927 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:27:10.927 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:27:10.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 msgr2=0x7ffac406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 0x7ffac406b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffad4004890 tx=0x7ffad4009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:10.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 msgr2=0x7ffad8112b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffac800eb40 tx=0x7ffac800b040 comp rx=0 tx=0).stop 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 shutdown_connections 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ffac40692a0 0x7ffac406b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffad80fbcd0 0x7ffad8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 --2- 172.21.15.103:0/1612030158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffad80fcf80 0x7ffad8112b80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 >> 172.21.15.103:0/1612030158 conn(0x7ffad80f7430 msgr2=0x7ffad80f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 shutdown_connections 2024-03-22T02:27:10.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:10.928+0000 7ffadf617700 1 -- 172.21.15.103:0/1612030158 wait complete. 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 -- 172.21.15.103:0/99725284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 msgr2=0x7fd0180ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 --2- 172.21.15.103:0/99725284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd0180ffa80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd008004940 tx=0x7fd008009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 -- 172.21.15.103:0/99725284 shutdown_connections 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 --2- 172.21.15.103:0/99725284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd0181023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 --2- 172.21.15.103:0/99725284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd0180ffa80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 -- 172.21.15.103:0/99725284 >> 172.21.15.103:0/99725284 conn(0x7fd0180f7430 msgr2=0x7fd0180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 -- 172.21.15.103:0/99725284 shutdown_connections 2024-03-22T02:27:11.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.050+0000 7fd01d291700 1 -- 172.21.15.103:0/99725284 wait complete. 2024-03-22T02:27:11.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 Processor -- start 2024-03-22T02:27:11.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 -- start start 2024-03-22T02:27:11.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd01810da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:11.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:11.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01810e5b0 con 0x7fd018066220 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd01d291700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01810edb0 con 0x7fd0180fffc0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd0167fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd016ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd01810da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd0167fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:45446/0 (socket says 172.21.15.103:45446) 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd016ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd01810da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41958/0 (socket says 172.21.15.103:41958) 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.051+0000 7fd0167fc700 1 -- 172.21.15.103:0/1223603083 learned_addr learned my addr 172.21.15.103:0/1223603083 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd0167fc700 1 -- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 msgr2=0x7fd01810da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd0167fc700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd01810da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd0167fc700 1 -- 172.21.15.103:0/1223603083 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008005040 con 0x7fd0180fffc0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd0167fc700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd00c010e50 tx=0x7fd00c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd00c015070 con 0x7fd0180fffc0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd00c00be00 con 0x7fd0180fffc0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01810f030 con 0x7fd0180fffc0 2024-03-22T02:27:11.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd00c01b470 con 0x7fd0180fffc0 2024-03-22T02:27:11.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.052+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01810f570 con 0x7fd0180fffc0 2024-03-22T02:27:11.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.054+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01805f5f0 con 0x7fd0180fffc0 2024-03-22T02:27:11.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.054+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd00c00b930 con 0x7fd0180fffc0 2024-03-22T02:27:11.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.054+0000 7fcfff7fe700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 0x7fd00006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:11.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.054+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd00c08b4c0 con 0x7fd0180fffc0 2024-03-22T02:27:11.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.054+0000 7fd016ffd700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 0x7fd00006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:11.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.055+0000 7fd016ffd700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 0x7fd00006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd008004890 tx=0x7fd008009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:11.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.062+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd00c05b1b0 con 0x7fd0180fffc0 2024-03-22T02:27:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:10 smithi122 ceph-mon[36403]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:10 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:11.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.396+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd01810f790 con 0x7fd0000692a0 2024-03-22T02:27:11.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.398+0000 7fcfff7fe700 1 -- 172.21.15.103:0/1223603083 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fd01810f790 con 0x7fd0000692a0 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "0/2 daemons upgraded", 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 image on host smithi122", 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:27:11.399 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 msgr2=0x7fd00006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 0x7fd00006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd008004890 tx=0x7fd008009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 msgr2=0x7fd01810dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd00c010e50 tx=0x7fd00c00b040 comp rx=0 tx=0).stop 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 shutdown_connections 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fd0000692a0 0x7fd00006b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd018066220 0x7fd01810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 --2- 172.21.15.103:0/1223603083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0180fffc0 0x7fd01810dfb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:11.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 >> 172.21.15.103:0/1223603083 conn(0x7fd0180f7430 msgr2=0x7fd0180f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:11.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 shutdown_connections 2024-03-22T02:27:11.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:11.401+0000 7fd01d291700 1 -- 172.21.15.103:0/1223603083 wait complete. 2024-03-22T02:27:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: Upgrade: Target is version 19.0.0-2344-gc3a04c0e (unknown) 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0'] 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: Upgrade: Need to upgrade myself (mgr.smithi103.pfvmui) 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 on smithi122 2024-03-22T02:27:12.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:11 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1612030158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: Upgrade: Target is version 19.0.0-2344-gc3a04c0e (unknown) 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0'] 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: Upgrade: Need to upgrade myself (mgr.smithi103.pfvmui) 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 on smithi122 2024-03-22T02:27:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:11 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1612030158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:27:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:12 smithi122 ceph-mon[36403]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:12 smithi103 ceph-mon[29258]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:14.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:13 smithi122 ceph-mon[36403]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:13 smithi103 ceph-mon[29258]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:16.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:15 smithi122 ceph-mon[36403]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:15 smithi103 ceph-mon[29258]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:17 smithi122 ceph-mon[36403]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:17 smithi103 ceph-mon[29258]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:19 smithi122 ceph-mon[36403]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:19 smithi103 ceph-mon[29258]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:20 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:21.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:20 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:21 smithi122 ceph-mon[36403]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:22.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:21 smithi103 ceph-mon[29258]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:23 smithi122 ceph-mon[36403]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:24.123 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:23 smithi103 ceph-mon[29258]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:25 smithi122 ceph-mon[36403]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:25 smithi103 ceph-mon[29258]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:27 smithi122 ceph-mon[36403]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:27 smithi103 ceph-mon[29258]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:29 smithi122 ceph-mon[36403]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:29 smithi103 ceph-mon[29258]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:31 smithi122 ceph-mon[36403]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:31 smithi103 ceph-mon[29258]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:33 smithi122 ceph-mon[36403]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:34.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:33 smithi103 ceph-mon[29258]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:35 smithi122 ceph-mon[36403]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:35 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:35 smithi103 ceph-mon[29258]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:35 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:38.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:37 smithi122 ceph-mon[36403]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:38.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:37 smithi103 ceph-mon[29258]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:39 smithi122 ceph-mon[36403]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:39 smithi103 ceph-mon[29258]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 -- 172.21.15.103:0/1276327802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fcf80 msgr2=0x7ff8f80fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/1276327802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fcf80 0x7ff8f80fd3f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8004940 tx=0x7ff8e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 -- 172.21.15.103:0/1276327802 shutdown_connections 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/1276327802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fcf80 0x7ff8f80fd3f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/1276327802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 -- 172.21.15.103:0/1276327802 >> 172.21.15.103:0/1276327802 conn(0x7ff8f80f7430 msgr2=0x7ff8f80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 -- 172.21.15.103:0/1276327802 shutdown_connections 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.508+0000 7ff8fe64f700 1 -- 172.21.15.103:0/1276327802 wait complete. 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 Processor -- start 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 -- start start 2024-03-22T02:27:41.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 0x7ff8f810cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f810d350 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8fe64f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f810d4c0 con 0x7ff8f80fcf80 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8f7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8f7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44792/0 (socket says 172.21.15.103:44792) 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8f7fff700 1 -- 172.21.15.103:0/3270739233 learned_addr learned my addr 172.21.15.103:0/3270739233 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.509+0000 7ff8f77fe700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 0x7ff8f810cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f7fff700 1 -- 172.21.15.103:0/3270739233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 msgr2=0x7ff8f810cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f7fff700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 0x7ff8f810cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f7fff700 1 -- 172.21.15.103:0/3270739233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8005040 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f77fe700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 0x7ff8f810cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f7fff700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00eb40 tx=0x7ff8ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8ec007cc0 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8ec00f040 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8111e70 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8ec00ca10 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.510+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f8112320 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.511+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f804c710 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.512+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff8ec00cc80 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.512+0000 7ff8f4ff9700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 0x7ff8e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.512+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff8ec086df0 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.512+0000 7ff8f77fe700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 0x7ff8e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.513+0000 7ff8f77fe700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 0x7ff8e006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009b30 tx=0x7ff8e8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:41.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.517+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff8ec04e190 con 0x7ff8f80fbcd0 2024-03-22T02:27:41.582 INFO:tasks.workunit.client.0.smithi103.stderr:Note: switching to 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'. 2024-03-22T02:27:41.582 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.582 INFO:tasks.workunit.client.0.smithi103.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:state without impacting any branches by switching back to a branch. 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: git switch -c 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:Or undo this operation with: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: git switch - 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-22T02:27:41.583 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-03-22T02:27:41.584 INFO:tasks.workunit.client.0.smithi103.stderr:HEAD is now at c3a04c0e75f Merge PR #54581 into wip-batrick-testing-20240321.172515 2024-03-22T02:27:41.588 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-22T02:27:41.614 INFO:tasks.workunit.client.0.smithi103.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-22T02:27:41.616 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T02:27:41.616 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-22T02:27:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.725+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8f81011c0 con 0x7ff8e00692a0 2024-03-22T02:27:41.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.726+0000 7ff8f4ff9700 1 -- 172.21.15.103:0/3270739233 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ff8f81011c0 con 0x7ff8e00692a0 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 msgr2=0x7ff8e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 0x7ff8e006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009b30 tx=0x7ff8e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 msgr2=0x7ff8f810c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00eb40 tx=0x7ff8ec00b040 comp rx=0 tx=0).stop 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 shutdown_connections 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7ff8e00692a0 0x7ff8e006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8f80fbcd0 0x7ff8f810c810 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 --2- 172.21.15.103:0/3270739233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8f80fcf80 0x7ff8f810cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 >> 172.21.15.103:0/3270739233 conn(0x7ff8f80f7430 msgr2=0x7ff8f80f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:41.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 shutdown_connections 2024-03-22T02:27:41.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.728+0000 7ff8fe64f700 1 -- 172.21.15.103:0/3270739233 wait complete. 2024-03-22T02:27:41.740 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:27:41.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 -- 172.21.15.103:0/1314808101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 msgr2=0x7fe9d80ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 --2- 172.21.15.103:0/1314808101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d80ffa50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8004940 tx=0x7fe9c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 -- 172.21.15.103:0/1314808101 shutdown_connections 2024-03-22T02:27:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 --2- 172.21.15.103:0/1314808101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 0x7fe9d8102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 --2- 172.21.15.103:0/1314808101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d80ffa50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 -- 172.21.15.103:0/1314808101 >> 172.21.15.103:0/1314808101 conn(0x7fe9d80f7440 msgr2=0x7fe9d80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:41.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:41 smithi103 ceph-mon[29258]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 -- 172.21.15.103:0/1314808101 shutdown_connections 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.831+0000 7fe9de086700 1 -- 172.21.15.103:0/1314808101 wait complete. 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 Processor -- start 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 -- start start 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 0x7fe9d810dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d810e5b0 con 0x7fe9d8066af0 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9de086700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d810ee00 con 0x7fe9d80fff90 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9d77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9d77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44808/0 (socket says 172.21.15.103:44808) 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9d77fe700 1 -- 172.21.15.103:0/792932557 learned_addr learned my addr 172.21.15.103:0/792932557 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.832+0000 7fe9d6ffd700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 0x7fe9d810dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9d77fe700 1 -- 172.21.15.103:0/792932557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 msgr2=0x7fe9d810dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9d77fe700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 0x7fe9d810dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9d77fe700 1 -- 172.21.15.103:0/792932557 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8005040 con 0x7fe9d8066af0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9d77fe700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800b970 tx=0x7fe9c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c800e040 con 0x7fe9d8066af0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d810f080 con 0x7fe9d8066af0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9c8004290 con 0x7fe9d8066af0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c801c3f0 con 0x7fe9d8066af0 2024-03-22T02:27:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.833+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d810f5c0 con 0x7fe9d8066af0 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.834+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d805f5f0 con 0x7fe9d8066af0 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.834+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe9c800bbf0 con 0x7fe9d8066af0 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.835+0000 7fe9bffff700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 0x7fe9c006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.835+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe9c80884f0 con 0x7fe9d8066af0 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.835+0000 7fe9d6ffd700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 0x7fe9c006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.835+0000 7fe9d6ffd700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 0x7fe9c006b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc004f60 tx=0x7fe9cc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:41.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:41.839+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe9c8058200 con 0x7fe9d8066af0 2024-03-22T02:27:42.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.047+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9d810f940 con 0x7fe9c00692b0 2024-03-22T02:27:42.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.048+0000 7fe9bffff700 1 -- 172.21.15.103:0/792932557 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe9d810f940 con 0x7fe9c00692b0 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 msgr2=0x7fe9c006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 0x7fe9c006b760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc004f60 tx=0x7fe9cc00b040 comp rx=0 tx=0).stop 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 msgr2=0x7fe9d810da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800b970 tx=0x7fe9c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 shutdown_connections 2024-03-22T02:27:42.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fe9c00692b0 0x7fe9c006b760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9d8066af0 0x7fe9d810da70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 --2- 172.21.15.103:0/792932557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9d80fff90 0x7fe9d810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 >> 172.21.15.103:0/792932557 conn(0x7fe9d80f7440 msgr2=0x7fe9d80f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 shutdown_connections 2024-03-22T02:27:42.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.050+0000 7fe9de086700 1 -- 172.21.15.103:0/792932557 wait complete. 2024-03-22T02:27:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:41 smithi122 ceph-mon[36403]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 -- 172.21.15.103:0/1613413495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040feda0 msgr2=0x7f7e040faee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 --2- 172.21.15.103:0/1613413495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040feda0 0x7f7e040faee0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009ce0 tx=0x7f7df8004d10 comp rx=0 tx=0).stop 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 -- 172.21.15.103:0/1613413495 shutdown_connections 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 --2- 172.21.15.103:0/1613413495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040feda0 0x7f7e040faee0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 --2- 172.21.15.103:0/1613413495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040fe450 0x7f7e040fe860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 -- 172.21.15.103:0/1613413495 >> 172.21.15.103:0/1613413495 conn(0x7f7e040f6c10 msgr2=0x7f7e040f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 -- 172.21.15.103:0/1613413495 shutdown_connections 2024-03-22T02:27:42.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.153+0000 7f7e094a8700 1 -- 172.21.15.103:0/1613413495 wait complete. 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 Processor -- start 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 -- start start 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040feda0 0x7f7e0410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0410e5e0 con 0x7f7e040fe450 2024-03-22T02:27:42.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e094a8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0410e750 con 0x7f7e040feda0 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44828/0 (socket says 172.21.15.103:44828) 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 -- 172.21.15.103:0/2029936880 learned_addr learned my addr 172.21.15.103:0/2029936880 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e027fc700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040feda0 0x7f7e0410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 -- 172.21.15.103:0/2029936880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040feda0 msgr2=0x7f7e0410dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040feda0 0x7f7e0410dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 -- 172.21.15.103:0/2029936880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8009940 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.154+0000 7f7e02ffd700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc002a50 tx=0x7f7dfc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.155+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dfc007cc0 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.155+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dfc00f040 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.155+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e041131b0 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.155+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dfc00da30 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.155+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e04113770 con 0x7f7e040fe450 2024-03-22T02:27:42.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.156+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0404c710 con 0x7f7e040fe450 2024-03-22T02:27:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.156+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7dfc00db90 con 0x7f7e040fe450 2024-03-22T02:27:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.157+0000 7f7deb7fe700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 0x7f7dec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.157+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7dfc086f90 con 0x7f7e040fe450 2024-03-22T02:27:42.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.157+0000 7f7e027fc700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 0x7f7dec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.157+0000 7f7e027fc700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 0x7f7dec06b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7df8015040 tx=0x7f7df8004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.161+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7dfc056c80 con 0x7f7e040fe450 2024-03-22T02:27:42.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.362+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e041024f0 con 0x7f7dec0692a0 2024-03-22T02:27:42.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.372+0000 7f7deb7fe700 1 -- 172.21.15.103:0/2029936880 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f7e041024f0 con 0x7f7dec0692a0 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (6m) 90s ago 7m 24.8M - 0.25.0 c8568f914cd2 31b45ef21cbd 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (7m) 90s ago 7m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (6m) 95s ago 6m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (5m) 90s ago 7m 84.7M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (116s) 90s ago 115s 10.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (109s) 90s ago 108s 14.1M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (106s) 95s ago 106s 11.9M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (113s) 95s ago 112s 16.8M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (8m) 90s ago 8m 481M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (6m) 95s ago 6m 432M - 17.2.7-726-g9c4e0c34 837124da804f 29f8988d41c7 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (8m) 90s ago 9m 48.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:27:42.374 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (6m) 95s ago 6m 36.4M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (7m) 90s ago 7m 16.2M - 1.5.0 0da6a335fe13 136d30f21984 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 95s ago 6m 16.4M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (5m) 90s ago 5m 45.4M 4288M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (5m) 90s ago 5m 45.3M 4288M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (4m) 90s ago 4m 47.6M 4288M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (4m) 95s ago 4m 45.6M 5312M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (3m) 95s ago 3m 44.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (3m) 95s ago 3m 41.3M 5312M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:27:42.375 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (5m) 90s ago 6m 54.3M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:27:42.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 msgr2=0x7f7dec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 0x7f7dec06b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7df8015040 tx=0x7f7df8004d10 comp rx=0 tx=0).stop 2024-03-22T02:27:42.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 msgr2=0x7f7e0410daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc002a50 tx=0x7f7dfc0046b0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 shutdown_connections 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f7dec0692a0 0x7f7dec06b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e040fe450 0x7f7e0410daa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 --2- 172.21.15.103:0/2029936880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7e040feda0 0x7f7e0410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 >> 172.21.15.103:0/2029936880 conn(0x7f7e040f6c10 msgr2=0x7f7e040f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 shutdown_connections 2024-03-22T02:27:42.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.374+0000 7f7e094a8700 1 -- 172.21.15.103:0/2029936880 wait complete. 2024-03-22T02:27:42.408 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-22T02:27:42.475 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-22T02:27:42.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 -- 172.21.15.103:0/2940555072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 msgr2=0x7fecc80fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 --2- 172.21.15.103:0/2940555072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc80fd410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fecbc004940 tx=0x7fecbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:42.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 -- 172.21.15.103:0/2940555072 shutdown_connections 2024-03-22T02:27:42.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 --2- 172.21.15.103:0/2940555072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc80fd410 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 --2- 172.21.15.103:0/2940555072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 0x7fecc80fc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 -- 172.21.15.103:0/2940555072 >> 172.21.15.103:0/2940555072 conn(0x7fecc80f7480 msgr2=0x7fecc80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 -- 172.21.15.103:0/2940555072 shutdown_connections 2024-03-22T02:27:42.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.478+0000 7feccf5bd700 1 -- 172.21.15.103:0/2940555072 wait complete. 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 Processor -- start 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 -- start start 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 0x7fecc81106f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8066b90 con 0x7fecc80fcfa0 2024-03-22T02:27:42.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7feccf5bd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8066cd0 con 0x7fecc80fbe30 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.479+0000 7fecccb58700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44852/0 (socket says 172.21.15.103:44852) 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 -- 172.21.15.103:0/750232986 learned_addr learned my addr 172.21.15.103:0/750232986 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7feccd359700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 0x7fecc81106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 -- 172.21.15.103:0/750232986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 msgr2=0x7fecc81106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 0x7fecc81106f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 -- 172.21.15.103:0/750232986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc005040 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecccb58700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009150 tx=0x7fecbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecbc00e040 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc8113180 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecbc004290 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.480+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecbc01c630 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.481+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc81136f0 con 0x7fecc80fcfa0 2024-03-22T02:27:42.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.482+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc805f5f0 con 0x7fecc80fcfa0 2024-03-22T02:27:42.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.482+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fecbc00bbf0 con 0x7fecc80fcfa0 2024-03-22T02:27:42.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.483+0000 7fecc5ffb700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 0x7fecb006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.483+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fecbc087f50 con 0x7fecc80fcfa0 2024-03-22T02:27:42.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.483+0000 7feccd359700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 0x7fecb006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.483+0000 7feccd359700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 0x7fecb006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fecc8066810 tx=0x7fecc0009e20 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.487+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fecbc054190 con 0x7fecc80fcfa0 2024-03-22T02:27:42.517 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T02:27:42.518 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T02:27:42.518 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-22T02:27:42.558 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T02:27:42.561 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:27:42.561 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-22T02:27:42.617 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-22T02:27:42.620 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-22T02:27:42.620 DEBUG:teuthology.orchestra.run.smithi103:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-22T02:27:42.683 INFO:tasks.workunit.client.0.smithi103.stderr:+ mkdir -p fsstress 2024-03-22T02:27:42.685 INFO:tasks.workunit.client.0.smithi103.stderr:+ pushd fsstress 2024-03-22T02:27:42.686 INFO:tasks.workunit.client.0.smithi103.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-22T02:27:42.687 INFO:tasks.workunit.client.0.smithi103.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-22T02:27:42.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.770+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fecc804bed0 con 0x7fecc80fcfa0 2024-03-22T02:27:42.773 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:42 smithi103 ceph-mon[29258]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:42.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.770+0000 7fecc5ffb700 1 -- 172.21.15.103:0/750232986 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fecbc0577b0 con 0x7fecc80fcfa0 2024-03-22T02:27:42.774 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:27:42.774 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:27:42.774 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:27:42.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:27:42.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 msgr2=0x7fecb006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 0x7fecb006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fecc8066810 tx=0x7fecc0009e20 comp rx=0 tx=0).stop 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 msgr2=0x7fecc8112c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009150 tx=0x7fecbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 shutdown_connections 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fecb00692a0 0x7fecb006b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc80fbe30 0x7fecc81106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 --2- 172.21.15.103:0/750232986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc80fcfa0 0x7fecc8112c40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.772+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 >> 172.21.15.103:0/750232986 conn(0x7fecc80f7480 msgr2=0x7fecc80f8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.773+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 shutdown_connections 2024-03-22T02:27:42.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.773+0000 7feccf5bd700 1 -- 172.21.15.103:0/750232986 wait complete. 2024-03-22T02:27:42.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 -- 172.21.15.103:0/2845761099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fbce0 msgr2=0x7f9a040fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/2845761099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fbce0 0x7f9a040fc0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f99f4004940 tx=0x7f99f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:42.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 -- 172.21.15.103:0/2845761099 shutdown_connections 2024-03-22T02:27:42.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/2845761099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fcf90 0x7f9a040fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/2845761099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fbce0 0x7f9a040fc0f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 -- 172.21.15.103:0/2845761099 >> 172.21.15.103:0/2845761099 conn(0x7f9a040f7480 msgr2=0x7f9a040f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:42.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 -- 172.21.15.103:0/2845761099 shutdown_connections 2024-03-22T02:27:42.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.878+0000 7f9a09cc5700 1 -- 172.21.15.103:0/2845761099 wait complete. 2024-03-22T02:27:42.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 Processor -- start 2024-03-22T02:27:42.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 -- start start 2024-03-22T02:27:42.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fbce0 0x7f9a04066d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04112c40 con 0x7f9a040fcf90 2024-03-22T02:27:42.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a09cc5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04112d80 con 0x7f9a040fbce0 2024-03-22T02:27:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44874/0 (socket says 172.21.15.103:44874) 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a02ffd700 1 -- 172.21.15.103:0/74947220 learned_addr learned my addr 172.21.15.103:0/74947220 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.879+0000 7f9a037fe700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fbce0 0x7f9a04066d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a02ffd700 1 -- 172.21.15.103:0/74947220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fbce0 msgr2=0x7f9a04066d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a02ffd700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fbce0 0x7f9a04066d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a02ffd700 1 -- 172.21.15.103:0/74947220 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f4005040 con 0x7f9a040fcf90 2024-03-22T02:27:42.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a02ffd700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f99f8002a50 tx=0x7f99f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f8007cc0 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99f800f040 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a04113060 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f800da70 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.880+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a04113620 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.881+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0404c710 con 0x7f9a040fcf90 2024-03-22T02:27:42.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.881+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f99f800dc10 con 0x7f9a040fcf90 2024-03-22T02:27:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.882+0000 7f99ebfff700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 0x7f99ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.882+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f99f8086870 con 0x7f9a040fcf90 2024-03-22T02:27:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.882+0000 7f9a037fe700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 0x7f99ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.882+0000 7f9a037fe700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 0x7f99ec06b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f400b970 tx=0x7f99f4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:42.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:42.886+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f99f8056560 con 0x7f9a040fcf90 2024-03-22T02:27:43.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.094+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a041012a0 con 0x7f99ec0692a0 2024-03-22T02:27:43.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.095+0000 7f99ebfff700 1 -- 172.21.15.103:0/74947220 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f9a041012a0 con 0x7f99ec0692a0 2024-03-22T02:27:43.096 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "0/2 daemons upgraded", 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 image on host smithi122", 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:27:43.097 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:27:43.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:42 smithi122 ceph-mon[36403]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 msgr2=0x7f99ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 0x7f99ec06b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f400b970 tx=0x7f99f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:27:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 msgr2=0x7f9a04112700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:43.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f99f8002a50 tx=0x7f99f80046b0 comp rx=0 tx=0).stop 2024-03-22T02:27:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 shutdown_connections 2024-03-22T02:27:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f99ec0692a0 0x7f99ec06b750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a040fbce0 0x7f9a04066d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 --2- 172.21.15.103:0/74947220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a040fcf90 0x7f9a04112700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:43.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 >> 172.21.15.103:0/74947220 conn(0x7f9a040f7480 msgr2=0x7f9a040f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 shutdown_connections 2024-03-22T02:27:43.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:27:43.097+0000 7f9a09cc5700 1 -- 172.21.15.103:0/74947220 wait complete. 2024-03-22T02:27:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:43 smithi122 ceph-mon[36403]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:43 smithi122 ceph-mon[36403]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:43 smithi122 ceph-mon[36403]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:44.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:43 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/750232986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:27:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:43 smithi103 ceph-mon[29258]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:43 smithi103 ceph-mon[29258]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:43 smithi103 ceph-mon[29258]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:43 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/750232986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:27:44.175 INFO:tasks.workunit.client.0.smithi103.stderr:+ tar xzf ltp-full.tgz 2024-03-22T02:27:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:44 smithi122 ceph-mon[36403]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:44 smithi103 ceph-mon[29258]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:27:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:45 smithi122 ceph-mon[36403]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:46.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:45 smithi103 ceph-mon[29258]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:27:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:47 smithi122 ceph-mon[36403]: pgmap v242: 65 pgs: 65 active+clean; 4.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 370 KiB/s wr, 16 op/s 2024-03-22T02:27:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:47 smithi103 ceph-mon[29258]: pgmap v242: 65 pgs: 65 active+clean; 4.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 370 KiB/s wr, 16 op/s 2024-03-22T02:27:48.660 INFO:tasks.workunit.client.1.smithi122.stderr:Updating files: 87% (10782/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-22T02:27:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: Upgrade: Updating mgr.smithi122.prgtti 2024-03-22T02:27:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:27:49.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:27:49.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:49.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: Deploying daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:27:49.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:49 smithi122 ceph-mon[36403]: pgmap v243: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s wr, 72 op/s 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:Note: switching to 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'. 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:state without impacting any branches by switching back to a branch. 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr: git switch -c 2024-03-22T02:27:49.372 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr:Or undo this operation with: 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr: git switch - 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr: 2024-03-22T02:27:49.373 INFO:tasks.workunit.client.1.smithi122.stderr:HEAD is now at c3a04c0e75f Merge PR #54581 into wip-batrick-testing-20240321.172515 2024-03-22T02:27:49.377 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-22T02:27:49.477 INFO:tasks.workunit.client.1.smithi122.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-22T02:27:49.479 INFO:tasks.workunit.client.1.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-22T02:27:49.479 INFO:tasks.workunit.client.1.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: Upgrade: Updating mgr.smithi122.prgtti 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: Deploying daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:27:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:49 smithi103 ceph-mon[29258]: pgmap v243: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s wr, 72 op/s 2024-03-22T02:27:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:50 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:50 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:27:51.139 INFO:tasks.workunit.client.1.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-22T02:27:51.261 INFO:tasks.workunit.client.1.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-22T02:27:51.305 INFO:tasks.workunit.client.1.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-22T02:27:51.307 INFO:tasks.workunit.client.1.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-22T02:27:51.307 INFO:tasks.workunit.client.1.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-22T02:27:51.363 INFO:tasks.workunit.client.1.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-22T02:27:51.366 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-22T02:27:51.366 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-22T02:27:51.382 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-22T02:27:51.383 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-22T02:27:51.383 DEBUG:teuthology.orchestra.run.smithi122:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-22T02:27:51.446 INFO:tasks.workunit.client.1.smithi122.stderr:+ mkdir -p fsstress 2024-03-22T02:27:51.449 INFO:tasks.workunit.client.1.smithi122.stderr:+ pushd fsstress 2024-03-22T02:27:51.450 INFO:tasks.workunit.client.1.smithi122.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-22T02:27:51.450 INFO:tasks.workunit.client.1.smithi122.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-22T02:27:51.799 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:51 smithi122 ceph-mon[36403]: pgmap v244: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 97 op/s 2024-03-22T02:27:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:51 smithi103 ceph-mon[29258]: pgmap v244: 65 pgs: 65 active+clean; 16 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 97 op/s 2024-03-22T02:27:52.138 INFO:tasks.workunit.client.1.smithi122.stderr:+ tar xzf ltp-full.tgz 2024-03-22T02:27:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:53 smithi103 ceph-mon[29258]: pgmap v245: 65 pgs: 65 active+clean; 22 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s wr, 187 op/s 2024-03-22T02:27:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:53 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:53 smithi122 ceph-mon[36403]: pgmap v245: 65 pgs: 65 active+clean; 22 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s wr, 187 op/s 2024-03-22T02:27:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:27:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:53 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:56.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:55 smithi122 ceph-mon[36403]: pgmap v246: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s wr, 225 op/s 2024-03-22T02:27:56.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:55 smithi103 ceph-mon[29258]: pgmap v246: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s wr, 225 op/s 2024-03-22T02:27:58.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:57 smithi122 ceph-mon[36403]: pgmap v247: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s wr, 288 op/s 2024-03-22T02:27:58.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:57 smithi103 ceph-mon[29258]: pgmap v247: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s wr, 288 op/s 2024-03-22T02:27:59.998 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:59 smithi122 ceph-mon[36403]: pgmap v248: 65 pgs: 65 active+clean; 62 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.0 MiB/s wr, 407 op/s 2024-03-22T02:27:59.999 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:27:59 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:28:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:59 smithi103 ceph-mon[29258]: pgmap v248: 65 pgs: 65 active+clean; 62 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.0 MiB/s wr, 407 op/s 2024-03-22T02:28:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:27:59 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:28:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: pgmap v249: 65 pgs: 65 active+clean; 63 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 382 op/s 2024-03-22T02:28:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:28:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:28:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:28:02.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:28:02.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:28:02.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:01 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: pgmap v249: 65 pgs: 65 active+clean; 63 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 382 op/s 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:28:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:01 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:28:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:02 smithi122 ceph-mon[36403]: mgrmap e20: smithi103.pfvmui(active, since 7m), standbys: smithi122.prgtti 2024-03-22T02:28:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:02 smithi103 ceph-mon[29258]: mgrmap e20: smithi103.pfvmui(active, since 7m), standbys: smithi122.prgtti 2024-03-22T02:28:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:03 smithi122 ceph-mon[36403]: pgmap v250: 65 pgs: 65 active+clean; 75 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 467 op/s 2024-03-22T02:28:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:03 smithi103 ceph-mon[29258]: pgmap v250: 65 pgs: 65 active+clean; 75 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 467 op/s 2024-03-22T02:28:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:05 smithi122 ceph-mon[36403]: pgmap v251: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 425 op/s 2024-03-22T02:28:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:05 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:05 smithi103 ceph-mon[29258]: pgmap v251: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 425 op/s 2024-03-22T02:28:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:05 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:07 smithi122 ceph-mon[36403]: pgmap v252: 65 pgs: 65 active+clean; 88 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 449 op/s 2024-03-22T02:28:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:07 smithi103 ceph-mon[29258]: pgmap v252: 65 pgs: 65 active+clean; 88 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 449 op/s 2024-03-22T02:28:09.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:09 smithi122 ceph-mon[36403]: pgmap v253: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s wr, 500 op/s 2024-03-22T02:28:09.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:09 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:09 smithi103 ceph-mon[29258]: pgmap v253: 65 pgs: 65 active+clean; 93 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s wr, 500 op/s 2024-03-22T02:28:09.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:09.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:09 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:11 smithi122 ceph-mon[36403]: pgmap v254: 65 pgs: 65 active+clean; 100 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 401 op/s 2024-03-22T02:28:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:11 smithi103 ceph-mon[29258]: pgmap v254: 65 pgs: 65 active+clean; 100 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 401 op/s 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 -- 172.21.15.103:0/969932821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 msgr2=0x7fa4ac0fd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 --2- 172.21.15.103:0/969932821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac0fd090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa498004940 tx=0x7fa498009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 -- 172.21.15.103:0/969932821 shutdown_connections 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 --2- 172.21.15.103:0/969932821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac0fd090 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 --2- 172.21.15.103:0/969932821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac0fc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 -- 172.21.15.103:0/969932821 >> 172.21.15.103:0/969932821 conn(0x7fa4ac0f6ba0 msgr2=0x7fa4ac0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 -- 172.21.15.103:0/969932821 shutdown_connections 2024-03-22T02:28:13.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.248+0000 7fa4b3259700 1 -- 172.21.15.103:0/969932821 wait complete. 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 Processor -- start 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 -- start start 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac111660 con 0x7fa4ac0fcc20 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b3259700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1117d0 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.249+0000 7fa4b0ff5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:41000/0 (socket says 172.21.15.103:41000) 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 -- 172.21.15.103:0/4168729359 learned_addr learned my addr 172.21.15.103:0/4168729359 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4abfff700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 -- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 msgr2=0x7fa4ac1023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac1023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 -- 172.21.15.103:0/4168729359 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498005040 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b0ff5700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000eb40 tx=0x7fa4a000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a0007cc0 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1029f0 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.250+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac102ee0 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.251+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4a000f040 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.251+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a000ca10 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.252+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4900052e0 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.253+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 20) v1 ==== 89278+0+0 (secure 0 0 0) 0x7fa4a000cc80 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.253+0000 7fa4a97fa700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 0x7fa49c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.253+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa4a0089d30 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.253+0000 7fa4abfff700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 0x7fa49c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.254+0000 7fa4abfff700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 0x7fa49c06dc50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa498009b30 tx=0x7fa498009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:13.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.259+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa4a0056040 con 0x7fa4ac0fc2d0 2024-03-22T02:28:13.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.563+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa490000bf0 con 0x7fa49c06b7a0 2024-03-22T02:28:13.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.565+0000 7fa4a97fa700 1 -- 172.21.15.103:0/4168729359 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fa490000bf0 con 0x7fa49c06b7a0 2024-03-22T02:28:13.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.568+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 msgr2=0x7fa49c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.568+0000 7fa4b3259700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 0x7fa49c06dc50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa498009b30 tx=0x7fa498009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.568+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 msgr2=0x7fa4ac103f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.568+0000 7fa4b3259700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000eb40 tx=0x7fa4a000b040 comp rx=0 tx=0).stop 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 shutdown_connections 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fa49c06b7a0 0x7fa49c06dc50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc2d0 0x7fa4ac103f20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 --2- 172.21.15.103:0/4168729359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fcc20 0x7fa4ac1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 >> 172.21.15.103:0/4168729359 conn(0x7fa4ac0f6ba0 msgr2=0x7fa4ac0f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 shutdown_connections 2024-03-22T02:28:13.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.569+0000 7fa4b3259700 1 -- 172.21.15.103:0/4168729359 wait complete. 2024-03-22T02:28:13.586 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 -- 172.21.15.103:0/991424478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 msgr2=0x7f49140fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 --2- 172.21.15.103:0/991424478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49140fd7c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f490c009ce0 tx=0x7f490c004d10 comp rx=0 tx=0).stop 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 -- 172.21.15.103:0/991424478 shutdown_connections 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 --2- 172.21.15.103:0/991424478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 0x7f4914100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 --2- 172.21.15.103:0/991424478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49140fd7c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 -- 172.21.15.103:0/991424478 >> 172.21.15.103:0/991424478 conn(0x7f49140f7440 msgr2=0x7f49140f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 -- 172.21.15.103:0/991424478 shutdown_connections 2024-03-22T02:28:13.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.725+0000 7f491b6fa700 1 -- 172.21.15.103:0/991424478 wait complete. 2024-03-22T02:28:13.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 Processor -- start 2024-03-22T02:28:13.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 -- start start 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 0x7f4914114600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4914114c00 con 0x7f4914066b70 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f491b6fa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4914115450 con 0x7f49140fdd00 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38924/0 (socket says 172.21.15.103:38924) 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 -- 172.21.15.103:0/1283337736 learned_addr learned my addr 172.21.15.103:0/1283337736 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4918c95700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 0x7f4914114600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 -- 172.21.15.103:0/1283337736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 msgr2=0x7f4914114600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 0x7f4914114600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.726+0000 7f4919496700 1 -- 172.21.15.103:0/1283337736 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c009940 con 0x7f4914066b70 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f4919496700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f490c009380 tx=0x7f490c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c00e040 con 0x7f4914066b70 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f490c01cb20 con 0x7f4914066b70 2024-03-22T02:28:13.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49141156d0 con 0x7f4914066b70 2024-03-22T02:28:13.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f490c012b10 con 0x7f4914066b70 2024-03-22T02:28:13.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.727+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4914115c10 con 0x7f4914066b70 2024-03-22T02:28:13.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.728+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491405f5f0 con 0x7f4914066b70 2024-03-22T02:28:13.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.729+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 20) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f490c01cc90 con 0x7f4914066b70 2024-03-22T02:28:13.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.729+0000 7f4909ffb700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 0x7f490006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:13.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.730+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f490c08c3f0 con 0x7f4914066b70 2024-03-22T02:28:13.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.730+0000 7f4918c95700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 0x7f490006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:13.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.730+0000 7f4918c95700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 0x7f490006dc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4910009b80 tx=0x7f491000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:13.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:13.736+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f490c05c160 con 0x7f4914066b70 2024-03-22T02:28:13.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:13 smithi122 ceph-mon[36403]: pgmap v255: 65 pgs: 65 active+clean; 118 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 515 op/s 2024-03-22T02:28:13.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:13 smithi103 ceph-mon[29258]: pgmap v255: 65 pgs: 65 active+clean; 118 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 515 op/s 2024-03-22T02:28:14.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.073+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4914115fb0 con 0x7f490006b7b0 2024-03-22T02:28:14.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.075+0000 7f4909ffb700 1 -- 172.21.15.103:0/1283337736 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f4914115fb0 con 0x7f490006b7b0 2024-03-22T02:28:14.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 msgr2=0x7f490006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 0x7f490006dc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4910009b80 tx=0x7f491000b040 comp rx=0 tx=0).stop 2024-03-22T02:28:14.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 msgr2=0x7f49141140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f490c009380 tx=0x7f490c004d10 comp rx=0 tx=0).stop 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 shutdown_connections 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f490006b7b0 0x7f490006dc60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4914066b70 0x7f49141140c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 --2- 172.21.15.103:0/1283337736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49140fdd00 0x7f4914114600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 >> 172.21.15.103:0/1283337736 conn(0x7f49140f7440 msgr2=0x7f49140f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 shutdown_connections 2024-03-22T02:28:14.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.078+0000 7f491b6fa700 1 -- 172.21.15.103:0/1283337736 wait complete. 2024-03-22T02:28:14.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 -- 172.21.15.103:0/3027221196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fbcd0 msgr2=0x7f2a240fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 --2- 172.21.15.103:0/3027221196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fbcd0 0x7f2a240fc0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2a14004940 tx=0x7f2a14009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 -- 172.21.15.103:0/3027221196 shutdown_connections 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 --2- 172.21.15.103:0/3027221196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fcf80 0x7f2a240fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 --2- 172.21.15.103:0/3027221196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fbcd0 0x7f2a240fc0e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.225+0000 7f2a2910d700 1 -- 172.21.15.103:0/3027221196 >> 172.21.15.103:0/3027221196 conn(0x7f2a240f7430 msgr2=0x7f2a240f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.226+0000 7f2a2910d700 1 -- 172.21.15.103:0/3027221196 shutdown_connections 2024-03-22T02:28:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.226+0000 7f2a2910d700 1 -- 172.21.15.103:0/3027221196 wait complete. 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.226+0000 7f2a2910d700 1 Processor -- start 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.226+0000 7f2a2910d700 1 -- start start 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2910d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2910d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2910d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a24111600 con 0x7f2a240fcf80 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2910d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a24111770 con 0x7f2a240fbcd0 2024-03-22T02:28:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:55562/0 (socket says 172.21.15.103:55562) 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 -- 172.21.15.103:0/1343426899 learned_addr learned my addr 172.21.15.103:0/1343426899 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38942/0 (socket says 172.21.15.103:38942) 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 -- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 msgr2=0x7f2a240fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 -- 172.21.15.103:0/1343426899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a14005040 con 0x7f2a240fbcd0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a2259c700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.227+0000 7f2a22d9d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a14013870 tx=0x7f2a14009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.228+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a14021070 con 0x7f2a240fbcd0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.228+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240fe8c0 con 0x7f2a240fbcd0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.228+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a14009430 con 0x7f2a240fbcd0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.228+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1400e070 con 0x7f2a240fbcd0 2024-03-22T02:28:14.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.228+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240fedb0 con 0x7f2a240fbcd0 2024-03-22T02:28:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.229+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2405f5f0 con 0x7f2a240fbcd0 2024-03-22T02:28:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.229+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 20) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f2a1401c750 con 0x7f2a240fbcd0 2024-03-22T02:28:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.230+0000 7f2a0b7fe700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 0x7f2a0c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.230+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2a1408b380 con 0x7f2a240fbcd0 2024-03-22T02:28:14.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.230+0000 7f2a2259c700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 0x7f2a0c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.230+0000 7f2a2259c700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 0x7f2a0c06dbe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009b80 tx=0x7f2a1800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:14.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.235+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2a14057600 con 0x7f2a240fbcd0 2024-03-22T02:28:14.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.564+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a241047c0 con 0x7f2a0c06b730 2024-03-22T02:28:14.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.580+0000 7f2a0b7fe700 1 -- 172.21.15.103:0/1343426899 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f2a241047c0 con 0x7f2a0c06b730 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (6m) 2m ago 7m 24.8M - 0.25.0 c8568f914cd2 31b45ef21cbd 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (8m) 2m ago 8m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (7m) 6s ago 7m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (6m) 2m ago 7m 84.7M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (2m) 2m ago 2m 10.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (2m) 2m ago 2m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (2m) 6s ago 2m 12.4M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (2m) 6s ago 2m 20.5M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:28:14.582 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (9m) 2m ago 9m 481M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (22s) 6s ago 6m 48.3M - 19.0.0-2344-gc3a04c0e 9d92c22378ed c01e6142c155 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (9m) 2m ago 9m 48.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (6m) 6s ago 6m 36.6M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (8m) 2m ago 8m 16.2M - 1.5.0 0da6a335fe13 136d30f21984 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 6s ago 7m 16.2M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (6m) 2m ago 6m 45.4M 4288M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (5m) 2m ago 5m 45.3M 4288M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (5m) 2m ago 5m 47.6M 4288M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (4m) 6s ago 4m 67.9M 5312M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (4m) 6s ago 4m 66.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (3m) 6s ago 3m 63.2M 5312M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:28:14.583 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (6m) 2m ago 7m 54.3M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:28:14.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 msgr2=0x7f2a0c06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 0x7f2a0c06dbe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009b80 tx=0x7f2a1800b040 comp rx=0 tx=0).stop 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 msgr2=0x7f2a240ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a14013870 tx=0x7f2a14009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 shutdown_connections 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f2a0c06b730 0x7f2a0c06dbe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a240fbcd0 0x7f2a240ffdf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 --2- 172.21.15.103:0/1343426899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a240fcf80 0x7f2a240fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 >> 172.21.15.103:0/1343426899 conn(0x7f2a240f7430 msgr2=0x7f2a240f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 shutdown_connections 2024-03-22T02:28:14.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.583+0000 7f2a2910d700 1 -- 172.21.15.103:0/1343426899 wait complete. 2024-03-22T02:28:14.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.717+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2037260298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fcf80 msgr2=0x7fc1740fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.717+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2037260298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fcf80 0x7fc1740fd3f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc164004940 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2037260298 shutdown_connections 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2037260298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fcf80 0x7fc1740fd3f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2037260298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fbcd0 0x7fc1740fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2037260298 >> 172.21.15.103:0/2037260298 conn(0x7fc1740f7430 msgr2=0x7fc1740f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2037260298 shutdown_connections 2024-03-22T02:28:14.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2037260298 wait complete. 2024-03-22T02:28:14.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.718+0000 7fc17bdf0700 1 Processor -- start 2024-03-22T02:28:14.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17bdf0700 1 -- start start 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17bdf0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17bdf0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fcf80 0x7fc174115070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17bdf0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc174066240 con 0x7fc1740fbcd0 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17bdf0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc174066380 con 0x7fc1740fcf80 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38968/0 (socket says 172.21.15.103:38968) 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 -- 172.21.15.103:0/2908184128 learned_addr learned my addr 172.21.15.103:0/2908184128 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc17938b700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fcf80 0x7fc174115070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 -- 172.21.15.103:0/2908184128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fcf80 msgr2=0x7fc174115070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fcf80 0x7fc174115070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 -- 172.21.15.103:0/2908184128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc1740fbcd0 2024-03-22T02:28:14.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.719+0000 7fc179b8c700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc170002a50 tx=0x7fc1700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:14.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.720+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc170007cc0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.720+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc17000f040 con 0x7fc1740fbcd0 2024-03-22T02:28:14.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.720+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1741155b0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.720+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17000da30 con 0x7fc1740fbcd0 2024-03-22T02:28:14.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.720+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1741159d0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.721+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17405f5f0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.721+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 20) v1 ==== 89278+0+0 (secure 0 0 0) 0x7fc17000dbb0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.722+0000 7fc16a7fc700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 0x7fc16006dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.722+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc170089eb0 con 0x7fc1740fbcd0 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.722+0000 7fc17938b700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 0x7fc16006dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:14.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.722+0000 7fc17938b700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 0x7fc16006dc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b3b0 tx=0x7fc164009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:14.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:14.726+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc170056190 con 0x7fc1740fbcd0 2024-03-22T02:28:14.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:14 smithi122 ceph-mon[36403]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:14.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:14 smithi103 ceph-mon[29258]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.111+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc174115d70 con 0x7fc1740fbcd0 2024-03-22T02:28:15.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.112+0000 7fc16a7fc700 1 -- 172.21.15.103:0/2908184128 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7fc1700597b0 con 0x7fc1740fbcd0 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:15.114 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:28:15.115 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 msgr2=0x7fc16006dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 0x7fc16006dc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b3b0 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 msgr2=0x7fc174112b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc170002a50 tx=0x7fc1700046b0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 shutdown_connections 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7fc16006b7a0 0x7fc16006dc50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fbcd0 0x7fc174112b20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 --2- 172.21.15.103:0/2908184128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fcf80 0x7fc174115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 >> 172.21.15.103:0/2908184128 conn(0x7fc1740f7430 msgr2=0x7fc1740f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:15.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 shutdown_connections 2024-03-22T02:28:15.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.115+0000 7fc17bdf0700 1 -- 172.21.15.103:0/2908184128 wait complete. 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 -- 172.21.15.103:0/868220878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fd240 msgr2=0x7f47d80fd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/868220878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fd240 0x7f47d80fd650 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f47c8004940 tx=0x7f47c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 -- 172.21.15.103:0/868220878 shutdown_connections 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/868220878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fdb90 0x7f47d80fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/868220878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fd240 0x7f47d80fd650 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 -- 172.21.15.103:0/868220878 >> 172.21.15.103:0/868220878 conn(0x7f47d80f6b20 msgr2=0x7f47d80f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 -- 172.21.15.103:0/868220878 shutdown_connections 2024-03-22T02:28:15.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.258+0000 7f47dd4a7700 1 -- 172.21.15.103:0/868220878 wait complete. 2024-03-22T02:28:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 Processor -- start 2024-03-22T02:28:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 -- start start 2024-03-22T02:28:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:15.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d810e5b0 con 0x7f47d80fdb90 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47dd4a7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d810ed60 con 0x7f47d80fd240 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d67fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d6ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d6ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:55580/0 (socket says 172.21.15.103:55580) 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d67fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38990/0 (socket says 172.21.15.103:38990) 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d6ffd700 1 -- 172.21.15.103:0/220891834 learned_addr learned my addr 172.21.15.103:0/220891834 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d6ffd700 1 -- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 msgr2=0x7f47d810dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.259+0000 7f47d6ffd700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47d6ffd700 1 -- 172.21.15.103:0/220891834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8005040 con 0x7f47d80fd240 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47d67fc700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47d6ffd700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47c8013870 tx=0x7f47c8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c800e070 con 0x7f47d80fd240 2024-03-22T02:28:15.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47c8013de0 con 0x7f47d80fd240 2024-03-22T02:28:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47c801cea0 con 0x7f47d80fd240 2024-03-22T02:28:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d810efe0 con 0x7f47d80fd240 2024-03-22T02:28:15.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.260+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d810f4c0 con 0x7f47d80fd240 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.261+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d805f5f0 con 0x7f47d80fd240 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.262+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 20) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f47c801c690 con 0x7f47d80fd240 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.262+0000 7f47bf7fe700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 0x7f47c0072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.262+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f47c808b8f0 con 0x7f47d80fd240 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.262+0000 7f47d67fc700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 0x7f47c0072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:15.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.263+0000 7f47d67fc700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 0x7f47c0072070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b80 tx=0x7f47cc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:15.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.267+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f47c805b640 con 0x7f47d80fd240 2024-03-22T02:28:15.583 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:15 smithi122 ceph-mon[36403]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:15.584 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:15 smithi122 ceph-mon[36403]: pgmap v256: 65 pgs: 65 active+clean; 122 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 464 op/s 2024-03-22T02:28:15.584 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:15 smithi122 ceph-mon[36403]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:15.584 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:15 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2908184128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:28:15.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.645+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 --> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47d810f730 con 0x7f47c006fbc0 2024-03-22T02:28:15.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.647+0000 7f47bf7fe700 1 -- 172.21.15.103:0/220891834 <== mgr.14208 v2:172.21.15.103:6800/3344821418 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f47d810f730 con 0x7f47c006fbc0 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "1/2 daemons upgraded", 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mgr daemons", 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:28:15.649 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 msgr2=0x7f47c0072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 0x7f47c0072070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009b80 tx=0x7f47cc00b040 comp rx=0 tx=0).stop 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 msgr2=0x7f47d810da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47c8013870 tx=0x7f47c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 shutdown_connections 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:6800/3344821418,v1:172.21.15.103:6801/3344821418] conn(0x7f47c006fbc0 0x7f47c0072070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47d80fd240 0x7f47d810da70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 --2- 172.21.15.103:0/220891834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47d80fdb90 0x7f47d810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 >> 172.21.15.103:0/220891834 conn(0x7f47d80f6b20 msgr2=0x7f47d80fb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 shutdown_connections 2024-03-22T02:28:15.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:15.650+0000 7f47dd4a7700 1 -- 172.21.15.103:0/220891834 wait complete. 2024-03-22T02:28:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:15 smithi103 ceph-mon[29258]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:15 smithi103 ceph-mon[29258]: pgmap v256: 65 pgs: 65 active+clean; 122 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 464 op/s 2024-03-22T02:28:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:15 smithi103 ceph-mon[29258]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:15 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2908184128' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:28:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:16 smithi122 ceph-mon[36403]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:16 smithi103 ceph-mon[29258]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:17.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:17 smithi122 ceph-mon[36403]: pgmap v257: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 499 op/s 2024-03-22T02:28:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:17 smithi103 ceph-mon[29258]: pgmap v257: 65 pgs: 65 active+clean; 132 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 499 op/s 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr fail", "who": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:28:19.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: osdmap e47: 6 total, 6 up, 6 in 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "mgr fail", "who": "smithi103.pfvmui"}]': finished 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: mgrmap e21: smithi122.prgtti(active, starting, since 0.0074683s) 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:28:19.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:28:19.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: Manager daemon smithi122.prgtti is now available 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/trash_purge_schedule"}]: dispatch 2024-03-22T02:28:19.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:18 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/trash_purge_schedule"}]: dispatch 2024-03-22T02:28:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr fail", "who": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:28:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: osdmap e47: 6 total, 6 up, 6 in 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.14208 172.21.15.103:0/1957994966' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "mgr fail", "who": "smithi103.pfvmui"}]': finished 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: mgrmap e21: smithi122.prgtti(active, starting, since 0.0074683s) 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:28:19.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: Manager daemon smithi122.prgtti is now available 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:28:19.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/trash_purge_schedule"}]: dispatch 2024-03-22T02:28:19.128 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:18 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.prgtti/trash_purge_schedule"}]: dispatch 2024-03-22T02:28:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: Deploying cephadm binary to smithi122 2024-03-22T02:28:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: Deploying cephadm binary to smithi103 2024-03-22T02:28:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: mgrmap e22: smithi122.prgtti(active, since 1.00906s) 2024-03-22T02:28:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: pgmap v3: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:20.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:19 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: Deploying cephadm binary to smithi122 2024-03-22T02:28:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: Deploying cephadm binary to smithi103 2024-03-22T02:28:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: mgrmap e22: smithi122.prgtti(active, since 1.00906s) 2024-03-22T02:28:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: pgmap v3: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:19 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: [22/Mar/2024:02:28:19] ENGINE Bus STARTING 2024-03-22T02:28:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: [22/Mar/2024:02:28:19] ENGINE Serving on http://172.21.15.122:8765 2024-03-22T02:28:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: [22/Mar/2024:02:28:19] ENGINE Serving on https://172.21.15.122:7150 2024-03-22T02:28:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: [22/Mar/2024:02:28:19] ENGINE Bus STARTED 2024-03-22T02:28:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: pgmap v4: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:21 smithi122 ceph-mon[36403]: mgrmap e23: smithi122.prgtti(active, since 2s) 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: [22/Mar/2024:02:28:19] ENGINE Bus STARTING 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: [22/Mar/2024:02:28:19] ENGINE Serving on http://172.21.15.122:8765 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: [22/Mar/2024:02:28:19] ENGINE Serving on https://172.21.15.122:7150 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: [22/Mar/2024:02:28:19] ENGINE Bus STARTED 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: pgmap v4: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:21 smithi103 ceph-mon[29258]: mgrmap e23: smithi122.prgtti(active, since 2s) 2024-03-22T02:28:23.324 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:23 smithi103 ceph-mon[29258]: pgmap v5: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:23.324 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:23 smithi103 ceph-mon[29258]: mgrmap e24: smithi122.prgtti(active, since 4s) 2024-03-22T02:28:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:23 smithi122 ceph-mon[36403]: pgmap v5: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:28:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:23 smithi122 ceph-mon[36403]: mgrmap e24: smithi122.prgtti(active, since 4s) 2024-03-22T02:28:24.727 INFO:tasks.workunit.client.0.smithi103.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-22T02:28:24.736 INFO:tasks.workunit.client.0.smithi103.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-22T02:28:24.736 INFO:tasks.workunit.client.0.smithi103.stderr:+ make 2024-03-22T02:28:24.845 INFO:tasks.workunit.client.0.smithi103.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-22T02:28:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:25 smithi103 ceph-mon[29258]: pgmap v6: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s wr, 87 op/s 2024-03-22T02:28:25.237 INFO:tasks.workunit.client.0.smithi103.stderr:++ readlink -f fsstress 2024-03-22T02:28:25.238 INFO:tasks.workunit.client.0.smithi103.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-22T02:28:25.238 INFO:tasks.workunit.client.0.smithi103.stderr:+ popd 2024-03-22T02:28:25.239 INFO:tasks.workunit.client.0.smithi103.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-22T02:28:25.240 INFO:tasks.workunit.client.0.smithi103.stderr:+ popd 2024-03-22T02:28:25.240 INFO:tasks.workunit.client.0.smithi103.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-22T02:28:25.240 INFO:tasks.workunit.client.0.smithi103.stderr:++ mktemp -d -p . 2024-03-22T02:28:25.242 INFO:tasks.workunit.client.0.smithi103.stderr:+ T=./tmp.6FLcEpU5kx 2024-03-22T02:28:25.242 INFO:tasks.workunit.client.0.smithi103.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.6FLcEpU5kx -l 1 -n 1000 -p 10 -v 2024-03-22T02:28:25.246 INFO:tasks.workunit.client.0.smithi103.stdout:seed = 1711221009 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/0: dread - no filename 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/1: write - no filename 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/2: rmdir - no directory 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/3: fsync - no filename 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/4: rename - no filename 2024-03-22T02:28:25.251 INFO:tasks.workunit.client.0.smithi103.stdout:0/5: chown . 1345 1 2024-03-22T02:28:25.253 INFO:tasks.workunit.client.0.smithi103.stdout:1/0: link - no file 2024-03-22T02:28:25.253 INFO:tasks.workunit.client.0.smithi103.stdout:0/6: creat f0 x:0 0 0 2024-03-22T02:28:25.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/0: dwrite - no filename 2024-03-22T02:28:25.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/1: dwrite - no filename 2024-03-22T02:28:25.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/2: stat - no entries 2024-03-22T02:28:25.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/3: stat - no entries 2024-03-22T02:28:25.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/4: rename - no filename 2024-03-22T02:28:25.256 INFO:tasks.workunit.client.0.smithi103.stdout:0/7: symlink l1 0 2024-03-22T02:28:25.257 INFO:tasks.workunit.client.0.smithi103.stdout:0/8: truncate f0 118213 0 2024-03-22T02:28:25.257 INFO:tasks.workunit.client.0.smithi103.stdout:1/1: mknod c0 0 2024-03-22T02:28:25.257 INFO:tasks.workunit.client.0.smithi103.stdout:1/2: chown c0 133 1 2024-03-22T02:28:25.258 INFO:tasks.workunit.client.0.smithi103.stdout:1/3: chown c0 4428561 1 2024-03-22T02:28:25.258 INFO:tasks.workunit.client.0.smithi103.stdout:3/0: dwrite - no filename 2024-03-22T02:28:25.258 INFO:tasks.workunit.client.0.smithi103.stdout:3/1: stat - no entries 2024-03-22T02:28:25.259 INFO:tasks.workunit.client.0.smithi103.stdout:0/9: creat f2 x:0 0 0 2024-03-22T02:28:25.260 INFO:tasks.workunit.client.0.smithi103.stdout:2/5: mkdir d0 0 2024-03-22T02:28:25.260 INFO:tasks.workunit.client.0.smithi103.stdout:2/6: dwrite - no filename 2024-03-22T02:28:25.260 INFO:tasks.workunit.client.0.smithi103.stdout:2/7: unlink - no file 2024-03-22T02:28:25.261 INFO:tasks.workunit.client.0.smithi103.stdout:1/4: symlink l1 0 2024-03-22T02:28:25.262 INFO:tasks.workunit.client.0.smithi103.stdout:4/0: truncate - no filename 2024-03-22T02:28:25.262 INFO:tasks.workunit.client.0.smithi103.stdout:4/1: dwrite - no filename 2024-03-22T02:28:25.263 INFO:tasks.workunit.client.0.smithi103.stdout:4/2: rmdir - no directory 2024-03-22T02:28:25.263 INFO:tasks.workunit.client.0.smithi103.stdout:4/3: write - no filename 2024-03-22T02:28:25.264 INFO:tasks.workunit.client.0.smithi103.stdout:0/10: rename l1 to l3 0 2024-03-22T02:28:25.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/2: mknod c0 0 2024-03-22T02:28:25.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/3: read - no filename 2024-03-22T02:28:25.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/4: readlink - no filename 2024-03-22T02:28:25.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/5: readlink - no filename 2024-03-22T02:28:25.270 INFO:tasks.workunit.client.0.smithi103.stdout:3/6: creat f1 x:0 0 0 2024-03-22T02:28:25.271 INFO:tasks.workunit.client.0.smithi103.stdout:4/4: creat f0 x:0 0 0 2024-03-22T02:28:25.274 INFO:tasks.workunit.client.0.smithi103.stdout:2/8: rmdir d0 0 2024-03-22T02:28:25.276 INFO:tasks.workunit.client.0.smithi103.stdout:4/5: rename f0 to f1 0 2024-03-22T02:28:25.277 INFO:tasks.workunit.client.0.smithi103.stdout:5/0: mkdir d0 0 2024-03-22T02:28:25.277 INFO:tasks.workunit.client.0.smithi103.stdout:5/1: dread - no filename 2024-03-22T02:28:25.277 INFO:tasks.workunit.client.0.smithi103.stdout:4/6: stat f1 0 2024-03-22T02:28:25.280 INFO:tasks.workunit.client.0.smithi103.stdout:2/9: creat f1 x:0 0 0 2024-03-22T02:28:25.282 INFO:tasks.workunit.client.0.smithi103.stdout:3/7: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.283 INFO:tasks.workunit.client.0.smithi103.stdout:6/0: mkdir d0 0 2024-03-22T02:28:25.283 INFO:tasks.workunit.client.0.smithi103.stdout:6/1: dwrite - no filename 2024-03-22T02:28:25.288 INFO:tasks.workunit.client.0.smithi103.stdout:3/8: creat f2 x:0 0 0 2024-03-22T02:28:25.292 INFO:tasks.workunit.client.0.smithi103.stdout:5/2: creat d0/f1 x:0 0 0 2024-03-22T02:28:25.292 INFO:tasks.workunit.client.0.smithi103.stdout:4/7: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.294 INFO:tasks.workunit.client.0.smithi103.stdout:7/0: symlink l0 0 2024-03-22T02:28:25.294 INFO:tasks.workunit.client.0.smithi103.stdout:7/1: write - no filename 2024-03-22T02:28:25.294 INFO:tasks.workunit.client.0.smithi103.stdout:7/2: rmdir - no directory 2024-03-22T02:28:25.295 INFO:tasks.workunit.client.0.smithi103.stdout:5/3: write d0/f1 [355571,5174] 0 2024-03-22T02:28:25.296 INFO:tasks.workunit.client.0.smithi103.stdout:5/4: stat d0/f1 0 2024-03-22T02:28:25.302 INFO:tasks.workunit.client.0.smithi103.stdout:3/9: symlink l3 0 2024-03-22T02:28:25.303 INFO:tasks.workunit.client.0.smithi103.stdout:6/2: mknod d0/c1 0 2024-03-22T02:28:25.303 INFO:tasks.workunit.client.0.smithi103.stdout:5/5: fdatasync d0/f1 0 2024-03-22T02:28:25.304 INFO:tasks.workunit.client.0.smithi103.stdout:4/8: creat f2 x:0 0 0 2024-03-22T02:28:25.305 INFO:tasks.workunit.client.0.smithi103.stdout:7/3: mkdir d1 0 2024-03-22T02:28:25.305 INFO:tasks.workunit.client.0.smithi103.stdout:7/4: write - no filename 2024-03-22T02:28:25.307 INFO:tasks.workunit.client.0.smithi103.stdout:8/0: mknod c0 0 2024-03-22T02:28:25.308 INFO:tasks.workunit.client.0.smithi103.stdout:8/1: fdatasync - no filename 2024-03-22T02:28:25.308 INFO:tasks.workunit.client.0.smithi103.stdout:0/11: getdents . 0 2024-03-22T02:28:25.309 INFO:tasks.workunit.client.0.smithi103.stdout:0/12: dread - f2 zero size 2024-03-22T02:28:25.310 INFO:tasks.workunit.client.0.smithi103.stdout:0/13: write f2 [415947,83908] 0 2024-03-22T02:28:25.311 INFO:tasks.workunit.client.0.smithi103.stdout:3/10: mknod c4 0 2024-03-22T02:28:25.312 INFO:tasks.workunit.client.0.smithi103.stdout:3/11: truncate f2 286067 0 2024-03-22T02:28:25.313 INFO:tasks.workunit.client.0.smithi103.stdout:6/3: creat d0/f2 x:0 0 0 2024-03-22T02:28:25.315 INFO:tasks.workunit.client.0.smithi103.stdout:5/6: mknod d0/c2 0 2024-03-22T02:28:25.316 INFO:tasks.workunit.client.0.smithi103.stdout:4/9: mkdir d3 0 2024-03-22T02:28:25.317 INFO:tasks.workunit.client.0.smithi103.stdout:4/10: write f1 [1552099,33485] 0 2024-03-22T02:28:25.318 INFO:tasks.workunit.client.0.smithi103.stdout:4/11: rename d3 to d3/d4 22 2024-03-22T02:28:25.318 INFO:tasks.workunit.client.0.smithi103.stdout:4/12: dread - f2 zero size 2024-03-22T02:28:25.320 INFO:tasks.workunit.client.0.smithi103.stdout:8/2: creat f1 x:0 0 0 2024-03-22T02:28:25.321 INFO:tasks.workunit.client.0.smithi103.stdout:9/0: getdents . 0 2024-03-22T02:28:25.321 INFO:tasks.workunit.client.0.smithi103.stdout:9/1: truncate - no filename 2024-03-22T02:28:25.322 INFO:tasks.workunit.client.0.smithi103.stdout:0/14: chown l3 3573 1 2024-03-22T02:28:25.323 INFO:tasks.workunit.client.0.smithi103.stdout:3/12: symlink l5 0 2024-03-22T02:28:25.325 INFO:tasks.workunit.client.0.smithi103.stdout:6/4: mkdir d0/d3 0 2024-03-22T02:28:25.327 INFO:tasks.workunit.client.0.smithi103.stdout:5/7: creat d0/f3 x:0 0 0 2024-03-22T02:28:25.329 INFO:tasks.workunit.client.0.smithi103.stdout:7/5: mkdir d1/d2 0 2024-03-22T02:28:25.329 INFO:tasks.workunit.client.0.smithi103.stdout:7/6: dread - no filename 2024-03-22T02:28:25.332 INFO:tasks.workunit.client.0.smithi103.stdout:5/8: fdatasync d0/f3 0 2024-03-22T02:28:25.333 INFO:tasks.workunit.client.0.smithi103.stdout:8/3: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.335 INFO:tasks.workunit.client.0.smithi103.stdout:5/9: dread - d0/f3 zero size 2024-03-22T02:28:25.336 INFO:tasks.workunit.client.0.smithi103.stdout:4/13: mknod d3/c5 0 2024-03-22T02:28:25.338 INFO:tasks.workunit.client.0.smithi103.stdout:9/2: creat f0 x:0 0 0 2024-03-22T02:28:25.339 INFO:tasks.workunit.client.0.smithi103.stdout:4/14: chown d3/c5 48 1 2024-03-22T02:28:25.340 INFO:tasks.workunit.client.0.smithi103.stdout:3/13: dread f1 [0,4194304] 0 2024-03-22T02:28:25.341 INFO:tasks.workunit.client.0.smithi103.stdout:0/15: dwrite f0 [0,4194304] 0 2024-03-22T02:28:25.343 INFO:tasks.workunit.client.0.smithi103.stdout:0/16: truncate f2 1105855 0 2024-03-22T02:28:25.343 INFO:tasks.workunit.client.0.smithi103.stdout:0/17: chown f2 25408 1 2024-03-22T02:28:25.344 INFO:tasks.workunit.client.0.smithi103.stdout:3/14: dread f2 [0,4194304] 0 2024-03-22T02:28:25.344 INFO:tasks.workunit.client.0.smithi103.stdout:5/10: mknod d0/c4 0 2024-03-22T02:28:25.347 INFO:tasks.workunit.client.0.smithi103.stdout:6/5: link d0/f2 d0/d3/f4 0 2024-03-22T02:28:25.348 INFO:tasks.workunit.client.0.smithi103.stdout:6/6: stat d0/d3/f4 0 2024-03-22T02:28:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:25 smithi122 ceph-mon[36403]: pgmap v6: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.0 MiB/s wr, 87 op/s 2024-03-22T02:28:25.349 INFO:tasks.workunit.client.0.smithi103.stdout:9/3: creat f1 x:0 0 0 2024-03-22T02:28:25.349 INFO:tasks.workunit.client.0.smithi103.stdout:8/4: fdatasync f1 0 2024-03-22T02:28:25.350 INFO:tasks.workunit.client.0.smithi103.stdout:4/15: rmdir d3 39 2024-03-22T02:28:25.351 INFO:tasks.workunit.client.0.smithi103.stdout:4/16: write f2 [95786,88713] 0 2024-03-22T02:28:25.353 INFO:tasks.workunit.client.0.smithi103.stdout:7/7: creat d1/d2/f3 x:0 0 0 2024-03-22T02:28:25.355 INFO:tasks.workunit.client.0.smithi103.stdout:7/8: dread - d1/d2/f3 zero size 2024-03-22T02:28:25.358 INFO:tasks.workunit.client.0.smithi103.stdout:3/15: write f1 [3123674,20774] 0 2024-03-22T02:28:25.363 INFO:tasks.workunit.client.0.smithi103.stdout:9/4: mkdir d2 0 2024-03-22T02:28:25.364 INFO:tasks.workunit.client.0.smithi103.stdout:4/17: dwrite f2 [0,4194304] 0 2024-03-22T02:28:25.373 INFO:tasks.workunit.client.0.smithi103.stdout:7/9: creat d1/d2/f4 x:0 0 0 2024-03-22T02:28:25.373 INFO:tasks.workunit.client.0.smithi103.stdout:5/11: rename d0/f3 to d0/f5 0 2024-03-22T02:28:25.374 INFO:tasks.workunit.client.0.smithi103.stdout:0/18: dwrite f0 [4194304,4194304] 0 2024-03-22T02:28:25.374 INFO:tasks.workunit.client.0.smithi103.stdout:3/16: mknod c6 0 2024-03-22T02:28:25.378 INFO:tasks.workunit.client.0.smithi103.stdout:6/7: mknod d0/d3/c5 0 2024-03-22T02:28:25.379 INFO:tasks.workunit.client.0.smithi103.stdout:7/10: symlink d1/l5 0 2024-03-22T02:28:25.381 INFO:tasks.workunit.client.0.smithi103.stdout:4/18: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.382 INFO:tasks.workunit.client.0.smithi103.stdout:9/5: creat d2/f3 x:0 0 0 2024-03-22T02:28:25.383 INFO:tasks.workunit.client.0.smithi103.stdout:3/17: dread f2 [0,4194304] 0 2024-03-22T02:28:25.383 INFO:tasks.workunit.client.0.smithi103.stdout:3/18: rmdir - no directory 2024-03-22T02:28:25.384 INFO:tasks.workunit.client.0.smithi103.stdout:8/5: getdents . 0 2024-03-22T02:28:25.384 INFO:tasks.workunit.client.0.smithi103.stdout:9/6: dread - d2/f3 zero size 2024-03-22T02:28:25.386 INFO:tasks.workunit.client.0.smithi103.stdout:8/6: write f1 [4915711,67892] 0 2024-03-22T02:28:25.386 INFO:tasks.workunit.client.0.smithi103.stdout:9/7: truncate d2/f3 496830 0 2024-03-22T02:28:25.386 INFO:tasks.workunit.client.0.smithi103.stdout:3/19: write f1 [4831872,52753] 0 2024-03-22T02:28:25.387 INFO:tasks.workunit.client.0.smithi103.stdout:9/8: read - f0 zero size 2024-03-22T02:28:25.391 INFO:tasks.workunit.client.0.smithi103.stdout:7/11: rename d1/l5 to d1/l6 0 2024-03-22T02:28:25.416 INFO:tasks.workunit.client.0.smithi103.stdout:8/7: creat f2 x:0 0 0 2024-03-22T02:28:25.416 INFO:tasks.workunit.client.0.smithi103.stdout:9/9: mknod d2/c4 0 2024-03-22T02:28:25.416 INFO:tasks.workunit.client.0.smithi103.stdout:3/20: creat f7 x:0 0 0 2024-03-22T02:28:25.417 INFO:tasks.workunit.client.0.smithi103.stdout:6/8: link d0/c1 d0/d3/c6 0 2024-03-22T02:28:25.418 INFO:tasks.workunit.client.0.smithi103.stdout:4/19: symlink d3/l6 0 2024-03-22T02:28:25.418 INFO:tasks.workunit.client.0.smithi103.stdout:9/10: stat d2/c4 0 2024-03-22T02:28:25.418 INFO:tasks.workunit.client.0.smithi103.stdout:8/8: chown f1 15036721 1 2024-03-22T02:28:25.420 INFO:tasks.workunit.client.0.smithi103.stdout:3/21: write f7 [312046,117941] 0 2024-03-22T02:28:25.421 INFO:tasks.workunit.client.0.smithi103.stdout:0/19: truncate f0 2405714 0 2024-03-22T02:28:25.422 INFO:tasks.workunit.client.0.smithi103.stdout:8/9: chown c0 8131965 1 2024-03-22T02:28:25.424 INFO:tasks.workunit.client.0.smithi103.stdout:8/10: read - f2 zero size 2024-03-22T02:28:25.424 INFO:tasks.workunit.client.0.smithi103.stdout:3/22: dread f2 [0,4194304] 0 2024-03-22T02:28:25.424 INFO:tasks.workunit.client.0.smithi103.stdout:9/11: write f0 [307563,30605] 0 2024-03-22T02:28:25.426 INFO:tasks.workunit.client.0.smithi103.stdout:7/12: dwrite d1/d2/f4 [0,4194304] 0 2024-03-22T02:28:25.426 INFO:tasks.workunit.client.0.smithi103.stdout:3/23: write f7 [227713,71699] 0 2024-03-22T02:28:25.436 INFO:tasks.workunit.client.0.smithi103.stdout:9/12: dwrite f0 [0,4194304] 0 2024-03-22T02:28:25.674 INFO:tasks.workunit.client.0.smithi103.stdout:4/20: creat d3/f7 x:0 0 0 2024-03-22T02:28:25.675 INFO:tasks.workunit.client.0.smithi103.stdout:6/9: truncate d0/f2 536070 0 2024-03-22T02:28:25.675 INFO:tasks.workunit.client.0.smithi103.stdout:4/21: chown f2 2 1 2024-03-22T02:28:25.675 INFO:tasks.workunit.client.0.smithi103.stdout:4/22: chown d3/c5 631222 1 2024-03-22T02:28:25.680 INFO:tasks.workunit.client.0.smithi103.stdout:7/13: symlink d1/l7 0 2024-03-22T02:28:25.682 INFO:tasks.workunit.client.0.smithi103.stdout:9/13: mkdir d2/d5 0 2024-03-22T02:28:25.684 INFO:tasks.workunit.client.0.smithi103.stdout:8/11: fsync f2 0 2024-03-22T02:28:25.685 INFO:tasks.workunit.client.0.smithi103.stdout:4/23: creat d3/f8 x:0 0 0 2024-03-22T02:28:25.686 INFO:tasks.workunit.client.0.smithi103.stdout:4/24: write f1 [460763,69608] 0 2024-03-22T02:28:25.688 INFO:tasks.workunit.client.0.smithi103.stdout:3/24: link f7 f8 0 2024-03-22T02:28:25.691 INFO:tasks.workunit.client.0.smithi103.stdout:7/14: creat d1/d2/f8 x:0 0 0 2024-03-22T02:28:25.694 INFO:tasks.workunit.client.0.smithi103.stdout:0/20: rename f0 to f4 0 2024-03-22T02:28:25.694 INFO:tasks.workunit.client.0.smithi103.stdout:7/15: read d1/d2/f4 [2226032,67672] 0 2024-03-22T02:28:25.696 INFO:tasks.workunit.client.0.smithi103.stdout:8/12: mknod c3 0 2024-03-22T02:28:25.699 INFO:tasks.workunit.client.0.smithi103.stdout:3/25: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.702 INFO:tasks.workunit.client.0.smithi103.stdout:9/14: creat d2/d5/f6 x:0 0 0 2024-03-22T02:28:25.707 INFO:tasks.workunit.client.0.smithi103.stdout:7/16: mknod d1/c9 0 2024-03-22T02:28:25.708 INFO:tasks.workunit.client.0.smithi103.stdout:0/21: dwrite f2 [0,4194304] 0 2024-03-22T02:28:25.710 INFO:tasks.workunit.client.0.smithi103.stdout:8/13: dwrite f1 [4194304,4194304] 0 2024-03-22T02:28:25.716 INFO:tasks.workunit.client.0.smithi103.stdout:3/26: symlink l9 0 2024-03-22T02:28:25.723 INFO:tasks.workunit.client.0.smithi103.stdout:7/17: dwrite d1/d2/f8 [0,4194304] 0 2024-03-22T02:28:25.723 INFO:tasks.workunit.client.0.smithi103.stdout:8/14: mkdir d4 0 2024-03-22T02:28:25.724 INFO:tasks.workunit.client.0.smithi103.stdout:8/15: write f2 [606815,2843] 0 2024-03-22T02:28:25.741 INFO:tasks.workunit.client.0.smithi103.stdout:8/16: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.741 INFO:tasks.workunit.client.0.smithi103.stdout:3/27: dwrite f8 [0,4194304] 0 2024-03-22T02:28:25.744 INFO:tasks.workunit.client.0.smithi103.stdout:8/17: write f1 [6731118,14789] 0 2024-03-22T02:28:25.745 INFO:tasks.workunit.client.0.smithi103.stdout:3/28: creat fa x:0 0 0 2024-03-22T02:28:25.752 INFO:tasks.workunit.client.0.smithi103.stdout:8/18: creat d4/f5 x:0 0 0 2024-03-22T02:28:25.756 INFO:tasks.workunit.client.0.smithi103.stdout:3/29: dwrite f1 [0,4194304] 0 2024-03-22T02:28:25.756 INFO:tasks.workunit.client.0.smithi103.stdout:8/19: write d4/f5 [395129,107462] 0 2024-03-22T02:28:25.764 INFO:tasks.workunit.client.0.smithi103.stdout:3/30: dread f7 [0,4194304] 0 2024-03-22T02:28:25.767 INFO:tasks.workunit.client.0.smithi103.stdout:3/31: write fa [25355,65304] 0 2024-03-22T02:28:25.770 INFO:tasks.workunit.client.0.smithi103.stdout:8/20: dwrite f1 [8388608,4194304] 0 2024-03-22T02:28:25.770 INFO:tasks.workunit.client.0.smithi103.stdout:3/32: creat fb x:0 0 0 2024-03-22T02:28:25.773 INFO:tasks.workunit.client.0.smithi103.stdout:8/21: symlink d4/l6 0 2024-03-22T02:28:25.777 INFO:tasks.workunit.client.0.smithi103.stdout:8/22: creat d4/f7 x:0 0 0 2024-03-22T02:28:25.781 INFO:tasks.workunit.client.0.smithi103.stdout:8/23: dread f1 [0,4194304] 0 2024-03-22T02:28:25.782 INFO:tasks.workunit.client.0.smithi103.stdout:8/24: write d4/f7 [36060,123086] 0 2024-03-22T02:28:25.785 INFO:tasks.workunit.client.0.smithi103.stdout:8/25: creat d4/f8 x:0 0 0 2024-03-22T02:28:25.787 INFO:tasks.workunit.client.0.smithi103.stdout:8/26: creat d4/f9 x:0 0 0 2024-03-22T02:28:25.788 INFO:tasks.workunit.client.0.smithi103.stdout:8/27: chown d4/f7 1457924249 1 2024-03-22T02:28:25.788 INFO:tasks.workunit.client.0.smithi103.stdout:8/28: chown f2 32484 1 2024-03-22T02:28:25.850 INFO:tasks.workunit.client.0.smithi103.stdout:7/18: rename d1/d2 to d1/da 0 2024-03-22T02:28:25.853 INFO:tasks.workunit.client.0.smithi103.stdout:7/19: creat d1/da/fb x:0 0 0 2024-03-22T02:28:25.854 INFO:tasks.workunit.client.0.smithi103.stdout:7/20: symlink d1/lc 0 2024-03-22T02:28:25.858 INFO:tasks.workunit.client.0.smithi103.stdout:7/21: rename l0 to d1/da/ld 0 2024-03-22T02:28:25.859 INFO:tasks.workunit.client.0.smithi103.stdout:4/25: truncate f2 2502781 0 2024-03-22T02:28:25.861 INFO:tasks.workunit.client.0.smithi103.stdout:0/22: read f4 [419515,6173] 0 2024-03-22T02:28:25.863 INFO:tasks.workunit.client.0.smithi103.stdout:4/26: unlink f1 0 2024-03-22T02:28:25.867 INFO:tasks.workunit.client.0.smithi103.stdout:4/27: symlink d3/l9 0 2024-03-22T02:28:25.868 INFO:tasks.workunit.client.0.smithi103.stdout:0/23: dwrite f2 [0,4194304] 0 2024-03-22T02:28:25.871 INFO:tasks.workunit.client.0.smithi103.stdout:0/24: fdatasync f4 0 2024-03-22T02:28:25.871 INFO:tasks.workunit.client.0.smithi103.stdout:0/25: rmdir - no directory 2024-03-22T02:28:25.873 INFO:tasks.workunit.client.0.smithi103.stdout:4/28: chown f2 11595589 1 2024-03-22T02:28:25.879 INFO:tasks.workunit.client.0.smithi103.stdout:3/33: truncate f7 773974 0 2024-03-22T02:28:25.881 INFO:tasks.workunit.client.0.smithi103.stdout:3/34: creat fc x:0 0 0 2024-03-22T02:28:25.882 INFO:tasks.workunit.client.0.smithi103.stdout:8/29: getdents d4 0 2024-03-22T02:28:25.883 INFO:tasks.workunit.client.0.smithi103.stdout:6/10: write d0/d3/f4 [1083811,60144] 0 2024-03-22T02:28:25.884 INFO:tasks.workunit.client.0.smithi103.stdout:8/30: write d4/f8 [1007567,34215] 0 2024-03-22T02:28:25.884 INFO:tasks.workunit.client.0.smithi103.stdout:3/35: creat fd x:0 0 0 2024-03-22T02:28:25.885 INFO:tasks.workunit.client.0.smithi103.stdout:8/31: rename d4 to d4/da 22 2024-03-22T02:28:25.887 INFO:tasks.workunit.client.0.smithi103.stdout:6/11: chown d0/d3/c6 35748 1 2024-03-22T02:28:25.888 INFO:tasks.workunit.client.0.smithi103.stdout:3/36: mkdir de 0 2024-03-22T02:28:25.890 INFO:tasks.workunit.client.0.smithi103.stdout:8/32: rename d4/f5 to d4/fb 0 2024-03-22T02:28:25.892 INFO:tasks.workunit.client.0.smithi103.stdout:6/12: mkdir d0/d3/d7 0 2024-03-22T02:28:25.896 INFO:tasks.workunit.client.0.smithi103.stdout:4/29: dread f2 [0,4194304] 0 2024-03-22T02:28:25.906 INFO:tasks.workunit.client.0.smithi103.stdout:0/26: dwrite f4 [0,4194304] 0 2024-03-22T02:28:25.908 INFO:tasks.workunit.client.0.smithi103.stdout:6/13: rmdir d0/d3/d7 0 2024-03-22T02:28:25.910 INFO:tasks.workunit.client.0.smithi103.stdout:0/27: mknod c5 0 2024-03-22T02:28:25.912 INFO:tasks.workunit.client.0.smithi103.stdout:8/33: getdents d4 0 2024-03-22T02:28:25.914 INFO:tasks.workunit.client.0.smithi103.stdout:6/14: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:25.915 INFO:tasks.workunit.client.0.smithi103.stdout:0/28: dread f2 [0,4194304] 0 2024-03-22T02:28:25.920 INFO:tasks.workunit.client.0.smithi103.stdout:8/34: rmdir d4 39 2024-03-22T02:28:25.923 INFO:tasks.workunit.client.0.smithi103.stdout:3/37: dwrite f2 [0,4194304] 0 2024-03-22T02:28:25.924 INFO:tasks.workunit.client.0.smithi103.stdout:0/29: fsync f2 0 2024-03-22T02:28:25.924 INFO:tasks.workunit.client.0.smithi103.stdout:6/15: creat d0/f8 x:0 0 0 2024-03-22T02:28:25.931 INFO:tasks.workunit.client.0.smithi103.stdout:6/16: creat d0/d3/f9 x:0 0 0 2024-03-22T02:28:25.932 INFO:tasks.workunit.client.0.smithi103.stdout:8/35: dwrite d4/f8 [0,4194304] 0 2024-03-22T02:28:25.932 INFO:tasks.workunit.client.0.smithi103.stdout:0/30: fdatasync f4 0 2024-03-22T02:28:25.933 INFO:tasks.workunit.client.0.smithi103.stdout:3/38: mknod de/cf 0 2024-03-22T02:28:25.935 INFO:tasks.workunit.client.0.smithi103.stdout:8/36: write d4/fb [805986,79435] 0 2024-03-22T02:28:25.935 INFO:tasks.workunit.client.0.smithi103.stdout:6/17: truncate d0/f2 1765409 0 2024-03-22T02:28:25.936 INFO:tasks.workunit.client.0.smithi103.stdout:8/37: chown d4/f7 404731182 1 2024-03-22T02:28:25.936 INFO:tasks.workunit.client.0.smithi103.stdout:6/18: chown d0/d3 0 1 2024-03-22T02:28:25.938 INFO:tasks.workunit.client.0.smithi103.stdout:0/31: creat f6 x:0 0 0 2024-03-22T02:28:25.940 INFO:tasks.workunit.client.0.smithi103.stdout:6/19: mkdir d0/d3/da 0 2024-03-22T02:28:25.942 INFO:tasks.workunit.client.0.smithi103.stdout:8/38: rename d4/fb to d4/fc 0 2024-03-22T02:28:25.943 INFO:tasks.workunit.client.0.smithi103.stdout:0/32: symlink l7 0 2024-03-22T02:28:25.943 INFO:tasks.workunit.client.0.smithi103.stdout:6/20: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:25.944 INFO:tasks.workunit.client.0.smithi103.stdout:0/33: dread - f6 zero size 2024-03-22T02:28:25.947 INFO:tasks.workunit.client.0.smithi103.stdout:6/21: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:25.950 INFO:tasks.workunit.client.0.smithi103.stdout:0/34: dread f4 [0,4194304] 0 2024-03-22T02:28:25.950 INFO:tasks.workunit.client.0.smithi103.stdout:8/39: unlink d4/f7 0 2024-03-22T02:28:25.957 INFO:tasks.workunit.client.0.smithi103.stdout:6/22: dwrite d0/f8 [0,4194304] 0 2024-03-22T02:28:25.958 INFO:tasks.workunit.client.0.smithi103.stdout:0/35: dread f4 [0,4194304] 0 2024-03-22T02:28:25.960 INFO:tasks.workunit.client.0.smithi103.stdout:0/36: creat f8 x:0 0 0 2024-03-22T02:28:25.963 INFO:tasks.workunit.client.0.smithi103.stdout:0/37: rename l3 to l9 0 2024-03-22T02:28:25.964 INFO:tasks.workunit.client.0.smithi103.stdout:0/38: read - f8 zero size 2024-03-22T02:28:25.965 INFO:tasks.workunit.client.0.smithi103.stdout:8/40: truncate d4/f8 454418 0 2024-03-22T02:28:25.966 INFO:tasks.workunit.client.0.smithi103.stdout:0/39: write f4 [1425848,10351] 0 2024-03-22T02:28:25.966 INFO:tasks.workunit.client.0.smithi103.stdout:8/41: fdatasync d4/f9 0 2024-03-22T02:28:25.970 INFO:tasks.workunit.client.0.smithi103.stdout:0/40: unlink c5 0 2024-03-22T02:28:25.971 INFO:tasks.workunit.client.0.smithi103.stdout:3/39: dwrite f8 [0,4194304] 0 2024-03-22T02:28:25.971 INFO:tasks.workunit.client.0.smithi103.stdout:3/40: fdatasync fa 0 2024-03-22T02:28:25.973 INFO:tasks.workunit.client.0.smithi103.stdout:0/41: creat fa x:0 0 0 2024-03-22T02:28:25.979 INFO:tasks.workunit.client.0.smithi103.stdout:0/42: dread f4 [0,4194304] 0 2024-03-22T02:28:25.980 INFO:tasks.workunit.client.0.smithi103.stdout:0/43: fdatasync f6 0 2024-03-22T02:28:25.983 INFO:tasks.workunit.client.0.smithi103.stdout:0/44: symlink lb 0 2024-03-22T02:28:25.983 INFO:tasks.workunit.client.0.smithi103.stdout:0/45: stat f8 0 2024-03-22T02:28:25.984 INFO:tasks.workunit.client.0.smithi103.stdout:8/42: getdents d4 0 2024-03-22T02:28:25.985 INFO:tasks.workunit.client.0.smithi103.stdout:3/41: getdents de 0 2024-03-22T02:28:25.987 INFO:tasks.workunit.client.0.smithi103.stdout:0/46: creat fc x:0 0 0 2024-03-22T02:28:25.988 INFO:tasks.workunit.client.0.smithi103.stdout:8/43: write d4/fc [1513318,72275] 0 2024-03-22T02:28:25.989 INFO:tasks.workunit.client.0.smithi103.stdout:3/42: creat de/f10 x:0 0 0 2024-03-22T02:28:25.989 INFO:tasks.workunit.client.0.smithi103.stdout:8/44: write f2 [1457721,90918] 0 2024-03-22T02:28:25.989 INFO:tasks.workunit.client.0.smithi103.stdout:3/43: readlink l9 0 2024-03-22T02:28:25.992 INFO:tasks.workunit.client.0.smithi103.stdout:8/45: creat d4/fd x:0 0 0 2024-03-22T02:28:25.998 INFO:tasks.workunit.client.0.smithi103.stdout:8/46: dwrite d4/f9 [0,4194304] 0 2024-03-22T02:28:25.999 INFO:tasks.workunit.client.0.smithi103.stdout:3/44: truncate f1 4822059 0 2024-03-22T02:28:25.999 INFO:tasks.workunit.client.0.smithi103.stdout:8/47: write d4/fc [2489285,34702] 0 2024-03-22T02:28:26.005 INFO:tasks.workunit.client.0.smithi103.stdout:3/45: rename l5 to de/l11 0 2024-03-22T02:28:26.008 INFO:tasks.workunit.client.0.smithi103.stdout:3/46: mkdir de/d12 0 2024-03-22T02:28:26.009 INFO:tasks.workunit.client.0.smithi103.stdout:8/48: dwrite f2 [0,4194304] 0 2024-03-22T02:28:26.009 INFO:tasks.workunit.client.0.smithi103.stdout:3/47: chown fb 127 1 2024-03-22T02:28:26.012 INFO:tasks.workunit.client.0.smithi103.stdout:8/49: write d4/f8 [288896,113036] 0 2024-03-22T02:28:26.013 INFO:tasks.workunit.client.0.smithi103.stdout:3/48: mknod de/d12/c13 0 2024-03-22T02:28:26.020 INFO:tasks.workunit.client.0.smithi103.stdout:8/50: dread d4/f9 [0,4194304] 0 2024-03-22T02:28:26.022 INFO:tasks.workunit.client.0.smithi103.stdout:3/49: dwrite fa [0,4194304] 0 2024-03-22T02:28:26.023 INFO:tasks.workunit.client.0.smithi103.stdout:3/50: truncate fb 953588 0 2024-03-22T02:28:26.027 INFO:tasks.workunit.client.0.smithi103.stdout:3/51: rename l9 to de/l14 0 2024-03-22T02:28:26.032 INFO:tasks.workunit.client.0.smithi103.stdout:8/51: dwrite d4/f9 [4194304,4194304] 0 2024-03-22T02:28:26.033 INFO:tasks.workunit.client.0.smithi103.stdout:8/52: chown f1 23562 1 2024-03-22T02:28:26.034 INFO:tasks.workunit.client.0.smithi103.stdout:8/53: readlink d4/l6 0 2024-03-22T02:28:26.036 INFO:tasks.workunit.client.0.smithi103.stdout:8/54: creat d4/fe x:0 0 0 2024-03-22T02:28:26.037 INFO:tasks.workunit.client.0.smithi103.stdout:8/55: write d4/fd [295707,124948] 0 2024-03-22T02:28:26.041 INFO:tasks.workunit.client.0.smithi103.stdout:8/56: dread f1 [8388608,4194304] 0 2024-03-22T02:28:26.048 INFO:tasks.workunit.client.0.smithi103.stdout:8/57: dwrite f1 [4194304,4194304] 0 2024-03-22T02:28:26.056 INFO:tasks.workunit.client.0.smithi103.stdout:8/58: truncate d4/f9 7502931 0 2024-03-22T02:28:26.058 INFO:tasks.workunit.client.0.smithi103.stdout:8/59: creat d4/ff x:0 0 0 2024-03-22T02:28:26.058 INFO:tasks.workunit.client.0.smithi103.stdout:8/60: readlink d4/l6 0 2024-03-22T02:28:26.059 INFO:tasks.workunit.client.0.smithi103.stdout:8/61: write f1 [10956387,45325] 0 2024-03-22T02:28:26.065 INFO:tasks.workunit.client.0.smithi103.stdout:8/62: dwrite d4/fc [0,4194304] 0 2024-03-22T02:28:26.067 INFO:tasks.workunit.client.0.smithi103.stdout:8/63: write d4/fe [791374,91135] 0 2024-03-22T02:28:26.074 INFO:tasks.workunit.client.0.smithi103.stdout:8/64: dread d4/f9 [4194304,4194304] 0 2024-03-22T02:28:26.076 INFO:tasks.workunit.client.0.smithi103.stdout:8/65: mknod d4/c10 0 2024-03-22T02:28:26.077 INFO:tasks.workunit.client.0.smithi103.stdout:8/66: chown d4/fc 968022 1 2024-03-22T02:28:26.077 INFO:tasks.workunit.client.0.smithi103.stdout:8/67: stat d4/ff 0 2024-03-22T02:28:26.078 INFO:tasks.workunit.client.0.smithi103.stdout:8/68: truncate d4/f8 1356144 0 2024-03-22T02:28:26.078 INFO:tasks.workunit.client.0.smithi103.stdout:8/69: stat d4 0 2024-03-22T02:28:26.079 INFO:tasks.workunit.client.0.smithi103.stdout:8/70: write f1 [6554900,33630] 0 2024-03-22T02:28:26.085 INFO:tasks.workunit.client.0.smithi103.stdout:8/71: dwrite d4/f9 [0,4194304] 0 2024-03-22T02:28:26.090 INFO:tasks.workunit.client.0.smithi103.stdout:8/72: dread d4/f8 [0,4194304] 0 2024-03-22T02:28:26.093 INFO:tasks.workunit.client.0.smithi103.stdout:8/73: dread d4/f8 [0,4194304] 0 2024-03-22T02:28:26.095 INFO:tasks.workunit.client.0.smithi103.stdout:8/74: mknod d4/c11 0 2024-03-22T02:28:26.095 INFO:tasks.workunit.client.0.smithi103.stdout:8/75: chown d4/l6 23 1 2024-03-22T02:28:26.103 INFO:tasks.workunit.client.0.smithi103.stdout:8/76: dwrite d4/fd [0,4194304] 0 2024-03-22T02:28:26.109 INFO:tasks.workunit.client.0.smithi103.stdout:8/77: dwrite f2 [0,4194304] 0 2024-03-22T02:28:26.114 INFO:tasks.workunit.client.0.smithi103.stdout:8/78: write d4/f8 [2213642,123000] 0 2024-03-22T02:28:26.114 INFO:tasks.workunit.client.0.smithi103.stdout:8/79: chown d4/ff 28435 1 2024-03-22T02:28:26.116 INFO:tasks.workunit.client.0.smithi103.stdout:8/80: creat d4/f12 x:0 0 0 2024-03-22T02:28:26.118 INFO:tasks.workunit.client.0.smithi103.stdout:8/81: mkdir d4/d13 0 2024-03-22T02:28:26.120 INFO:tasks.workunit.client.0.smithi103.stdout:8/82: mkdir d4/d14 0 2024-03-22T02:28:26.120 INFO:tasks.workunit.client.0.smithi103.stdout:8/83: write d4/f12 [413582,91149] 0 2024-03-22T02:28:26.121 INFO:tasks.workunit.client.0.smithi103.stdout:8/84: stat d4/fd 0 2024-03-22T02:28:26.124 INFO:tasks.workunit.client.0.smithi103.stdout:8/85: dread f2 [0,4194304] 0 2024-03-22T02:28:26.125 INFO:tasks.workunit.client.0.smithi103.stdout:8/86: write d4/ff [773211,2374] 0 2024-03-22T02:28:26.127 INFO:tasks.workunit.client.0.smithi103.stdout:8/87: creat d4/f15 x:0 0 0 2024-03-22T02:28:26.130 INFO:tasks.workunit.client.0.smithi103.stdout:8/88: dread f1 [0,4194304] 0 2024-03-22T02:28:26.130 INFO:tasks.workunit.client.0.smithi103.stdout:8/89: fsync d4/fd 0 2024-03-22T02:28:26.138 INFO:tasks.workunit.client.0.smithi103.stdout:8/90: dwrite d4/fc [4194304,4194304] 0 2024-03-22T02:28:26.138 INFO:tasks.workunit.client.0.smithi103.stdout:8/91: readlink d4/l6 0 2024-03-22T02:28:26.140 INFO:tasks.workunit.client.0.smithi103.stdout:8/92: mknod d4/d14/c16 0 2024-03-22T02:28:26.141 INFO:tasks.workunit.client.0.smithi103.stdout:8/93: fdatasync d4/f8 0 2024-03-22T02:28:26.143 INFO:tasks.workunit.client.0.smithi103.stdout:8/94: symlink d4/d14/l17 0 2024-03-22T02:28:26.147 INFO:tasks.workunit.client.0.smithi103.stdout:8/95: link f2 d4/d13/f18 0 2024-03-22T02:28:26.147 INFO:tasks.workunit.client.0.smithi103.stdout:8/96: fdatasync d4/fd 0 2024-03-22T02:28:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: pgmap v7: 65 pgs: 65 active+clean; 185 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 4.8 MiB/s wr, 387 op/s 2024-03-22T02:28:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: Standby manager daemon smithi103.pfvmui started 2024-03-22T02:28:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/crt"}]: dispatch 2024-03-22T02:28:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:28:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/key"}]: dispatch 2024-03-22T02:28:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:27 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: pgmap v7: 65 pgs: 65 active+clean; 185 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 4.8 MiB/s wr, 387 op/s 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: Standby manager daemon smithi103.pfvmui started 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/crt"}]: dispatch 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/key"}]: dispatch 2024-03-22T02:28:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:27 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/3423799772' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:28:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:28 smithi122 ceph-mon[36403]: mgrmap e25: smithi122.prgtti(active, since 9s), standbys: smithi103.pfvmui 2024-03-22T02:28:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:28 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:28:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:28 smithi103 ceph-mon[29258]: mgrmap e25: smithi122.prgtti(active, since 9s), standbys: smithi103.pfvmui 2024-03-22T02:28:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:28 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:28:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:29 smithi122 ceph-mon[36403]: pgmap v8: 65 pgs: 65 active+clean; 185 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 3.7 MiB/s wr, 301 op/s 2024-03-22T02:28:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:29 smithi103 ceph-mon[29258]: pgmap v8: 65 pgs: 65 active+clean; 185 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 3.7 MiB/s wr, 301 op/s 2024-03-22T02:28:29.378 INFO:tasks.workunit.client.0.smithi103.stdout:1/5: sync 2024-03-22T02:28:29.379 INFO:tasks.workunit.client.0.smithi103.stdout:1/6: dwrite - no filename 2024-03-22T02:28:29.379 INFO:tasks.workunit.client.0.smithi103.stdout:1/7: write - no filename 2024-03-22T02:28:29.381 INFO:tasks.workunit.client.0.smithi103.stdout:1/8: rename l1 to l2 0 2024-03-22T02:28:29.384 INFO:tasks.workunit.client.0.smithi103.stdout:1/9: link c0 c3 0 2024-03-22T02:28:29.384 INFO:tasks.workunit.client.0.smithi103.stdout:1/10: dread - no filename 2024-03-22T02:28:29.389 INFO:tasks.workunit.client.0.smithi103.stdout:1/11: mknod c4 0 2024-03-22T02:28:29.390 INFO:tasks.workunit.client.0.smithi103.stdout:1/12: chown c4 3774054 1 2024-03-22T02:28:29.392 INFO:tasks.workunit.client.0.smithi103.stdout:1/13: creat f5 x:0 0 0 2024-03-22T02:28:29.393 INFO:tasks.workunit.client.0.smithi103.stdout:1/14: mknod c6 0 2024-03-22T02:28:29.394 INFO:tasks.workunit.client.0.smithi103.stdout:1/15: creat f7 x:0 0 0 2024-03-22T02:28:29.453 INFO:tasks.workunit.client.0.smithi103.stdout:2/10: sync 2024-03-22T02:28:29.455 INFO:tasks.workunit.client.0.smithi103.stdout:2/11: creat f2 x:0 0 0 2024-03-22T02:28:29.455 INFO:tasks.workunit.client.0.smithi103.stdout:2/12: dread - f1 zero size 2024-03-22T02:28:29.456 INFO:tasks.workunit.client.0.smithi103.stdout:2/13: write f1 [606770,56540] 0 2024-03-22T02:28:29.568 INFO:tasks.workunit.client.0.smithi103.stdout:9/15: sync 2024-03-22T02:28:29.568 INFO:tasks.workunit.client.0.smithi103.stdout:5/12: sync 2024-03-22T02:28:29.582 INFO:tasks.workunit.client.0.smithi103.stdout:9/16: write d2/d5/f6 [692851,99009] 0 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:3/52: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:2/14: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:4/30: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:6/23: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:0/47: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:7/22: sync 2024-03-22T02:28:29.601 INFO:tasks.workunit.client.0.smithi103.stdout:1/16: sync 2024-03-22T02:28:29.602 INFO:tasks.workunit.client.0.smithi103.stdout:8/97: sync 2024-03-22T02:28:29.602 INFO:tasks.workunit.client.0.smithi103.stdout:0/48: chown lb 2126440 1 2024-03-22T02:28:29.602 INFO:tasks.workunit.client.0.smithi103.stdout:7/23: readlink d1/l7 0 2024-03-22T02:28:29.603 INFO:tasks.workunit.client.0.smithi103.stdout:8/98: read d4/fd [3129722,116857] 0 2024-03-22T02:28:29.603 INFO:tasks.workunit.client.0.smithi103.stdout:1/17: truncate f7 453224 0 2024-03-22T02:28:29.604 INFO:tasks.workunit.client.0.smithi103.stdout:7/24: fdatasync d1/da/fb 0 2024-03-22T02:28:29.606 INFO:tasks.workunit.client.0.smithi103.stdout:8/99: chown d4/d13/f18 310354250 1 2024-03-22T02:28:29.609 INFO:tasks.workunit.client.0.smithi103.stdout:3/53: dwrite fd [0,4194304] 0 2024-03-22T02:28:29.685 INFO:tasks.workunit.client.0.smithi103.stdout:5/13: unlink d0/f1 0 2024-03-22T02:28:29.686 INFO:tasks.workunit.client.0.smithi103.stdout:9/17: symlink d2/l7 0 2024-03-22T02:28:29.687 INFO:tasks.workunit.client.0.smithi103.stdout:4/31: symlink d3/la 0 2024-03-22T02:28:29.689 INFO:tasks.workunit.client.0.smithi103.stdout:7/25: creat d1/da/fe x:0 0 0 2024-03-22T02:28:29.691 INFO:tasks.workunit.client.0.smithi103.stdout:8/100: mkdir d4/d19 0 2024-03-22T02:28:29.695 INFO:tasks.workunit.client.0.smithi103.stdout:4/32: dwrite d3/f8 [0,4194304] 0 2024-03-22T02:28:29.701 INFO:tasks.workunit.client.0.smithi103.stdout:7/26: dwrite d1/da/f8 [0,4194304] 0 2024-03-22T02:28:29.703 INFO:tasks.workunit.client.0.smithi103.stdout:3/54: rename de/l14 to de/d12/l15 0 2024-03-22T02:28:29.711 INFO:tasks.workunit.client.0.smithi103.stdout:8/101: mkdir d4/d19/d1a 0 2024-03-22T02:28:29.712 INFO:tasks.workunit.client.0.smithi103.stdout:7/27: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:29.713 INFO:tasks.workunit.client.0.smithi103.stdout:1/18: getdents . 0 2024-03-22T02:28:29.713 INFO:tasks.workunit.client.0.smithi103.stdout:8/102: read d4/fd [1800252,119358] 0 2024-03-22T02:28:29.715 INFO:tasks.workunit.client.0.smithi103.stdout:1/19: read f7 [297274,4102] 0 2024-03-22T02:28:29.717 INFO:tasks.workunit.client.0.smithi103.stdout:4/33: link d3/f7 d3/fb 0 2024-03-22T02:28:29.719 INFO:tasks.workunit.client.0.smithi103.stdout:4/34: chown d3/l6 98305032 1 2024-03-22T02:28:29.723 INFO:tasks.workunit.client.0.smithi103.stdout:8/103: dwrite d4/fd [0,4194304] 0 2024-03-22T02:28:29.731 INFO:tasks.workunit.client.0.smithi103.stdout:1/20: getdents . 0 2024-03-22T02:28:29.732 INFO:tasks.workunit.client.0.smithi103.stdout:7/28: link d1/lc d1/lf 0 2024-03-22T02:28:29.733 INFO:tasks.workunit.client.0.smithi103.stdout:7/29: dread - d1/da/f3 zero size 2024-03-22T02:28:29.736 INFO:tasks.workunit.client.0.smithi103.stdout:1/21: link c4 c8 0 2024-03-22T02:28:29.740 INFO:tasks.workunit.client.0.smithi103.stdout:1/22: chown c4 371192866 1 2024-03-22T02:28:29.740 INFO:tasks.workunit.client.0.smithi103.stdout:1/23: rmdir - no directory 2024-03-22T02:28:29.741 INFO:tasks.workunit.client.0.smithi103.stdout:7/30: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:29.742 INFO:tasks.workunit.client.0.smithi103.stdout:7/31: fdatasync d1/da/f3 0 2024-03-22T02:28:29.743 INFO:tasks.workunit.client.0.smithi103.stdout:1/24: creat f9 x:0 0 0 2024-03-22T02:28:29.744 INFO:tasks.workunit.client.0.smithi103.stdout:1/25: chown c6 4 1 2024-03-22T02:28:29.747 INFO:tasks.workunit.client.0.smithi103.stdout:1/26: symlink la 0 2024-03-22T02:28:29.752 INFO:tasks.workunit.client.0.smithi103.stdout:7/32: dwrite d1/da/f8 [0,4194304] 0 2024-03-22T02:28:29.760 INFO:tasks.workunit.client.0.smithi103.stdout:1/27: dwrite f9 [0,4194304] 0 2024-03-22T02:28:29.761 INFO:tasks.workunit.client.0.smithi103.stdout:1/28: dread - f5 zero size 2024-03-22T02:28:29.764 INFO:tasks.workunit.client.0.smithi103.stdout:7/33: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:29.765 INFO:tasks.workunit.client.0.smithi103.stdout:7/34: dread - d1/da/fe zero size 2024-03-22T02:28:29.765 INFO:tasks.workunit.client.0.smithi103.stdout:1/29: mkdir db 0 2024-03-22T02:28:29.771 INFO:tasks.workunit.client.0.smithi103.stdout:7/35: mkdir d1/d10 0 2024-03-22T02:28:29.772 INFO:tasks.workunit.client.0.smithi103.stdout:1/30: rename f9 to db/fc 0 2024-03-22T02:28:29.779 INFO:tasks.workunit.client.0.smithi103.stdout:7/36: chown d1/lf 29936 1 2024-03-22T02:28:29.780 INFO:tasks.workunit.client.0.smithi103.stdout:7/37: write d1/da/f3 [998504,75195] 0 2024-03-22T02:28:29.783 INFO:tasks.workunit.client.0.smithi103.stdout:7/38: unlink d1/l7 0 2024-03-22T02:28:29.785 INFO:tasks.workunit.client.0.smithi103.stdout:7/39: chown d1/da/ld 0 1 2024-03-22T02:28:29.787 INFO:tasks.workunit.client.0.smithi103.stdout:7/40: stat d1/lc 0 2024-03-22T02:28:29.789 INFO:tasks.workunit.client.0.smithi103.stdout:7/41: mkdir d1/d10/d11 0 2024-03-22T02:28:29.794 INFO:tasks.workunit.client.0.smithi103.stdout:7/42: dread d1/da/f4 [0,4194304] 0 2024-03-22T02:28:29.798 INFO:tasks.workunit.client.0.smithi103.stdout:7/43: symlink d1/d10/d11/l12 0 2024-03-22T02:28:29.799 INFO:tasks.workunit.client.0.smithi103.stdout:7/44: rename d1/da to d1/da/d13 22 2024-03-22T02:28:29.801 INFO:tasks.workunit.client.0.smithi103.stdout:7/45: creat d1/d10/d11/f14 x:0 0 0 2024-03-22T02:28:29.805 INFO:tasks.workunit.client.0.smithi103.stdout:7/46: dread d1/da/f4 [0,4194304] 0 2024-03-22T02:28:29.812 INFO:tasks.workunit.client.0.smithi103.stdout:7/47: dwrite d1/da/f3 [0,4194304] 0 2024-03-22T02:28:29.813 INFO:tasks.workunit.client.0.smithi103.stdout:7/48: rename d1/d10 to d1/d10/d15 22 2024-03-22T02:28:29.844 INFO:tasks.workunit.client.0.smithi103.stdout:9/18: write f0 [5066349,49304] 0 2024-03-22T02:28:29.844 INFO:tasks.workunit.client.0.smithi103.stdout:2/15: truncate f1 115057 0 2024-03-22T02:28:29.846 INFO:tasks.workunit.client.0.smithi103.stdout:2/16: chown f2 1 1 2024-03-22T02:28:29.850 INFO:tasks.workunit.client.0.smithi103.stdout:3/55: write f1 [2905285,67942] 0 2024-03-22T02:28:29.851 INFO:tasks.workunit.client.0.smithi103.stdout:2/17: chown f1 815451 1 2024-03-22T02:28:29.852 INFO:tasks.workunit.client.0.smithi103.stdout:6/24: sync 2024-03-22T02:28:29.852 INFO:tasks.workunit.client.0.smithi103.stdout:0/49: sync 2024-03-22T02:28:29.854 INFO:tasks.workunit.client.0.smithi103.stdout:5/14: dwrite d0/f5 [0,4194304] 0 2024-03-22T02:28:29.857 INFO:tasks.workunit.client.0.smithi103.stdout:2/18: write f2 [307642,16515] 0 2024-03-22T02:28:29.859 INFO:tasks.workunit.client.0.smithi103.stdout:3/56: creat de/d12/f16 x:0 0 0 2024-03-22T02:28:29.862 INFO:tasks.workunit.client.0.smithi103.stdout:9/19: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:29.863 INFO:tasks.workunit.client.0.smithi103.stdout:1/31: truncate db/fc 4054398 0 2024-03-22T02:28:29.866 INFO:tasks.workunit.client.0.smithi103.stdout:8/104: dwrite f2 [0,4194304] 0 2024-03-22T02:28:29.870 INFO:tasks.workunit.client.0.smithi103.stdout:5/15: dread d0/f5 [0,4194304] 0 2024-03-22T02:28:29.870 INFO:tasks.workunit.client.0.smithi103.stdout:6/25: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:29.871 INFO:tasks.workunit.client.0.smithi103.stdout:3/57: unlink f2 0 2024-03-22T02:28:29.873 INFO:tasks.workunit.client.0.smithi103.stdout:3/58: dread - fc zero size 2024-03-22T02:28:29.877 INFO:tasks.workunit.client.0.smithi103.stdout:7/49: truncate d1/da/fb 1555126 0 2024-03-22T02:28:29.877 INFO:tasks.workunit.client.0.smithi103.stdout:9/20: dwrite f1 [0,4194304] 0 2024-03-22T02:28:29.882 INFO:tasks.workunit.client.0.smithi103.stdout:3/59: dread f7 [0,4194304] 0 2024-03-22T02:28:29.883 INFO:tasks.workunit.client.0.smithi103.stdout:9/21: dread d2/f3 [0,4194304] 0 2024-03-22T02:28:29.884 INFO:tasks.workunit.client.0.smithi103.stdout:9/22: read d2/f3 [1107124,84531] 0 2024-03-22T02:28:29.886 INFO:tasks.workunit.client.0.smithi103.stdout:0/50: getdents . 0 2024-03-22T02:28:29.889 INFO:tasks.workunit.client.0.smithi103.stdout:1/32: rename c8 to db/cd 0 2024-03-22T02:28:29.890 INFO:tasks.workunit.client.0.smithi103.stdout:2/19: dread f1 [0,4194304] 0 2024-03-22T02:28:29.891 INFO:tasks.workunit.client.0.smithi103.stdout:7/50: creat d1/f16 x:0 0 0 2024-03-22T02:28:29.892 INFO:tasks.workunit.client.0.smithi103.stdout:2/20: write f2 [1278491,5721] 0 2024-03-22T02:28:29.894 INFO:tasks.workunit.client.0.smithi103.stdout:9/23: dwrite d2/d5/f6 [0,4194304] 0 2024-03-22T02:28:29.895 INFO:tasks.workunit.client.0.smithi103.stdout:7/51: write d1/da/f3 [2644367,43486] 0 2024-03-22T02:28:29.901 INFO:tasks.workunit.client.0.smithi103.stdout:3/60: creat de/f17 x:0 0 0 2024-03-22T02:28:29.902 INFO:tasks.workunit.client.0.smithi103.stdout:6/26: rename d0/f8 to d0/fb 0 2024-03-22T02:28:29.905 INFO:tasks.workunit.client.0.smithi103.stdout:0/51: mkdir dd 0 2024-03-22T02:28:29.906 INFO:tasks.workunit.client.0.smithi103.stdout:1/33: fdatasync f7 0 2024-03-22T02:28:29.907 INFO:tasks.workunit.client.0.smithi103.stdout:0/52: stat lb 0 2024-03-22T02:28:29.909 INFO:tasks.workunit.client.0.smithi103.stdout:2/21: dwrite f2 [0,4194304] 0 2024-03-22T02:28:29.910 INFO:tasks.workunit.client.0.smithi103.stdout:9/24: dread f1 [0,4194304] 0 2024-03-22T02:28:29.912 INFO:tasks.workunit.client.0.smithi103.stdout:8/105: truncate f1 3117932 0 2024-03-22T02:28:29.912 INFO:tasks.workunit.client.0.smithi103.stdout:3/61: dwrite fa [0,4194304] 0 2024-03-22T02:28:29.913 INFO:tasks.workunit.client.0.smithi103.stdout:5/16: truncate d0/f5 1445386 0 2024-03-22T02:28:29.915 INFO:tasks.workunit.client.0.smithi103.stdout:2/22: dread f2 [0,4194304] 0 2024-03-22T02:28:29.916 INFO:tasks.workunit.client.0.smithi103.stdout:7/52: symlink d1/da/l17 0 2024-03-22T02:28:29.921 INFO:tasks.workunit.client.0.smithi103.stdout:0/53: write f4 [4166719,4588] 0 2024-03-22T02:28:29.922 INFO:tasks.workunit.client.0.smithi103.stdout:0/54: readlink l9 0 2024-03-22T02:28:29.925 INFO:tasks.workunit.client.0.smithi103.stdout:9/25: mknod d2/c8 0 2024-03-22T02:28:29.928 INFO:tasks.workunit.client.0.smithi103.stdout:4/35: sync 2024-03-22T02:28:29.930 INFO:tasks.workunit.client.0.smithi103.stdout:0/55: dwrite fa [0,4194304] 0 2024-03-22T02:28:29.932 INFO:tasks.workunit.client.0.smithi103.stdout:2/23: creat f3 x:0 0 0 2024-03-22T02:28:29.932 INFO:tasks.workunit.client.0.smithi103.stdout:7/53: rmdir d1/d10 39 2024-03-22T02:28:29.933 INFO:tasks.workunit.client.0.smithi103.stdout:7/54: fsync d1/f16 0 2024-03-22T02:28:29.933 INFO:tasks.workunit.client.0.smithi103.stdout:3/62: dread f7 [0,4194304] 0 2024-03-22T02:28:29.934 INFO:tasks.workunit.client.0.smithi103.stdout:6/27: mknod d0/d3/da/cc 0 2024-03-22T02:28:29.941 INFO:tasks.workunit.client.0.smithi103.stdout:8/106: creat d4/d13/f1b x:0 0 0 2024-03-22T02:28:29.942 INFO:tasks.workunit.client.0.smithi103.stdout:6/28: dwrite d0/d3/f9 [0,4194304] 0 2024-03-22T02:28:29.943 INFO:tasks.workunit.client.0.smithi103.stdout:8/107: chown d4/d14/l17 516206 1 2024-03-22T02:28:29.943 INFO:tasks.workunit.client.0.smithi103.stdout:1/34: truncate f7 1242319 0 2024-03-22T02:28:29.948 INFO:tasks.workunit.client.0.smithi103.stdout:3/63: fdatasync f8 0 2024-03-22T02:28:29.949 INFO:tasks.workunit.client.0.smithi103.stdout:3/64: stat f7 0 2024-03-22T02:28:29.950 INFO:tasks.workunit.client.0.smithi103.stdout:3/65: chown fd 0 1 2024-03-22T02:28:29.951 INFO:tasks.workunit.client.0.smithi103.stdout:6/29: rename d0/d3/c6 to d0/cd 0 2024-03-22T02:28:29.953 INFO:tasks.workunit.client.0.smithi103.stdout:8/108: creat d4/d13/f1c x:0 0 0 2024-03-22T02:28:29.954 INFO:tasks.workunit.client.0.smithi103.stdout:2/24: dwrite f1 [0,4194304] 0 2024-03-22T02:28:29.954 INFO:tasks.workunit.client.0.smithi103.stdout:6/30: truncate d0/d3/f9 4706515 0 2024-03-22T02:28:29.954 INFO:tasks.workunit.client.0.smithi103.stdout:1/35: unlink l2 0 2024-03-22T02:28:29.957 INFO:tasks.workunit.client.0.smithi103.stdout:7/55: creat d1/d10/d11/f18 x:0 0 0 2024-03-22T02:28:29.961 INFO:tasks.workunit.client.0.smithi103.stdout:3/66: symlink de/d12/l18 0 2024-03-22T02:28:29.961 INFO:tasks.workunit.client.0.smithi103.stdout:8/109: dwrite d4/f12 [0,4194304] 0 2024-03-22T02:28:29.963 INFO:tasks.workunit.client.0.smithi103.stdout:8/110: fdatasync d4/f9 0 2024-03-22T02:28:29.964 INFO:tasks.workunit.client.0.smithi103.stdout:1/36: stat c3 0 2024-03-22T02:28:29.965 INFO:tasks.workunit.client.0.smithi103.stdout:1/37: readlink la 0 2024-03-22T02:28:29.967 INFO:tasks.workunit.client.0.smithi103.stdout:9/26: getdents d2 0 2024-03-22T02:28:29.970 INFO:tasks.workunit.client.0.smithi103.stdout:5/17: dwrite d0/f5 [0,4194304] 0 2024-03-22T02:28:29.973 INFO:tasks.workunit.client.0.smithi103.stdout:8/111: mkdir d4/d13/d1d 0 2024-03-22T02:28:29.974 INFO:tasks.workunit.client.0.smithi103.stdout:6/31: rmdir d0 39 2024-03-22T02:28:29.979 INFO:tasks.workunit.client.0.smithi103.stdout:9/27: creat d2/f9 x:0 0 0 2024-03-22T02:28:29.981 INFO:tasks.workunit.client.0.smithi103.stdout:5/18: mknod d0/c6 0 2024-03-22T02:28:29.982 INFO:tasks.workunit.client.0.smithi103.stdout:2/25: dwrite f2 [0,4194304] 0 2024-03-22T02:28:29.983 INFO:tasks.workunit.client.0.smithi103.stdout:4/36: dwrite d3/f7 [0,4194304] 0 2024-03-22T02:28:29.984 INFO:tasks.workunit.client.0.smithi103.stdout:5/19: stat d0/f5 0 2024-03-22T02:28:29.985 INFO:tasks.workunit.client.0.smithi103.stdout:8/112: creat d4/d14/f1e x:0 0 0 2024-03-22T02:28:29.988 INFO:tasks.workunit.client.0.smithi103.stdout:2/26: creat f4 x:0 0 0 2024-03-22T02:28:29.988 INFO:tasks.workunit.client.0.smithi103.stdout:4/37: creat d3/fc x:0 0 0 2024-03-22T02:28:29.989 INFO:tasks.workunit.client.0.smithi103.stdout:8/113: chown d4/d14/c16 1588 1 2024-03-22T02:28:29.992 INFO:tasks.workunit.client.0.smithi103.stdout:5/20: creat d0/f7 x:0 0 0 2024-03-22T02:28:29.993 INFO:tasks.workunit.client.0.smithi103.stdout:6/32: write d0/d3/f9 [2621230,54800] 0 2024-03-22T02:28:29.995 INFO:tasks.workunit.client.0.smithi103.stdout:2/27: write f3 [644579,86500] 0 2024-03-22T02:28:29.995 INFO:tasks.workunit.client.0.smithi103.stdout:5/21: stat d0/c6 0 2024-03-22T02:28:29.997 INFO:tasks.workunit.client.0.smithi103.stdout:9/28: dwrite d2/f9 [0,4194304] 0 2024-03-22T02:28:29.998 INFO:tasks.workunit.client.0.smithi103.stdout:8/114: unlink d4/f8 0 2024-03-22T02:28:30.000 INFO:tasks.workunit.client.0.smithi103.stdout:5/22: write d0/f7 [888661,36463] 0 2024-03-22T02:28:30.000 INFO:tasks.workunit.client.0.smithi103.stdout:3/67: dwrite f8 [0,4194304] 0 2024-03-22T02:28:30.003 INFO:tasks.workunit.client.0.smithi103.stdout:2/28: creat f5 x:0 0 0 2024-03-22T02:28:30.008 INFO:tasks.workunit.client.0.smithi103.stdout:8/115: symlink d4/d19/l1f 0 2024-03-22T02:28:30.009 INFO:tasks.workunit.client.0.smithi103.stdout:2/29: creat f6 x:0 0 0 2024-03-22T02:28:30.010 INFO:tasks.workunit.client.0.smithi103.stdout:8/116: dread - d4/d13/f1c zero size 2024-03-22T02:28:30.012 INFO:tasks.workunit.client.0.smithi103.stdout:5/23: rename d0/c6 to d0/c8 0 2024-03-22T02:28:30.014 INFO:tasks.workunit.client.0.smithi103.stdout:2/30: creat f7 x:0 0 0 2024-03-22T02:28:30.015 INFO:tasks.workunit.client.0.smithi103.stdout:1/38: dwrite f7 [0,4194304] 0 2024-03-22T02:28:30.017 INFO:tasks.workunit.client.0.smithi103.stdout:8/117: symlink d4/l20 0 2024-03-22T02:28:30.018 INFO:tasks.workunit.client.0.smithi103.stdout:6/33: getdents d0 0 2024-03-22T02:28:30.019 INFO:tasks.workunit.client.0.smithi103.stdout:3/68: getdents de/d12 0 2024-03-22T02:28:30.021 INFO:tasks.workunit.client.0.smithi103.stdout:2/31: rename f6 to f8 0 2024-03-22T02:28:30.022 INFO:tasks.workunit.client.0.smithi103.stdout:5/24: dwrite d0/f5 [0,4194304] 0 2024-03-22T02:28:30.024 INFO:tasks.workunit.client.0.smithi103.stdout:8/118: symlink d4/l21 0 2024-03-22T02:28:30.024 INFO:tasks.workunit.client.0.smithi103.stdout:4/38: fdatasync d3/f7 0 2024-03-22T02:28:30.025 INFO:tasks.workunit.client.0.smithi103.stdout:2/32: read - f8 zero size 2024-03-22T02:28:30.026 INFO:tasks.workunit.client.0.smithi103.stdout:8/119: write d4/ff [68275,57331] 0 2024-03-22T02:28:30.026 INFO:tasks.workunit.client.0.smithi103.stdout:2/33: write f5 [762395,24504] 0 2024-03-22T02:28:30.027 INFO:tasks.workunit.client.0.smithi103.stdout:3/69: mkdir de/d12/d19 0 2024-03-22T02:28:30.027 INFO:tasks.workunit.client.0.smithi103.stdout:2/34: stat f1 0 2024-03-22T02:28:30.028 INFO:tasks.workunit.client.0.smithi103.stdout:3/70: chown de/cf 1970231967 1 2024-03-22T02:28:30.028 INFO:tasks.workunit.client.0.smithi103.stdout:2/35: chown f7 2484 1 2024-03-22T02:28:30.029 INFO:tasks.workunit.client.0.smithi103.stdout:6/34: creat d0/d3/da/fe x:0 0 0 2024-03-22T02:28:30.030 INFO:tasks.workunit.client.0.smithi103.stdout:5/25: dread d0/f5 [0,4194304] 0 2024-03-22T02:28:30.031 INFO:tasks.workunit.client.0.smithi103.stdout:9/29: write f1 [4154539,82096] 0 2024-03-22T02:28:30.032 INFO:tasks.workunit.client.0.smithi103.stdout:6/35: write d0/d3/f9 [3322110,78378] 0 2024-03-22T02:28:30.034 INFO:tasks.workunit.client.0.smithi103.stdout:8/120: unlink d4/fc 0 2024-03-22T02:28:30.036 INFO:tasks.workunit.client.0.smithi103.stdout:3/71: mkdir de/d1a 0 2024-03-22T02:28:30.038 INFO:tasks.workunit.client.0.smithi103.stdout:2/36: creat f9 x:0 0 0 2024-03-22T02:28:30.039 INFO:tasks.workunit.client.0.smithi103.stdout:8/121: dread d4/f9 [4194304,4194304] 0 2024-03-22T02:28:30.044 INFO:tasks.workunit.client.0.smithi103.stdout:3/72: readlink de/l11 0 2024-03-22T02:28:30.046 INFO:tasks.workunit.client.0.smithi103.stdout:3/73: write fd [1082263,32233] 0 2024-03-22T02:28:30.046 INFO:tasks.workunit.client.0.smithi103.stdout:5/26: unlink d0/c8 0 2024-03-22T02:28:30.047 INFO:tasks.workunit.client.0.smithi103.stdout:8/122: dwrite d4/fd [0,4194304] 0 2024-03-22T02:28:30.048 INFO:tasks.workunit.client.0.smithi103.stdout:8/123: fsync d4/ff 0 2024-03-22T02:28:30.050 INFO:tasks.workunit.client.0.smithi103.stdout:4/39: getdents d3 0 2024-03-22T02:28:30.051 INFO:tasks.workunit.client.0.smithi103.stdout:5/27: creat d0/f9 x:0 0 0 2024-03-22T02:28:30.052 INFO:tasks.workunit.client.0.smithi103.stdout:3/74: rename de/f10 to de/d12/f1b 0 2024-03-22T02:28:30.054 INFO:tasks.workunit.client.0.smithi103.stdout:9/30: getdents d2/d5 0 2024-03-22T02:28:30.056 INFO:tasks.workunit.client.0.smithi103.stdout:8/124: mknod d4/d19/c22 0 2024-03-22T02:28:30.057 INFO:tasks.workunit.client.0.smithi103.stdout:4/40: creat d3/fd x:0 0 0 2024-03-22T02:28:30.059 INFO:tasks.workunit.client.0.smithi103.stdout:4/41: readlink d3/l6 0 2024-03-22T02:28:30.059 INFO:tasks.workunit.client.0.smithi103.stdout:8/125: truncate d4/d13/f1c 944114 0 2024-03-22T02:28:30.060 INFO:tasks.workunit.client.0.smithi103.stdout:4/42: dread - d3/fd zero size 2024-03-22T02:28:30.060 INFO:tasks.workunit.client.0.smithi103.stdout:3/75: dwrite de/d12/f16 [0,4194304] 0 2024-03-22T02:28:30.061 INFO:tasks.workunit.client.0.smithi103.stdout:9/31: mkdir d2/da 0 2024-03-22T02:28:30.062 INFO:tasks.workunit.client.0.smithi103.stdout:4/43: write d3/fb [3817318,12696] 0 2024-03-22T02:28:30.064 INFO:tasks.workunit.client.0.smithi103.stdout:5/28: link d0/f5 d0/fa 0 2024-03-22T02:28:30.065 INFO:tasks.workunit.client.0.smithi103.stdout:8/126: rmdir d4/d19 39 2024-03-22T02:28:30.067 INFO:tasks.workunit.client.0.smithi103.stdout:3/76: creat de/d12/f1c x:0 0 0 2024-03-22T02:28:30.069 INFO:tasks.workunit.client.0.smithi103.stdout:9/32: creat d2/d5/fb x:0 0 0 2024-03-22T02:28:30.070 INFO:tasks.workunit.client.0.smithi103.stdout:4/44: symlink d3/le 0 2024-03-22T02:28:30.072 INFO:tasks.workunit.client.0.smithi103.stdout:5/29: dread d0/fa [0,4194304] 0 2024-03-22T02:28:30.075 INFO:tasks.workunit.client.0.smithi103.stdout:4/45: symlink d3/lf 0 2024-03-22T02:28:30.077 INFO:tasks.workunit.client.0.smithi103.stdout:6/36: dwrite d0/fb [0,4194304] 0 2024-03-22T02:28:30.077 INFO:tasks.workunit.client.0.smithi103.stdout:8/127: mknod d4/d19/c23 0 2024-03-22T02:28:30.079 INFO:tasks.workunit.client.0.smithi103.stdout:9/33: rename d2/d5/f6 to d2/da/fc 0 2024-03-22T02:28:30.080 INFO:tasks.workunit.client.0.smithi103.stdout:5/30: mknod d0/cb 0 2024-03-22T02:28:30.084 INFO:tasks.workunit.client.0.smithi103.stdout:9/34: write d2/f9 [57279,14056] 0 2024-03-22T02:28:30.086 INFO:tasks.workunit.client.0.smithi103.stdout:3/77: dwrite de/d12/f16 [0,4194304] 0 2024-03-22T02:28:30.089 INFO:tasks.workunit.client.0.smithi103.stdout:5/31: creat d0/fc x:0 0 0 2024-03-22T02:28:30.091 INFO:tasks.workunit.client.0.smithi103.stdout:6/37: dread d0/fb [0,4194304] 0 2024-03-22T02:28:30.091 INFO:tasks.workunit.client.0.smithi103.stdout:4/46: dwrite d3/fc [0,4194304] 0 2024-03-22T02:28:30.092 INFO:tasks.workunit.client.0.smithi103.stdout:5/32: read d0/f5 [1354402,88979] 0 2024-03-22T02:28:30.095 INFO:tasks.workunit.client.0.smithi103.stdout:9/35: creat d2/da/fd x:0 0 0 2024-03-22T02:28:30.096 INFO:tasks.workunit.client.0.smithi103.stdout:3/78: dread f8 [0,4194304] 0 2024-03-22T02:28:30.097 INFO:tasks.workunit.client.0.smithi103.stdout:6/38: fsync d0/f2 0 2024-03-22T02:28:30.099 INFO:tasks.workunit.client.0.smithi103.stdout:4/47: mknod d3/c10 0 2024-03-22T02:28:30.102 INFO:tasks.workunit.client.0.smithi103.stdout:5/33: creat d0/fd x:0 0 0 2024-03-22T02:28:30.103 INFO:tasks.workunit.client.0.smithi103.stdout:5/34: truncate d0/fc 566313 0 2024-03-22T02:28:30.103 INFO:tasks.workunit.client.0.smithi103.stdout:9/36: mkdir d2/da/de 0 2024-03-22T02:28:30.105 INFO:tasks.workunit.client.0.smithi103.stdout:8/128: link d4/d19/c22 d4/c24 0 2024-03-22T02:28:30.105 INFO:tasks.workunit.client.0.smithi103.stdout:9/37: chown d2/d5 1280890803 1 2024-03-22T02:28:30.105 INFO:tasks.workunit.client.0.smithi103.stdout:5/35: read d0/fc [402195,121669] 0 2024-03-22T02:28:30.106 INFO:tasks.workunit.client.0.smithi103.stdout:4/48: dread d3/fb [0,4194304] 0 2024-03-22T02:28:30.107 INFO:tasks.workunit.client.0.smithi103.stdout:6/39: mkdir d0/d3/da/df 0 2024-03-22T02:28:30.108 INFO:tasks.workunit.client.0.smithi103.stdout:5/36: write d0/f7 [790934,114279] 0 2024-03-22T02:28:30.111 INFO:tasks.workunit.client.0.smithi103.stdout:9/38: creat d2/d5/ff x:0 0 0 2024-03-22T02:28:30.115 INFO:tasks.workunit.client.0.smithi103.stdout:6/40: mknod d0/c10 0 2024-03-22T02:28:30.117 INFO:tasks.workunit.client.0.smithi103.stdout:4/49: creat d3/f11 x:0 0 0 2024-03-22T02:28:30.118 INFO:tasks.workunit.client.0.smithi103.stdout:9/39: mkdir d2/d10 0 2024-03-22T02:28:30.120 INFO:tasks.workunit.client.0.smithi103.stdout:5/37: fsync d0/fd 0 2024-03-22T02:28:30.121 INFO:tasks.workunit.client.0.smithi103.stdout:9/40: read d2/da/fc [1345237,10918] 0 2024-03-22T02:28:30.124 INFO:tasks.workunit.client.0.smithi103.stdout:6/41: write d0/fb [491485,121732] 0 2024-03-22T02:28:30.124 INFO:tasks.workunit.client.0.smithi103.stdout:4/50: read d3/f7 [2130645,29035] 0 2024-03-22T02:28:30.124 INFO:tasks.workunit.client.0.smithi103.stdout:3/79: dwrite f8 [4194304,4194304] 0 2024-03-22T02:28:30.124 INFO:tasks.workunit.client.0.smithi103.stdout:8/129: dwrite d4/d13/f1b [0,4194304] 0 2024-03-22T02:28:30.126 INFO:tasks.workunit.client.0.smithi103.stdout:5/38: creat d0/fe x:0 0 0 2024-03-22T02:28:30.127 INFO:tasks.workunit.client.0.smithi103.stdout:9/41: write d2/da/fd [719993,130651] 0 2024-03-22T02:28:30.132 INFO:tasks.workunit.client.0.smithi103.stdout:5/39: mknod d0/cf 0 2024-03-22T02:28:30.133 INFO:tasks.workunit.client.0.smithi103.stdout:3/80: mknod de/d12/c1d 0 2024-03-22T02:28:30.135 INFO:tasks.workunit.client.0.smithi103.stdout:6/42: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:30.135 INFO:tasks.workunit.client.0.smithi103.stdout:5/40: write d0/f7 [1635478,55552] 0 2024-03-22T02:28:30.135 INFO:tasks.workunit.client.0.smithi103.stdout:3/81: chown de/d12/f16 8 1 2024-03-22T02:28:30.136 INFO:tasks.workunit.client.0.smithi103.stdout:9/42: symlink d2/da/de/l11 0 2024-03-22T02:28:30.137 INFO:tasks.workunit.client.0.smithi103.stdout:5/41: fsync d0/fe 0 2024-03-22T02:28:30.137 INFO:tasks.workunit.client.0.smithi103.stdout:5/42: stat d0 0 2024-03-22T02:28:30.138 INFO:tasks.workunit.client.0.smithi103.stdout:8/130: dread d4/f12 [0,4194304] 0 2024-03-22T02:28:30.139 INFO:tasks.workunit.client.0.smithi103.stdout:3/82: write de/d12/f1b [606812,66298] 0 2024-03-22T02:28:30.141 INFO:tasks.workunit.client.0.smithi103.stdout:6/43: creat d0/d3/da/df/f11 x:0 0 0 2024-03-22T02:28:30.143 INFO:tasks.workunit.client.0.smithi103.stdout:3/83: creat de/f1e x:0 0 0 2024-03-22T02:28:30.143 INFO:tasks.workunit.client.0.smithi103.stdout:8/131: mknod d4/d19/d1a/c25 0 2024-03-22T02:28:30.146 INFO:tasks.workunit.client.0.smithi103.stdout:3/84: write fc [469930,90302] 0 2024-03-22T02:28:30.146 INFO:tasks.workunit.client.0.smithi103.stdout:8/132: mkdir d4/d26 0 2024-03-22T02:28:30.148 INFO:tasks.workunit.client.0.smithi103.stdout:9/43: dwrite d2/d5/fb [0,4194304] 0 2024-03-22T02:28:30.149 INFO:tasks.workunit.client.0.smithi103.stdout:3/85: creat de/f1f x:0 0 0 2024-03-22T02:28:30.154 INFO:tasks.workunit.client.0.smithi103.stdout:3/86: creat de/d12/f20 x:0 0 0 2024-03-22T02:28:30.154 INFO:tasks.workunit.client.0.smithi103.stdout:9/44: creat d2/da/de/f12 x:0 0 0 2024-03-22T02:28:30.155 INFO:tasks.workunit.client.0.smithi103.stdout:6/44: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:30.155 INFO:tasks.workunit.client.0.smithi103.stdout:3/87: fdatasync fa 0 2024-03-22T02:28:30.156 INFO:tasks.workunit.client.0.smithi103.stdout:6/45: chown d0/d3/da 0 1 2024-03-22T02:28:30.157 INFO:tasks.workunit.client.0.smithi103.stdout:5/43: write d0/fc [620034,44711] 0 2024-03-22T02:28:30.157 INFO:tasks.workunit.client.0.smithi103.stdout:3/88: chown fa 30 1 2024-03-22T02:28:30.160 INFO:tasks.workunit.client.0.smithi103.stdout:9/45: rename f0 to d2/d10/f13 0 2024-03-22T02:28:30.163 INFO:tasks.workunit.client.0.smithi103.stdout:5/44: rename d0/c2 to d0/c10 0 2024-03-22T02:28:30.163 INFO:tasks.workunit.client.0.smithi103.stdout:5/45: dread - d0/fd zero size 2024-03-22T02:28:30.164 INFO:tasks.workunit.client.0.smithi103.stdout:5/46: dread - d0/fd zero size 2024-03-22T02:28:30.165 INFO:tasks.workunit.client.0.smithi103.stdout:6/46: symlink d0/d3/l12 0 2024-03-22T02:28:30.167 INFO:tasks.workunit.client.0.smithi103.stdout:3/89: getdents de/d12/d19 0 2024-03-22T02:28:30.204 INFO:tasks.workunit.client.0.smithi103.stdout:6/47: rename d0/c1 to d0/d3/c13 0 2024-03-22T02:28:30.207 INFO:tasks.workunit.client.0.smithi103.stdout:5/47: dwrite d0/fe [0,4194304] 0 2024-03-22T02:28:30.207 INFO:tasks.workunit.client.0.smithi103.stdout:9/46: getdents d2/d10 0 2024-03-22T02:28:30.209 INFO:tasks.workunit.client.0.smithi103.stdout:5/48: dread - d0/fd zero size 2024-03-22T02:28:30.209 INFO:tasks.workunit.client.0.smithi103.stdout:6/48: write d0/d3/f4 [1839984,76232] 0 2024-03-22T02:28:30.210 INFO:tasks.workunit.client.0.smithi103.stdout:5/49: chown d0/fd 129189466 1 2024-03-22T02:28:30.210 INFO:tasks.workunit.client.0.smithi103.stdout:3/90: rename de/d12/f20 to de/f21 0 2024-03-22T02:28:30.211 INFO:tasks.workunit.client.0.smithi103.stdout:3/91: stat de/d12/f16 0 2024-03-22T02:28:30.212 INFO:tasks.workunit.client.0.smithi103.stdout:5/50: symlink d0/l11 0 2024-03-22T02:28:30.215 INFO:tasks.workunit.client.0.smithi103.stdout:9/47: rename d2/da/de/f12 to d2/f14 0 2024-03-22T02:28:30.216 INFO:tasks.workunit.client.0.smithi103.stdout:3/92: creat de/f22 x:0 0 0 2024-03-22T02:28:30.218 INFO:tasks.workunit.client.0.smithi103.stdout:9/48: chown d2/d5/fb 1684 1 2024-03-22T02:28:30.219 INFO:tasks.workunit.client.0.smithi103.stdout:3/93: write de/f1f [646296,38007] 0 2024-03-22T02:28:30.221 INFO:tasks.workunit.client.0.smithi103.stdout:6/49: rename d0/d3/da to d0/d14 0 2024-03-22T02:28:30.222 INFO:tasks.workunit.client.0.smithi103.stdout:5/51: dwrite d0/fd [0,4194304] 0 2024-03-22T02:28:30.223 INFO:tasks.workunit.client.0.smithi103.stdout:5/52: read d0/fa [766837,1432] 0 2024-03-22T02:28:30.225 INFO:tasks.workunit.client.0.smithi103.stdout:3/94: rename fc to de/f23 0 2024-03-22T02:28:30.226 INFO:tasks.workunit.client.0.smithi103.stdout:5/53: truncate d0/f7 2011662 0 2024-03-22T02:28:30.228 INFO:tasks.workunit.client.0.smithi103.stdout:3/95: write de/f17 [870048,98936] 0 2024-03-22T02:28:30.232 INFO:tasks.workunit.client.0.smithi103.stdout:6/50: dwrite d0/d3/f9 [0,4194304] 0 2024-03-22T02:28:30.233 INFO:tasks.workunit.client.0.smithi103.stdout:9/49: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:30.235 INFO:tasks.workunit.client.0.smithi103.stdout:5/54: getdents d0 0 2024-03-22T02:28:30.236 INFO:tasks.workunit.client.0.smithi103.stdout:6/51: chown d0/d14/df/f11 3533 1 2024-03-22T02:28:30.236 INFO:tasks.workunit.client.0.smithi103.stdout:9/50: chown d2/da/fd 19884487 1 2024-03-22T02:28:30.240 INFO:tasks.workunit.client.0.smithi103.stdout:9/51: chown d2/d10/f13 353 1 2024-03-22T02:28:30.246 INFO:tasks.workunit.client.0.smithi103.stdout:3/96: dwrite de/d12/f16 [0,4194304] 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:6/52: creat d0/d14/df/f15 x:0 0 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:9/52: symlink d2/da/l15 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:5/55: write d0/f5 [3996419,41622] 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:3/97: fsync de/f23 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:6/53: rmdir d0/d3 39 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:3/98: stat de/cf 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:3/99: fdatasync f7 0 2024-03-22T02:28:30.247 INFO:tasks.workunit.client.0.smithi103.stdout:3/100: dread - de/f21 zero size 2024-03-22T02:28:30.248 INFO:tasks.workunit.client.0.smithi103.stdout:3/101: truncate de/f21 85710 0 2024-03-22T02:28:30.249 INFO:tasks.workunit.client.0.smithi103.stdout:3/102: truncate fd 4628614 0 2024-03-22T02:28:30.253 INFO:tasks.workunit.client.0.smithi103.stdout:6/54: rename d0/cd to d0/d3/c16 0 2024-03-22T02:28:30.256 INFO:tasks.workunit.client.0.smithi103.stdout:3/103: dread de/f23 [0,4194304] 0 2024-03-22T02:28:30.257 INFO:tasks.workunit.client.0.smithi103.stdout:9/53: rmdir d2 39 2024-03-22T02:28:30.258 INFO:tasks.workunit.client.0.smithi103.stdout:5/56: truncate d0/fc 578072 0 2024-03-22T02:28:30.262 INFO:tasks.workunit.client.0.smithi103.stdout:6/55: dwrite d0/d14/df/f11 [0,4194304] 0 2024-03-22T02:28:30.262 INFO:tasks.workunit.client.0.smithi103.stdout:9/54: write d2/d5/fb [1570942,67786] 0 2024-03-22T02:28:30.262 INFO:tasks.workunit.client.0.smithi103.stdout:3/104: rename de/f17 to de/d1a/f24 0 2024-03-22T02:28:30.266 INFO:tasks.workunit.client.0.smithi103.stdout:5/57: rename d0/f9 to d0/f12 0 2024-03-22T02:28:30.268 INFO:tasks.workunit.client.0.smithi103.stdout:3/105: mkdir de/d1a/d25 0 2024-03-22T02:28:30.274 INFO:tasks.workunit.client.0.smithi103.stdout:3/106: mkdir de/d1a/d25/d26 0 2024-03-22T02:28:30.275 INFO:tasks.workunit.client.0.smithi103.stdout:6/56: write d0/d14/df/f11 [4763949,23044] 0 2024-03-22T02:28:30.275 INFO:tasks.workunit.client.0.smithi103.stdout:9/55: getdents d2 0 2024-03-22T02:28:30.277 INFO:tasks.workunit.client.0.smithi103.stdout:6/57: dread - d0/d14/df/f15 zero size 2024-03-22T02:28:30.278 INFO:tasks.workunit.client.0.smithi103.stdout:3/107: creat de/f27 x:0 0 0 2024-03-22T02:28:30.280 INFO:tasks.workunit.client.0.smithi103.stdout:9/56: symlink d2/d10/l16 0 2024-03-22T02:28:30.280 INFO:tasks.workunit.client.0.smithi103.stdout:6/58: mknod d0/d14/c17 0 2024-03-22T02:28:30.282 INFO:tasks.workunit.client.0.smithi103.stdout:5/58: getdents d0 0 2024-03-22T02:28:30.283 INFO:tasks.workunit.client.0.smithi103.stdout:9/57: mkdir d2/d10/d17 0 2024-03-22T02:28:30.284 INFO:tasks.workunit.client.0.smithi103.stdout:3/108: dread f7 [4194304,4194304] 0 2024-03-22T02:28:30.286 INFO:tasks.workunit.client.0.smithi103.stdout:9/58: mkdir d2/da/d18 0 2024-03-22T02:28:30.287 INFO:tasks.workunit.client.0.smithi103.stdout:9/59: fdatasync d2/d5/fb 0 2024-03-22T02:28:30.289 INFO:tasks.workunit.client.0.smithi103.stdout:3/109: dread de/d12/f1b [0,4194304] 0 2024-03-22T02:28:30.292 INFO:tasks.workunit.client.0.smithi103.stdout:5/59: dwrite d0/fa [4194304,4194304] 0 2024-03-22T02:28:30.292 INFO:tasks.workunit.client.0.smithi103.stdout:9/60: rename d2/l7 to d2/d5/l19 0 2024-03-22T02:28:30.292 INFO:tasks.workunit.client.0.smithi103.stdout:5/60: dread - d0/f12 zero size 2024-03-22T02:28:30.293 INFO:tasks.workunit.client.0.smithi103.stdout:9/61: write d2/d5/fb [794636,38345] 0 2024-03-22T02:28:30.294 INFO:tasks.workunit.client.0.smithi103.stdout:3/110: rmdir de/d12 39 2024-03-22T02:28:30.299 INFO:tasks.workunit.client.0.smithi103.stdout:9/62: write d2/f14 [1000600,127512] 0 2024-03-22T02:28:30.300 INFO:tasks.workunit.client.0.smithi103.stdout:5/61: dread d0/fd [0,4194304] 0 2024-03-22T02:28:30.301 INFO:tasks.workunit.client.0.smithi103.stdout:9/63: readlink d2/da/de/l11 0 2024-03-22T02:28:30.302 INFO:tasks.workunit.client.0.smithi103.stdout:5/62: chown d0/c4 46568 1 2024-03-22T02:28:30.302 INFO:tasks.workunit.client.0.smithi103.stdout:9/64: stat d2/d5/fb 0 2024-03-22T02:28:30.303 INFO:tasks.workunit.client.0.smithi103.stdout:3/111: dwrite fd [0,4194304] 0 2024-03-22T02:28:30.303 INFO:tasks.workunit.client.0.smithi103.stdout:9/65: readlink d2/da/l15 0 2024-03-22T02:28:30.304 INFO:tasks.workunit.client.0.smithi103.stdout:9/66: write d2/f3 [1917557,15191] 0 2024-03-22T02:28:30.307 INFO:tasks.workunit.client.0.smithi103.stdout:9/67: creat d2/da/f1a x:0 0 0 2024-03-22T02:28:30.309 INFO:tasks.workunit.client.0.smithi103.stdout:9/68: creat d2/da/de/f1b x:0 0 0 2024-03-22T02:28:30.315 INFO:tasks.workunit.client.0.smithi103.stdout:3/112: dwrite de/d12/f1b [0,4194304] 0 2024-03-22T02:28:30.319 INFO:tasks.workunit.client.0.smithi103.stdout:9/69: dwrite d2/da/fd [0,4194304] 0 2024-03-22T02:28:30.323 INFO:tasks.workunit.client.0.smithi103.stdout:9/70: unlink d2/da/f1a 0 2024-03-22T02:28:30.326 INFO:tasks.workunit.client.0.smithi103.stdout:3/113: dwrite f8 [4194304,4194304] 0 2024-03-22T02:28:30.328 INFO:tasks.workunit.client.0.smithi103.stdout:9/71: rename d2/c4 to d2/da/de/c1c 0 2024-03-22T02:28:30.330 INFO:tasks.workunit.client.0.smithi103.stdout:3/114: creat de/d12/d19/f28 x:0 0 0 2024-03-22T02:28:30.336 INFO:tasks.workunit.client.0.smithi103.stdout:3/115: dwrite fd [4194304,4194304] 0 2024-03-22T02:28:30.344 INFO:tasks.workunit.client.0.smithi103.stdout:3/116: dread de/d1a/f24 [0,4194304] 0 2024-03-22T02:28:30.344 INFO:tasks.workunit.client.0.smithi103.stdout:9/72: truncate d2/d10/f13 141161 0 2024-03-22T02:28:30.348 INFO:tasks.workunit.client.0.smithi103.stdout:3/117: dread fa [0,4194304] 0 2024-03-22T02:28:30.350 INFO:tasks.workunit.client.0.smithi103.stdout:3/118: creat de/d1a/d25/f29 x:0 0 0 2024-03-22T02:28:30.356 INFO:tasks.workunit.client.0.smithi103.stdout:3/119: dwrite de/f23 [0,4194304] 0 2024-03-22T02:28:30.359 INFO:tasks.workunit.client.0.smithi103.stdout:3/120: creat de/d1a/f2a x:0 0 0 2024-03-22T02:28:30.361 INFO:tasks.workunit.client.0.smithi103.stdout:9/73: dread d2/d10/f13 [0,4194304] 0 2024-03-22T02:28:30.362 INFO:tasks.workunit.client.0.smithi103.stdout:3/121: creat de/d12/f2b x:0 0 0 2024-03-22T02:28:30.398 INFO:tasks.workunit.client.0.smithi103.stdout:0/56: sync 2024-03-22T02:28:30.401 INFO:tasks.workunit.client.0.smithi103.stdout:0/57: mknod dd/ce 0 2024-03-22T02:28:30.402 INFO:tasks.workunit.client.0.smithi103.stdout:9/74: truncate d2/f14 1090169 0 2024-03-22T02:28:30.403 INFO:tasks.workunit.client.0.smithi103.stdout:0/58: mknod dd/cf 0 2024-03-22T02:28:30.404 INFO:tasks.workunit.client.0.smithi103.stdout:0/59: truncate fc 1020688 0 2024-03-22T02:28:30.405 INFO:tasks.workunit.client.0.smithi103.stdout:9/75: fdatasync d2/d10/f13 0 2024-03-22T02:28:30.407 INFO:tasks.workunit.client.0.smithi103.stdout:9/76: write d2/da/fd [1873694,27228] 0 2024-03-22T02:28:30.407 INFO:tasks.workunit.client.0.smithi103.stdout:3/122: truncate f1 1330598 0 2024-03-22T02:28:30.407 INFO:tasks.workunit.client.0.smithi103.stdout:3/123: chown de 248984121 1 2024-03-22T02:28:30.408 INFO:tasks.workunit.client.0.smithi103.stdout:9/77: readlink d2/da/l15 0 2024-03-22T02:28:30.414 INFO:tasks.workunit.client.0.smithi103.stdout:3/124: dwrite f8 [4194304,4194304] 0 2024-03-22T02:28:30.416 INFO:tasks.workunit.client.0.smithi103.stdout:3/125: mknod de/d12/c2c 0 2024-03-22T02:28:30.420 INFO:tasks.workunit.client.0.smithi103.stdout:9/78: truncate d2/d10/f13 1000266 0 2024-03-22T02:28:30.423 INFO:tasks.workunit.client.0.smithi103.stdout:3/126: rename de/d12/f1b to de/f2d 0 2024-03-22T02:28:30.424 INFO:tasks.workunit.client.0.smithi103.stdout:0/60: truncate f4 1358089 0 2024-03-22T02:28:30.427 INFO:tasks.workunit.client.0.smithi103.stdout:9/79: rename d2/c8 to d2/d10/c1d 0 2024-03-22T02:28:30.428 INFO:tasks.workunit.client.0.smithi103.stdout:0/61: symlink dd/l10 0 2024-03-22T02:28:30.429 INFO:tasks.workunit.client.0.smithi103.stdout:0/62: dread - f6 zero size 2024-03-22T02:28:30.429 INFO:tasks.workunit.client.0.smithi103.stdout:3/127: mknod de/d1a/d25/d26/c2e 0 2024-03-22T02:28:30.432 INFO:tasks.workunit.client.0.smithi103.stdout:0/63: creat dd/f11 x:0 0 0 2024-03-22T02:28:30.432 INFO:tasks.workunit.client.0.smithi103.stdout:3/128: creat de/d1a/d25/d26/f2f x:0 0 0 2024-03-22T02:28:30.433 INFO:tasks.workunit.client.0.smithi103.stdout:9/80: mkdir d2/da/d18/d1e 0 2024-03-22T02:28:30.433 INFO:tasks.workunit.client.0.smithi103.stdout:9/81: chown d2/da/de 58 1 2024-03-22T02:28:30.435 INFO:tasks.workunit.client.0.smithi103.stdout:9/82: write d2/da/fd [219549,3479] 0 2024-03-22T02:28:30.435 INFO:tasks.workunit.client.0.smithi103.stdout:3/129: write de/f2d [389870,55058] 0 2024-03-22T02:28:30.435 INFO:tasks.workunit.client.0.smithi103.stdout:9/83: chown d2/d5 2894 1 2024-03-22T02:28:30.442 INFO:tasks.workunit.client.0.smithi103.stdout:9/84: creat d2/d10/d17/f1f x:0 0 0 2024-03-22T02:28:30.444 INFO:tasks.workunit.client.0.smithi103.stdout:9/85: mknod d2/da/de/c20 0 2024-03-22T02:28:30.447 INFO:tasks.workunit.client.0.smithi103.stdout:3/130: getdents de/d1a/d25/d26 0 2024-03-22T02:28:30.448 INFO:tasks.workunit.client.0.smithi103.stdout:9/86: dread f1 [0,4194304] 0 2024-03-22T02:28:30.449 INFO:tasks.workunit.client.0.smithi103.stdout:0/64: truncate fc 929371 0 2024-03-22T02:28:30.449 INFO:tasks.workunit.client.0.smithi103.stdout:0/65: chown dd/l10 966 1 2024-03-22T02:28:30.449 INFO:tasks.workunit.client.0.smithi103.stdout:3/131: mkdir de/d12/d30 0 2024-03-22T02:28:30.450 INFO:tasks.workunit.client.0.smithi103.stdout:0/66: write f2 [21921,3152] 0 2024-03-22T02:28:30.451 INFO:tasks.workunit.client.0.smithi103.stdout:9/87: rmdir d2/d10 39 2024-03-22T02:28:30.451 INFO:tasks.workunit.client.0.smithi103.stdout:0/67: dread - f6 zero size 2024-03-22T02:28:30.453 INFO:tasks.workunit.client.0.smithi103.stdout:3/132: rename f7 to de/d1a/f31 0 2024-03-22T02:28:30.454 INFO:tasks.workunit.client.0.smithi103.stdout:9/88: creat d2/da/d18/f21 x:0 0 0 2024-03-22T02:28:30.456 INFO:tasks.workunit.client.0.smithi103.stdout:3/133: rename de/d1a/d25/d26/c2e to de/d1a/d25/c32 0 2024-03-22T02:28:30.457 INFO:tasks.workunit.client.0.smithi103.stdout:9/89: write d2/da/fc [1207653,70457] 0 2024-03-22T02:28:30.458 INFO:tasks.workunit.client.0.smithi103.stdout:3/134: mkdir de/d1a/d33 0 2024-03-22T02:28:30.458 INFO:tasks.workunit.client.0.smithi103.stdout:9/90: chown d2/f3 214420082 1 2024-03-22T02:28:30.460 INFO:tasks.workunit.client.0.smithi103.stdout:3/135: rmdir de/d12/d19 39 2024-03-22T02:28:30.461 INFO:tasks.workunit.client.0.smithi103.stdout:3/136: fsync de/f23 0 2024-03-22T02:28:30.462 INFO:tasks.workunit.client.0.smithi103.stdout:9/91: rename d2/da/fc to d2/d5/f22 0 2024-03-22T02:28:30.467 INFO:tasks.workunit.client.0.smithi103.stdout:3/137: rmdir de/d12 39 2024-03-22T02:28:30.468 INFO:tasks.workunit.client.0.smithi103.stdout:9/92: dwrite d2/f9 [0,4194304] 0 2024-03-22T02:28:30.469 INFO:tasks.workunit.client.0.smithi103.stdout:9/93: chown d2/da/de/f1b 1032 1 2024-03-22T02:28:30.471 INFO:tasks.workunit.client.0.smithi103.stdout:3/138: mknod de/c34 0 2024-03-22T02:28:30.474 INFO:tasks.workunit.client.0.smithi103.stdout:3/139: dread fb [0,4194304] 0 2024-03-22T02:28:30.475 INFO:tasks.workunit.client.0.smithi103.stdout:9/94: dread d2/d5/f22 [0,4194304] 0 2024-03-22T02:28:30.479 INFO:tasks.workunit.client.0.smithi103.stdout:3/140: creat de/d12/d30/f35 x:0 0 0 2024-03-22T02:28:30.481 INFO:tasks.workunit.client.0.smithi103.stdout:3/141: dread - de/d12/d30/f35 zero size 2024-03-22T02:28:30.482 INFO:tasks.workunit.client.0.smithi103.stdout:9/95: dwrite d2/da/de/f1b [0,4194304] 0 2024-03-22T02:28:30.483 INFO:tasks.workunit.client.0.smithi103.stdout:9/96: dread - d2/d5/ff zero size 2024-03-22T02:28:30.484 INFO:tasks.workunit.client.0.smithi103.stdout:9/97: chown d2/da/l15 161832 1 2024-03-22T02:28:30.488 INFO:tasks.workunit.client.0.smithi103.stdout:9/98: creat d2/da/f23 x:0 0 0 2024-03-22T02:28:30.492 INFO:tasks.workunit.client.0.smithi103.stdout:3/142: dwrite de/d1a/f24 [0,4194304] 0 2024-03-22T02:28:30.496 INFO:tasks.workunit.client.0.smithi103.stdout:3/143: mknod de/c36 0 2024-03-22T02:28:30.498 INFO:tasks.workunit.client.0.smithi103.stdout:9/99: dwrite d2/f9 [4194304,4194304] 0 2024-03-22T02:28:30.500 INFO:tasks.workunit.client.0.smithi103.stdout:3/144: unlink de/d1a/f24 0 2024-03-22T02:28:30.501 INFO:tasks.workunit.client.0.smithi103.stdout:3/145: fsync de/d1a/d25/d26/f2f 0 2024-03-22T02:28:30.509 INFO:tasks.workunit.client.0.smithi103.stdout:3/146: dwrite de/d12/f16 [0,4194304] 0 2024-03-22T02:28:30.512 INFO:tasks.workunit.client.0.smithi103.stdout:9/100: write d2/f14 [1521097,35528] 0 2024-03-22T02:28:30.514 INFO:tasks.workunit.client.0.smithi103.stdout:9/101: dread - d2/da/d18/f21 zero size 2024-03-22T02:28:30.514 INFO:tasks.workunit.client.0.smithi103.stdout:3/147: getdents de/d1a/d25 0 2024-03-22T02:28:30.515 INFO:tasks.workunit.client.0.smithi103.stdout:9/102: write d2/da/de/f1b [3864444,102140] 0 2024-03-22T02:28:30.518 INFO:tasks.workunit.client.0.smithi103.stdout:9/103: creat d2/da/f24 x:0 0 0 2024-03-22T02:28:30.522 INFO:tasks.workunit.client.0.smithi103.stdout:3/148: dwrite de/d12/d30/f35 [0,4194304] 0 2024-03-22T02:28:30.524 INFO:tasks.workunit.client.0.smithi103.stdout:3/149: fdatasync de/d1a/f31 0 2024-03-22T02:28:30.531 INFO:tasks.workunit.client.0.smithi103.stdout:9/104: write d2/d10/f13 [1202841,118573] 0 2024-03-22T02:28:30.531 INFO:tasks.workunit.client.0.smithi103.stdout:3/150: write f1 [1553077,48586] 0 2024-03-22T02:28:30.539 INFO:tasks.workunit.client.0.smithi103.stdout:9/105: dwrite d2/da/de/f1b [0,4194304] 0 2024-03-22T02:28:30.539 INFO:tasks.workunit.client.0.smithi103.stdout:3/151: dwrite de/d1a/f31 [4194304,4194304] 0 2024-03-22T02:28:30.542 INFO:tasks.workunit.client.0.smithi103.stdout:9/106: creat d2/da/f25 x:0 0 0 2024-03-22T02:28:30.543 INFO:tasks.workunit.client.0.smithi103.stdout:3/152: mknod de/d12/d19/c37 0 2024-03-22T02:28:30.554 INFO:tasks.workunit.client.0.smithi103.stdout:9/107: dwrite d2/d5/ff [0,4194304] 0 2024-03-22T02:28:30.554 INFO:tasks.workunit.client.0.smithi103.stdout:3/153: dwrite de/f1f [0,4194304] 0 2024-03-22T02:28:30.555 INFO:tasks.workunit.client.0.smithi103.stdout:3/154: fdatasync f1 0 2024-03-22T02:28:30.568 INFO:tasks.workunit.client.0.smithi103.stdout:3/155: dwrite de/f27 [0,4194304] 0 2024-03-22T02:28:30.569 INFO:tasks.workunit.client.0.smithi103.stdout:9/108: dwrite d2/d5/f22 [0,4194304] 0 2024-03-22T02:28:30.570 INFO:tasks.workunit.client.0.smithi103.stdout:9/109: stat d2/f3 0 2024-03-22T02:28:30.575 INFO:tasks.workunit.client.0.smithi103.stdout:3/156: symlink de/d1a/d25/l38 0 2024-03-22T02:28:30.575 INFO:tasks.workunit.client.0.smithi103.stdout:9/110: rename d2/d5/f22 to d2/d10/d17/f26 0 2024-03-22T02:28:30.575 INFO:tasks.workunit.client.0.smithi103.stdout:3/157: write de/f27 [3318230,21349] 0 2024-03-22T02:28:30.578 INFO:tasks.workunit.client.0.smithi103.stdout:9/111: mkdir d2/d10/d27 0 2024-03-22T02:28:30.579 INFO:tasks.workunit.client.0.smithi103.stdout:3/158: rename fb to de/d1a/f39 0 2024-03-22T02:28:30.588 INFO:tasks.workunit.client.0.smithi103.stdout:3/159: mknod de/d1a/d25/c3a 0 2024-03-22T02:28:30.605 INFO:tasks.workunit.client.0.smithi103.stdout:3/160: symlink de/l3b 0 2024-03-22T02:28:30.611 INFO:tasks.workunit.client.0.smithi103.stdout:3/161: dwrite de/d12/f2b [0,4194304] 0 2024-03-22T02:28:30.614 INFO:tasks.workunit.client.0.smithi103.stdout:3/162: mknod de/d12/c3c 0 2024-03-22T02:28:30.622 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:30 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:30.622 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:30 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:30.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:30 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:30.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:30 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:30.969 INFO:tasks.workunit.client.0.smithi103.stdout:1/39: sync 2024-03-22T02:28:30.970 INFO:tasks.workunit.client.0.smithi103.stdout:2/37: sync 2024-03-22T02:28:30.970 INFO:tasks.workunit.client.0.smithi103.stdout:7/56: sync 2024-03-22T02:28:30.973 INFO:tasks.workunit.client.0.smithi103.stdout:7/57: mkdir d1/d10/d11/d19 0 2024-03-22T02:28:30.976 INFO:tasks.workunit.client.0.smithi103.stdout:7/58: write d1/da/f8 [2861129,127591] 0 2024-03-22T02:28:30.976 INFO:tasks.workunit.client.0.smithi103.stdout:2/38: dread f2 [0,4194304] 0 2024-03-22T02:28:30.977 INFO:tasks.workunit.client.0.smithi103.stdout:1/40: dwrite f7 [4194304,4194304] 0 2024-03-22T02:28:30.977 INFO:tasks.workunit.client.0.smithi103.stdout:7/59: rename d1/d10/d11 to d1/d10/d11/d1a 22 2024-03-22T02:28:30.979 INFO:tasks.workunit.client.0.smithi103.stdout:2/39: symlink la 0 2024-03-22T02:28:30.981 INFO:tasks.workunit.client.0.smithi103.stdout:1/41: chown c4 9784 1 2024-03-22T02:28:30.981 INFO:tasks.workunit.client.0.smithi103.stdout:1/42: stat c4 0 2024-03-22T02:28:30.982 INFO:tasks.workunit.client.0.smithi103.stdout:7/60: rmdir d1/d10 39 2024-03-22T02:28:30.983 INFO:tasks.workunit.client.0.smithi103.stdout:2/40: symlink lb 0 2024-03-22T02:28:30.984 INFO:tasks.workunit.client.0.smithi103.stdout:1/43: symlink db/le 0 2024-03-22T02:28:30.984 INFO:tasks.workunit.client.0.smithi103.stdout:1/44: read f7 [4625407,50228] 0 2024-03-22T02:28:30.985 INFO:tasks.workunit.client.0.smithi103.stdout:1/45: chown c6 394790947 1 2024-03-22T02:28:30.985 INFO:tasks.workunit.client.0.smithi103.stdout:1/46: stat db/cd 0 2024-03-22T02:28:30.986 INFO:tasks.workunit.client.0.smithi103.stdout:7/61: truncate d1/d10/d11/f18 1018260 0 2024-03-22T02:28:30.987 INFO:tasks.workunit.client.0.smithi103.stdout:7/62: read d1/da/f4 [2977247,95515] 0 2024-03-22T02:28:30.987 INFO:tasks.workunit.client.0.smithi103.stdout:1/47: write db/fc [846452,18164] 0 2024-03-22T02:28:30.988 INFO:tasks.workunit.client.0.smithi103.stdout:1/48: read f7 [5397503,48441] 0 2024-03-22T02:28:30.990 INFO:tasks.workunit.client.0.smithi103.stdout:7/63: fdatasync d1/da/fb 0 2024-03-22T02:28:30.992 INFO:tasks.workunit.client.0.smithi103.stdout:7/64: mkdir d1/d10/d1b 0 2024-03-22T02:28:30.993 INFO:tasks.workunit.client.0.smithi103.stdout:1/49: dread f7 [0,4194304] 0 2024-03-22T02:28:30.994 INFO:tasks.workunit.client.0.smithi103.stdout:2/41: getdents . 0 2024-03-22T02:28:30.996 INFO:tasks.workunit.client.0.smithi103.stdout:1/50: rename la to db/lf 0 2024-03-22T02:28:30.996 INFO:tasks.workunit.client.0.smithi103.stdout:8/133: sync 2024-03-22T02:28:30.997 INFO:tasks.workunit.client.0.smithi103.stdout:4/51: sync 2024-03-22T02:28:30.997 INFO:tasks.workunit.client.0.smithi103.stdout:1/51: mkdir db/d10 0 2024-03-22T02:28:30.998 INFO:tasks.workunit.client.0.smithi103.stdout:4/52: write d3/fc [4551618,32005] 0 2024-03-22T02:28:30.999 INFO:tasks.workunit.client.0.smithi103.stdout:4/53: chown d3/l6 1280 1 2024-03-22T02:28:31.008 INFO:tasks.workunit.client.0.smithi103.stdout:2/42: dwrite f2 [0,4194304] 0 2024-03-22T02:28:31.011 INFO:tasks.workunit.client.0.smithi103.stdout:8/134: dwrite d4/f15 [0,4194304] 0 2024-03-22T02:28:31.013 INFO:tasks.workunit.client.0.smithi103.stdout:1/52: dwrite f5 [0,4194304] 0 2024-03-22T02:28:31.015 INFO:tasks.workunit.client.0.smithi103.stdout:8/135: mkdir d4/d26/d27 0 2024-03-22T02:28:31.016 INFO:tasks.workunit.client.0.smithi103.stdout:1/53: read f7 [788307,129703] 0 2024-03-22T02:28:31.018 INFO:tasks.workunit.client.0.smithi103.stdout:4/54: dwrite d3/f8 [0,4194304] 0 2024-03-22T02:28:31.022 INFO:tasks.workunit.client.0.smithi103.stdout:4/55: chown d3/f7 18 1 2024-03-22T02:28:31.024 INFO:tasks.workunit.client.0.smithi103.stdout:2/43: dwrite f3 [0,4194304] 0 2024-03-22T02:28:31.025 INFO:tasks.workunit.client.0.smithi103.stdout:2/44: dread - f4 zero size 2024-03-22T02:28:31.028 INFO:tasks.workunit.client.0.smithi103.stdout:4/56: mknod d3/c12 0 2024-03-22T02:28:31.028 INFO:tasks.workunit.client.0.smithi103.stdout:1/54: link f7 db/f11 0 2024-03-22T02:28:31.032 INFO:tasks.workunit.client.0.smithi103.stdout:8/136: link d4/c11 d4/c28 0 2024-03-22T02:28:31.036 INFO:tasks.workunit.client.0.smithi103.stdout:8/137: chown d4/d14/l17 374704119 1 2024-03-22T02:28:31.036 INFO:tasks.workunit.client.0.smithi103.stdout:2/45: dread f5 [0,4194304] 0 2024-03-22T02:28:31.038 INFO:tasks.workunit.client.0.smithi103.stdout:4/57: dwrite d3/f11 [0,4194304] 0 2024-03-22T02:28:31.039 INFO:tasks.workunit.client.0.smithi103.stdout:2/46: dread f5 [0,4194304] 0 2024-03-22T02:28:31.040 INFO:tasks.workunit.client.0.smithi103.stdout:2/47: readlink lb 0 2024-03-22T02:28:31.041 INFO:tasks.workunit.client.0.smithi103.stdout:1/55: dread f7 [4194304,4194304] 0 2024-03-22T02:28:31.043 INFO:tasks.workunit.client.0.smithi103.stdout:2/48: symlink lc 0 2024-03-22T02:28:31.046 INFO:tasks.workunit.client.0.smithi103.stdout:1/56: dread f7 [4194304,4194304] 0 2024-03-22T02:28:31.047 INFO:tasks.workunit.client.0.smithi103.stdout:2/49: creat fd x:0 0 0 2024-03-22T02:28:31.047 INFO:tasks.workunit.client.0.smithi103.stdout:8/138: link d4/c24 d4/d26/c29 0 2024-03-22T02:28:31.047 INFO:tasks.workunit.client.0.smithi103.stdout:4/58: link d3/le d3/l13 0 2024-03-22T02:28:31.049 INFO:tasks.workunit.client.0.smithi103.stdout:4/59: creat d3/f14 x:0 0 0 2024-03-22T02:28:31.050 INFO:tasks.workunit.client.0.smithi103.stdout:8/139: mkdir d4/d2a 0 2024-03-22T02:28:31.051 INFO:tasks.workunit.client.0.smithi103.stdout:4/60: symlink d3/l15 0 2024-03-22T02:28:31.055 INFO:tasks.workunit.client.0.smithi103.stdout:2/50: dwrite f8 [0,4194304] 0 2024-03-22T02:28:31.056 INFO:tasks.workunit.client.0.smithi103.stdout:8/140: dread d4/f15 [0,4194304] 0 2024-03-22T02:28:31.057 INFO:tasks.workunit.client.0.smithi103.stdout:8/141: write f2 [3268372,123752] 0 2024-03-22T02:28:31.069 INFO:tasks.workunit.client.0.smithi103.stdout:8/142: dread d4/f9 [0,4194304] 0 2024-03-22T02:28:31.069 INFO:tasks.workunit.client.0.smithi103.stdout:8/143: stat d4/l20 0 2024-03-22T02:28:31.076 INFO:tasks.workunit.client.0.smithi103.stdout:4/61: unlink d3/c12 0 2024-03-22T02:28:31.077 INFO:tasks.workunit.client.0.smithi103.stdout:4/62: chown d3/f7 64459 1 2024-03-22T02:28:31.078 INFO:tasks.workunit.client.0.smithi103.stdout:2/51: mknod ce 0 2024-03-22T02:28:31.081 INFO:tasks.workunit.client.0.smithi103.stdout:8/144: write d4/f15 [2651397,127170] 0 2024-03-22T02:28:31.081 INFO:tasks.workunit.client.0.smithi103.stdout:4/63: mknod d3/c16 0 2024-03-22T02:28:31.082 INFO:tasks.workunit.client.0.smithi103.stdout:4/64: write d3/fd [484447,33271] 0 2024-03-22T02:28:31.083 INFO:tasks.workunit.client.0.smithi103.stdout:2/52: mknod cf 0 2024-03-22T02:28:31.086 INFO:tasks.workunit.client.0.smithi103.stdout:4/65: creat d3/f17 x:0 0 0 2024-03-22T02:28:31.087 INFO:tasks.workunit.client.0.smithi103.stdout:2/53: rename f3 to f10 0 2024-03-22T02:28:31.088 INFO:tasks.workunit.client.0.smithi103.stdout:2/54: truncate f2 4873735 0 2024-03-22T02:28:31.088 INFO:tasks.workunit.client.0.smithi103.stdout:4/66: dread d3/fd [0,4194304] 0 2024-03-22T02:28:31.092 INFO:tasks.workunit.client.0.smithi103.stdout:2/55: mkdir d11 0 2024-03-22T02:28:31.093 INFO:tasks.workunit.client.0.smithi103.stdout:8/145: rmdir d4/d13/d1d 0 2024-03-22T02:28:31.098 INFO:tasks.workunit.client.0.smithi103.stdout:8/146: rmdir d4/d26 39 2024-03-22T02:28:31.098 INFO:tasks.workunit.client.0.smithi103.stdout:2/56: dwrite f2 [0,4194304] 0 2024-03-22T02:28:31.099 INFO:tasks.workunit.client.0.smithi103.stdout:8/147: rename d4/d19 to d4/d19/d1a/d2b 22 2024-03-22T02:28:31.101 INFO:tasks.workunit.client.0.smithi103.stdout:8/148: unlink d4/f12 0 2024-03-22T02:28:31.102 INFO:tasks.workunit.client.0.smithi103.stdout:2/57: dread f10 [0,4194304] 0 2024-03-22T02:28:31.103 INFO:tasks.workunit.client.0.smithi103.stdout:4/67: rmdir d3 39 2024-03-22T02:28:31.104 INFO:tasks.workunit.client.0.smithi103.stdout:2/58: dread f5 [0,4194304] 0 2024-03-22T02:28:31.110 INFO:tasks.workunit.client.0.smithi103.stdout:4/68: mknod d3/c18 0 2024-03-22T02:28:31.111 INFO:tasks.workunit.client.0.smithi103.stdout:2/59: creat d11/f12 x:0 0 0 2024-03-22T02:28:31.112 INFO:tasks.workunit.client.0.smithi103.stdout:4/69: symlink d3/l19 0 2024-03-22T02:28:31.114 INFO:tasks.workunit.client.0.smithi103.stdout:2/60: rename f9 to d11/f13 0 2024-03-22T02:28:31.115 INFO:tasks.workunit.client.0.smithi103.stdout:2/61: write d11/f12 [3652,14324] 0 2024-03-22T02:28:31.116 INFO:tasks.workunit.client.0.smithi103.stdout:4/70: unlink d3/l15 0 2024-03-22T02:28:31.116 INFO:tasks.workunit.client.0.smithi103.stdout:2/62: dread - fd zero size 2024-03-22T02:28:31.117 INFO:tasks.workunit.client.0.smithi103.stdout:2/63: stat d11/f12 0 2024-03-22T02:28:31.119 INFO:tasks.workunit.client.0.smithi103.stdout:4/71: dread d3/fd [0,4194304] 0 2024-03-22T02:28:31.119 INFO:tasks.workunit.client.0.smithi103.stdout:8/149: dwrite f1 [0,4194304] 0 2024-03-22T02:28:31.123 INFO:tasks.workunit.client.0.smithi103.stdout:4/72: creat d3/f1a x:0 0 0 2024-03-22T02:28:31.124 INFO:tasks.workunit.client.0.smithi103.stdout:4/73: fsync d3/f8 0 2024-03-22T02:28:31.126 INFO:tasks.workunit.client.0.smithi103.stdout:2/64: dwrite f1 [4194304,4194304] 0 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:4/74: symlink d3/l1b 0 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:3/163: sync 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:6/59: sync 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:9/112: sync 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:0/68: sync 2024-03-22T02:28:31.127 INFO:tasks.workunit.client.0.smithi103.stdout:5/63: sync 2024-03-22T02:28:31.129 INFO:tasks.workunit.client.0.smithi103.stdout:2/65: creat d11/f14 x:0 0 0 2024-03-22T02:28:31.129 INFO:tasks.workunit.client.0.smithi103.stdout:5/64: rename d0 to d0/d13 22 2024-03-22T02:28:31.130 INFO:tasks.workunit.client.0.smithi103.stdout:4/75: write d3/fc [2976429,16080] 0 2024-03-22T02:28:31.131 INFO:tasks.workunit.client.0.smithi103.stdout:9/113: stat d2/d5/ff 0 2024-03-22T02:28:31.131 INFO:tasks.workunit.client.0.smithi103.stdout:3/164: write de/f1e [492552,131026] 0 2024-03-22T02:28:31.131 INFO:tasks.workunit.client.0.smithi103.stdout:6/60: write d0/d3/f4 [4270241,111286] 0 2024-03-22T02:28:31.133 INFO:tasks.workunit.client.0.smithi103.stdout:5/65: mkdir d0/d14 0 2024-03-22T02:28:31.134 INFO:tasks.workunit.client.0.smithi103.stdout:8/150: dread d4/ff [0,4194304] 0 2024-03-22T02:28:31.135 INFO:tasks.workunit.client.0.smithi103.stdout:4/76: fsync d3/f1a 0 2024-03-22T02:28:31.136 INFO:tasks.workunit.client.0.smithi103.stdout:2/66: mkdir d11/d15 0 2024-03-22T02:28:31.138 INFO:tasks.workunit.client.0.smithi103.stdout:3/165: creat de/d12/d30/f3d x:0 0 0 2024-03-22T02:28:31.139 INFO:tasks.workunit.client.0.smithi103.stdout:6/61: unlink d0/f2 0 2024-03-22T02:28:31.141 INFO:tasks.workunit.client.0.smithi103.stdout:9/114: symlink d2/da/d18/l28 0 2024-03-22T02:28:31.141 INFO:tasks.workunit.client.0.smithi103.stdout:4/77: dread d3/fd [0,4194304] 0 2024-03-22T02:28:31.142 INFO:tasks.workunit.client.0.smithi103.stdout:0/69: dwrite f8 [0,4194304] 0 2024-03-22T02:28:31.145 INFO:tasks.workunit.client.0.smithi103.stdout:9/115: truncate d2/d10/d17/f26 4897526 0 2024-03-22T02:28:31.145 INFO:tasks.workunit.client.0.smithi103.stdout:2/67: unlink f8 0 2024-03-22T02:28:31.145 INFO:tasks.workunit.client.0.smithi103.stdout:5/66: dread d0/fa [0,4194304] 0 2024-03-22T02:28:31.146 INFO:tasks.workunit.client.0.smithi103.stdout:8/151: rmdir d4/d26 39 2024-03-22T02:28:31.146 INFO:tasks.workunit.client.0.smithi103.stdout:9/116: read - d2/d10/d17/f1f zero size 2024-03-22T02:28:31.180 INFO:tasks.workunit.client.0.smithi103.stdout:4/78: creat d3/f1c x:0 0 0 2024-03-22T02:28:31.182 INFO:tasks.workunit.client.0.smithi103.stdout:9/117: unlink d2/f14 0 2024-03-22T02:28:31.185 INFO:tasks.workunit.client.0.smithi103.stdout:2/68: dwrite f7 [0,4194304] 0 2024-03-22T02:28:31.186 INFO:tasks.workunit.client.0.smithi103.stdout:4/79: truncate f2 614806 0 2024-03-22T02:28:31.187 INFO:tasks.workunit.client.0.smithi103.stdout:3/166: truncate de/f1f 482521 0 2024-03-22T02:28:31.188 INFO:tasks.workunit.client.0.smithi103.stdout:3/167: fdatasync de/d1a/f31 0 2024-03-22T02:28:31.189 INFO:tasks.workunit.client.0.smithi103.stdout:5/67: symlink d0/d14/l15 0 2024-03-22T02:28:31.196 INFO:tasks.workunit.client.0.smithi103.stdout:2/69: creat d11/f16 x:0 0 0 2024-03-22T02:28:31.197 INFO:tasks.workunit.client.0.smithi103.stdout:4/80: creat d3/f1d x:0 0 0 2024-03-22T02:28:31.200 INFO:tasks.workunit.client.0.smithi103.stdout:8/152: link f2 d4/d26/d27/f2c 0 2024-03-22T02:28:31.201 INFO:tasks.workunit.client.0.smithi103.stdout:8/153: dread d4/ff [0,4194304] 0 2024-03-22T02:28:31.205 INFO:tasks.workunit.client.0.smithi103.stdout:4/81: mkdir d3/d1e 0 2024-03-22T02:28:31.209 INFO:tasks.workunit.client.0.smithi103.stdout:4/82: creat d3/f1f x:0 0 0 2024-03-22T02:28:31.210 INFO:tasks.workunit.client.0.smithi103.stdout:9/118: link d2/d10/c1d d2/d5/c29 0 2024-03-22T02:28:31.214 INFO:tasks.workunit.client.0.smithi103.stdout:9/119: dread d2/d5/ff [0,4194304] 0 2024-03-22T02:28:31.214 INFO:tasks.workunit.client.0.smithi103.stdout:3/168: getdents de/d12/d19 0 2024-03-22T02:28:31.219 INFO:tasks.workunit.client.0.smithi103.stdout:2/70: dread f1 [0,4194304] 0 2024-03-22T02:28:31.219 INFO:tasks.workunit.client.0.smithi103.stdout:8/154: truncate d4/d13/f1c 620861 0 2024-03-22T02:28:31.220 INFO:tasks.workunit.client.0.smithi103.stdout:4/83: creat d3/d1e/f20 x:0 0 0 2024-03-22T02:28:31.221 INFO:tasks.workunit.client.0.smithi103.stdout:9/120: mknod d2/d10/d27/c2a 0 2024-03-22T02:28:31.222 INFO:tasks.workunit.client.0.smithi103.stdout:3/169: symlink de/d12/l3e 0 2024-03-22T02:28:31.222 INFO:tasks.workunit.client.0.smithi103.stdout:2/71: fdatasync f1 0 2024-03-22T02:28:31.223 INFO:tasks.workunit.client.0.smithi103.stdout:3/170: fdatasync de/d1a/d25/f29 0 2024-03-22T02:28:31.223 INFO:tasks.workunit.client.0.smithi103.stdout:2/72: chown f4 16 1 2024-03-22T02:28:31.223 INFO:tasks.workunit.client.0.smithi103.stdout:8/155: fsync d4/d26/d27/f2c 0 2024-03-22T02:28:31.225 INFO:tasks.workunit.client.0.smithi103.stdout:9/121: write f1 [4940189,115438] 0 2024-03-22T02:28:31.226 INFO:tasks.workunit.client.0.smithi103.stdout:9/122: fsync d2/da/f23 0 2024-03-22T02:28:31.226 INFO:tasks.workunit.client.0.smithi103.stdout:2/73: symlink d11/l17 0 2024-03-22T02:28:31.227 INFO:tasks.workunit.client.0.smithi103.stdout:9/123: write f1 [3571883,69280] 0 2024-03-22T02:28:31.229 INFO:tasks.workunit.client.0.smithi103.stdout:8/156: fsync d4/d26/d27/f2c 0 2024-03-22T02:28:31.230 INFO:tasks.workunit.client.0.smithi103.stdout:4/84: rmdir d3/d1e 39 2024-03-22T02:28:31.230 INFO:tasks.workunit.client.0.smithi103.stdout:4/85: chown d3/f1d 17957033 1 2024-03-22T02:28:31.232 INFO:tasks.workunit.client.0.smithi103.stdout:2/74: fsync d11/f13 0 2024-03-22T02:28:31.234 INFO:tasks.workunit.client.0.smithi103.stdout:9/124: symlink d2/da/d18/l2b 0 2024-03-22T02:28:31.234 INFO:tasks.workunit.client.0.smithi103.stdout:9/125: dread - d2/d10/d17/f1f zero size 2024-03-22T02:28:31.235 INFO:tasks.workunit.client.0.smithi103.stdout:9/126: write d2/da/de/f1b [902964,82032] 0 2024-03-22T02:28:31.238 INFO:tasks.workunit.client.0.smithi103.stdout:3/171: rmdir de/d1a/d33 0 2024-03-22T02:28:31.240 INFO:tasks.workunit.client.0.smithi103.stdout:3/172: readlink de/l11 0 2024-03-22T02:28:31.242 INFO:tasks.workunit.client.0.smithi103.stdout:4/86: creat d3/f21 x:0 0 0 2024-03-22T02:28:31.243 INFO:tasks.workunit.client.0.smithi103.stdout:8/157: getdents d4/d14 0 2024-03-22T02:28:31.247 INFO:tasks.workunit.client.0.smithi103.stdout:9/127: dwrite d2/da/d18/f21 [0,4194304] 0 2024-03-22T02:28:31.247 INFO:tasks.workunit.client.0.smithi103.stdout:2/75: dwrite f5 [0,4194304] 0 2024-03-22T02:28:31.247 INFO:tasks.workunit.client.0.smithi103.stdout:4/87: fsync d3/fb 0 2024-03-22T02:28:31.249 INFO:tasks.workunit.client.0.smithi103.stdout:4/88: chown d3/fd 47439 1 2024-03-22T02:28:31.250 INFO:tasks.workunit.client.0.smithi103.stdout:2/76: chown ce 817 1 2024-03-22T02:28:31.255 INFO:tasks.workunit.client.0.smithi103.stdout:2/77: dread d11/f12 [0,4194304] 0 2024-03-22T02:28:31.258 INFO:tasks.workunit.client.0.smithi103.stdout:8/158: dwrite d4/ff [0,4194304] 0 2024-03-22T02:28:31.259 INFO:tasks.workunit.client.0.smithi103.stdout:2/78: truncate d11/f13 311038 0 2024-03-22T02:28:31.260 INFO:tasks.workunit.client.0.smithi103.stdout:9/128: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:31.260 INFO:tasks.workunit.client.0.smithi103.stdout:4/89: getdents d3 0 2024-03-22T02:28:31.261 INFO:tasks.workunit.client.0.smithi103.stdout:4/90: read d3/fc [1994411,99449] 0 2024-03-22T02:28:31.262 INFO:tasks.workunit.client.0.smithi103.stdout:8/159: creat d4/d19/d1a/f2d x:0 0 0 2024-03-22T02:28:31.265 INFO:tasks.workunit.client.0.smithi103.stdout:9/129: dread d2/da/d18/f21 [0,4194304] 0 2024-03-22T02:28:31.265 INFO:tasks.workunit.client.0.smithi103.stdout:4/91: mknod d3/c22 0 2024-03-22T02:28:31.266 INFO:tasks.workunit.client.0.smithi103.stdout:2/79: link f1 d11/f18 0 2024-03-22T02:28:31.266 INFO:tasks.workunit.client.0.smithi103.stdout:7/65: sync 2024-03-22T02:28:31.267 INFO:tasks.workunit.client.0.smithi103.stdout:8/160: creat d4/d19/f2e x:0 0 0 2024-03-22T02:28:31.271 INFO:tasks.workunit.client.0.smithi103.stdout:8/161: chown d4/d26/d27/f2c 0 1 2024-03-22T02:28:31.273 INFO:tasks.workunit.client.0.smithi103.stdout:9/130: getdents d2/d10 0 2024-03-22T02:28:31.278 INFO:tasks.workunit.client.0.smithi103.stdout:9/131: write d2/da/f25 [423365,122506] 0 2024-03-22T02:28:31.278 INFO:tasks.workunit.client.0.smithi103.stdout:2/80: dwrite d11/f13 [0,4194304] 0 2024-03-22T02:28:31.279 INFO:tasks.workunit.client.0.smithi103.stdout:7/66: dwrite d1/d10/d11/f14 [0,4194304] 0 2024-03-22T02:28:31.286 INFO:tasks.workunit.client.0.smithi103.stdout:2/81: mkdir d11/d19 0 2024-03-22T02:28:31.286 INFO:tasks.workunit.client.0.smithi103.stdout:7/67: mkdir d1/da/d1c 0 2024-03-22T02:28:31.293 INFO:tasks.workunit.client.0.smithi103.stdout:8/162: truncate d4/d13/f1b 1216515 0 2024-03-22T02:28:31.294 INFO:tasks.workunit.client.0.smithi103.stdout:9/132: dwrite d2/da/f25 [0,4194304] 0 2024-03-22T02:28:31.297 INFO:tasks.workunit.client.0.smithi103.stdout:8/163: mknod d4/d26/c2f 0 2024-03-22T02:28:31.301 INFO:tasks.workunit.client.0.smithi103.stdout:4/92: dwrite d3/f21 [0,4194304] 0 2024-03-22T02:28:31.301 INFO:tasks.workunit.client.0.smithi103.stdout:7/68: dwrite d1/da/f3 [0,4194304] 0 2024-03-22T02:28:31.303 INFO:tasks.workunit.client.0.smithi103.stdout:9/133: creat d2/f2c x:0 0 0 2024-03-22T02:28:31.305 INFO:tasks.workunit.client.0.smithi103.stdout:7/69: write d1/da/fe [925008,89630] 0 2024-03-22T02:28:31.305 INFO:tasks.workunit.client.0.smithi103.stdout:8/164: write d4/d26/d27/f2c [489721,69243] 0 2024-03-22T02:28:31.310 INFO:tasks.workunit.client.0.smithi103.stdout:8/165: write d4/d26/d27/f2c [429825,338] 0 2024-03-22T02:28:31.314 INFO:tasks.workunit.client.0.smithi103.stdout:7/70: write d1/d10/d11/f18 [780332,29081] 0 2024-03-22T02:28:31.314 INFO:tasks.workunit.client.0.smithi103.stdout:4/93: rename d3/la to d3/l23 0 2024-03-22T02:28:31.315 INFO:tasks.workunit.client.0.smithi103.stdout:8/166: truncate d4/d26/d27/f2c 4815124 0 2024-03-22T02:28:31.317 INFO:tasks.workunit.client.0.smithi103.stdout:9/134: dwrite d2/d10/d17/f1f [0,4194304] 0 2024-03-22T02:28:31.319 INFO:tasks.workunit.client.0.smithi103.stdout:7/71: mkdir d1/d10/d1b/d1d 0 2024-03-22T02:28:31.323 INFO:tasks.workunit.client.0.smithi103.stdout:8/167: dwrite d4/f9 [4194304,4194304] 0 2024-03-22T02:28:31.328 INFO:tasks.workunit.client.0.smithi103.stdout:9/135: dwrite d2/da/f25 [0,4194304] 0 2024-03-22T02:28:31.333 INFO:tasks.workunit.client.0.smithi103.stdout:8/168: dwrite f2 [0,4194304] 0 2024-03-22T02:28:31.340 INFO:tasks.workunit.client.0.smithi103.stdout:4/94: creat d3/f24 x:0 0 0 2024-03-22T02:28:31.340 INFO:tasks.workunit.client.0.smithi103.stdout:9/136: link d2/da/de/c1c d2/da/de/c2d 0 2024-03-22T02:28:31.341 INFO:tasks.workunit.client.0.smithi103.stdout:9/137: fdatasync d2/f9 0 2024-03-22T02:28:31.342 INFO:tasks.workunit.client.0.smithi103.stdout:4/95: truncate d3/f17 278474 0 2024-03-22T02:28:31.345 INFO:tasks.workunit.client.0.smithi103.stdout:4/96: write d3/f8 [4082349,124932] 0 2024-03-22T02:28:31.348 INFO:tasks.workunit.client.0.smithi103.stdout:4/97: mkdir d3/d1e/d25 0 2024-03-22T02:28:31.349 INFO:tasks.workunit.client.0.smithi103.stdout:4/98: chown d3/d1e 243 1 2024-03-22T02:28:31.351 INFO:tasks.workunit.client.0.smithi103.stdout:4/99: write d3/f11 [702338,5623] 0 2024-03-22T02:28:31.354 INFO:tasks.workunit.client.0.smithi103.stdout:4/100: rename d3/c5 to d3/d1e/c26 0 2024-03-22T02:28:31.359 INFO:tasks.workunit.client.0.smithi103.stdout:4/101: symlink d3/d1e/d25/l27 0 2024-03-22T02:28:31.361 INFO:tasks.workunit.client.0.smithi103.stdout:4/102: creat d3/d1e/f28 x:0 0 0 2024-03-22T02:28:31.362 INFO:tasks.workunit.client.0.smithi103.stdout:8/169: write d4/d13/f1b [1912692,78055] 0 2024-03-22T02:28:31.363 INFO:tasks.workunit.client.0.smithi103.stdout:4/103: write d3/f24 [606252,38724] 0 2024-03-22T02:28:31.365 INFO:tasks.workunit.client.0.smithi103.stdout:9/138: dwrite d2/d5/ff [0,4194304] 0 2024-03-22T02:28:31.365 INFO:tasks.workunit.client.0.smithi103.stdout:8/170: write d4/d13/f1b [2858803,15167] 0 2024-03-22T02:28:31.367 INFO:tasks.workunit.client.0.smithi103.stdout:4/104: read - d3/f1d zero size 2024-03-22T02:28:31.370 INFO:tasks.workunit.client.0.smithi103.stdout:9/139: unlink f1 0 2024-03-22T02:28:31.370 INFO:tasks.workunit.client.0.smithi103.stdout:4/105: truncate d3/d1e/f28 679995 0 2024-03-22T02:28:31.371 INFO:tasks.workunit.client.0.smithi103.stdout:4/106: fdatasync d3/d1e/f20 0 2024-03-22T02:28:31.374 INFO:tasks.workunit.client.0.smithi103.stdout:4/107: write d3/f21 [5182251,104685] 0 2024-03-22T02:28:31.375 INFO:tasks.workunit.client.0.smithi103.stdout:8/171: dwrite f1 [0,4194304] 0 2024-03-22T02:28:31.379 INFO:tasks.workunit.client.0.smithi103.stdout:8/172: rmdir d4/d19/d1a 39 2024-03-22T02:28:31.382 INFO:tasks.workunit.client.0.smithi103.stdout:8/173: rmdir d4/d19 39 2024-03-22T02:28:31.384 INFO:tasks.workunit.client.0.smithi103.stdout:4/108: dwrite d3/f1a [0,4194304] 0 2024-03-22T02:28:31.388 INFO:tasks.workunit.client.0.smithi103.stdout:4/109: creat d3/d1e/d25/f29 x:0 0 0 2024-03-22T02:28:31.389 INFO:tasks.workunit.client.0.smithi103.stdout:4/110: write d3/d1e/f28 [1679607,21065] 0 2024-03-22T02:28:31.392 INFO:tasks.workunit.client.0.smithi103.stdout:8/174: fdatasync f1 0 2024-03-22T02:28:31.395 INFO:tasks.workunit.client.0.smithi103.stdout:8/175: truncate d4/d13/f1c 763666 0 2024-03-22T02:28:31.397 INFO:tasks.workunit.client.0.smithi103.stdout:8/176: mknod d4/d14/c30 0 2024-03-22T02:28:31.398 INFO:tasks.workunit.client.0.smithi103.stdout:8/177: write d4/f9 [8978953,70114] 0 2024-03-22T02:28:31.405 INFO:tasks.workunit.client.0.smithi103.stdout:8/178: dwrite d4/ff [4194304,4194304] 0 2024-03-22T02:28:31.406 INFO:tasks.workunit.client.0.smithi103.stdout:8/179: chown d4/d14/c16 11969814 1 2024-03-22T02:28:31.411 INFO:tasks.workunit.client.0.smithi103.stdout:8/180: link d4/d19/d1a/c25 d4/d26/c31 0 2024-03-22T02:28:31.425 INFO:tasks.workunit.client.0.smithi103.stdout:1/57: sync 2024-03-22T02:28:31.425 INFO:tasks.workunit.client.0.smithi103.stdout:1/58: chown db/le 62921314 1 2024-03-22T02:28:31.428 INFO:tasks.workunit.client.0.smithi103.stdout:1/59: rename db/f11 to db/d10/f12 0 2024-03-22T02:28:31.436 INFO:tasks.workunit.client.0.smithi103.stdout:1/60: write db/d10/f12 [5737873,45266] 0 2024-03-22T02:28:31.437 INFO:tasks.workunit.client.0.smithi103.stdout:8/181: dread d4/fd [0,4194304] 0 2024-03-22T02:28:31.443 INFO:tasks.workunit.client.0.smithi103.stdout:1/61: dwrite f5 [0,4194304] 0 2024-03-22T02:28:31.445 INFO:tasks.workunit.client.0.smithi103.stdout:1/62: mknod db/c13 0 2024-03-22T02:28:31.446 INFO:tasks.workunit.client.0.smithi103.stdout:1/63: write f7 [4601535,28849] 0 2024-03-22T02:28:31.450 INFO:tasks.workunit.client.0.smithi103.stdout:8/182: dread d4/d13/f1b [0,4194304] 0 2024-03-22T02:28:31.452 INFO:tasks.workunit.client.0.smithi103.stdout:1/64: dread f5 [0,4194304] 0 2024-03-22T02:28:31.454 INFO:tasks.workunit.client.0.smithi103.stdout:8/183: dread d4/f15 [0,4194304] 0 2024-03-22T02:28:31.455 INFO:tasks.workunit.client.0.smithi103.stdout:1/65: mkdir db/d10/d14 0 2024-03-22T02:28:31.455 INFO:tasks.workunit.client.0.smithi103.stdout:1/66: chown db/d10/d14 975 1 2024-03-22T02:28:31.456 INFO:tasks.workunit.client.0.smithi103.stdout:1/67: chown f5 887891 1 2024-03-22T02:28:31.456 INFO:tasks.workunit.client.0.smithi103.stdout:8/184: mknod d4/d2a/c32 0 2024-03-22T02:28:31.458 INFO:tasks.workunit.client.0.smithi103.stdout:1/68: symlink db/d10/l15 0 2024-03-22T02:28:31.460 INFO:tasks.workunit.client.0.smithi103.stdout:8/185: dread d4/d13/f1c [0,4194304] 0 2024-03-22T02:28:31.463 INFO:tasks.workunit.client.0.smithi103.stdout:8/186: getdents d4/d19/d1a 0 2024-03-22T02:28:31.465 INFO:tasks.workunit.client.0.smithi103.stdout:8/187: chown d4/c28 435 1 2024-03-22T02:28:31.466 INFO:tasks.workunit.client.0.smithi103.stdout:1/69: getdents db/d10 0 2024-03-22T02:28:31.471 INFO:tasks.workunit.client.0.smithi103.stdout:1/70: dwrite db/d10/f12 [4194304,4194304] 0 2024-03-22T02:28:31.476 INFO:tasks.workunit.client.0.smithi103.stdout:1/71: rename db/fc to db/d10/d14/f16 0 2024-03-22T02:28:31.476 INFO:tasks.workunit.client.0.smithi103.stdout:1/72: stat f7 0 2024-03-22T02:28:31.479 INFO:tasks.workunit.client.0.smithi103.stdout:1/73: creat db/f17 x:0 0 0 2024-03-22T02:28:31.480 INFO:tasks.workunit.client.0.smithi103.stdout:1/74: write db/f17 [82288,22670] 0 2024-03-22T02:28:31.485 INFO:tasks.workunit.client.0.smithi103.stdout:1/75: dwrite f7 [4194304,4194304] 0 2024-03-22T02:28:31.488 INFO:tasks.workunit.client.0.smithi103.stdout:1/76: mkdir db/d10/d18 0 2024-03-22T02:28:31.492 INFO:tasks.workunit.client.0.smithi103.stdout:1/77: dread f7 [0,4194304] 0 2024-03-22T02:28:31.492 INFO:tasks.workunit.client.0.smithi103.stdout:1/78: rename db to db/d10/d18/d19 22 2024-03-22T02:28:31.494 INFO:tasks.workunit.client.0.smithi103.stdout:1/79: write db/d10/d14/f16 [361480,81628] 0 2024-03-22T02:28:31.495 INFO:tasks.workunit.client.0.smithi103.stdout:1/80: read db/d10/f12 [4007504,85166] 0 2024-03-22T02:28:31.500 INFO:tasks.workunit.client.0.smithi103.stdout:1/81: dwrite db/f17 [0,4194304] 0 2024-03-22T02:28:31.503 INFO:tasks.workunit.client.0.smithi103.stdout:1/82: symlink db/d10/d14/l1a 0 2024-03-22T02:28:31.505 INFO:tasks.workunit.client.0.smithi103.stdout:1/83: dread f5 [0,4194304] 0 2024-03-22T02:28:31.507 INFO:tasks.workunit.client.0.smithi103.stdout:1/84: unlink db/d10/l15 0 2024-03-22T02:28:31.522 INFO:tasks.workunit.client.0.smithi103.stdout:1/85: getdents db 0 2024-03-22T02:28:31.524 INFO:tasks.workunit.client.0.smithi103.stdout:1/86: mknod db/c1b 0 2024-03-22T02:28:31.524 INFO:tasks.workunit.client.0.smithi103.stdout:1/87: read f7 [2044626,15455] 0 2024-03-22T02:28:31.526 INFO:tasks.workunit.client.0.smithi103.stdout:1/88: creat db/d10/f1c x:0 0 0 2024-03-22T02:28:31.528 INFO:tasks.workunit.client.0.smithi103.stdout:1/89: symlink db/d10/d18/l1d 0 2024-03-22T02:28:31.531 INFO:tasks.workunit.client.0.smithi103.stdout:1/90: write db/f17 [4250823,55697] 0 2024-03-22T02:28:31.533 INFO:tasks.workunit.client.0.smithi103.stdout:1/91: mknod db/c1e 0 2024-03-22T02:28:31.534 INFO:tasks.workunit.client.0.smithi103.stdout:1/92: rename db/c1e to db/c1f 0 2024-03-22T02:28:31.535 INFO:tasks.workunit.client.0.smithi103.stdout:1/93: readlink db/d10/d14/l1a 0 2024-03-22T02:28:31.537 INFO:tasks.workunit.client.0.smithi103.stdout:1/94: dread f5 [0,4194304] 0 2024-03-22T02:28:31.543 INFO:tasks.workunit.client.0.smithi103.stdout:1/95: dwrite f5 [0,4194304] 0 2024-03-22T02:28:31.547 INFO:tasks.workunit.client.0.smithi103.stdout:1/96: dread f7 [0,4194304] 0 2024-03-22T02:28:31.559 INFO:tasks.workunit.client.0.smithi103.stdout:1/97: getdents db 0 2024-03-22T02:28:31.559 INFO:tasks.workunit.client.0.smithi103.stdout:1/98: stat db/d10/f1c 0 2024-03-22T02:28:31.560 INFO:tasks.workunit.client.0.smithi103.stdout:1/99: creat db/d10/d18/f20 x:0 0 0 2024-03-22T02:28:31.562 INFO:tasks.workunit.client.0.smithi103.stdout:1/100: mkdir db/d10/d21 0 2024-03-22T02:28:31.563 INFO:tasks.workunit.client.0.smithi103.stdout:1/101: creat db/d10/d21/f22 x:0 0 0 2024-03-22T02:28:31.564 INFO:tasks.workunit.client.0.smithi103.stdout:1/102: chown db/d10/d18 390236742 1 2024-03-22T02:28:31.565 INFO:tasks.workunit.client.0.smithi103.stdout:1/103: creat db/d10/d21/f23 x:0 0 0 2024-03-22T02:28:31.582 INFO:tasks.workunit.client.0.smithi103.stdout:0/70: sync 2024-03-22T02:28:31.582 INFO:tasks.workunit.client.0.smithi103.stdout:6/62: sync 2024-03-22T02:28:31.583 INFO:tasks.workunit.client.0.smithi103.stdout:0/71: creat dd/f12 x:0 0 0 2024-03-22T02:28:31.591 INFO:tasks.workunit.client.0.smithi103.stdout:6/63: dwrite d0/d14/fe [0,4194304] 0 2024-03-22T02:28:31.593 INFO:tasks.workunit.client.0.smithi103.stdout:0/72: dwrite f2 [4194304,4194304] 0 2024-03-22T02:28:31.593 INFO:tasks.workunit.client.0.smithi103.stdout:6/64: chown d0/d3 256712640 1 2024-03-22T02:28:31.597 INFO:tasks.workunit.client.0.smithi103.stdout:6/65: rmdir d0/d3 39 2024-03-22T02:28:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:31 smithi122 ceph-mon[36403]: pgmap v9: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 115 KiB/s rd, 17 MiB/s wr, 414 op/s 2024-03-22T02:28:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:31 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:31 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:31.599 INFO:tasks.workunit.client.0.smithi103.stdout:6/66: mknod d0/d14/c18 0 2024-03-22T02:28:31.605 INFO:tasks.workunit.client.0.smithi103.stdout:6/67: dwrite d0/d14/df/f11 [4194304,4194304] 0 2024-03-22T02:28:31.608 INFO:tasks.workunit.client.0.smithi103.stdout:0/73: write f4 [398389,42376] 0 2024-03-22T02:28:31.608 INFO:tasks.workunit.client.0.smithi103.stdout:0/74: dread - dd/f12 zero size 2024-03-22T02:28:31.609 INFO:tasks.workunit.client.0.smithi103.stdout:0/75: chown f2 36 1 2024-03-22T02:28:31.610 INFO:tasks.workunit.client.0.smithi103.stdout:6/68: dread d0/d14/fe [0,4194304] 0 2024-03-22T02:28:31.610 INFO:tasks.workunit.client.0.smithi103.stdout:0/76: readlink dd/l10 0 2024-03-22T02:28:31.610 INFO:tasks.workunit.client.0.smithi103.stdout:0/77: readlink l9 0 2024-03-22T02:28:31.613 INFO:tasks.workunit.client.0.smithi103.stdout:0/78: creat dd/f13 x:0 0 0 2024-03-22T02:28:31.615 INFO:tasks.workunit.client.0.smithi103.stdout:0/79: symlink dd/l14 0 2024-03-22T02:28:31.615 INFO:tasks.workunit.client.0.smithi103.stdout:0/80: write f2 [6194075,21698] 0 2024-03-22T02:28:31.616 INFO:tasks.workunit.client.0.smithi103.stdout:6/69: rename d0/d3/f4 to d0/f19 0 2024-03-22T02:28:31.688 INFO:tasks.workunit.client.0.smithi103.stdout:6/70: creat d0/f1a x:0 0 0 2024-03-22T02:28:31.689 INFO:tasks.workunit.client.0.smithi103.stdout:6/71: chown d0/d14/c18 18062 1 2024-03-22T02:28:31.689 INFO:tasks.workunit.client.0.smithi103.stdout:0/81: link fc dd/f15 0 2024-03-22T02:28:31.691 INFO:tasks.workunit.client.0.smithi103.stdout:0/82: mkdir dd/d16 0 2024-03-22T02:28:31.694 INFO:tasks.workunit.client.0.smithi103.stdout:0/83: dread f8 [0,4194304] 0 2024-03-22T02:28:31.696 INFO:tasks.workunit.client.0.smithi103.stdout:0/84: unlink lb 0 2024-03-22T02:28:31.701 INFO:tasks.workunit.client.0.smithi103.stdout:0/85: getdents dd 0 2024-03-22T02:28:31.703 INFO:tasks.workunit.client.0.smithi103.stdout:0/86: mknod dd/d16/c17 0 2024-03-22T02:28:31.704 INFO:tasks.workunit.client.0.smithi103.stdout:0/87: chown l9 165785644 1 2024-03-22T02:28:31.709 INFO:tasks.workunit.client.0.smithi103.stdout:0/88: dwrite f8 [0,4194304] 0 2024-03-22T02:28:31.709 INFO:tasks.workunit.client.0.smithi103.stdout:0/89: truncate f4 1560361 0 2024-03-22T02:28:31.712 INFO:tasks.workunit.client.0.smithi103.stdout:0/90: rename dd/ce to dd/d16/c18 0 2024-03-22T02:28:31.712 INFO:tasks.workunit.client.0.smithi103.stdout:0/91: readlink l7 0 2024-03-22T02:28:31.713 INFO:tasks.workunit.client.0.smithi103.stdout:0/92: creat dd/d16/f19 x:0 0 0 2024-03-22T02:28:31.716 INFO:tasks.workunit.client.0.smithi103.stdout:0/93: link l7 dd/d16/l1a 0 2024-03-22T02:28:31.717 INFO:tasks.workunit.client.0.smithi103.stdout:0/94: symlink dd/d16/l1b 0 2024-03-22T02:28:31.730 INFO:tasks.workunit.client.0.smithi103.stdout:0/95: write fc [666435,43746] 0 2024-03-22T02:28:31.730 INFO:tasks.workunit.client.0.smithi103.stdout:0/96: dread - f6 zero size 2024-03-22T02:28:31.734 INFO:tasks.workunit.client.0.smithi103.stdout:0/97: dwrite dd/f15 [0,4194304] 0 2024-03-22T02:28:31.738 INFO:tasks.workunit.client.0.smithi103.stdout:0/98: getdents dd 0 2024-03-22T02:28:31.779 INFO:tasks.workunit.client.0.smithi103.stdout:0/99: rename f8 to dd/f1c 0 2024-03-22T02:28:31.779 INFO:tasks.workunit.client.0.smithi103.stdout:0/100: chown fa 1 1 2024-03-22T02:28:31.783 INFO:tasks.workunit.client.0.smithi103.stdout:0/101: dread dd/f15 [0,4194304] 0 2024-03-22T02:28:31.784 INFO:tasks.workunit.client.0.smithi103.stdout:0/102: creat dd/f1d x:0 0 0 2024-03-22T02:28:31.790 INFO:tasks.workunit.client.0.smithi103.stdout:0/103: dwrite dd/f12 [0,4194304] 0 2024-03-22T02:28:31.792 INFO:tasks.workunit.client.0.smithi103.stdout:0/104: mknod dd/c1e 0 2024-03-22T02:28:31.794 INFO:tasks.workunit.client.0.smithi103.stdout:0/105: mknod dd/d16/c1f 0 2024-03-22T02:28:31.796 INFO:tasks.workunit.client.0.smithi103.stdout:0/106: mkdir dd/d16/d20 0 2024-03-22T02:28:31.798 INFO:tasks.workunit.client.0.smithi103.stdout:0/107: creat dd/d16/d20/f21 x:0 0 0 2024-03-22T02:28:31.803 INFO:tasks.workunit.client.0.smithi103.stdout:0/108: dwrite dd/f15 [0,4194304] 0 2024-03-22T02:28:31.809 INFO:tasks.workunit.client.0.smithi103.stdout:0/109: rmdir dd/d16 39 2024-03-22T02:28:31.815 INFO:tasks.workunit.client.0.smithi103.stdout:0/110: dwrite dd/f1c [0,4194304] 0 2024-03-22T02:28:31.819 INFO:tasks.workunit.client.0.smithi103.stdout:0/111: unlink dd/l14 0 2024-03-22T02:28:31.823 INFO:tasks.workunit.client.0.smithi103.stdout:0/112: creat dd/d16/d20/f22 x:0 0 0 2024-03-22T02:28:31.823 INFO:tasks.workunit.client.0.smithi103.stdout:0/113: fdatasync dd/f13 0 2024-03-22T02:28:31.826 INFO:tasks.workunit.client.0.smithi103.stdout:0/114: link dd/f1c dd/d16/d20/f23 0 2024-03-22T02:28:31.828 INFO:tasks.workunit.client.0.smithi103.stdout:0/115: fsync fc 0 2024-03-22T02:28:31.833 INFO:tasks.workunit.client.0.smithi103.stdout:0/116: rename f6 to dd/f24 0 2024-03-22T02:28:31.836 INFO:tasks.workunit.client.0.smithi103.stdout:0/117: dread dd/f12 [0,4194304] 0 2024-03-22T02:28:31.838 INFO:tasks.workunit.client.0.smithi103.stdout:0/118: mkdir dd/d25 0 2024-03-22T02:28:31.839 INFO:tasks.workunit.client.0.smithi103.stdout:0/119: fsync dd/d16/f19 0 2024-03-22T02:28:31.840 INFO:tasks.workunit.client.0.smithi103.stdout:0/120: dread - dd/d16/d20/f22 zero size 2024-03-22T02:28:31.841 INFO:tasks.workunit.client.0.smithi103.stdout:0/121: write dd/d16/f19 [190703,127701] 0 2024-03-22T02:28:31.842 INFO:tasks.workunit.client.0.smithi103.stdout:0/122: write f2 [8937573,118329] 0 2024-03-22T02:28:31.843 INFO:tasks.workunit.client.0.smithi103.stdout:0/123: write fa [4678325,26847] 0 2024-03-22T02:28:31.845 INFO:tasks.workunit.client.0.smithi103.stdout:0/124: creat dd/d16/d20/f26 x:0 0 0 2024-03-22T02:28:31.847 INFO:tasks.workunit.client.0.smithi103.stdout:0/125: unlink f2 0 2024-03-22T02:28:31.848 INFO:tasks.workunit.client.0.smithi103.stdout:0/126: write dd/d16/f19 [928110,81046] 0 2024-03-22T02:28:31.849 INFO:tasks.workunit.client.0.smithi103.stdout:0/127: chown dd/f13 36317 1 2024-03-22T02:28:31.851 INFO:tasks.workunit.client.0.smithi103.stdout:0/128: mknod dd/c27 0 2024-03-22T02:28:31.851 INFO:tasks.workunit.client.0.smithi103.stdout:0/129: chown dd/f1c 7879 1 2024-03-22T02:28:31.852 INFO:tasks.workunit.client.0.smithi103.stdout:0/130: write dd/f1c [4121225,72768] 0 2024-03-22T02:28:31.854 INFO:tasks.workunit.client.0.smithi103.stdout:0/131: mknod dd/d16/d20/c28 0 2024-03-22T02:28:31.855 INFO:tasks.workunit.client.0.smithi103.stdout:0/132: readlink dd/d16/l1a 0 2024-03-22T02:28:31.855 INFO:tasks.workunit.client.0.smithi103.stdout:0/133: chown dd/d25 2070086 1 2024-03-22T02:28:31.858 INFO:tasks.workunit.client.0.smithi103.stdout:5/68: sync 2024-03-22T02:28:31.858 INFO:tasks.workunit.client.0.smithi103.stdout:3/173: sync 2024-03-22T02:28:31.859 INFO:tasks.workunit.client.0.smithi103.stdout:5/69: chown d0/f12 1341 1 2024-03-22T02:28:31.861 INFO:tasks.workunit.client.0.smithi103.stdout:3/174: mkdir de/d1a/d25/d3f 0 2024-03-22T02:28:31.863 INFO:tasks.workunit.client.0.smithi103.stdout:5/70: symlink d0/d14/l16 0 2024-03-22T02:28:31.863 INFO:tasks.workunit.client.0.smithi103.stdout:5/71: readlink d0/d14/l15 0 2024-03-22T02:28:31.865 INFO:tasks.workunit.client.0.smithi103.stdout:3/175: symlink de/d1a/d25/d3f/l40 0 2024-03-22T02:28:31.865 INFO:tasks.workunit.client.0.smithi103.stdout:3/176: fdatasync de/f1e 0 2024-03-22T02:28:31.867 INFO:tasks.workunit.client.0.smithi103.stdout:5/72: creat d0/f17 x:0 0 0 2024-03-22T02:28:31.868 INFO:tasks.workunit.client.0.smithi103.stdout:0/134: truncate dd/d16/f19 380351 0 2024-03-22T02:28:31.870 INFO:tasks.workunit.client.0.smithi103.stdout:5/73: symlink d0/d14/l18 0 2024-03-22T02:28:31.872 INFO:tasks.workunit.client.0.smithi103.stdout:0/135: mkdir dd/d16/d20/d29 0 2024-03-22T02:28:31.872 INFO:tasks.workunit.client.0.smithi103.stdout:3/177: link de/d12/c2c de/d1a/d25/c41 0 2024-03-22T02:28:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:31 smithi103 ceph-mon[29258]: pgmap v9: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 115 KiB/s rd, 17 MiB/s wr, 414 op/s 2024-03-22T02:28:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:31 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:31 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:31.874 INFO:tasks.workunit.client.0.smithi103.stdout:3/178: readlink de/d12/l3e 0 2024-03-22T02:28:31.877 INFO:tasks.workunit.client.0.smithi103.stdout:5/74: dwrite d0/fe [4194304,4194304] 0 2024-03-22T02:28:31.878 INFO:tasks.workunit.client.0.smithi103.stdout:3/179: creat de/d1a/d25/f42 x:0 0 0 2024-03-22T02:28:31.881 INFO:tasks.workunit.client.0.smithi103.stdout:0/136: dwrite dd/f12 [0,4194304] 0 2024-03-22T02:28:31.883 INFO:tasks.workunit.client.0.smithi103.stdout:5/75: mkdir d0/d14/d19 0 2024-03-22T02:28:31.887 INFO:tasks.workunit.client.0.smithi103.stdout:5/76: write d0/f12 [307604,53999] 0 2024-03-22T02:28:31.889 INFO:tasks.workunit.client.0.smithi103.stdout:5/77: creat d0/d14/d19/f1a x:0 0 0 2024-03-22T02:28:31.895 INFO:tasks.workunit.client.0.smithi103.stdout:5/78: dwrite d0/f7 [0,4194304] 0 2024-03-22T02:28:31.898 INFO:tasks.workunit.client.0.smithi103.stdout:5/79: rename d0/cf to d0/c1b 0 2024-03-22T02:28:31.901 INFO:tasks.workunit.client.0.smithi103.stdout:0/137: dread f4 [0,4194304] 0 2024-03-22T02:28:31.902 INFO:tasks.workunit.client.0.smithi103.stdout:0/138: write dd/f1c [598594,96783] 0 2024-03-22T02:28:31.903 INFO:tasks.workunit.client.0.smithi103.stdout:0/139: read fc [18350,59605] 0 2024-03-22T02:28:31.903 INFO:tasks.workunit.client.0.smithi103.stdout:0/140: fdatasync dd/f11 0 2024-03-22T02:28:31.904 INFO:tasks.workunit.client.0.smithi103.stdout:0/141: rmdir dd/d16 39 2024-03-22T02:28:31.930 INFO:tasks.workunit.client.0.smithi103.stdout:5/80: dread d0/fe [0,4194304] 0 2024-03-22T02:28:31.930 INFO:tasks.workunit.client.0.smithi103.stdout:5/81: read - d0/f17 zero size 2024-03-22T02:28:31.931 INFO:tasks.workunit.client.0.smithi103.stdout:5/82: chown d0/d14/d19 876 1 2024-03-22T02:28:31.932 INFO:tasks.workunit.client.0.smithi103.stdout:5/83: write d0/fd [3138618,28063] 0 2024-03-22T02:28:31.933 INFO:tasks.workunit.client.0.smithi103.stdout:5/84: truncate d0/f17 390384 0 2024-03-22T02:28:31.934 INFO:tasks.workunit.client.0.smithi103.stdout:5/85: write d0/f12 [960513,58734] 0 2024-03-22T02:28:31.935 INFO:tasks.workunit.client.0.smithi103.stdout:5/86: dread d0/f17 [0,4194304] 0 2024-03-22T02:28:31.936 INFO:tasks.workunit.client.0.smithi103.stdout:5/87: readlink d0/l11 0 2024-03-22T02:28:31.938 INFO:tasks.workunit.client.0.smithi103.stdout:5/88: fsync d0/fc 0 2024-03-22T02:28:31.939 INFO:tasks.workunit.client.0.smithi103.stdout:5/89: readlink d0/d14/l15 0 2024-03-22T02:28:31.940 INFO:tasks.workunit.client.0.smithi103.stdout:5/90: creat d0/d14/d19/f1c x:0 0 0 2024-03-22T02:28:31.943 INFO:tasks.workunit.client.0.smithi103.stdout:5/91: link d0/fe d0/f1d 0 2024-03-22T02:28:31.945 INFO:tasks.workunit.client.0.smithi103.stdout:5/92: creat d0/f1e x:0 0 0 2024-03-22T02:28:31.946 INFO:tasks.workunit.client.0.smithi103.stdout:5/93: fdatasync d0/fe 0 2024-03-22T02:28:31.948 INFO:tasks.workunit.client.0.smithi103.stdout:5/94: mkdir d0/d14/d19/d1f 0 2024-03-22T02:28:31.949 INFO:tasks.workunit.client.0.smithi103.stdout:5/95: read d0/fc [549306,27351] 0 2024-03-22T02:28:31.958 INFO:tasks.workunit.client.0.smithi103.stdout:5/96: dwrite d0/f17 [0,4194304] 0 2024-03-22T02:28:31.961 INFO:tasks.workunit.client.0.smithi103.stdout:5/97: rename d0/fa to d0/d14/d19/d1f/f20 0 2024-03-22T02:28:31.966 INFO:tasks.workunit.client.0.smithi103.stdout:5/98: dwrite d0/f1d [0,4194304] 0 2024-03-22T02:28:31.968 INFO:tasks.workunit.client.0.smithi103.stdout:5/99: mknod d0/d14/c21 0 2024-03-22T02:28:31.973 INFO:tasks.workunit.client.0.smithi103.stdout:5/100: link d0/f1e d0/d14/f22 0 2024-03-22T02:28:31.976 INFO:tasks.workunit.client.0.smithi103.stdout:5/101: dread d0/d14/d19/d1f/f20 [4194304,4194304] 0 2024-03-22T02:28:31.990 INFO:tasks.workunit.client.0.smithi103.stdout:5/102: dwrite d0/fc [0,4194304] 0 2024-03-22T02:28:31.997 INFO:tasks.workunit.client.0.smithi103.stdout:5/103: dwrite d0/f7 [0,4194304] 0 2024-03-22T02:28:31.999 INFO:tasks.workunit.client.0.smithi103.stdout:5/104: mkdir d0/d14/d23 0 2024-03-22T02:28:32.000 INFO:tasks.workunit.client.0.smithi103.stdout:9/140: sync 2024-03-22T02:28:32.000 INFO:tasks.workunit.client.0.smithi103.stdout:2/82: sync 2024-03-22T02:28:32.000 INFO:tasks.workunit.client.0.smithi103.stdout:7/72: sync 2024-03-22T02:28:32.000 INFO:tasks.workunit.client.0.smithi103.stdout:5/105: readlink d0/d14/l18 0 2024-03-22T02:28:32.002 INFO:tasks.workunit.client.0.smithi103.stdout:9/141: chown d2/da/de/c2d 103110 1 2024-03-22T02:28:32.003 INFO:tasks.workunit.client.0.smithi103.stdout:9/142: chown d2/da/de/c1c 32256 1 2024-03-22T02:28:32.003 INFO:tasks.workunit.client.0.smithi103.stdout:9/143: chown d2/f3 0 1 2024-03-22T02:28:32.004 INFO:tasks.workunit.client.0.smithi103.stdout:5/106: rename d0/f1d to d0/d14/d23/f24 0 2024-03-22T02:28:32.005 INFO:tasks.workunit.client.0.smithi103.stdout:2/83: symlink d11/d19/l1a 0 2024-03-22T02:28:32.007 INFO:tasks.workunit.client.0.smithi103.stdout:9/144: symlink d2/d5/l2e 0 2024-03-22T02:28:32.009 INFO:tasks.workunit.client.0.smithi103.stdout:2/84: dread d11/f13 [0,4194304] 0 2024-03-22T02:28:32.009 INFO:tasks.workunit.client.0.smithi103.stdout:5/107: mkdir d0/d14/d19/d1f/d25 0 2024-03-22T02:28:32.011 INFO:tasks.workunit.client.0.smithi103.stdout:2/85: symlink d11/d15/l1b 0 2024-03-22T02:28:32.017 INFO:tasks.workunit.client.0.smithi103.stdout:2/86: dwrite f10 [0,4194304] 0 2024-03-22T02:28:32.018 INFO:tasks.workunit.client.0.smithi103.stdout:2/87: chown la 2366 1 2024-03-22T02:28:32.027 INFO:tasks.workunit.client.0.smithi103.stdout:2/88: dwrite fd [0,4194304] 0 2024-03-22T02:28:32.030 INFO:tasks.workunit.client.0.smithi103.stdout:2/89: symlink d11/l1c 0 2024-03-22T02:28:32.033 INFO:tasks.workunit.client.0.smithi103.stdout:2/90: unlink cf 0 2024-03-22T02:28:32.034 INFO:tasks.workunit.client.0.smithi103.stdout:2/91: write f10 [3120619,96440] 0 2024-03-22T02:28:32.034 INFO:tasks.workunit.client.0.smithi103.stdout:2/92: dread - f4 zero size 2024-03-22T02:28:32.035 INFO:tasks.workunit.client.0.smithi103.stdout:2/93: write f2 [924749,1860] 0 2024-03-22T02:28:32.037 INFO:tasks.workunit.client.0.smithi103.stdout:2/94: symlink d11/d19/l1d 0 2024-03-22T02:28:32.038 INFO:tasks.workunit.client.0.smithi103.stdout:2/95: mknod d11/c1e 0 2024-03-22T02:28:32.040 INFO:tasks.workunit.client.0.smithi103.stdout:2/96: creat d11/d19/f1f x:0 0 0 2024-03-22T02:28:32.040 INFO:tasks.workunit.client.0.smithi103.stdout:2/97: stat ce 0 2024-03-22T02:28:32.041 INFO:tasks.workunit.client.0.smithi103.stdout:2/98: dread - d11/f14 zero size 2024-03-22T02:28:32.041 INFO:tasks.workunit.client.0.smithi103.stdout:2/99: read - d11/f14 zero size 2024-03-22T02:28:32.048 INFO:tasks.workunit.client.0.smithi103.stdout:2/100: dwrite fd [4194304,4194304] 0 2024-03-22T02:28:32.051 INFO:tasks.workunit.client.0.smithi103.stdout:2/101: unlink d11/d19/l1d 0 2024-03-22T02:28:32.052 INFO:tasks.workunit.client.0.smithi103.stdout:2/102: chown f2 2151530 1 2024-03-22T02:28:32.054 INFO:tasks.workunit.client.0.smithi103.stdout:2/103: creat d11/f20 x:0 0 0 2024-03-22T02:28:32.055 INFO:tasks.workunit.client.0.smithi103.stdout:2/104: write d11/f12 [553136,17261] 0 2024-03-22T02:28:32.063 INFO:tasks.workunit.client.0.smithi103.stdout:2/105: getdents d11/d19 0 2024-03-22T02:28:32.064 INFO:tasks.workunit.client.0.smithi103.stdout:2/106: fdatasync d11/f13 0 2024-03-22T02:28:32.065 INFO:tasks.workunit.client.0.smithi103.stdout:2/107: fdatasync f2 0 2024-03-22T02:28:32.066 INFO:tasks.workunit.client.0.smithi103.stdout:2/108: write f7 [2415216,5236] 0 2024-03-22T02:28:32.162 INFO:tasks.workunit.client.0.smithi103.stdout:6/72: sync 2024-03-22T02:28:32.162 INFO:tasks.workunit.client.0.smithi103.stdout:8/188: sync 2024-03-22T02:28:32.163 INFO:tasks.workunit.client.0.smithi103.stdout:4/111: sync 2024-03-22T02:28:32.163 INFO:tasks.workunit.client.0.smithi103.stdout:1/104: sync 2024-03-22T02:28:32.163 INFO:tasks.workunit.client.0.smithi103.stdout:6/73: chown d0 364 1 2024-03-22T02:28:32.163 INFO:tasks.workunit.client.0.smithi103.stdout:4/112: chown d3/d1e/d25/f29 3134057 1 2024-03-22T02:28:32.164 INFO:tasks.workunit.client.0.smithi103.stdout:6/74: fsync d0/d3/f9 0 2024-03-22T02:28:32.165 INFO:tasks.workunit.client.0.smithi103.stdout:1/105: mknod db/d10/d18/c24 0 2024-03-22T02:28:32.165 INFO:tasks.workunit.client.0.smithi103.stdout:1/106: fdatasync db/d10/d14/f16 0 2024-03-22T02:28:32.166 INFO:tasks.workunit.client.0.smithi103.stdout:4/113: mknod d3/d1e/c2a 0 2024-03-22T02:28:32.166 INFO:tasks.workunit.client.0.smithi103.stdout:4/114: stat d3/d1e 0 2024-03-22T02:28:32.167 INFO:tasks.workunit.client.0.smithi103.stdout:1/107: write db/d10/d21/f23 [408782,31643] 0 2024-03-22T02:28:32.168 INFO:tasks.workunit.client.0.smithi103.stdout:6/75: rmdir d0 39 2024-03-22T02:28:32.168 INFO:tasks.workunit.client.0.smithi103.stdout:4/115: dread d3/f17 [0,4194304] 0 2024-03-22T02:28:32.169 INFO:tasks.workunit.client.0.smithi103.stdout:1/108: truncate db/d10/d18/f20 63206 0 2024-03-22T02:28:32.171 INFO:tasks.workunit.client.0.smithi103.stdout:1/109: creat db/f25 x:0 0 0 2024-03-22T02:28:32.173 INFO:tasks.workunit.client.0.smithi103.stdout:4/116: rename d3/f21 to d3/d1e/d25/f2b 0 2024-03-22T02:28:32.177 INFO:tasks.workunit.client.0.smithi103.stdout:1/110: dwrite db/f17 [0,4194304] 0 2024-03-22T02:28:32.178 INFO:tasks.workunit.client.0.smithi103.stdout:6/76: creat d0/d3/f1b x:0 0 0 2024-03-22T02:28:32.179 INFO:tasks.workunit.client.0.smithi103.stdout:1/111: creat db/d10/f26 x:0 0 0 2024-03-22T02:28:32.180 INFO:tasks.workunit.client.0.smithi103.stdout:6/77: write d0/d14/df/f11 [7377905,111230] 0 2024-03-22T02:28:32.181 INFO:tasks.workunit.client.0.smithi103.stdout:1/112: mkdir db/d27 0 2024-03-22T02:28:32.182 INFO:tasks.workunit.client.0.smithi103.stdout:1/113: write db/d10/d21/f22 [1020823,70164] 0 2024-03-22T02:28:32.183 INFO:tasks.workunit.client.0.smithi103.stdout:6/78: rename d0/d14/fe to d0/d14/df/f1c 0 2024-03-22T02:28:32.184 INFO:tasks.workunit.client.0.smithi103.stdout:6/79: stat d0/d14/c17 0 2024-03-22T02:28:32.184 INFO:tasks.workunit.client.0.smithi103.stdout:6/80: dread - d0/d14/df/f15 zero size 2024-03-22T02:28:32.186 INFO:tasks.workunit.client.0.smithi103.stdout:6/81: mknod d0/d14/c1d 0 2024-03-22T02:28:32.187 INFO:tasks.workunit.client.0.smithi103.stdout:1/114: dread db/d10/d18/f20 [0,4194304] 0 2024-03-22T02:28:32.189 INFO:tasks.workunit.client.0.smithi103.stdout:1/115: mkdir db/d10/d28 0 2024-03-22T02:28:32.190 INFO:tasks.workunit.client.0.smithi103.stdout:6/82: unlink d0/d3/c5 0 2024-03-22T02:28:32.195 INFO:tasks.workunit.client.0.smithi103.stdout:6/83: rename d0/fb to d0/f1e 0 2024-03-22T02:28:32.197 INFO:tasks.workunit.client.0.smithi103.stdout:4/117: dwrite d3/fd [0,4194304] 0 2024-03-22T02:28:32.199 INFO:tasks.workunit.client.0.smithi103.stdout:1/116: dwrite f7 [0,4194304] 0 2024-03-22T02:28:32.202 INFO:tasks.workunit.client.0.smithi103.stdout:1/117: mkdir db/d10/d29 0 2024-03-22T02:28:32.205 INFO:tasks.workunit.client.0.smithi103.stdout:1/118: unlink db/c1b 0 2024-03-22T02:28:32.206 INFO:tasks.workunit.client.0.smithi103.stdout:4/118: dwrite d3/fd [0,4194304] 0 2024-03-22T02:28:32.210 INFO:tasks.workunit.client.0.smithi103.stdout:1/119: symlink db/d10/d21/l2a 0 2024-03-22T02:28:32.213 INFO:tasks.workunit.client.0.smithi103.stdout:6/84: dread d0/f19 [0,4194304] 0 2024-03-22T02:28:32.213 INFO:tasks.workunit.client.0.smithi103.stdout:1/120: creat db/f2b x:0 0 0 2024-03-22T02:28:32.214 INFO:tasks.workunit.client.0.smithi103.stdout:1/121: stat db 0 2024-03-22T02:28:32.221 INFO:tasks.workunit.client.0.smithi103.stdout:4/119: getdents d3 0 2024-03-22T02:28:32.221 INFO:tasks.workunit.client.0.smithi103.stdout:6/85: creat d0/d3/f1f x:0 0 0 2024-03-22T02:28:32.223 INFO:tasks.workunit.client.0.smithi103.stdout:1/122: dwrite db/d10/f1c [0,4194304] 0 2024-03-22T02:28:32.225 INFO:tasks.workunit.client.0.smithi103.stdout:6/86: mkdir d0/d14/d20 0 2024-03-22T02:28:32.228 INFO:tasks.workunit.client.0.smithi103.stdout:6/87: stat d0/d3/c16 0 2024-03-22T02:28:32.228 INFO:tasks.workunit.client.0.smithi103.stdout:1/123: symlink db/d10/d28/l2c 0 2024-03-22T02:28:32.231 INFO:tasks.workunit.client.0.smithi103.stdout:4/120: dwrite d3/f24 [0,4194304] 0 2024-03-22T02:28:32.232 INFO:tasks.workunit.client.0.smithi103.stdout:1/124: write db/d10/d18/f20 [346388,101602] 0 2024-03-22T02:28:32.233 INFO:tasks.workunit.client.0.smithi103.stdout:4/121: unlink d3/c22 0 2024-03-22T02:28:32.236 INFO:tasks.workunit.client.0.smithi103.stdout:1/125: dread f7 [4194304,4194304] 0 2024-03-22T02:28:32.243 INFO:tasks.workunit.client.0.smithi103.stdout:1/126: dwrite db/d10/f26 [0,4194304] 0 2024-03-22T02:28:32.245 INFO:tasks.workunit.client.0.smithi103.stdout:0/142: sync 2024-03-22T02:28:32.245 INFO:tasks.workunit.client.0.smithi103.stdout:3/180: sync 2024-03-22T02:28:32.246 INFO:tasks.workunit.client.0.smithi103.stdout:0/143: write dd/f11 [682004,130062] 0 2024-03-22T02:28:32.247 INFO:tasks.workunit.client.0.smithi103.stdout:1/127: rename db/d10/d14 to db/d10/d18/d2d 0 2024-03-22T02:28:32.248 INFO:tasks.workunit.client.0.smithi103.stdout:1/128: write db/d10/d18/f20 [1449094,127590] 0 2024-03-22T02:28:32.248 INFO:tasks.workunit.client.0.smithi103.stdout:3/181: creat de/d12/d19/f43 x:0 0 0 2024-03-22T02:28:32.251 INFO:tasks.workunit.client.0.smithi103.stdout:3/182: creat de/d1a/d25/d26/f44 x:0 0 0 2024-03-22T02:28:32.253 INFO:tasks.workunit.client.0.smithi103.stdout:0/144: truncate dd/d16/f19 432208 0 2024-03-22T02:28:32.293 INFO:tasks.workunit.client.0.smithi103.stdout:6/88: rmdir d0/d3 39 2024-03-22T02:28:32.294 INFO:tasks.workunit.client.0.smithi103.stdout:3/183: link de/d12/c1d de/d1a/d25/c45 0 2024-03-22T02:28:32.295 INFO:tasks.workunit.client.0.smithi103.stdout:6/89: chown d0/d3/f1f 742 1 2024-03-22T02:28:32.296 INFO:tasks.workunit.client.0.smithi103.stdout:3/184: mknod de/d1a/d25/c46 0 2024-03-22T02:28:32.310 INFO:tasks.workunit.client.0.smithi103.stdout:4/122: mkdir d3/d1e/d25/d2c 0 2024-03-22T02:28:32.311 INFO:tasks.workunit.client.0.smithi103.stdout:4/123: write d3/f1d [504416,5729] 0 2024-03-22T02:28:32.314 INFO:tasks.workunit.client.0.smithi103.stdout:4/124: unlink d3/l23 0 2024-03-22T02:28:32.320 INFO:tasks.workunit.client.0.smithi103.stdout:4/125: dwrite d3/d1e/f28 [0,4194304] 0 2024-03-22T02:28:32.324 INFO:tasks.workunit.client.0.smithi103.stdout:3/185: dread fa [0,4194304] 0 2024-03-22T02:28:32.326 INFO:tasks.workunit.client.0.smithi103.stdout:4/126: link d3/d1e/d25/f2b d3/f2d 0 2024-03-22T02:28:32.328 INFO:tasks.workunit.client.0.smithi103.stdout:3/186: mkdir de/d12/d47 0 2024-03-22T02:28:32.329 INFO:tasks.workunit.client.0.smithi103.stdout:4/127: symlink d3/d1e/l2e 0 2024-03-22T02:28:32.330 INFO:tasks.workunit.client.0.smithi103.stdout:3/187: creat de/d12/d19/f48 x:0 0 0 2024-03-22T02:28:32.331 INFO:tasks.workunit.client.0.smithi103.stdout:4/128: stat d3/l19 0 2024-03-22T02:28:32.331 INFO:tasks.workunit.client.0.smithi103.stdout:3/188: dread - de/f22 zero size 2024-03-22T02:28:32.331 INFO:tasks.workunit.client.0.smithi103.stdout:3/189: fsync de/d1a/f31 0 2024-03-22T02:28:32.333 INFO:tasks.workunit.client.0.smithi103.stdout:3/190: chown de/d12/d30/f35 8479 1 2024-03-22T02:28:32.335 INFO:tasks.workunit.client.0.smithi103.stdout:4/129: link d3/f7 d3/d1e/f2f 0 2024-03-22T02:28:32.337 INFO:tasks.workunit.client.0.smithi103.stdout:4/130: fsync d3/f24 0 2024-03-22T02:28:32.339 INFO:tasks.workunit.client.0.smithi103.stdout:4/131: mkdir d3/d30 0 2024-03-22T02:28:32.342 INFO:tasks.workunit.client.0.smithi103.stdout:4/132: unlink d3/d1e/d25/f29 0 2024-03-22T02:28:32.348 INFO:tasks.workunit.client.0.smithi103.stdout:4/133: truncate d3/f2d 1443973 0 2024-03-22T02:28:32.352 INFO:tasks.workunit.client.0.smithi103.stdout:3/191: dread f8 [0,4194304] 0 2024-03-22T02:28:32.353 INFO:tasks.workunit.client.0.smithi103.stdout:4/134: dread d3/d1e/f28 [0,4194304] 0 2024-03-22T02:28:32.354 INFO:tasks.workunit.client.0.smithi103.stdout:3/192: read de/f21 [26560,117090] 0 2024-03-22T02:28:32.354 INFO:tasks.workunit.client.0.smithi103.stdout:3/193: write fa [2097037,115260] 0 2024-03-22T02:28:32.358 INFO:tasks.workunit.client.0.smithi103.stdout:4/135: mkdir d3/d31 0 2024-03-22T02:28:32.362 INFO:tasks.workunit.client.0.smithi103.stdout:3/194: rename de/d1a/d25/c45 to de/d1a/d25/d26/c49 0 2024-03-22T02:28:32.362 INFO:tasks.workunit.client.0.smithi103.stdout:3/195: fsync de/d12/d19/f43 0 2024-03-22T02:28:32.364 INFO:tasks.workunit.client.0.smithi103.stdout:3/196: read - de/d1a/d25/f42 zero size 2024-03-22T02:28:32.364 INFO:tasks.workunit.client.0.smithi103.stdout:3/197: readlink de/d12/l15 0 2024-03-22T02:28:32.365 INFO:tasks.workunit.client.0.smithi103.stdout:4/136: rename f2 to d3/d31/f32 0 2024-03-22T02:28:32.366 INFO:tasks.workunit.client.0.smithi103.stdout:3/198: symlink de/d12/l4a 0 2024-03-22T02:28:32.369 INFO:tasks.workunit.client.0.smithi103.stdout:7/73: sync 2024-03-22T02:28:32.369 INFO:tasks.workunit.client.0.smithi103.stdout:5/108: sync 2024-03-22T02:28:32.369 INFO:tasks.workunit.client.0.smithi103.stdout:8/189: sync 2024-03-22T02:28:32.370 INFO:tasks.workunit.client.0.smithi103.stdout:2/109: sync 2024-03-22T02:28:32.370 INFO:tasks.workunit.client.0.smithi103.stdout:9/145: sync 2024-03-22T02:28:32.370 INFO:tasks.workunit.client.0.smithi103.stdout:2/110: readlink lb 0 2024-03-22T02:28:32.371 INFO:tasks.workunit.client.0.smithi103.stdout:2/111: truncate d11/f20 119992 0 2024-03-22T02:28:32.372 INFO:tasks.workunit.client.0.smithi103.stdout:5/109: rmdir d0/d14 39 2024-03-22T02:28:32.372 INFO:tasks.workunit.client.0.smithi103.stdout:2/112: dread d11/f20 [0,4194304] 0 2024-03-22T02:28:32.373 INFO:tasks.workunit.client.0.smithi103.stdout:9/146: chown d2/d10/c1d 216 1 2024-03-22T02:28:32.373 INFO:tasks.workunit.client.0.smithi103.stdout:4/137: truncate d3/d1e/f28 3588025 0 2024-03-22T02:28:32.374 INFO:tasks.workunit.client.0.smithi103.stdout:8/190: fdatasync d4/d13/f1b 0 2024-03-22T02:28:32.377 INFO:tasks.workunit.client.0.smithi103.stdout:2/113: rmdir d11 39 2024-03-22T02:28:32.379 INFO:tasks.workunit.client.0.smithi103.stdout:4/138: write d3/fd [56074,15714] 0 2024-03-22T02:28:32.383 INFO:tasks.workunit.client.0.smithi103.stdout:7/74: creat d1/f1e x:0 0 0 2024-03-22T02:28:32.384 INFO:tasks.workunit.client.0.smithi103.stdout:5/110: dwrite d0/d14/d19/f1a [0,4194304] 0 2024-03-22T02:28:32.385 INFO:tasks.workunit.client.0.smithi103.stdout:8/191: symlink d4/d13/l33 0 2024-03-22T02:28:32.393 INFO:tasks.workunit.client.0.smithi103.stdout:9/147: dread d2/d10/d17/f26 [0,4194304] 0 2024-03-22T02:28:32.394 INFO:tasks.workunit.client.0.smithi103.stdout:5/111: rmdir d0/d14/d19 39 2024-03-22T02:28:32.395 INFO:tasks.workunit.client.0.smithi103.stdout:5/112: read d0/d14/d23/f24 [5801421,57974] 0 2024-03-22T02:28:32.396 INFO:tasks.workunit.client.0.smithi103.stdout:8/192: unlink d4/d13/l33 0 2024-03-22T02:28:32.398 INFO:tasks.workunit.client.0.smithi103.stdout:2/114: write f1 [2035598,74360] 0 2024-03-22T02:28:32.402 INFO:tasks.workunit.client.0.smithi103.stdout:7/75: mknod d1/da/d1c/c1f 0 2024-03-22T02:28:32.404 INFO:tasks.workunit.client.0.smithi103.stdout:9/148: unlink d2/da/d18/l2b 0 2024-03-22T02:28:32.408 INFO:tasks.workunit.client.0.smithi103.stdout:4/139: write d3/f2d [2169943,6859] 0 2024-03-22T02:28:32.409 INFO:tasks.workunit.client.0.smithi103.stdout:7/76: mknod d1/da/d1c/c20 0 2024-03-22T02:28:32.411 INFO:tasks.workunit.client.0.smithi103.stdout:2/115: mkdir d11/d21 0 2024-03-22T02:28:32.412 INFO:tasks.workunit.client.0.smithi103.stdout:9/149: dwrite d2/da/f24 [0,4194304] 0 2024-03-22T02:28:32.414 INFO:tasks.workunit.client.0.smithi103.stdout:4/140: truncate d3/f1c 744933 0 2024-03-22T02:28:32.414 INFO:tasks.workunit.client.0.smithi103.stdout:5/113: rename d0/c4 to d0/c26 0 2024-03-22T02:28:32.416 INFO:tasks.workunit.client.0.smithi103.stdout:7/77: readlink d1/l6 0 2024-03-22T02:28:32.417 INFO:tasks.workunit.client.0.smithi103.stdout:8/193: creat d4/f34 x:0 0 0 2024-03-22T02:28:32.420 INFO:tasks.workunit.client.0.smithi103.stdout:2/116: rename d11/d19/l1a to d11/d19/l22 0 2024-03-22T02:28:32.423 INFO:tasks.workunit.client.0.smithi103.stdout:9/150: unlink d2/da/d18/l28 0 2024-03-22T02:28:32.427 INFO:tasks.workunit.client.0.smithi103.stdout:8/194: mkdir d4/d13/d35 0 2024-03-22T02:28:32.428 INFO:tasks.workunit.client.0.smithi103.stdout:9/151: mknod d2/d10/d17/c2f 0 2024-03-22T02:28:32.429 INFO:tasks.workunit.client.0.smithi103.stdout:5/114: mkdir d0/d14/d19/d1f/d27 0 2024-03-22T02:28:32.430 INFO:tasks.workunit.client.0.smithi103.stdout:4/141: mkdir d3/d1e/d25/d2c/d33 0 2024-03-22T02:28:32.434 INFO:tasks.workunit.client.0.smithi103.stdout:8/195: symlink d4/d13/l36 0 2024-03-22T02:28:32.436 INFO:tasks.workunit.client.0.smithi103.stdout:4/142: unlink d3/f1c 0 2024-03-22T02:28:32.441 INFO:tasks.workunit.client.0.smithi103.stdout:8/196: dwrite d4/ff [4194304,4194304] 0 2024-03-22T02:28:32.444 INFO:tasks.workunit.client.0.smithi103.stdout:7/78: getdents d1/da 0 2024-03-22T02:28:32.446 INFO:tasks.workunit.client.0.smithi103.stdout:9/152: creat d2/f30 x:0 0 0 2024-03-22T02:28:32.448 INFO:tasks.workunit.client.0.smithi103.stdout:5/115: creat d0/f28 x:0 0 0 2024-03-22T02:28:32.451 INFO:tasks.workunit.client.0.smithi103.stdout:8/197: dread d4/fe [0,4194304] 0 2024-03-22T02:28:32.452 INFO:tasks.workunit.client.0.smithi103.stdout:9/153: mknod d2/d10/d27/c31 0 2024-03-22T02:28:32.453 INFO:tasks.workunit.client.0.smithi103.stdout:5/116: mknod d0/d14/d19/d1f/c29 0 2024-03-22T02:28:32.455 INFO:tasks.workunit.client.0.smithi103.stdout:4/143: rmdir d3/d1e/d25/d2c/d33 0 2024-03-22T02:28:32.456 INFO:tasks.workunit.client.0.smithi103.stdout:2/117: truncate fd 6141689 0 2024-03-22T02:28:32.457 INFO:tasks.workunit.client.0.smithi103.stdout:2/118: chown lb 1 1 2024-03-22T02:28:32.459 INFO:tasks.workunit.client.0.smithi103.stdout:8/198: creat d4/d19/d1a/f37 x:0 0 0 2024-03-22T02:28:32.459 INFO:tasks.workunit.client.0.smithi103.stdout:4/144: dread d3/d1e/d25/f2b [0,4194304] 0 2024-03-22T02:28:32.463 INFO:tasks.workunit.client.0.smithi103.stdout:2/119: rename d11/l17 to d11/l23 0 2024-03-22T02:28:32.468 INFO:tasks.workunit.client.0.smithi103.stdout:5/117: rename d0/d14/d19 to d0/d14/d23/d2a 0 2024-03-22T02:28:32.470 INFO:tasks.workunit.client.0.smithi103.stdout:2/120: read d11/f20 [63175,20146] 0 2024-03-22T02:28:32.471 INFO:tasks.workunit.client.0.smithi103.stdout:5/118: write d0/d14/d23/d2a/f1a [2460106,61726] 0 2024-03-22T02:28:32.471 INFO:tasks.workunit.client.0.smithi103.stdout:7/79: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:32.478 INFO:tasks.workunit.client.0.smithi103.stdout:7/80: mkdir d1/d10/d1b/d21 0 2024-03-22T02:28:32.480 INFO:tasks.workunit.client.0.smithi103.stdout:9/154: truncate d2/da/d18/f21 3515866 0 2024-03-22T02:28:32.482 INFO:tasks.workunit.client.0.smithi103.stdout:8/199: dwrite d4/fe [0,4194304] 0 2024-03-22T02:28:32.483 INFO:tasks.workunit.client.0.smithi103.stdout:4/145: creat d3/f34 x:0 0 0 2024-03-22T02:28:32.483 INFO:tasks.workunit.client.0.smithi103.stdout:8/200: stat d4/ff 0 2024-03-22T02:28:32.484 INFO:tasks.workunit.client.0.smithi103.stdout:2/121: mkdir d11/d21/d24 0 2024-03-22T02:28:32.486 INFO:tasks.workunit.client.0.smithi103.stdout:2/122: truncate d11/d19/f1f 343831 0 2024-03-22T02:28:32.489 INFO:tasks.workunit.client.0.smithi103.stdout:5/119: symlink d0/d14/d23/d2a/d1f/d27/l2b 0 2024-03-22T02:28:32.489 INFO:tasks.workunit.client.0.smithi103.stdout:5/120: readlink d0/d14/l15 0 2024-03-22T02:28:32.489 INFO:tasks.workunit.client.0.smithi103.stdout:9/155: mknod d2/d5/c32 0 2024-03-22T02:28:32.491 INFO:tasks.workunit.client.0.smithi103.stdout:8/201: mkdir d4/d2a/d38 0 2024-03-22T02:28:32.492 INFO:tasks.workunit.client.0.smithi103.stdout:7/81: mknod d1/d10/d1b/d21/c22 0 2024-03-22T02:28:32.493 INFO:tasks.workunit.client.0.smithi103.stdout:7/82: chown d1/c9 84589689 1 2024-03-22T02:28:32.493 INFO:tasks.workunit.client.0.smithi103.stdout:5/121: mkdir d0/d14/d23/d2a/d2c 0 2024-03-22T02:28:32.495 INFO:tasks.workunit.client.0.smithi103.stdout:4/146: fsync d3/d31/f32 0 2024-03-22T02:28:32.497 INFO:tasks.workunit.client.0.smithi103.stdout:7/83: mkdir d1/d10/d1b/d21/d23 0 2024-03-22T02:28:32.498 INFO:tasks.workunit.client.0.smithi103.stdout:9/156: symlink d2/l33 0 2024-03-22T02:28:32.499 INFO:tasks.workunit.client.0.smithi103.stdout:9/157: readlink d2/d10/l16 0 2024-03-22T02:28:32.500 INFO:tasks.workunit.client.0.smithi103.stdout:4/147: write d3/f1a [3256552,6331] 0 2024-03-22T02:28:32.505 INFO:tasks.workunit.client.0.smithi103.stdout:7/84: creat d1/d10/d1b/f24 x:0 0 0 2024-03-22T02:28:32.508 INFO:tasks.workunit.client.0.smithi103.stdout:4/148: creat d3/d1e/f35 x:0 0 0 2024-03-22T02:28:32.509 INFO:tasks.workunit.client.0.smithi103.stdout:2/123: getdents d11/d19 0 2024-03-22T02:28:32.511 INFO:tasks.workunit.client.0.smithi103.stdout:5/122: creat d0/f2d x:0 0 0 2024-03-22T02:28:32.512 INFO:tasks.workunit.client.0.smithi103.stdout:7/85: rename d1/d10/d1b/d21/c22 to d1/d10/d1b/d21/c25 0 2024-03-22T02:28:32.513 INFO:tasks.workunit.client.0.smithi103.stdout:8/202: getdents d4/d2a 0 2024-03-22T02:28:32.513 INFO:tasks.workunit.client.0.smithi103.stdout:8/203: chown d4/d19/d1a/f2d 0 1 2024-03-22T02:28:32.514 INFO:tasks.workunit.client.0.smithi103.stdout:4/149: creat d3/d1e/f36 x:0 0 0 2024-03-22T02:28:32.515 INFO:tasks.workunit.client.0.smithi103.stdout:2/124: mkdir d11/d25 0 2024-03-22T02:28:32.518 INFO:tasks.workunit.client.0.smithi103.stdout:7/86: rename d1/d10/d11/f14 to d1/d10/d11/d19/f26 0 2024-03-22T02:28:32.519 INFO:tasks.workunit.client.0.smithi103.stdout:8/204: write d4/f15 [4523348,24345] 0 2024-03-22T02:28:32.520 INFO:tasks.workunit.client.0.smithi103.stdout:2/125: symlink d11/d15/l26 0 2024-03-22T02:28:32.521 INFO:tasks.workunit.client.0.smithi103.stdout:2/126: chown d11/d15/l1b 8253477 1 2024-03-22T02:28:32.522 INFO:tasks.workunit.client.0.smithi103.stdout:5/123: mkdir d0/d14/d2e 0 2024-03-22T02:28:32.522 INFO:tasks.workunit.client.0.smithi103.stdout:7/87: dread d1/da/fb [0,4194304] 0 2024-03-22T02:28:32.523 INFO:tasks.workunit.client.0.smithi103.stdout:5/124: chown d0/d14/d23/d2a/d2c 1887 1 2024-03-22T02:28:32.526 INFO:tasks.workunit.client.0.smithi103.stdout:8/205: fdatasync d4/fd 0 2024-03-22T02:28:32.530 INFO:tasks.workunit.client.0.smithi103.stdout:7/88: dwrite d1/f16 [0,4194304] 0 2024-03-22T02:28:32.530 INFO:tasks.workunit.client.0.smithi103.stdout:5/125: symlink d0/d14/d23/d2a/d1f/l2f 0 2024-03-22T02:28:32.533 INFO:tasks.workunit.client.0.smithi103.stdout:4/150: rename d3/f17 to d3/d31/f37 0 2024-03-22T02:28:32.538 INFO:tasks.workunit.client.0.smithi103.stdout:2/127: truncate fd 4709893 0 2024-03-22T02:28:32.538 INFO:tasks.workunit.client.0.smithi103.stdout:2/128: readlink d11/d19/l22 0 2024-03-22T02:28:32.541 INFO:tasks.workunit.client.0.smithi103.stdout:5/126: creat d0/d14/d23/d2a/d1f/d27/f30 x:0 0 0 2024-03-22T02:28:32.542 INFO:tasks.workunit.client.0.smithi103.stdout:5/127: write d0/fd [4092718,47656] 0 2024-03-22T02:28:32.544 INFO:tasks.workunit.client.0.smithi103.stdout:2/129: mknod d11/d21/c27 0 2024-03-22T02:28:32.546 INFO:tasks.workunit.client.0.smithi103.stdout:5/128: write d0/fc [3915249,25599] 0 2024-03-22T02:28:32.547 INFO:tasks.workunit.client.0.smithi103.stdout:7/89: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:32.551 INFO:tasks.workunit.client.0.smithi103.stdout:2/130: unlink f5 0 2024-03-22T02:28:32.553 INFO:tasks.workunit.client.0.smithi103.stdout:5/129: mknod d0/d14/d23/d2a/c31 0 2024-03-22T02:28:32.555 INFO:tasks.workunit.client.0.smithi103.stdout:5/130: write d0/f17 [3821487,3170] 0 2024-03-22T02:28:32.556 INFO:tasks.workunit.client.0.smithi103.stdout:7/90: mknod d1/d10/d1b/d21/c27 0 2024-03-22T02:28:32.557 INFO:tasks.workunit.client.0.smithi103.stdout:7/91: chown d1/da/d1c/c1f 297 1 2024-03-22T02:28:32.558 INFO:tasks.workunit.client.0.smithi103.stdout:2/131: symlink d11/d19/l28 0 2024-03-22T02:28:32.559 INFO:tasks.workunit.client.0.smithi103.stdout:8/206: truncate d4/ff 6993573 0 2024-03-22T02:28:32.561 INFO:tasks.workunit.client.0.smithi103.stdout:4/151: dread d3/f8 [0,4194304] 0 2024-03-22T02:28:32.564 INFO:tasks.workunit.client.0.smithi103.stdout:7/92: dwrite d1/f16 [0,4194304] 0 2024-03-22T02:28:32.566 INFO:tasks.workunit.client.0.smithi103.stdout:8/207: mkdir d4/d14/d39 0 2024-03-22T02:28:32.567 INFO:tasks.workunit.client.0.smithi103.stdout:8/208: chown d4/d14/f1e 411787538 1 2024-03-22T02:28:32.568 INFO:tasks.workunit.client.0.smithi103.stdout:8/209: write f1 [3443364,43547] 0 2024-03-22T02:28:32.572 INFO:tasks.workunit.client.0.smithi103.stdout:7/93: creat d1/d10/d11/f28 x:0 0 0 2024-03-22T02:28:32.573 INFO:tasks.workunit.client.0.smithi103.stdout:8/210: dread d4/fe [0,4194304] 0 2024-03-22T02:28:32.574 INFO:tasks.workunit.client.0.smithi103.stdout:5/131: rmdir d0/d14/d2e 0 2024-03-22T02:28:32.574 INFO:tasks.workunit.client.0.smithi103.stdout:8/211: fsync d4/d19/d1a/f2d 0 2024-03-22T02:28:32.575 INFO:tasks.workunit.client.0.smithi103.stdout:8/212: fsync d4/d26/d27/f2c 0 2024-03-22T02:28:32.578 INFO:tasks.workunit.client.0.smithi103.stdout:4/152: mkdir d3/d38 0 2024-03-22T02:28:32.582 INFO:tasks.workunit.client.0.smithi103.stdout:8/213: dwrite d4/f9 [0,4194304] 0 2024-03-22T02:28:32.587 INFO:tasks.workunit.client.0.smithi103.stdout:2/132: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:32.588 INFO:tasks.workunit.client.0.smithi103.stdout:2/133: dread - f4 zero size 2024-03-22T02:28:32.593 INFO:tasks.workunit.client.0.smithi103.stdout:8/214: symlink d4/d26/l3a 0 2024-03-22T02:28:32.595 INFO:tasks.workunit.client.0.smithi103.stdout:2/134: creat d11/d15/f29 x:0 0 0 2024-03-22T02:28:32.597 INFO:tasks.workunit.client.0.smithi103.stdout:7/94: rmdir d1/d10/d1b/d21/d23 0 2024-03-22T02:28:32.597 INFO:tasks.workunit.client.0.smithi103.stdout:2/135: write f7 [3060610,128818] 0 2024-03-22T02:28:32.598 INFO:tasks.workunit.client.0.smithi103.stdout:5/132: link d0/d14/f22 d0/d14/d23/d2a/d1f/f32 0 2024-03-22T02:28:32.599 INFO:tasks.workunit.client.0.smithi103.stdout:8/215: dread f1 [0,4194304] 0 2024-03-22T02:28:32.603 INFO:tasks.workunit.client.0.smithi103.stdout:5/133: mkdir d0/d14/d23/d2a/d33 0 2024-03-22T02:28:32.604 INFO:tasks.workunit.client.0.smithi103.stdout:2/136: dwrite f2 [0,4194304] 0 2024-03-22T02:28:32.606 INFO:tasks.workunit.client.0.smithi103.stdout:8/216: link d4/d14/c30 d4/d26/c3b 0 2024-03-22T02:28:32.608 INFO:tasks.workunit.client.0.smithi103.stdout:2/137: readlink d11/l23 0 2024-03-22T02:28:32.608 INFO:tasks.workunit.client.0.smithi103.stdout:8/217: read d4/d13/f1b [1822666,91885] 0 2024-03-22T02:28:32.609 INFO:tasks.workunit.client.0.smithi103.stdout:5/134: dread d0/f5 [4194304,4194304] 0 2024-03-22T02:28:32.610 INFO:tasks.workunit.client.0.smithi103.stdout:5/135: read - d0/f2d zero size 2024-03-22T02:28:32.612 INFO:tasks.workunit.client.0.smithi103.stdout:5/136: chown d0/d14/l18 9 1 2024-03-22T02:28:32.617 INFO:tasks.workunit.client.0.smithi103.stdout:8/218: dwrite d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:32.618 INFO:tasks.workunit.client.0.smithi103.stdout:7/95: truncate d1/da/f3 1042764 0 2024-03-22T02:28:32.619 INFO:tasks.workunit.client.0.smithi103.stdout:8/219: chown d4/d13/d35 1481 1 2024-03-22T02:28:32.620 INFO:tasks.workunit.client.0.smithi103.stdout:4/153: dwrite d3/d1e/d25/f2b [0,4194304] 0 2024-03-22T02:28:32.621 INFO:tasks.workunit.client.0.smithi103.stdout:7/96: mkdir d1/d10/d11/d19/d29 0 2024-03-22T02:28:32.624 INFO:tasks.workunit.client.0.smithi103.stdout:4/154: creat d3/d1e/f39 x:0 0 0 2024-03-22T02:28:32.625 INFO:tasks.workunit.client.0.smithi103.stdout:7/97: unlink d1/f16 0 2024-03-22T02:28:32.625 INFO:tasks.workunit.client.0.smithi103.stdout:4/155: stat d3/fd 0 2024-03-22T02:28:32.626 INFO:tasks.workunit.client.0.smithi103.stdout:4/156: stat d3/d1e/f36 0 2024-03-22T02:28:32.628 INFO:tasks.workunit.client.0.smithi103.stdout:1/129: sync 2024-03-22T02:28:32.636 INFO:tasks.workunit.client.0.smithi103.stdout:2/138: dread f7 [0,4194304] 0 2024-03-22T02:28:32.668 INFO:tasks.workunit.client.0.smithi103.stdout:7/98: rmdir d1/d10/d1b 39 2024-03-22T02:28:32.670 INFO:tasks.workunit.client.0.smithi103.stdout:1/130: mkdir db/d10/d2e 0 2024-03-22T02:28:32.672 INFO:tasks.workunit.client.0.smithi103.stdout:2/139: creat d11/d21/f2a x:0 0 0 2024-03-22T02:28:32.672 INFO:tasks.workunit.client.0.smithi103.stdout:5/137: truncate d0/f7 41990 0 2024-03-22T02:28:32.675 INFO:tasks.workunit.client.0.smithi103.stdout:5/138: write d0/d14/d23/d2a/d1f/d27/f30 [58016,19422] 0 2024-03-22T02:28:32.676 INFO:tasks.workunit.client.0.smithi103.stdout:1/131: dread db/d10/d21/f22 [0,4194304] 0 2024-03-22T02:28:32.676 INFO:tasks.workunit.client.0.smithi103.stdout:5/139: chown d0/f5 99785612 1 2024-03-22T02:28:32.679 INFO:tasks.workunit.client.0.smithi103.stdout:2/140: mkdir d11/d25/d2b 0 2024-03-22T02:28:32.680 INFO:tasks.workunit.client.0.smithi103.stdout:2/141: chown d11/f18 366652 1 2024-03-22T02:28:32.682 INFO:tasks.workunit.client.0.smithi103.stdout:4/157: write d3/f7 [3347385,108927] 0 2024-03-22T02:28:32.683 INFO:tasks.workunit.client.0.smithi103.stdout:5/140: rename d0/d14/d23/d2a/c31 to d0/d14/d23/d2a/d1f/d27/c34 0 2024-03-22T02:28:32.687 INFO:tasks.workunit.client.0.smithi103.stdout:1/132: rmdir db/d27 0 2024-03-22T02:28:32.689 INFO:tasks.workunit.client.0.smithi103.stdout:5/141: mknod d0/d14/d23/d2a/d1f/d27/c35 0 2024-03-22T02:28:32.689 INFO:tasks.workunit.client.0.smithi103.stdout:4/158: creat d3/d30/f3a x:0 0 0 2024-03-22T02:28:32.690 INFO:tasks.workunit.client.0.smithi103.stdout:7/99: dread d1/da/f4 [0,4194304] 0 2024-03-22T02:28:32.692 INFO:tasks.workunit.client.0.smithi103.stdout:5/142: mkdir d0/d14/d23/d36 0 2024-03-22T02:28:32.693 INFO:tasks.workunit.client.0.smithi103.stdout:4/159: creat d3/d1e/d25/d2c/f3b x:0 0 0 2024-03-22T02:28:32.694 INFO:tasks.workunit.client.0.smithi103.stdout:7/100: truncate d1/da/f4 52749 0 2024-03-22T02:28:32.695 INFO:tasks.workunit.client.0.smithi103.stdout:7/101: chown d1/d10/d11 25 1 2024-03-22T02:28:32.695 INFO:tasks.workunit.client.0.smithi103.stdout:1/133: link db/c13 db/d10/d18/c2f 0 2024-03-22T02:28:32.696 INFO:tasks.workunit.client.0.smithi103.stdout:5/143: symlink d0/d14/d23/d2a/d33/l37 0 2024-03-22T02:28:32.699 INFO:tasks.workunit.client.0.smithi103.stdout:4/160: creat d3/f3c x:0 0 0 2024-03-22T02:28:32.701 INFO:tasks.workunit.client.0.smithi103.stdout:5/144: creat d0/d14/d23/d2a/d1f/f38 x:0 0 0 2024-03-22T02:28:32.705 INFO:tasks.workunit.client.0.smithi103.stdout:4/161: rmdir d3/d1e/d25/d2c 39 2024-03-22T02:28:32.706 INFO:tasks.workunit.client.0.smithi103.stdout:4/162: chown d3/f34 116972226 1 2024-03-22T02:28:32.708 INFO:tasks.workunit.client.0.smithi103.stdout:5/145: dwrite d0/f5 [4194304,4194304] 0 2024-03-22T02:28:32.713 INFO:tasks.workunit.client.0.smithi103.stdout:1/134: truncate db/d10/d18/f20 1308237 0 2024-03-22T02:28:32.714 INFO:tasks.workunit.client.0.smithi103.stdout:5/146: mknod d0/d14/d23/d2a/d2c/c39 0 2024-03-22T02:28:32.717 INFO:tasks.workunit.client.0.smithi103.stdout:1/135: mknod db/d10/d29/c30 0 2024-03-22T02:28:32.720 INFO:tasks.workunit.client.0.smithi103.stdout:4/163: dwrite d3/d1e/d25/d2c/f3b [0,4194304] 0 2024-03-22T02:28:32.720 INFO:tasks.workunit.client.0.smithi103.stdout:1/136: chown db/d10/f12 0 1 2024-03-22T02:28:32.722 INFO:tasks.workunit.client.0.smithi103.stdout:5/147: rename d0/f28 to d0/d14/f3a 0 2024-03-22T02:28:32.723 INFO:tasks.workunit.client.0.smithi103.stdout:4/164: chown d3/f1a 1353357 1 2024-03-22T02:28:32.723 INFO:tasks.workunit.client.0.smithi103.stdout:7/102: dwrite d1/da/f4 [0,4194304] 0 2024-03-22T02:28:32.726 INFO:tasks.workunit.client.0.smithi103.stdout:7/103: chown d1/d10/d11/f18 58 1 2024-03-22T02:28:32.730 INFO:tasks.workunit.client.0.smithi103.stdout:1/137: creat db/d10/d18/f31 x:0 0 0 2024-03-22T02:28:32.733 INFO:tasks.workunit.client.0.smithi103.stdout:7/104: getdents d1/d10/d11/d19/d29 0 2024-03-22T02:28:32.734 INFO:tasks.workunit.client.0.smithi103.stdout:5/148: dwrite d0/d14/d23/f24 [0,4194304] 0 2024-03-22T02:28:32.737 INFO:tasks.workunit.client.0.smithi103.stdout:1/138: rename db/c1f to db/d10/d28/c32 0 2024-03-22T02:28:32.740 INFO:tasks.workunit.client.0.smithi103.stdout:5/149: mkdir d0/d14/d23/d2a/d1f/d27/d3b 0 2024-03-22T02:28:32.746 INFO:tasks.workunit.client.0.smithi103.stdout:5/150: symlink d0/d14/d23/l3c 0 2024-03-22T02:28:32.746 INFO:tasks.workunit.client.0.smithi103.stdout:7/105: getdents d1/d10/d1b 0 2024-03-22T02:28:32.746 INFO:tasks.workunit.client.0.smithi103.stdout:4/165: dread d3/f11 [0,4194304] 0 2024-03-22T02:28:32.747 INFO:tasks.workunit.client.0.smithi103.stdout:5/151: symlink d0/d14/d23/d2a/d33/l3d 0 2024-03-22T02:28:32.748 INFO:tasks.workunit.client.0.smithi103.stdout:7/106: stat d1/lc 0 2024-03-22T02:28:32.749 INFO:tasks.workunit.client.0.smithi103.stdout:1/139: dwrite db/d10/f12 [4194304,4194304] 0 2024-03-22T02:28:32.751 INFO:tasks.workunit.client.0.smithi103.stdout:1/140: write db/d10/d21/f23 [377679,34095] 0 2024-03-22T02:28:32.754 INFO:tasks.workunit.client.0.smithi103.stdout:7/107: symlink d1/d10/d1b/d21/l2a 0 2024-03-22T02:28:32.754 INFO:tasks.workunit.client.0.smithi103.stdout:4/166: creat d3/d1e/d25/f3d x:0 0 0 2024-03-22T02:28:32.755 INFO:tasks.workunit.client.0.smithi103.stdout:1/141: creat db/d10/d21/f33 x:0 0 0 2024-03-22T02:28:32.757 INFO:tasks.workunit.client.0.smithi103.stdout:7/108: symlink d1/da/l2b 0 2024-03-22T02:28:32.760 INFO:tasks.workunit.client.0.smithi103.stdout:1/142: creat db/d10/d2e/f34 x:0 0 0 2024-03-22T02:28:32.762 INFO:tasks.workunit.client.0.smithi103.stdout:1/143: write db/d10/d18/d2d/f16 [4449745,91935] 0 2024-03-22T02:28:32.765 INFO:tasks.workunit.client.0.smithi103.stdout:4/167: rename d3/l6 to d3/d1e/d25/l3e 0 2024-03-22T02:28:32.767 INFO:tasks.workunit.client.0.smithi103.stdout:4/168: symlink d3/d31/l3f 0 2024-03-22T02:28:32.768 INFO:tasks.workunit.client.0.smithi103.stdout:4/169: stat d3/d1e/f39 0 2024-03-22T02:28:32.771 INFO:tasks.workunit.client.0.smithi103.stdout:4/170: symlink d3/d31/l40 0 2024-03-22T02:28:32.773 INFO:tasks.workunit.client.0.smithi103.stdout:5/152: dwrite d0/f7 [0,4194304] 0 2024-03-22T02:28:32.775 INFO:tasks.workunit.client.0.smithi103.stdout:7/109: write d1/da/f3 [1257459,121421] 0 2024-03-22T02:28:32.776 INFO:tasks.workunit.client.0.smithi103.stdout:1/144: truncate f7 986375 0 2024-03-22T02:28:32.777 INFO:tasks.workunit.client.0.smithi103.stdout:1/145: dread - db/d10/d21/f33 zero size 2024-03-22T02:28:32.778 INFO:tasks.workunit.client.0.smithi103.stdout:7/110: mknod d1/da/c2c 0 2024-03-22T02:28:32.782 INFO:tasks.workunit.client.0.smithi103.stdout:5/153: rename d0/d14/d23/d2a/d1f/d27 to d0/d3e 0 2024-03-22T02:28:32.783 INFO:tasks.workunit.client.0.smithi103.stdout:3/199: sync 2024-03-22T02:28:32.783 INFO:tasks.workunit.client.0.smithi103.stdout:0/145: sync 2024-03-22T02:28:32.784 INFO:tasks.workunit.client.0.smithi103.stdout:9/158: sync 2024-03-22T02:28:32.784 INFO:tasks.workunit.client.0.smithi103.stdout:6/90: sync 2024-03-22T02:28:32.784 INFO:tasks.workunit.client.0.smithi103.stdout:7/111: creat d1/f2d x:0 0 0 2024-03-22T02:28:32.785 INFO:tasks.workunit.client.0.smithi103.stdout:6/91: stat d0/d3/f1f 0 2024-03-22T02:28:32.785 INFO:tasks.workunit.client.0.smithi103.stdout:7/112: dread - d1/f1e zero size 2024-03-22T02:28:32.787 INFO:tasks.workunit.client.0.smithi103.stdout:3/200: chown de/d12/c1d 1314118 1 2024-03-22T02:28:32.788 INFO:tasks.workunit.client.0.smithi103.stdout:7/113: truncate d1/d10/d1b/f24 550291 0 2024-03-22T02:28:32.788 INFO:tasks.workunit.client.0.smithi103.stdout:9/159: mkdir d2/da/d18/d1e/d34 0 2024-03-22T02:28:32.791 INFO:tasks.workunit.client.0.smithi103.stdout:5/154: rmdir d0/d14/d23/d2a/d1f/d25 0 2024-03-22T02:28:32.792 INFO:tasks.workunit.client.0.smithi103.stdout:7/114: truncate d1/da/fe 1640871 0 2024-03-22T02:28:32.795 INFO:tasks.workunit.client.0.smithi103.stdout:0/146: link dd/f13 dd/d16/d20/d29/f2a 0 2024-03-22T02:28:32.796 INFO:tasks.workunit.client.0.smithi103.stdout:3/201: dwrite de/f2d [4194304,4194304] 0 2024-03-22T02:28:32.796 INFO:tasks.workunit.client.0.smithi103.stdout:0/147: read - dd/d16/d20/f21 zero size 2024-03-22T02:28:32.797 INFO:tasks.workunit.client.0.smithi103.stdout:9/160: unlink d2/d5/c29 0 2024-03-22T02:28:32.798 INFO:tasks.workunit.client.0.smithi103.stdout:9/161: readlink d2/l33 0 2024-03-22T02:28:32.798 INFO:tasks.workunit.client.0.smithi103.stdout:6/92: mknod d0/d3/c21 0 2024-03-22T02:28:32.801 INFO:tasks.workunit.client.0.smithi103.stdout:5/155: rename d0/fd to d0/d14/d23/d2a/d2c/f3f 0 2024-03-22T02:28:32.802 INFO:tasks.workunit.client.0.smithi103.stdout:5/156: write d0/fc [4203325,73370] 0 2024-03-22T02:28:32.803 INFO:tasks.workunit.client.0.smithi103.stdout:7/115: symlink d1/d10/d1b/d21/l2e 0 2024-03-22T02:28:32.804 INFO:tasks.workunit.client.0.smithi103.stdout:7/116: stat d1/lf 0 2024-03-22T02:28:32.805 INFO:tasks.workunit.client.0.smithi103.stdout:3/202: mknod de/d1a/d25/d3f/c4b 0 2024-03-22T02:28:32.807 INFO:tasks.workunit.client.0.smithi103.stdout:3/203: chown de/d12/l18 32300146 1 2024-03-22T02:28:32.807 INFO:tasks.workunit.client.0.smithi103.stdout:7/117: write d1/da/f4 [1761489,127602] 0 2024-03-22T02:28:32.810 INFO:tasks.workunit.client.0.smithi103.stdout:6/93: write d0/d14/df/f1c [1696920,58257] 0 2024-03-22T02:28:32.812 INFO:tasks.workunit.client.0.smithi103.stdout:1/146: dwrite db/d10/f12 [0,4194304] 0 2024-03-22T02:28:32.812 INFO:tasks.workunit.client.0.smithi103.stdout:5/157: creat d0/d14/d23/f40 x:0 0 0 2024-03-22T02:28:32.814 INFO:tasks.workunit.client.0.smithi103.stdout:1/147: read db/d10/f12 [3174947,52524] 0 2024-03-22T02:28:32.814 INFO:tasks.workunit.client.0.smithi103.stdout:9/162: symlink d2/da/d18/d1e/d34/l35 0 2024-03-22T02:28:32.817 INFO:tasks.workunit.client.0.smithi103.stdout:7/118: mkdir d1/da/d2f 0 2024-03-22T02:28:32.817 INFO:tasks.workunit.client.0.smithi103.stdout:3/204: symlink de/d12/d19/l4c 0 2024-03-22T02:28:32.819 INFO:tasks.workunit.client.0.smithi103.stdout:7/119: write d1/da/fb [149625,2281] 0 2024-03-22T02:28:32.821 INFO:tasks.workunit.client.0.smithi103.stdout:3/205: truncate de/d1a/d25/d26/f2f 341072 0 2024-03-22T02:28:32.822 INFO:tasks.workunit.client.0.smithi103.stdout:5/158: fdatasync d0/d14/d23/d2a/d2c/f3f 0 2024-03-22T02:28:32.825 INFO:tasks.workunit.client.0.smithi103.stdout:1/148: rename db/d10/d2e/f34 to db/f35 0 2024-03-22T02:28:32.825 INFO:tasks.workunit.client.0.smithi103.stdout:7/120: dread d1/da/fb [0,4194304] 0 2024-03-22T02:28:32.829 INFO:tasks.workunit.client.0.smithi103.stdout:0/148: truncate dd/f1c 1788935 0 2024-03-22T02:28:32.829 INFO:tasks.workunit.client.0.smithi103.stdout:0/149: chown dd/f11 746253 1 2024-03-22T02:28:32.830 INFO:tasks.workunit.client.0.smithi103.stdout:0/150: dread - dd/d16/d20/f21 zero size 2024-03-22T02:28:32.833 INFO:tasks.workunit.client.0.smithi103.stdout:9/163: dread d2/da/f25 [0,4194304] 0 2024-03-22T02:28:32.833 INFO:tasks.workunit.client.0.smithi103.stdout:5/159: write d0/d14/f3a [692491,124315] 0 2024-03-22T02:28:32.835 INFO:tasks.workunit.client.0.smithi103.stdout:1/149: mknod db/d10/d18/d2d/c36 0 2024-03-22T02:28:32.836 INFO:tasks.workunit.client.0.smithi103.stdout:1/150: fdatasync db/d10/f26 0 2024-03-22T02:28:32.841 INFO:tasks.workunit.client.0.smithi103.stdout:6/94: getdents d0/d3 0 2024-03-22T02:28:32.842 INFO:tasks.workunit.client.0.smithi103.stdout:6/95: chown d0/d3/f1f 8 1 2024-03-22T02:28:32.844 INFO:tasks.workunit.client.0.smithi103.stdout:6/96: write d0/d14/df/f11 [3295543,108692] 0 2024-03-22T02:28:32.846 INFO:tasks.workunit.client.0.smithi103.stdout:7/121: rename d1/da/f4 to d1/da/f30 0 2024-03-22T02:28:32.847 INFO:tasks.workunit.client.0.smithi103.stdout:1/151: creat db/d10/d18/f37 x:0 0 0 2024-03-22T02:28:32.849 INFO:tasks.workunit.client.0.smithi103.stdout:7/122: dread d1/d10/d1b/f24 [0,4194304] 0 2024-03-22T02:28:32.853 INFO:tasks.workunit.client.0.smithi103.stdout:6/97: rename d0/d14/df to d0/d3/d22 0 2024-03-22T02:28:32.854 INFO:tasks.workunit.client.0.smithi103.stdout:9/164: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:32.855 INFO:tasks.workunit.client.0.smithi103.stdout:1/152: symlink db/d10/d29/l38 0 2024-03-22T02:28:32.856 INFO:tasks.workunit.client.0.smithi103.stdout:7/123: mknod d1/d10/d1b/c31 0 2024-03-22T02:28:32.858 INFO:tasks.workunit.client.0.smithi103.stdout:9/165: chown d2/d10/d27/c31 10799 1 2024-03-22T02:28:32.859 INFO:tasks.workunit.client.0.smithi103.stdout:3/206: dwrite de/d1a/f39 [0,4194304] 0 2024-03-22T02:28:32.859 INFO:tasks.workunit.client.0.smithi103.stdout:5/160: link d0/f5 d0/d14/f41 0 2024-03-22T02:28:32.860 INFO:tasks.workunit.client.0.smithi103.stdout:0/151: dwrite dd/d16/d20/d29/f2a [0,4194304] 0 2024-03-22T02:28:32.862 INFO:tasks.workunit.client.0.smithi103.stdout:9/166: write d2/f3 [2489750,61449] 0 2024-03-22T02:28:32.866 INFO:tasks.workunit.client.0.smithi103.stdout:1/153: write db/d10/d21/f22 [1316202,111779] 0 2024-03-22T02:28:32.866 INFO:tasks.workunit.client.0.smithi103.stdout:1/154: dread - db/f25 zero size 2024-03-22T02:28:32.868 INFO:tasks.workunit.client.0.smithi103.stdout:1/155: readlink db/d10/d29/l38 0 2024-03-22T02:28:32.869 INFO:tasks.workunit.client.0.smithi103.stdout:7/124: creat d1/d10/d1b/d21/f32 x:0 0 0 2024-03-22T02:28:32.870 INFO:tasks.workunit.client.0.smithi103.stdout:7/125: chown d1/d10/d1b/c31 44528 1 2024-03-22T02:28:32.871 INFO:tasks.workunit.client.0.smithi103.stdout:0/152: rename dd/d16/l1b to dd/d16/d20/l2b 0 2024-03-22T02:28:32.874 INFO:tasks.workunit.client.0.smithi103.stdout:3/207: mkdir de/d12/d4d 0 2024-03-22T02:28:32.876 INFO:tasks.workunit.client.0.smithi103.stdout:3/208: truncate de/d1a/f2a 247338 0 2024-03-22T02:28:32.876 INFO:tasks.workunit.client.0.smithi103.stdout:6/98: creat d0/f23 x:0 0 0 2024-03-22T02:28:32.877 INFO:tasks.workunit.client.0.smithi103.stdout:3/209: write de/d1a/f2a [743219,17462] 0 2024-03-22T02:28:32.880 INFO:tasks.workunit.client.0.smithi103.stdout:1/156: read db/d10/d21/f22 [1199231,78334] 0 2024-03-22T02:28:32.881 INFO:tasks.workunit.client.0.smithi103.stdout:7/126: dread d1/d10/d1b/f24 [0,4194304] 0 2024-03-22T02:28:32.881 INFO:tasks.workunit.client.0.smithi103.stdout:5/161: truncate d0/d14/f41 4791925 0 2024-03-22T02:28:32.882 INFO:tasks.workunit.client.0.smithi103.stdout:0/153: symlink dd/d16/l2c 0 2024-03-22T02:28:32.886 INFO:tasks.workunit.client.0.smithi103.stdout:1/157: dread db/d10/f1c [0,4194304] 0 2024-03-22T02:28:32.891 INFO:tasks.workunit.client.0.smithi103.stdout:5/162: dwrite d0/d14/d23/d2a/f1a [0,4194304] 0 2024-03-22T02:28:32.895 INFO:tasks.workunit.client.0.smithi103.stdout:6/99: fsync d0/d3/d22/f11 0 2024-03-22T02:28:32.897 INFO:tasks.workunit.client.0.smithi103.stdout:6/100: write d0/d3/d22/f15 [92685,127268] 0 2024-03-22T02:28:32.897 INFO:tasks.workunit.client.0.smithi103.stdout:1/158: creat db/d10/d29/f39 x:0 0 0 2024-03-22T02:28:32.899 INFO:tasks.workunit.client.0.smithi103.stdout:3/210: fsync de/f1f 0 2024-03-22T02:28:32.903 INFO:tasks.workunit.client.0.smithi103.stdout:7/127: mkdir d1/d10/d11/d19/d29/d33 0 2024-03-22T02:28:32.906 INFO:tasks.workunit.client.0.smithi103.stdout:0/154: write f4 [1563899,56165] 0 2024-03-22T02:28:32.908 INFO:tasks.workunit.client.0.smithi103.stdout:0/155: read - dd/d16/d20/f22 zero size 2024-03-22T02:28:32.911 INFO:tasks.workunit.client.0.smithi103.stdout:1/159: symlink db/d10/d2e/l3a 0 2024-03-22T02:28:32.912 INFO:tasks.workunit.client.0.smithi103.stdout:7/128: dwrite d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:32.914 INFO:tasks.workunit.client.0.smithi103.stdout:5/163: mknod d0/d14/c42 0 2024-03-22T02:28:32.914 INFO:tasks.workunit.client.0.smithi103.stdout:6/101: symlink d0/d3/l24 0 2024-03-22T02:28:32.914 INFO:tasks.workunit.client.0.smithi103.stdout:0/156: rmdir dd/d16/d20/d29 39 2024-03-22T02:28:32.914 INFO:tasks.workunit.client.0.smithi103.stdout:6/102: stat d0/d14/c17 0 2024-03-22T02:28:32.915 INFO:tasks.workunit.client.0.smithi103.stdout:0/157: dread - dd/f1d zero size 2024-03-22T02:28:32.916 INFO:tasks.workunit.client.0.smithi103.stdout:1/160: creat db/d10/d18/f3b x:0 0 0 2024-03-22T02:28:32.918 INFO:tasks.workunit.client.0.smithi103.stdout:5/164: rmdir d0/d3e 39 2024-03-22T02:28:32.919 INFO:tasks.workunit.client.0.smithi103.stdout:5/165: readlink d0/d14/l16 0 2024-03-22T02:28:32.921 INFO:tasks.workunit.client.0.smithi103.stdout:3/211: rename l3 to de/d12/l4e 0 2024-03-22T02:28:32.924 INFO:tasks.workunit.client.0.smithi103.stdout:3/212: read de/f1e [168004,9388] 0 2024-03-22T02:28:32.925 INFO:tasks.workunit.client.0.smithi103.stdout:1/161: mknod db/d10/d29/c3c 0 2024-03-22T02:28:32.926 INFO:tasks.workunit.client.0.smithi103.stdout:0/158: dwrite dd/f12 [0,4194304] 0 2024-03-22T02:28:32.926 INFO:tasks.workunit.client.0.smithi103.stdout:1/162: readlink db/le 0 2024-03-22T02:28:32.927 INFO:tasks.workunit.client.0.smithi103.stdout:3/213: write de/d12/d19/f43 [158334,109548] 0 2024-03-22T02:28:32.929 INFO:tasks.workunit.client.0.smithi103.stdout:5/166: dread d0/fe [4194304,4194304] 0 2024-03-22T02:28:32.929 INFO:tasks.workunit.client.0.smithi103.stdout:0/159: fsync dd/d16/d20/d29/f2a 0 2024-03-22T02:28:32.932 INFO:tasks.workunit.client.0.smithi103.stdout:5/167: readlink d0/d14/l16 0 2024-03-22T02:28:32.933 INFO:tasks.workunit.client.0.smithi103.stdout:6/103: link d0/d3/c13 d0/d14/d20/c25 0 2024-03-22T02:28:32.935 INFO:tasks.workunit.client.0.smithi103.stdout:0/160: mknod dd/d16/d20/c2d 0 2024-03-22T02:28:32.937 INFO:tasks.workunit.client.0.smithi103.stdout:3/214: mkdir de/d12/d47/d4f 0 2024-03-22T02:28:32.939 INFO:tasks.workunit.client.0.smithi103.stdout:6/104: readlink d0/d3/l12 0 2024-03-22T02:28:32.940 INFO:tasks.workunit.client.0.smithi103.stdout:3/215: read de/d12/d30/f35 [3467124,123133] 0 2024-03-22T02:28:32.941 INFO:tasks.workunit.client.0.smithi103.stdout:3/216: chown de/d1a/d25/c46 0 1 2024-03-22T02:28:32.941 INFO:tasks.workunit.client.0.smithi103.stdout:1/163: link db/d10/d21/f22 db/d10/d18/d2d/f3d 0 2024-03-22T02:28:32.943 INFO:tasks.workunit.client.0.smithi103.stdout:5/168: symlink d0/d14/l43 0 2024-03-22T02:28:32.950 INFO:tasks.workunit.client.0.smithi103.stdout:5/169: creat d0/d14/d23/f44 x:0 0 0 2024-03-22T02:28:32.951 INFO:tasks.workunit.client.0.smithi103.stdout:3/217: dwrite de/d12/d19/f28 [0,4194304] 0 2024-03-22T02:28:32.953 INFO:tasks.workunit.client.0.smithi103.stdout:6/105: unlink d0/d3/f1f 0 2024-03-22T02:28:32.953 INFO:tasks.workunit.client.0.smithi103.stdout:3/218: write de/d1a/f31 [6525279,48647] 0 2024-03-22T02:28:32.955 INFO:tasks.workunit.client.0.smithi103.stdout:3/219: dread - de/d12/d19/f48 zero size 2024-03-22T02:28:32.955 INFO:tasks.workunit.client.0.smithi103.stdout:1/164: dwrite db/f35 [0,4194304] 0 2024-03-22T02:28:32.956 INFO:tasks.workunit.client.0.smithi103.stdout:0/161: rmdir dd/d25 0 2024-03-22T02:28:32.957 INFO:tasks.workunit.client.0.smithi103.stdout:3/220: fsync de/d12/d30/f3d 0 2024-03-22T02:28:32.962 INFO:tasks.workunit.client.0.smithi103.stdout:6/106: write d0/f19 [4755441,56732] 0 2024-03-22T02:28:32.964 INFO:tasks.workunit.client.0.smithi103.stdout:3/221: rename de/d1a/d25/c46 to de/d12/d47/d4f/c50 0 2024-03-22T02:28:32.965 INFO:tasks.workunit.client.0.smithi103.stdout:6/107: chown d0/d14/c1d 2293 1 2024-03-22T02:28:32.966 INFO:tasks.workunit.client.0.smithi103.stdout:0/162: dwrite dd/d16/d20/d29/f2a [0,4194304] 0 2024-03-22T02:28:32.967 INFO:tasks.workunit.client.0.smithi103.stdout:6/108: chown d0/d3/d22 452422127 1 2024-03-22T02:28:32.968 INFO:tasks.workunit.client.0.smithi103.stdout:5/170: getdents d0 0 2024-03-22T02:28:32.971 INFO:tasks.workunit.client.0.smithi103.stdout:5/171: write d0/fe [432278,68331] 0 2024-03-22T02:28:32.971 INFO:tasks.workunit.client.0.smithi103.stdout:6/109: mknod d0/c26 0 2024-03-22T02:28:32.972 INFO:tasks.workunit.client.0.smithi103.stdout:6/110: fsync d0/d3/d22/f15 0 2024-03-22T02:28:32.974 INFO:tasks.workunit.client.0.smithi103.stdout:0/163: getdents dd/d16/d20/d29 0 2024-03-22T02:28:32.975 INFO:tasks.workunit.client.0.smithi103.stdout:0/164: stat l7 0 2024-03-22T02:28:32.979 INFO:tasks.workunit.client.0.smithi103.stdout:0/165: creat dd/d16/f2e x:0 0 0 2024-03-22T02:28:32.983 INFO:tasks.workunit.client.0.smithi103.stdout:6/111: creat d0/f27 x:0 0 0 2024-03-22T02:28:32.984 INFO:tasks.workunit.client.0.smithi103.stdout:0/166: truncate dd/f1c 915217 0 2024-03-22T02:28:32.984 INFO:tasks.workunit.client.0.smithi103.stdout:1/165: dwrite db/d10/d18/f20 [0,4194304] 0 2024-03-22T02:28:32.984 INFO:tasks.workunit.client.0.smithi103.stdout:6/112: chown d0/d3/d22/f11 495691262 1 2024-03-22T02:28:32.985 INFO:tasks.workunit.client.0.smithi103.stdout:1/166: readlink db/d10/d21/l2a 0 2024-03-22T02:28:33.014 INFO:tasks.workunit.client.0.smithi103.stdout:8/220: sync 2024-03-22T02:28:33.014 INFO:tasks.workunit.client.0.smithi103.stdout:0/167: write dd/d16/d20/f21 [547552,110044] 0 2024-03-22T02:28:33.016 INFO:tasks.workunit.client.0.smithi103.stdout:6/113: fsync d0/d3/f1b 0 2024-03-22T02:28:33.027 INFO:tasks.workunit.client.0.smithi103.stdout:5/172: dwrite d0/d14/d23/d2a/d1f/f20 [0,4194304] 0 2024-03-22T02:28:33.029 INFO:tasks.workunit.client.0.smithi103.stdout:0/168: rename dd/d16/f2e to dd/d16/f2f 0 2024-03-22T02:28:33.030 INFO:tasks.workunit.client.0.smithi103.stdout:6/114: creat d0/d14/d20/f28 x:0 0 0 2024-03-22T02:28:33.032 INFO:tasks.workunit.client.0.smithi103.stdout:0/169: fsync dd/f1d 0 2024-03-22T02:28:33.037 INFO:tasks.workunit.client.0.smithi103.stdout:1/167: dwrite db/d10/f1c [0,4194304] 0 2024-03-22T02:28:33.042 INFO:tasks.workunit.client.0.smithi103.stdout:8/221: dwrite d4/d13/f1c [0,4194304] 0 2024-03-22T02:28:33.045 INFO:tasks.workunit.client.0.smithi103.stdout:5/173: dwrite d0/d14/f3a [0,4194304] 0 2024-03-22T02:28:33.046 INFO:tasks.workunit.client.0.smithi103.stdout:8/222: dread - d4/d19/f2e zero size 2024-03-22T02:28:33.048 INFO:tasks.workunit.client.0.smithi103.stdout:6/115: dwrite d0/f1a [0,4194304] 0 2024-03-22T02:28:33.050 INFO:tasks.workunit.client.0.smithi103.stdout:1/168: getdents db/d10/d18/d2d 0 2024-03-22T02:28:33.054 INFO:tasks.workunit.client.0.smithi103.stdout:1/169: write f5 [3633043,46364] 0 2024-03-22T02:28:33.060 INFO:tasks.workunit.client.0.smithi103.stdout:5/174: dwrite d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:33.061 INFO:tasks.workunit.client.0.smithi103.stdout:0/170: write dd/d16/f19 [621705,97377] 0 2024-03-22T02:28:33.063 INFO:tasks.workunit.client.0.smithi103.stdout:1/170: rename db/d10/d28/c32 to db/c3e 0 2024-03-22T02:28:33.064 INFO:tasks.workunit.client.0.smithi103.stdout:1/171: chown f5 1 1 2024-03-22T02:28:33.066 INFO:tasks.workunit.client.0.smithi103.stdout:0/171: mknod dd/d16/d20/c30 0 2024-03-22T02:28:33.067 INFO:tasks.workunit.client.0.smithi103.stdout:0/172: chown dd/f15 117791314 1 2024-03-22T02:28:33.068 INFO:tasks.workunit.client.0.smithi103.stdout:0/173: fsync fa 0 2024-03-22T02:28:33.070 INFO:tasks.workunit.client.0.smithi103.stdout:5/175: dwrite d0/d14/d23/f24 [0,4194304] 0 2024-03-22T02:28:33.072 INFO:tasks.workunit.client.0.smithi103.stdout:5/176: rename d0/d14/d23/d36 to d0/d14/d23/d36/d45 22 2024-03-22T02:28:33.074 INFO:tasks.workunit.client.0.smithi103.stdout:0/174: mkdir dd/d16/d20/d31 0 2024-03-22T02:28:33.078 INFO:tasks.workunit.client.0.smithi103.stdout:0/175: dread f4 [0,4194304] 0 2024-03-22T02:28:33.081 INFO:tasks.workunit.client.0.smithi103.stdout:5/177: dwrite d0/d14/d23/d2a/d1f/f20 [0,4194304] 0 2024-03-22T02:28:33.084 INFO:tasks.workunit.client.0.smithi103.stdout:5/178: chown d0/d14/d23/d2a/d2c 750 1 2024-03-22T02:28:33.086 INFO:tasks.workunit.client.0.smithi103.stdout:5/179: write d0/f7 [4669174,92531] 0 2024-03-22T02:28:33.088 INFO:tasks.workunit.client.0.smithi103.stdout:0/176: dwrite dd/f15 [0,4194304] 0 2024-03-22T02:28:33.090 INFO:tasks.workunit.client.0.smithi103.stdout:6/116: dread d0/d3/d22/f11 [0,4194304] 0 2024-03-22T02:28:33.091 INFO:tasks.workunit.client.0.smithi103.stdout:6/117: chown d0/d14 24774 1 2024-03-22T02:28:33.092 INFO:tasks.workunit.client.0.smithi103.stdout:5/180: unlink d0/fc 0 2024-03-22T02:28:33.096 INFO:tasks.workunit.client.0.smithi103.stdout:6/118: creat d0/d14/f29 x:0 0 0 2024-03-22T02:28:33.099 INFO:tasks.workunit.client.0.smithi103.stdout:0/177: dwrite fc [4194304,4194304] 0 2024-03-22T02:28:33.101 INFO:tasks.workunit.client.0.smithi103.stdout:6/119: unlink d0/d3/c16 0 2024-03-22T02:28:33.105 INFO:tasks.workunit.client.0.smithi103.stdout:6/120: unlink d0/d14/cc 0 2024-03-22T02:28:33.105 INFO:tasks.workunit.client.0.smithi103.stdout:1/172: dread db/d10/d18/d2d/f16 [0,4194304] 0 2024-03-22T02:28:33.106 INFO:tasks.workunit.client.0.smithi103.stdout:6/121: symlink d0/l2a 0 2024-03-22T02:28:33.108 INFO:tasks.workunit.client.0.smithi103.stdout:1/173: dread f7 [0,4194304] 0 2024-03-22T02:28:33.111 INFO:tasks.workunit.client.0.smithi103.stdout:1/174: creat db/d10/d18/f3f x:0 0 0 2024-03-22T02:28:33.113 INFO:tasks.workunit.client.0.smithi103.stdout:1/175: unlink db/d10/d28/l2c 0 2024-03-22T02:28:33.113 INFO:tasks.workunit.client.0.smithi103.stdout:1/176: chown db/d10/d29/c3c 103436 1 2024-03-22T02:28:33.119 INFO:tasks.workunit.client.0.smithi103.stdout:5/181: mkdir d0/d14/d23/d46 0 2024-03-22T02:28:33.120 INFO:tasks.workunit.client.0.smithi103.stdout:5/182: chown d0/d3e/l2b 0 1 2024-03-22T02:28:33.121 INFO:tasks.workunit.client.0.smithi103.stdout:5/183: fdatasync d0/fe 0 2024-03-22T02:28:33.122 INFO:tasks.workunit.client.0.smithi103.stdout:0/178: dread dd/d16/f19 [0,4194304] 0 2024-03-22T02:28:33.124 INFO:tasks.workunit.client.0.smithi103.stdout:5/184: getdents d0/d3e 0 2024-03-22T02:28:33.125 INFO:tasks.workunit.client.0.smithi103.stdout:0/179: truncate dd/d16/d20/f23 557672 0 2024-03-22T02:28:33.126 INFO:tasks.workunit.client.0.smithi103.stdout:0/180: chown dd 124 1 2024-03-22T02:28:33.127 INFO:tasks.workunit.client.0.smithi103.stdout:0/181: write dd/d16/d20/f26 [155029,27262] 0 2024-03-22T02:28:33.128 INFO:tasks.workunit.client.0.smithi103.stdout:5/185: rmdir d0/d3e/d3b 0 2024-03-22T02:28:33.129 INFO:tasks.workunit.client.0.smithi103.stdout:0/182: dread dd/d16/d20/f26 [0,4194304] 0 2024-03-22T02:28:33.130 INFO:tasks.workunit.client.0.smithi103.stdout:5/186: creat d0/d14/d23/d46/f47 x:0 0 0 2024-03-22T02:28:33.139 INFO:tasks.workunit.client.0.smithi103.stdout:5/187: rmdir d0/d14/d23 39 2024-03-22T02:28:33.146 INFO:tasks.workunit.client.0.smithi103.stdout:0/183: dwrite dd/d16/d20/f23 [0,4194304] 0 2024-03-22T02:28:33.149 INFO:tasks.workunit.client.0.smithi103.stdout:5/188: dwrite d0/d14/d23/d46/f47 [0,4194304] 0 2024-03-22T02:28:33.150 INFO:tasks.workunit.client.0.smithi103.stdout:5/189: chown d0/d3e/c35 1373768917 1 2024-03-22T02:28:33.153 INFO:tasks.workunit.client.0.smithi103.stdout:0/184: rename dd/f1c to dd/d16/d20/d31/f32 0 2024-03-22T02:28:33.155 INFO:tasks.workunit.client.0.smithi103.stdout:0/185: unlink dd/d16/l2c 0 2024-03-22T02:28:33.155 INFO:tasks.workunit.client.0.smithi103.stdout:5/190: mknod d0/c48 0 2024-03-22T02:28:33.158 INFO:tasks.workunit.client.0.smithi103.stdout:5/191: stat d0/f1e 0 2024-03-22T02:28:33.159 INFO:tasks.workunit.client.0.smithi103.stdout:0/186: mkdir dd/d16/d20/d33 0 2024-03-22T02:28:33.162 INFO:tasks.workunit.client.0.smithi103.stdout:0/187: mkdir dd/d34 0 2024-03-22T02:28:33.164 INFO:tasks.workunit.client.0.smithi103.stdout:0/188: write f4 [692709,130016] 0 2024-03-22T02:28:33.166 INFO:tasks.workunit.client.0.smithi103.stdout:5/192: dread d0/d14/d23/f24 [4194304,4194304] 0 2024-03-22T02:28:33.168 INFO:tasks.workunit.client.0.smithi103.stdout:5/193: write d0/d14/d23/d2a/f1c [28227,66965] 0 2024-03-22T02:28:33.168 INFO:tasks.workunit.client.0.smithi103.stdout:0/189: mkdir dd/d16/d20/d35 0 2024-03-22T02:28:33.170 INFO:tasks.workunit.client.0.smithi103.stdout:5/194: mknod d0/d14/d23/d36/c49 0 2024-03-22T02:28:33.172 INFO:tasks.workunit.client.0.smithi103.stdout:5/195: fdatasync d0/f12 0 2024-03-22T02:28:33.174 INFO:tasks.workunit.client.0.smithi103.stdout:5/196: write d0/d14/d23/f24 [7212603,47123] 0 2024-03-22T02:28:33.175 INFO:tasks.workunit.client.0.smithi103.stdout:0/190: dwrite dd/d16/f19 [0,4194304] 0 2024-03-22T02:28:33.177 INFO:tasks.workunit.client.0.smithi103.stdout:5/197: stat d0/d3e/f30 0 2024-03-22T02:28:33.178 INFO:tasks.workunit.client.0.smithi103.stdout:0/191: creat dd/d16/f36 x:0 0 0 2024-03-22T02:28:33.184 INFO:tasks.workunit.client.0.smithi103.stdout:0/192: rename dd/d16/c18 to dd/d16/d20/d35/c37 0 2024-03-22T02:28:33.184 INFO:tasks.workunit.client.0.smithi103.stdout:5/198: dwrite d0/d14/d23/f44 [0,4194304] 0 2024-03-22T02:28:33.186 INFO:tasks.workunit.client.0.smithi103.stdout:5/199: fsync d0/d14/d23/d2a/d1f/f20 0 2024-03-22T02:28:33.192 INFO:tasks.workunit.client.0.smithi103.stdout:0/193: rmdir dd/d16/d20/d33 0 2024-03-22T02:28:33.193 INFO:tasks.workunit.client.0.smithi103.stdout:0/194: readlink dd/l10 0 2024-03-22T02:28:33.193 INFO:tasks.workunit.client.0.smithi103.stdout:5/200: link d0/c48 d0/d3e/c4a 0 2024-03-22T02:28:33.194 INFO:tasks.workunit.client.0.smithi103.stdout:5/201: fsync d0/d14/f3a 0 2024-03-22T02:28:33.195 INFO:tasks.workunit.client.0.smithi103.stdout:0/195: creat dd/d16/f38 x:0 0 0 2024-03-22T02:28:33.199 INFO:tasks.workunit.client.0.smithi103.stdout:0/196: rename dd/d16/d20/f22 to dd/d16/d20/f39 0 2024-03-22T02:28:33.199 INFO:tasks.workunit.client.0.smithi103.stdout:5/202: getdents d0/d14/d23/d2a/d2c 0 2024-03-22T02:28:33.201 INFO:tasks.workunit.client.0.smithi103.stdout:0/197: mknod dd/d16/d20/c3a 0 2024-03-22T02:28:33.202 INFO:tasks.workunit.client.0.smithi103.stdout:0/198: chown dd/d16/d20/d31/f32 59676 1 2024-03-22T02:28:33.203 INFO:tasks.workunit.client.0.smithi103.stdout:2/142: sync 2024-03-22T02:28:33.204 INFO:tasks.workunit.client.0.smithi103.stdout:5/203: rename d0/f5 to d0/d14/d23/d2a/d2c/f4b 0 2024-03-22T02:28:33.205 INFO:tasks.workunit.client.0.smithi103.stdout:2/143: symlink d11/d25/l2c 0 2024-03-22T02:28:33.208 INFO:tasks.workunit.client.0.smithi103.stdout:0/199: dwrite dd/d16/d20/f21 [0,4194304] 0 2024-03-22T02:28:33.209 INFO:tasks.workunit.client.0.smithi103.stdout:0/200: fdatasync fa 0 2024-03-22T02:28:33.210 INFO:tasks.workunit.client.0.smithi103.stdout:2/144: dread f7 [0,4194304] 0 2024-03-22T02:28:33.212 INFO:tasks.workunit.client.0.smithi103.stdout:0/201: getdents dd/d16/d20/d29 0 2024-03-22T02:28:33.230 INFO:tasks.workunit.client.0.smithi103.stdout:2/145: dread fd [0,4194304] 0 2024-03-22T02:28:33.329 INFO:tasks.workunit.client.0.smithi103.stdout:0/202: symlink dd/d16/d20/d35/l3b 0 2024-03-22T02:28:33.330 INFO:tasks.workunit.client.0.smithi103.stdout:2/146: symlink d11/d15/l2d 0 2024-03-22T02:28:33.334 INFO:tasks.workunit.client.0.smithi103.stdout:2/147: rmdir d11/d25/d2b 0 2024-03-22T02:28:33.335 INFO:tasks.workunit.client.0.smithi103.stdout:2/148: dread - d11/f16 zero size 2024-03-22T02:28:33.335 INFO:tasks.workunit.client.0.smithi103.stdout:0/203: dwrite fc [0,4194304] 0 2024-03-22T02:28:33.337 INFO:tasks.workunit.client.0.smithi103.stdout:2/149: truncate d11/f14 229978 0 2024-03-22T02:28:33.341 INFO:tasks.workunit.client.0.smithi103.stdout:2/150: rename d11/c1e to d11/d19/c2e 0 2024-03-22T02:28:33.341 INFO:tasks.workunit.client.0.smithi103.stdout:2/151: stat d11/f20 0 2024-03-22T02:28:33.342 INFO:tasks.workunit.client.0.smithi103.stdout:0/204: creat dd/d16/d20/f3c x:0 0 0 2024-03-22T02:28:33.342 INFO:tasks.workunit.client.0.smithi103.stdout:0/205: chown dd/d16/l1a 28099404 1 2024-03-22T02:28:33.345 INFO:tasks.workunit.client.0.smithi103.stdout:0/206: chown dd/d16/d20/d35/c37 290 1 2024-03-22T02:28:33.345 INFO:tasks.workunit.client.0.smithi103.stdout:2/152: link d11/d15/l1b d11/l2f 0 2024-03-22T02:28:33.346 INFO:tasks.workunit.client.0.smithi103.stdout:2/153: fsync d11/f14 0 2024-03-22T02:28:33.346 INFO:tasks.workunit.client.0.smithi103.stdout:2/154: chown d11/d15/f29 13657 1 2024-03-22T02:28:33.350 INFO:tasks.workunit.client.0.smithi103.stdout:0/207: dwrite fa [0,4194304] 0 2024-03-22T02:28:33.352 INFO:tasks.workunit.client.0.smithi103.stdout:0/208: symlink dd/d16/d20/d31/l3d 0 2024-03-22T02:28:33.355 INFO:tasks.workunit.client.0.smithi103.stdout:0/209: creat dd/d16/d20/d29/f3e x:0 0 0 2024-03-22T02:28:33.357 INFO:tasks.workunit.client.0.smithi103.stdout:0/210: write dd/d16/d20/f39 [833316,12385] 0 2024-03-22T02:28:33.359 INFO:tasks.workunit.client.0.smithi103.stdout:0/211: creat dd/d16/f3f x:0 0 0 2024-03-22T02:28:33.361 INFO:tasks.workunit.client.0.smithi103.stdout:0/212: fdatasync dd/f11 0 2024-03-22T02:28:33.361 INFO:tasks.workunit.client.0.smithi103.stdout:2/155: write fd [3416480,93538] 0 2024-03-22T02:28:33.363 INFO:tasks.workunit.client.0.smithi103.stdout:2/156: rename f1 to d11/f30 0 2024-03-22T02:28:33.366 INFO:tasks.workunit.client.0.smithi103.stdout:2/157: rmdir d11/d15 39 2024-03-22T02:28:33.369 INFO:tasks.workunit.client.0.smithi103.stdout:0/213: getdents dd/d16 0 2024-03-22T02:28:33.372 INFO:tasks.workunit.client.0.smithi103.stdout:0/214: rename l9 to dd/d34/l40 0 2024-03-22T02:28:33.373 INFO:tasks.workunit.client.0.smithi103.stdout:2/158: dwrite d11/f16 [0,4194304] 0 2024-03-22T02:28:33.374 INFO:tasks.workunit.client.0.smithi103.stdout:0/215: read - dd/d16/f2f zero size 2024-03-22T02:28:33.376 INFO:tasks.workunit.client.0.smithi103.stdout:9/167: sync 2024-03-22T02:28:33.377 INFO:tasks.workunit.client.0.smithi103.stdout:4/171: sync 2024-03-22T02:28:33.377 INFO:tasks.workunit.client.0.smithi103.stdout:2/159: rename d11/d15 to d11/d15/d31 22 2024-03-22T02:28:33.377 INFO:tasks.workunit.client.0.smithi103.stdout:0/216: symlink dd/d16/d20/d31/l41 0 2024-03-22T02:28:33.378 INFO:tasks.workunit.client.0.smithi103.stdout:0/217: chown dd/f12 1434470 1 2024-03-22T02:28:33.379 INFO:tasks.workunit.client.0.smithi103.stdout:2/160: write d11/f20 [1217239,68656] 0 2024-03-22T02:28:33.379 INFO:tasks.workunit.client.0.smithi103.stdout:4/172: write d3/f11 [730315,12986] 0 2024-03-22T02:28:33.384 INFO:tasks.workunit.client.0.smithi103.stdout:0/218: unlink dd/d16/d20/c28 0 2024-03-22T02:28:33.386 INFO:tasks.workunit.client.0.smithi103.stdout:2/161: dwrite f10 [0,4194304] 0 2024-03-22T02:28:33.386 INFO:tasks.workunit.client.0.smithi103.stdout:4/173: mknod d3/d38/c41 0 2024-03-22T02:28:33.387 INFO:tasks.workunit.client.0.smithi103.stdout:4/174: stat d3/f3c 0 2024-03-22T02:28:33.388 INFO:tasks.workunit.client.0.smithi103.stdout:0/219: creat dd/d16/d20/d29/f42 x:0 0 0 2024-03-22T02:28:33.389 INFO:tasks.workunit.client.0.smithi103.stdout:2/162: rmdir d11 39 2024-03-22T02:28:33.392 INFO:tasks.workunit.client.0.smithi103.stdout:0/220: creat dd/d34/f43 x:0 0 0 2024-03-22T02:28:33.393 INFO:tasks.workunit.client.0.smithi103.stdout:0/221: stat dd/d16/d20/c2d 0 2024-03-22T02:28:33.393 INFO:tasks.workunit.client.0.smithi103.stdout:0/222: fsync dd/f12 0 2024-03-22T02:28:33.395 INFO:tasks.workunit.client.0.smithi103.stdout:0/223: chown dd/d16/d20/d35/c37 23 1 2024-03-22T02:28:33.397 INFO:tasks.workunit.client.0.smithi103.stdout:4/175: link d3/lf d3/d1e/d25/d2c/l42 0 2024-03-22T02:28:33.398 INFO:tasks.workunit.client.0.smithi103.stdout:2/163: creat d11/d21/d24/f32 x:0 0 0 2024-03-22T02:28:33.400 INFO:tasks.workunit.client.0.smithi103.stdout:2/164: mknod d11/d21/c33 0 2024-03-22T02:28:33.401 INFO:tasks.workunit.client.0.smithi103.stdout:0/224: dread dd/f12 [0,4194304] 0 2024-03-22T02:28:33.401 INFO:tasks.workunit.client.0.smithi103.stdout:2/165: stat d11/d21/c27 0 2024-03-22T02:28:33.404 INFO:tasks.workunit.client.0.smithi103.stdout:4/176: dwrite d3/d1e/f36 [0,4194304] 0 2024-03-22T02:28:33.408 INFO:tasks.workunit.client.0.smithi103.stdout:4/177: creat d3/d1e/f43 x:0 0 0 2024-03-22T02:28:33.410 INFO:tasks.workunit.client.0.smithi103.stdout:9/168: dread d2/da/f24 [0,4194304] 0 2024-03-22T02:28:33.413 INFO:tasks.workunit.client.0.smithi103.stdout:2/166: dwrite d11/d21/d24/f32 [0,4194304] 0 2024-03-22T02:28:33.413 INFO:tasks.workunit.client.0.smithi103.stdout:2/167: chown d11/f20 81 1 2024-03-22T02:28:33.415 INFO:tasks.workunit.client.0.smithi103.stdout:4/178: creat d3/d1e/d25/d2c/f44 x:0 0 0 2024-03-22T02:28:33.415 INFO:tasks.workunit.client.0.smithi103.stdout:9/169: symlink d2/d5/l36 0 2024-03-22T02:28:33.418 INFO:tasks.workunit.client.0.smithi103.stdout:9/170: creat d2/d10/d27/f37 x:0 0 0 2024-03-22T02:28:33.420 INFO:tasks.workunit.client.0.smithi103.stdout:9/171: creat d2/d5/f38 x:0 0 0 2024-03-22T02:28:33.422 INFO:tasks.workunit.client.0.smithi103.stdout:0/225: getdents dd/d16/d20/d29 0 2024-03-22T02:28:33.423 INFO:tasks.workunit.client.0.smithi103.stdout:9/172: mkdir d2/d5/d39 0 2024-03-22T02:28:33.426 INFO:tasks.workunit.client.0.smithi103.stdout:9/173: write d2/d10/f13 [1989821,78171] 0 2024-03-22T02:28:33.429 INFO:tasks.workunit.client.0.smithi103.stdout:9/174: dread - d2/d10/d27/f37 zero size 2024-03-22T02:28:33.430 INFO:tasks.workunit.client.0.smithi103.stdout:0/226: dwrite dd/d16/f36 [0,4194304] 0 2024-03-22T02:28:33.430 INFO:tasks.workunit.client.0.smithi103.stdout:9/175: fdatasync d2/d5/f38 0 2024-03-22T02:28:33.432 INFO:tasks.workunit.client.0.smithi103.stdout:2/168: rename d11/d21 to d11/d15/d34 0 2024-03-22T02:28:33.433 INFO:tasks.workunit.client.0.smithi103.stdout:2/169: write f4 [749883,102507] 0 2024-03-22T02:28:33.437 INFO:tasks.workunit.client.0.smithi103.stdout:9/176: creat d2/d5/f3a x:0 0 0 2024-03-22T02:28:33.438 INFO:tasks.workunit.client.0.smithi103.stdout:2/170: fsync d11/f13 0 2024-03-22T02:28:33.438 INFO:tasks.workunit.client.0.smithi103.stdout:0/227: symlink dd/l44 0 2024-03-22T02:28:33.439 INFO:tasks.workunit.client.0.smithi103.stdout:9/177: write d2/d10/d17/f1f [2218831,27722] 0 2024-03-22T02:28:33.440 INFO:tasks.workunit.client.0.smithi103.stdout:0/228: write fc [4652058,77336] 0 2024-03-22T02:28:33.440 INFO:tasks.workunit.client.0.smithi103.stdout:9/178: chown d2/d10/d27 466 1 2024-03-22T02:28:33.441 INFO:tasks.workunit.client.0.smithi103.stdout:2/171: creat d11/d25/f35 x:0 0 0 2024-03-22T02:28:33.445 INFO:tasks.workunit.client.0.smithi103.stdout:9/179: dread d2/da/f25 [0,4194304] 0 2024-03-22T02:28:33.447 INFO:tasks.workunit.client.0.smithi103.stdout:0/229: mknod dd/d16/d20/d35/c45 0 2024-03-22T02:28:33.449 INFO:tasks.workunit.client.0.smithi103.stdout:0/230: write dd/f13 [693717,55832] 0 2024-03-22T02:28:33.452 INFO:tasks.workunit.client.0.smithi103.stdout:2/172: rename d11/f14 to d11/d15/d34/f36 0 2024-03-22T02:28:33.453 INFO:tasks.workunit.client.0.smithi103.stdout:4/179: dread d3/fd [0,4194304] 0 2024-03-22T02:28:33.455 INFO:tasks.workunit.client.0.smithi103.stdout:0/231: creat dd/d16/d20/d35/f46 x:0 0 0 2024-03-22T02:28:33.460 INFO:tasks.workunit.client.0.smithi103.stdout:9/180: creat d2/f3b x:0 0 0 2024-03-22T02:28:33.460 INFO:tasks.workunit.client.0.smithi103.stdout:4/180: dwrite d3/d1e/d25/f2b [4194304,4194304] 0 2024-03-22T02:28:33.463 INFO:tasks.workunit.client.0.smithi103.stdout:2/173: link f10 d11/d15/f37 0 2024-03-22T02:28:33.463 INFO:tasks.workunit.client.0.smithi103.stdout:2/174: stat d11/d19/l22 0 2024-03-22T02:28:33.466 INFO:tasks.workunit.client.0.smithi103.stdout:9/181: creat d2/d5/f3c x:0 0 0 2024-03-22T02:28:33.466 INFO:tasks.workunit.client.0.smithi103.stdout:2/175: read d11/f30 [3542615,90600] 0 2024-03-22T02:28:33.467 INFO:tasks.workunit.client.0.smithi103.stdout:4/181: mkdir d3/d30/d45 0 2024-03-22T02:28:33.469 INFO:tasks.workunit.client.0.smithi103.stdout:9/182: write d2/da/f25 [3421006,54259] 0 2024-03-22T02:28:33.471 INFO:tasks.workunit.client.0.smithi103.stdout:2/176: symlink d11/d15/d34/l38 0 2024-03-22T02:28:33.471 INFO:tasks.workunit.client.0.smithi103.stdout:9/183: write d2/da/f23 [861757,46611] 0 2024-03-22T02:28:33.471 INFO:tasks.workunit.client.0.smithi103.stdout:2/177: fsync d11/f16 0 2024-03-22T02:28:33.472 INFO:tasks.workunit.client.0.smithi103.stdout:0/232: getdents dd 0 2024-03-22T02:28:33.473 INFO:tasks.workunit.client.0.smithi103.stdout:9/184: write d2/d5/f38 [14649,75964] 0 2024-03-22T02:28:33.475 INFO:tasks.workunit.client.0.smithi103.stdout:2/178: write d11/f13 [3548089,18993] 0 2024-03-22T02:28:33.476 INFO:tasks.workunit.client.0.smithi103.stdout:0/233: truncate dd/d16/d20/f26 475187 0 2024-03-22T02:28:33.477 INFO:tasks.workunit.client.0.smithi103.stdout:2/179: stat d11/f18 0 2024-03-22T02:28:33.477 INFO:tasks.workunit.client.0.smithi103.stdout:0/234: read dd/d16/d20/d29/f2a [2158935,150] 0 2024-03-22T02:28:33.480 INFO:tasks.workunit.client.0.smithi103.stdout:4/182: getdents d3 0 2024-03-22T02:28:33.481 INFO:tasks.workunit.client.0.smithi103.stdout:0/235: symlink dd/d16/d20/d35/l47 0 2024-03-22T02:28:33.482 INFO:tasks.workunit.client.0.smithi103.stdout:2/180: dread d11/f16 [0,4194304] 0 2024-03-22T02:28:33.484 INFO:tasks.workunit.client.0.smithi103.stdout:4/183: rename d3/f1d to d3/d1e/d25/d2c/f46 0 2024-03-22T02:28:33.486 INFO:tasks.workunit.client.0.smithi103.stdout:0/236: mkdir dd/d16/d20/d29/d48 0 2024-03-22T02:28:33.488 INFO:tasks.workunit.client.0.smithi103.stdout:2/181: symlink d11/d15/d34/d24/l39 0 2024-03-22T02:28:33.491 INFO:tasks.workunit.client.0.smithi103.stdout:0/237: mkdir dd/d16/d20/d35/d49 0 2024-03-22T02:28:33.492 INFO:tasks.workunit.client.0.smithi103.stdout:0/238: write dd/d16/f3f [967462,89298] 0 2024-03-22T02:28:33.496 INFO:tasks.workunit.client.0.smithi103.stdout:0/239: rename dd/c1e to dd/d16/c4a 0 2024-03-22T02:28:33.497 INFO:tasks.workunit.client.0.smithi103.stdout:0/240: stat f4 0 2024-03-22T02:28:33.498 INFO:tasks.workunit.client.0.smithi103.stdout:4/184: fdatasync d3/f7 0 2024-03-22T02:28:33.500 INFO:tasks.workunit.client.0.smithi103.stdout:2/182: dwrite d11/d15/d34/f36 [0,4194304] 0 2024-03-22T02:28:33.507 INFO:tasks.workunit.client.0.smithi103.stdout:2/183: creat d11/f3a x:0 0 0 2024-03-22T02:28:33.509 INFO:tasks.workunit.client.0.smithi103.stdout:0/241: unlink dd/f11 0 2024-03-22T02:28:33.510 INFO:tasks.workunit.client.0.smithi103.stdout:4/185: link d3/f8 d3/d31/f47 0 2024-03-22T02:28:33.515 INFO:tasks.workunit.client.0.smithi103.stdout:4/186: mknod d3/d1e/d25/d2c/c48 0 2024-03-22T02:28:33.516 INFO:tasks.workunit.client.0.smithi103.stdout:2/184: rmdir d11 39 2024-03-22T02:28:33.518 INFO:tasks.workunit.client.0.smithi103.stdout:4/187: chown d3/f3c 11 1 2024-03-22T02:28:33.520 INFO:tasks.workunit.client.0.smithi103.stdout:2/185: chown d11/d15/d34/f2a 75 1 2024-03-22T02:28:33.520 INFO:tasks.workunit.client.0.smithi103.stdout:0/242: dwrite dd/d16/d20/d31/f32 [0,4194304] 0 2024-03-22T02:28:33.523 INFO:tasks.workunit.client.0.smithi103.stdout:4/188: creat d3/d30/f49 x:0 0 0 2024-03-22T02:28:33.524 INFO:tasks.workunit.client.0.smithi103.stdout:2/186: symlink d11/d19/l3b 0 2024-03-22T02:28:33.525 INFO:tasks.workunit.client.0.smithi103.stdout:0/243: mkdir dd/d16/d4b 0 2024-03-22T02:28:33.526 INFO:tasks.workunit.client.0.smithi103.stdout:2/187: fdatasync f10 0 2024-03-22T02:28:33.527 INFO:tasks.workunit.client.0.smithi103.stdout:0/244: rmdir dd/d16/d20/d29 39 2024-03-22T02:28:33.528 INFO:tasks.workunit.client.0.smithi103.stdout:4/189: fsync d3/d31/f37 0 2024-03-22T02:28:33.530 INFO:tasks.workunit.client.0.smithi103.stdout:4/190: chown d3/d30/f3a 1 1 2024-03-22T02:28:33.531 INFO:tasks.workunit.client.0.smithi103.stdout:2/188: dread d11/f12 [0,4194304] 0 2024-03-22T02:28:33.535 INFO:tasks.workunit.client.0.smithi103.stdout:4/191: readlink d3/l19 0 2024-03-22T02:28:33.537 INFO:tasks.workunit.client.0.smithi103.stdout:0/245: creat dd/d16/d20/d29/f4c x:0 0 0 2024-03-22T02:28:33.538 INFO:tasks.workunit.client.0.smithi103.stdout:2/189: rename d11/d15/d34/c33 to d11/d15/d34/c3c 0 2024-03-22T02:28:33.540 INFO:tasks.workunit.client.0.smithi103.stdout:0/246: creat dd/d16/d20/d29/f4d x:0 0 0 2024-03-22T02:28:33.541 INFO:tasks.workunit.client.0.smithi103.stdout:4/192: symlink d3/d30/d45/l4a 0 2024-03-22T02:28:33.541 INFO:tasks.workunit.client.0.smithi103.stdout:2/190: symlink d11/l3d 0 2024-03-22T02:28:33.542 INFO:tasks.workunit.client.0.smithi103.stdout:2/191: dread - d11/d15/f29 zero size 2024-03-22T02:28:33.543 INFO:tasks.workunit.client.0.smithi103.stdout:0/247: unlink dd/d16/d20/f26 0 2024-03-22T02:28:33.543 INFO:tasks.workunit.client.0.smithi103.stdout:2/192: write d11/d15/f29 [1034790,106281] 0 2024-03-22T02:28:33.544 INFO:tasks.workunit.client.0.smithi103.stdout:0/248: fsync f4 0 2024-03-22T02:28:33.545 INFO:tasks.workunit.client.0.smithi103.stdout:4/193: write d3/f2d [7903684,43674] 0 2024-03-22T02:28:33.545 INFO:tasks.workunit.client.0.smithi103.stdout:2/193: write f2 [459416,5195] 0 2024-03-22T02:28:33.548 INFO:tasks.workunit.client.0.smithi103.stdout:4/194: symlink d3/d30/d45/l4b 0 2024-03-22T02:28:33.550 INFO:tasks.workunit.client.0.smithi103.stdout:0/249: symlink dd/d34/l4e 0 2024-03-22T02:28:33.552 INFO:tasks.workunit.client.0.smithi103.stdout:0/250: dread - dd/f1d zero size 2024-03-22T02:28:33.554 INFO:tasks.workunit.client.0.smithi103.stdout:2/194: link d11/d19/l3b d11/d15/d34/l3e 0 2024-03-22T02:28:33.556 INFO:tasks.workunit.client.0.smithi103.stdout:0/251: dread fc [4194304,4194304] 0 2024-03-22T02:28:33.558 INFO:tasks.workunit.client.0.smithi103.stdout:4/195: rename d3/f1a to d3/d38/f4c 0 2024-03-22T02:28:33.564 INFO:tasks.workunit.client.0.smithi103.stdout:0/252: rename dd/d34/l4e to dd/d16/d20/d35/l4f 0 2024-03-22T02:28:33.565 INFO:tasks.workunit.client.0.smithi103.stdout:0/253: dread - dd/d16/f38 zero size 2024-03-22T02:28:33.566 INFO:tasks.workunit.client.0.smithi103.stdout:0/254: stat dd/d16/d20/d31/l41 0 2024-03-22T02:28:33.567 INFO:tasks.workunit.client.0.smithi103.stdout:0/255: write dd/d16/f38 [929423,70337] 0 2024-03-22T02:28:33.569 INFO:tasks.workunit.client.0.smithi103.stdout:4/196: write d3/f24 [4505610,18340] 0 2024-03-22T02:28:33.571 INFO:tasks.workunit.client.0.smithi103.stdout:0/256: creat dd/d16/d4b/f50 x:0 0 0 2024-03-22T02:28:33.572 INFO:tasks.workunit.client.0.smithi103.stdout:2/195: dread d11/f20 [0,4194304] 0 2024-03-22T02:28:33.573 INFO:tasks.workunit.client.0.smithi103.stdout:4/197: write d3/d1e/d25/f3d [792496,122980] 0 2024-03-22T02:28:33.573 INFO:tasks.workunit.client.0.smithi103.stdout:2/196: rmdir d11/d25 39 2024-03-22T02:28:33.575 INFO:tasks.workunit.client.0.smithi103.stdout:2/197: write d11/d15/d34/d24/f32 [4181236,112929] 0 2024-03-22T02:28:33.575 INFO:tasks.workunit.client.0.smithi103.stdout:0/257: unlink dd/d16/d20/l2b 0 2024-03-22T02:28:33.577 INFO:tasks.workunit.client.0.smithi103.stdout:2/198: write d11/d19/f1f [228889,91497] 0 2024-03-22T02:28:33.578 INFO:tasks.workunit.client.0.smithi103.stdout:4/198: write d3/d38/f4c [4938507,67568] 0 2024-03-22T02:28:33.579 INFO:tasks.workunit.client.0.smithi103.stdout:0/258: mknod dd/d16/d20/d29/d48/c51 0 2024-03-22T02:28:33.582 INFO:tasks.workunit.client.0.smithi103.stdout:2/199: unlink d11/d15/d34/d24/f32 0 2024-03-22T02:28:33.584 INFO:tasks.workunit.client.0.smithi103.stdout:4/199: dread - d3/d30/f49 zero size 2024-03-22T02:28:33.587 INFO:tasks.workunit.client.0.smithi103.stdout:2/200: symlink d11/d15/d34/l3f 0 2024-03-22T02:28:33.588 INFO:tasks.workunit.client.0.smithi103.stdout:2/201: truncate d11/d15/d34/f36 4333096 0 2024-03-22T02:28:33.590 INFO:tasks.workunit.client.0.smithi103.stdout:4/200: creat d3/d1e/d25/f4d x:0 0 0 2024-03-22T02:28:33.590 INFO:tasks.workunit.client.0.smithi103.stdout:0/259: getdents dd/d16/d20/d29 0 2024-03-22T02:28:33.591 INFO:tasks.workunit.client.0.smithi103.stdout:0/260: stat dd/d16/d4b/f50 0 2024-03-22T02:28:33.594 INFO:tasks.workunit.client.0.smithi103.stdout:0/261: dread - dd/d34/f43 zero size 2024-03-22T02:28:33.595 INFO:tasks.workunit.client.0.smithi103.stdout:2/202: write d11/d15/f37 [2393504,83857] 0 2024-03-22T02:28:33.597 INFO:tasks.workunit.client.0.smithi103.stdout:2/203: write d11/d15/d34/f36 [3109711,40059] 0 2024-03-22T02:28:33.597 INFO:tasks.workunit.client.0.smithi103.stdout:0/262: fdatasync dd/f13 0 2024-03-22T02:28:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:33 smithi122 ceph-mon[36403]: pgmap v10: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 106 KiB/s rd, 15 MiB/s wr, 379 op/s 2024-03-22T02:28:33.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:33 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:33.600 INFO:tasks.workunit.client.0.smithi103.stdout:0/263: truncate dd/d16/d20/d29/f42 692065 0 2024-03-22T02:28:33.600 INFO:tasks.workunit.client.0.smithi103.stdout:4/201: truncate d3/f7 410256 0 2024-03-22T02:28:33.603 INFO:tasks.workunit.client.0.smithi103.stdout:2/204: mkdir d11/d15/d40 0 2024-03-22T02:28:33.607 INFO:tasks.workunit.client.0.smithi103.stdout:2/205: creat d11/d15/d40/f41 x:0 0 0 2024-03-22T02:28:33.608 INFO:tasks.workunit.client.0.smithi103.stdout:0/264: dwrite dd/d16/f3f [0,4194304] 0 2024-03-22T02:28:33.609 INFO:tasks.workunit.client.0.smithi103.stdout:4/202: dwrite d3/d1e/f20 [0,4194304] 0 2024-03-22T02:28:33.612 INFO:tasks.workunit.client.0.smithi103.stdout:2/206: symlink d11/d15/d40/l42 0 2024-03-22T02:28:33.615 INFO:tasks.workunit.client.0.smithi103.stdout:0/265: rmdir dd/d16/d20/d31 39 2024-03-22T02:28:33.616 INFO:tasks.workunit.client.0.smithi103.stdout:0/266: fsync dd/d16/d20/f21 0 2024-03-22T02:28:33.620 INFO:tasks.workunit.client.0.smithi103.stdout:4/203: rename d3/f2d to d3/d31/f4e 0 2024-03-22T02:28:33.623 INFO:tasks.workunit.client.0.smithi103.stdout:0/267: mknod dd/c52 0 2024-03-22T02:28:33.624 INFO:tasks.workunit.client.0.smithi103.stdout:0/268: mkdir dd/d34/d53 0 2024-03-22T02:28:33.630 INFO:tasks.workunit.client.0.smithi103.stdout:0/269: dwrite dd/d16/d20/d35/f46 [0,4194304] 0 2024-03-22T02:28:33.633 INFO:tasks.workunit.client.0.smithi103.stdout:0/270: dread dd/d16/d20/f21 [0,4194304] 0 2024-03-22T02:28:33.640 INFO:tasks.workunit.client.0.smithi103.stdout:0/271: dwrite dd/f13 [4194304,4194304] 0 2024-03-22T02:28:33.641 INFO:tasks.workunit.client.0.smithi103.stdout:0/272: dread - dd/d16/d20/d29/f3e zero size 2024-03-22T02:28:33.644 INFO:tasks.workunit.client.0.smithi103.stdout:0/273: symlink dd/d16/l54 0 2024-03-22T02:28:33.646 INFO:tasks.workunit.client.0.smithi103.stdout:0/274: creat dd/d16/f55 x:0 0 0 2024-03-22T02:28:33.652 INFO:tasks.workunit.client.0.smithi103.stdout:0/275: dwrite dd/f13 [0,4194304] 0 2024-03-22T02:28:33.654 INFO:tasks.workunit.client.0.smithi103.stdout:0/276: chown dd/d16/d20/d31/l41 4 1 2024-03-22T02:28:33.654 INFO:tasks.workunit.client.0.smithi103.stdout:0/277: stat dd/d16/f36 0 2024-03-22T02:28:33.679 INFO:tasks.workunit.client.0.smithi103.stdout:7/129: sync 2024-03-22T02:28:33.679 INFO:tasks.workunit.client.0.smithi103.stdout:3/222: sync 2024-03-22T02:28:33.679 INFO:tasks.workunit.client.0.smithi103.stdout:8/223: sync 2024-03-22T02:28:33.681 INFO:tasks.workunit.client.0.smithi103.stdout:3/223: dread de/f21 [0,4194304] 0 2024-03-22T02:28:33.682 INFO:tasks.workunit.client.0.smithi103.stdout:2/207: getdents d11/d15 0 2024-03-22T02:28:33.683 INFO:tasks.workunit.client.0.smithi103.stdout:7/130: symlink d1/d10/d11/l34 0 2024-03-22T02:28:33.684 INFO:tasks.workunit.client.0.smithi103.stdout:7/131: chown d1/lc 2050105088 1 2024-03-22T02:28:33.685 INFO:tasks.workunit.client.0.smithi103.stdout:7/132: write d1/f1e [550578,105147] 0 2024-03-22T02:28:33.686 INFO:tasks.workunit.client.0.smithi103.stdout:0/278: dread dd/d16/f38 [0,4194304] 0 2024-03-22T02:28:33.686 INFO:tasks.workunit.client.0.smithi103.stdout:7/133: chown d1/lc 51736 1 2024-03-22T02:28:33.687 INFO:tasks.workunit.client.0.smithi103.stdout:7/134: chown d1/f2d 7026 1 2024-03-22T02:28:33.696 INFO:tasks.workunit.client.0.smithi103.stdout:2/208: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:33.698 INFO:tasks.workunit.client.0.smithi103.stdout:0/279: dread dd/d16/d20/f21 [0,4194304] 0 2024-03-22T02:28:33.699 INFO:tasks.workunit.client.0.smithi103.stdout:2/209: readlink d11/d19/l22 0 2024-03-22T02:28:33.699 INFO:tasks.workunit.client.0.smithi103.stdout:2/210: fdatasync f4 0 2024-03-22T02:28:33.700 INFO:tasks.workunit.client.0.smithi103.stdout:2/211: chown d11/f20 2 1 2024-03-22T02:28:33.701 INFO:tasks.workunit.client.0.smithi103.stdout:2/212: chown d11/d19/f1f 7106 1 2024-03-22T02:28:33.704 INFO:tasks.workunit.client.0.smithi103.stdout:8/224: dread d4/fe [0,4194304] 0 2024-03-22T02:28:33.704 INFO:tasks.workunit.client.0.smithi103.stdout:7/135: dread d1/d10/d11/f18 [0,4194304] 0 2024-03-22T02:28:33.707 INFO:tasks.workunit.client.0.smithi103.stdout:7/136: truncate d1/d10/d11/d19/f26 4752981 0 2024-03-22T02:28:33.707 INFO:tasks.workunit.client.0.smithi103.stdout:2/213: mkdir d11/d25/d43 0 2024-03-22T02:28:33.709 INFO:tasks.workunit.client.0.smithi103.stdout:6/122: sync 2024-03-22T02:28:33.709 INFO:tasks.workunit.client.0.smithi103.stdout:1/177: sync 2024-03-22T02:28:33.710 INFO:tasks.workunit.client.0.smithi103.stdout:9/185: sync 2024-03-22T02:28:33.710 INFO:tasks.workunit.client.0.smithi103.stdout:5/204: sync 2024-03-22T02:28:33.712 INFO:tasks.workunit.client.0.smithi103.stdout:9/186: dread - d2/d5/f3c zero size 2024-03-22T02:28:33.712 INFO:tasks.workunit.client.0.smithi103.stdout:7/137: creat d1/d10/d1b/f35 x:0 0 0 2024-03-22T02:28:33.713 INFO:tasks.workunit.client.0.smithi103.stdout:2/214: fsync d11/f18 0 2024-03-22T02:28:33.716 INFO:tasks.workunit.client.0.smithi103.stdout:8/225: unlink d4/d26/c3b 0 2024-03-22T02:28:33.717 INFO:tasks.workunit.client.0.smithi103.stdout:6/123: rmdir d0/d3/d22 39 2024-03-22T02:28:33.719 INFO:tasks.workunit.client.0.smithi103.stdout:1/178: dwrite db/d10/d18/f3b [0,4194304] 0 2024-03-22T02:28:33.723 INFO:tasks.workunit.client.0.smithi103.stdout:7/138: unlink d1/d10/d1b/d21/l2e 0 2024-03-22T02:28:33.727 INFO:tasks.workunit.client.0.smithi103.stdout:1/179: dwrite db/d10/d18/f3b [0,4194304] 0 2024-03-22T02:28:33.730 INFO:tasks.workunit.client.0.smithi103.stdout:7/139: dread - d1/d10/d1b/d21/f32 zero size 2024-03-22T02:28:33.732 INFO:tasks.workunit.client.0.smithi103.stdout:7/140: fsync d1/da/f8 0 2024-03-22T02:28:33.738 INFO:tasks.workunit.client.0.smithi103.stdout:6/124: mknod d0/d3/c2b 0 2024-03-22T02:28:33.738 INFO:tasks.workunit.client.0.smithi103.stdout:1/180: dwrite db/d10/f1c [4194304,4194304] 0 2024-03-22T02:28:33.742 INFO:tasks.workunit.client.0.smithi103.stdout:3/224: truncate de/d1a/f2a 519339 0 2024-03-22T02:28:33.744 INFO:tasks.workunit.client.0.smithi103.stdout:9/187: getdents d2/da/de 0 2024-03-22T02:28:33.746 INFO:tasks.workunit.client.0.smithi103.stdout:3/225: truncate de/d1a/d25/d26/f2f 918948 0 2024-03-22T02:28:33.748 INFO:tasks.workunit.client.0.smithi103.stdout:1/181: rename db/d10/d18/d2d/f3d to db/d10/d18/f40 0 2024-03-22T02:28:33.749 INFO:tasks.workunit.client.0.smithi103.stdout:7/141: mkdir d1/da/d2f/d36 0 2024-03-22T02:28:33.752 INFO:tasks.workunit.client.0.smithi103.stdout:0/280: truncate fc 2474668 0 2024-03-22T02:28:33.753 INFO:tasks.workunit.client.0.smithi103.stdout:9/188: chown d2/da/d18/f21 22 1 2024-03-22T02:28:33.754 INFO:tasks.workunit.client.0.smithi103.stdout:4/204: dwrite d3/d31/f37 [0,4194304] 0 2024-03-22T02:28:33.754 INFO:tasks.workunit.client.0.smithi103.stdout:3/226: creat de/d12/d19/f51 x:0 0 0 2024-03-22T02:28:33.755 INFO:tasks.workunit.client.0.smithi103.stdout:2/215: getdents d11/d25 0 2024-03-22T02:28:33.758 INFO:tasks.workunit.client.0.smithi103.stdout:1/182: unlink db/d10/d18/f31 0 2024-03-22T02:28:33.762 INFO:tasks.workunit.client.0.smithi103.stdout:6/125: creat d0/f2c x:0 0 0 2024-03-22T02:28:33.763 INFO:tasks.workunit.client.0.smithi103.stdout:0/281: symlink dd/d16/d4b/l56 0 2024-03-22T02:28:33.765 INFO:tasks.workunit.client.0.smithi103.stdout:7/142: dwrite d1/f2d [0,4194304] 0 2024-03-22T02:28:33.770 INFO:tasks.workunit.client.0.smithi103.stdout:3/227: dwrite f1 [0,4194304] 0 2024-03-22T02:28:33.770 INFO:tasks.workunit.client.0.smithi103.stdout:0/282: dread - dd/d16/d4b/f50 zero size 2024-03-22T02:28:33.770 INFO:tasks.workunit.client.0.smithi103.stdout:3/228: read - de/f22 zero size 2024-03-22T02:28:33.771 INFO:tasks.workunit.client.0.smithi103.stdout:0/283: chown dd/d16/d4b/l56 0 1 2024-03-22T02:28:33.771 INFO:tasks.workunit.client.0.smithi103.stdout:2/216: symlink d11/d19/l44 0 2024-03-22T02:28:33.772 INFO:tasks.workunit.client.0.smithi103.stdout:0/284: fdatasync fa 0 2024-03-22T02:28:33.772 INFO:tasks.workunit.client.0.smithi103.stdout:7/143: truncate d1/d10/d1b/f35 701619 0 2024-03-22T02:28:33.773 INFO:tasks.workunit.client.0.smithi103.stdout:1/183: mkdir db/d10/d29/d41 0 2024-03-22T02:28:33.774 INFO:tasks.workunit.client.0.smithi103.stdout:1/184: chown db/d10/d18/f3f 8139 1 2024-03-22T02:28:33.775 INFO:tasks.workunit.client.0.smithi103.stdout:1/185: write db/f17 [1733058,1000] 0 2024-03-22T02:28:33.776 INFO:tasks.workunit.client.0.smithi103.stdout:9/189: rename d2/d5/d39 to d2/da/d18/d1e/d34/d3d 0 2024-03-22T02:28:33.778 INFO:tasks.workunit.client.0.smithi103.stdout:6/126: creat d0/d14/d20/f2d x:0 0 0 2024-03-22T02:28:33.781 INFO:tasks.workunit.client.0.smithi103.stdout:4/205: rename d3/d1e/d25/d2c/f44 to d3/d1e/d25/d2c/f4f 0 2024-03-22T02:28:33.783 INFO:tasks.workunit.client.0.smithi103.stdout:3/229: symlink de/d12/d19/l52 0 2024-03-22T02:28:33.784 INFO:tasks.workunit.client.0.smithi103.stdout:1/186: dwrite db/d10/d18/f3f [0,4194304] 0 2024-03-22T02:28:33.784 INFO:tasks.workunit.client.0.smithi103.stdout:8/226: write d4/ff [6586710,11479] 0 2024-03-22T02:28:33.784 INFO:tasks.workunit.client.0.smithi103.stdout:7/144: fdatasync d1/f2d 0 2024-03-22T02:28:33.791 INFO:tasks.workunit.client.0.smithi103.stdout:7/145: dread d1/d10/d1b/f35 [0,4194304] 0 2024-03-22T02:28:33.792 INFO:tasks.workunit.client.0.smithi103.stdout:9/190: creat d2/da/de/f3e x:0 0 0 2024-03-22T02:28:33.795 INFO:tasks.workunit.client.0.smithi103.stdout:3/230: dwrite de/d12/d19/f43 [0,4194304] 0 2024-03-22T02:28:33.795 INFO:tasks.workunit.client.0.smithi103.stdout:6/127: write d0/d3/d22/f11 [8415928,126444] 0 2024-03-22T02:28:33.798 INFO:tasks.workunit.client.0.smithi103.stdout:2/217: mkdir d11/d25/d43/d45 0 2024-03-22T02:28:33.802 INFO:tasks.workunit.client.0.smithi103.stdout:6/128: dread d0/f1a [0,4194304] 0 2024-03-22T02:28:33.803 INFO:tasks.workunit.client.0.smithi103.stdout:8/227: unlink d4/l6 0 2024-03-22T02:28:33.805 INFO:tasks.workunit.client.0.smithi103.stdout:1/187: mknod db/d10/d18/c42 0 2024-03-22T02:28:33.807 INFO:tasks.workunit.client.0.smithi103.stdout:0/285: mknod dd/d16/d20/d35/d49/c57 0 2024-03-22T02:28:33.811 INFO:tasks.workunit.client.0.smithi103.stdout:3/231: creat de/d1a/d25/d26/f53 x:0 0 0 2024-03-22T02:28:33.815 INFO:tasks.workunit.client.0.smithi103.stdout:2/218: rename d11/d15 to d11/d25/d46 0 2024-03-22T02:28:33.818 INFO:tasks.workunit.client.0.smithi103.stdout:8/228: symlink d4/d2a/l3c 0 2024-03-22T02:28:33.819 INFO:tasks.workunit.client.0.smithi103.stdout:1/188: creat db/d10/d28/f43 x:0 0 0 2024-03-22T02:28:33.820 INFO:tasks.workunit.client.0.smithi103.stdout:1/189: chown c4 0 1 2024-03-22T02:28:33.820 INFO:tasks.workunit.client.0.smithi103.stdout:7/146: dread d1/da/f30 [0,4194304] 0 2024-03-22T02:28:33.820 INFO:tasks.workunit.client.0.smithi103.stdout:0/286: creat dd/d16/d20/d31/f58 x:0 0 0 2024-03-22T02:28:33.823 INFO:tasks.workunit.client.0.smithi103.stdout:3/232: fsync de/d12/d30/f35 0 2024-03-22T02:28:33.823 INFO:tasks.workunit.client.0.smithi103.stdout:3/233: chown de/f21 42273 1 2024-03-22T02:28:33.825 INFO:tasks.workunit.client.0.smithi103.stdout:2/219: rmdir d11/d25/d46/d40 39 2024-03-22T02:28:33.828 INFO:tasks.workunit.client.0.smithi103.stdout:2/220: truncate d11/d25/d46/f29 1644157 0 2024-03-22T02:28:33.831 INFO:tasks.workunit.client.0.smithi103.stdout:1/190: mknod db/d10/d29/c44 0 2024-03-22T02:28:33.831 INFO:tasks.workunit.client.0.smithi103.stdout:3/234: dwrite de/d1a/d25/f29 [0,4194304] 0 2024-03-22T02:28:33.834 INFO:tasks.workunit.client.0.smithi103.stdout:6/129: creat d0/f2e x:0 0 0 2024-03-22T02:28:33.835 INFO:tasks.workunit.client.0.smithi103.stdout:8/229: mknod d4/d14/d39/c3d 0 2024-03-22T02:28:33.838 INFO:tasks.workunit.client.0.smithi103.stdout:8/230: chown d4/d13/f1b 1013 1 2024-03-22T02:28:33.842 INFO:tasks.workunit.client.0.smithi103.stdout:7/147: symlink d1/da/d2f/d36/l37 0 2024-03-22T02:28:33.842 INFO:tasks.workunit.client.0.smithi103.stdout:3/235: dwrite de/d12/d30/f3d [0,4194304] 0 2024-03-22T02:28:33.845 INFO:tasks.workunit.client.0.smithi103.stdout:1/191: rename db/d10/d21/f22 to db/f45 0 2024-03-22T02:28:33.845 INFO:tasks.workunit.client.0.smithi103.stdout:6/130: dwrite d0/f2e [0,4194304] 0 2024-03-22T02:28:33.849 INFO:tasks.workunit.client.0.smithi103.stdout:8/231: mkdir d4/d2a/d3e 0 2024-03-22T02:28:33.854 INFO:tasks.workunit.client.0.smithi103.stdout:2/221: mkdir d11/d25/d43/d45/d47 0 2024-03-22T02:28:33.857 INFO:tasks.workunit.client.0.smithi103.stdout:3/236: unlink de/f27 0 2024-03-22T02:28:33.858 INFO:tasks.workunit.client.0.smithi103.stdout:7/148: creat d1/da/d2f/f38 x:0 0 0 2024-03-22T02:28:33.859 INFO:tasks.workunit.client.0.smithi103.stdout:1/192: mknod db/d10/d28/c46 0 2024-03-22T02:28:33.865 INFO:tasks.workunit.client.0.smithi103.stdout:3/237: creat de/d1a/d25/d26/f54 x:0 0 0 2024-03-22T02:28:33.867 INFO:tasks.workunit.client.0.smithi103.stdout:8/232: symlink d4/d19/l3f 0 2024-03-22T02:28:33.869 INFO:tasks.workunit.client.0.smithi103.stdout:1/193: symlink db/d10/d18/d2d/l47 0 2024-03-22T02:28:33.871 INFO:tasks.workunit.client.0.smithi103.stdout:2/222: dwrite f7 [0,4194304] 0 2024-03-22T02:28:33.871 INFO:tasks.workunit.client.0.smithi103.stdout:3/238: symlink de/d12/d47/d4f/l55 0 2024-03-22T02:28:33.872 INFO:tasks.workunit.client.0.smithi103.stdout:6/131: dread d0/f1e [0,4194304] 0 2024-03-22T02:28:33.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:33 smithi103 ceph-mon[29258]: pgmap v10: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 106 KiB/s rd, 15 MiB/s wr, 379 op/s 2024-03-22T02:28:33.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:33 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:33.875 INFO:tasks.workunit.client.0.smithi103.stdout:1/194: unlink db/cd 0 2024-03-22T02:28:33.877 INFO:tasks.workunit.client.0.smithi103.stdout:1/195: stat db/d10/d18/f20 0 2024-03-22T02:28:33.877 INFO:tasks.workunit.client.0.smithi103.stdout:8/233: mkdir d4/d13/d40 0 2024-03-22T02:28:33.878 INFO:tasks.workunit.client.0.smithi103.stdout:2/223: fdatasync d11/f12 0 2024-03-22T02:28:33.879 INFO:tasks.workunit.client.0.smithi103.stdout:8/234: fsync d4/d13/f18 0 2024-03-22T02:28:33.880 INFO:tasks.workunit.client.0.smithi103.stdout:6/132: rmdir d0/d3/d22 39 2024-03-22T02:28:33.882 INFO:tasks.workunit.client.0.smithi103.stdout:3/239: dwrite de/d1a/f31 [8388608,4194304] 0 2024-03-22T02:28:33.883 INFO:tasks.workunit.client.0.smithi103.stdout:0/287: dread dd/f15 [0,4194304] 0 2024-03-22T02:28:33.886 INFO:tasks.workunit.client.0.smithi103.stdout:1/196: creat db/d10/d18/d2d/f48 x:0 0 0 2024-03-22T02:28:33.886 INFO:tasks.workunit.client.0.smithi103.stdout:8/235: mknod d4/d14/c41 0 2024-03-22T02:28:33.889 INFO:tasks.workunit.client.0.smithi103.stdout:1/197: chown db/d10/d18/f3b 3923009 1 2024-03-22T02:28:33.889 INFO:tasks.workunit.client.0.smithi103.stdout:0/288: mknod dd/d34/c59 0 2024-03-22T02:28:33.891 INFO:tasks.workunit.client.0.smithi103.stdout:8/236: truncate d4/d19/d1a/f2d 631113 0 2024-03-22T02:28:33.891 INFO:tasks.workunit.client.0.smithi103.stdout:7/149: write d1/da/fb [2922654,120540] 0 2024-03-22T02:28:33.895 INFO:tasks.workunit.client.0.smithi103.stdout:3/240: getdents de/d12/d47 0 2024-03-22T02:28:33.896 INFO:tasks.workunit.client.0.smithi103.stdout:6/133: dwrite d0/f27 [0,4194304] 0 2024-03-22T02:28:33.896 INFO:tasks.workunit.client.0.smithi103.stdout:7/150: chown d1/da/l17 14504 1 2024-03-22T02:28:33.897 INFO:tasks.workunit.client.0.smithi103.stdout:3/241: readlink de/d12/l4a 0 2024-03-22T02:28:33.898 INFO:tasks.workunit.client.0.smithi103.stdout:7/151: write d1/f2d [4676713,19908] 0 2024-03-22T02:28:33.900 INFO:tasks.workunit.client.0.smithi103.stdout:0/289: getdents dd/d16/d20/d29 0 2024-03-22T02:28:33.905 INFO:tasks.workunit.client.0.smithi103.stdout:7/152: rename d1/d10/d1b to d1/d39 0 2024-03-22T02:28:33.906 INFO:tasks.workunit.client.0.smithi103.stdout:1/198: dwrite db/d10/f1c [0,4194304] 0 2024-03-22T02:28:33.907 INFO:tasks.workunit.client.0.smithi103.stdout:0/290: creat dd/d16/d20/d29/f5a x:0 0 0 2024-03-22T02:28:33.911 INFO:tasks.workunit.client.0.smithi103.stdout:3/242: dwrite de/d1a/d25/d26/f53 [0,4194304] 0 2024-03-22T02:28:33.911 INFO:tasks.workunit.client.0.smithi103.stdout:7/153: write d1/d10/d11/f18 [1354768,10904] 0 2024-03-22T02:28:33.913 INFO:tasks.workunit.client.0.smithi103.stdout:6/134: dwrite d0/d3/d22/f11 [4194304,4194304] 0 2024-03-22T02:28:33.914 INFO:tasks.workunit.client.0.smithi103.stdout:7/154: symlink d1/da/d1c/l3a 0 2024-03-22T02:28:33.915 INFO:tasks.workunit.client.0.smithi103.stdout:3/243: truncate de/d12/d30/f3d 4846114 0 2024-03-22T02:28:33.919 INFO:tasks.workunit.client.0.smithi103.stdout:6/135: write d0/d3/f9 [4680470,9527] 0 2024-03-22T02:28:33.921 INFO:tasks.workunit.client.0.smithi103.stdout:0/291: dwrite dd/d16/d20/d29/f42 [0,4194304] 0 2024-03-22T02:28:33.923 INFO:tasks.workunit.client.0.smithi103.stdout:2/224: dread d11/f18 [0,4194304] 0 2024-03-22T02:28:33.927 INFO:tasks.workunit.client.0.smithi103.stdout:6/136: mknod d0/d14/c2f 0 2024-03-22T02:28:33.927 INFO:tasks.workunit.client.0.smithi103.stdout:3/244: link c4 de/d12/d4d/c56 0 2024-03-22T02:28:33.933 INFO:tasks.workunit.client.0.smithi103.stdout:6/137: creat d0/d14/f30 x:0 0 0 2024-03-22T02:28:33.935 INFO:tasks.workunit.client.0.smithi103.stdout:3/245: symlink de/d12/d19/l57 0 2024-03-22T02:28:33.936 INFO:tasks.workunit.client.0.smithi103.stdout:0/292: creat dd/f5b x:0 0 0 2024-03-22T02:28:33.938 INFO:tasks.workunit.client.0.smithi103.stdout:0/293: write dd/d16/f55 [217043,28597] 0 2024-03-22T02:28:33.941 INFO:tasks.workunit.client.0.smithi103.stdout:6/138: mknod d0/d3/d22/c31 0 2024-03-22T02:28:33.943 INFO:tasks.workunit.client.0.smithi103.stdout:2/225: getdents d11/d25/d43/d45 0 2024-03-22T02:28:33.944 INFO:tasks.workunit.client.0.smithi103.stdout:0/294: mkdir dd/d16/d20/d29/d48/d5c 0 2024-03-22T02:28:33.944 INFO:tasks.workunit.client.0.smithi103.stdout:6/139: dread - d0/d14/d20/f2d zero size 2024-03-22T02:28:33.948 INFO:tasks.workunit.client.0.smithi103.stdout:3/246: dwrite de/f1e [0,4194304] 0 2024-03-22T02:28:33.948 INFO:tasks.workunit.client.0.smithi103.stdout:1/199: dread db/d10/d21/f23 [0,4194304] 0 2024-03-22T02:28:33.948 INFO:tasks.workunit.client.0.smithi103.stdout:2/226: mkdir d11/d25/d46/d48 0 2024-03-22T02:28:33.954 INFO:tasks.workunit.client.0.smithi103.stdout:0/295: creat dd/d16/d20/f5d x:0 0 0 2024-03-22T02:28:33.957 INFO:tasks.workunit.client.0.smithi103.stdout:6/140: getdents d0 0 2024-03-22T02:28:33.957 INFO:tasks.workunit.client.0.smithi103.stdout:1/200: mknod db/d10/d29/d41/c49 0 2024-03-22T02:28:33.958 INFO:tasks.workunit.client.0.smithi103.stdout:1/201: read - db/d10/d29/f39 zero size 2024-03-22T02:28:33.959 INFO:tasks.workunit.client.0.smithi103.stdout:1/202: chown db/d10/d18/f40 975 1 2024-03-22T02:28:33.960 INFO:tasks.workunit.client.0.smithi103.stdout:3/247: link de/d1a/d25/f29 de/d1a/d25/d26/f58 0 2024-03-22T02:28:33.961 INFO:tasks.workunit.client.0.smithi103.stdout:1/203: readlink db/d10/d18/l1d 0 2024-03-22T02:28:33.961 INFO:tasks.workunit.client.0.smithi103.stdout:2/227: rename fd to d11/d25/d43/d45/d47/f49 0 2024-03-22T02:28:33.962 INFO:tasks.workunit.client.0.smithi103.stdout:3/248: fdatasync de/d12/d19/f48 0 2024-03-22T02:28:33.962 INFO:tasks.workunit.client.0.smithi103.stdout:2/228: chown d11/d25/d46/d34 30 1 2024-03-22T02:28:33.964 INFO:tasks.workunit.client.0.smithi103.stdout:2/229: chown d11/d25/d46/d34/f36 15 1 2024-03-22T02:28:33.968 INFO:tasks.workunit.client.0.smithi103.stdout:7/155: dread d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:33.971 INFO:tasks.workunit.client.0.smithi103.stdout:1/204: dwrite db/d10/d18/f3b [4194304,4194304] 0 2024-03-22T02:28:33.971 INFO:tasks.workunit.client.0.smithi103.stdout:3/249: symlink de/l59 0 2024-03-22T02:28:33.971 INFO:tasks.workunit.client.0.smithi103.stdout:7/156: write d1/da/fb [697868,98814] 0 2024-03-22T02:28:33.978 INFO:tasks.workunit.client.0.smithi103.stdout:3/250: write de/d12/d19/f48 [872479,23368] 0 2024-03-22T02:28:33.978 INFO:tasks.workunit.client.0.smithi103.stdout:3/251: chown de/d1a/d25/d3f/l40 265191683 1 2024-03-22T02:28:33.978 INFO:tasks.workunit.client.0.smithi103.stdout:3/252: write de/d12/d19/f48 [999330,39809] 0 2024-03-22T02:28:33.980 INFO:tasks.workunit.client.0.smithi103.stdout:1/205: rmdir db 39 2024-03-22T02:28:33.981 INFO:tasks.workunit.client.0.smithi103.stdout:7/157: mknod d1/d10/d11/d19/c3b 0 2024-03-22T02:28:33.982 INFO:tasks.workunit.client.0.smithi103.stdout:7/158: chown d1/d39 257085 1 2024-03-22T02:28:33.982 INFO:tasks.workunit.client.0.smithi103.stdout:7/159: stat d1/d10/d11/l12 0 2024-03-22T02:28:33.984 INFO:tasks.workunit.client.0.smithi103.stdout:7/160: fdatasync d1/d10/d11/f18 0 2024-03-22T02:28:33.987 INFO:tasks.workunit.client.0.smithi103.stdout:3/253: chown de/d12/l4e 13 1 2024-03-22T02:28:33.988 INFO:tasks.workunit.client.0.smithi103.stdout:1/206: mkdir db/d4a 0 2024-03-22T02:28:33.989 INFO:tasks.workunit.client.0.smithi103.stdout:3/254: write de/d12/d19/f48 [670289,67811] 0 2024-03-22T02:28:33.990 INFO:tasks.workunit.client.0.smithi103.stdout:3/255: fdatasync fa 0 2024-03-22T02:28:34.018 INFO:tasks.workunit.client.0.smithi103.stdout:7/161: dwrite d1/f2d [4194304,4194304] 0 2024-03-22T02:28:34.019 INFO:tasks.workunit.client.0.smithi103.stdout:7/162: chown d1/da/f8 2023 1 2024-03-22T02:28:34.020 INFO:tasks.workunit.client.0.smithi103.stdout:3/256: mknod de/d1a/d25/d3f/c5a 0 2024-03-22T02:28:34.022 INFO:tasks.workunit.client.0.smithi103.stdout:7/163: symlink d1/d10/d11/l3c 0 2024-03-22T02:28:34.024 INFO:tasks.workunit.client.0.smithi103.stdout:3/257: creat de/d12/d47/d4f/f5b x:0 0 0 2024-03-22T02:28:34.025 INFO:tasks.workunit.client.0.smithi103.stdout:1/207: getdents db/d10/d29 0 2024-03-22T02:28:34.026 INFO:tasks.workunit.client.0.smithi103.stdout:7/164: symlink d1/d39/d21/l3d 0 2024-03-22T02:28:34.027 INFO:tasks.workunit.client.0.smithi103.stdout:1/208: rmdir db/d10 39 2024-03-22T02:28:34.030 INFO:tasks.workunit.client.0.smithi103.stdout:3/258: link de/f22 de/d12/d47/f5c 0 2024-03-22T02:28:34.031 INFO:tasks.workunit.client.0.smithi103.stdout:7/165: mknod d1/d10/d11/d19/d29/c3e 0 2024-03-22T02:28:34.031 INFO:tasks.workunit.client.0.smithi103.stdout:3/259: chown de/d12/d19/l52 24267 1 2024-03-22T02:28:34.032 INFO:tasks.workunit.client.0.smithi103.stdout:1/209: symlink db/d10/l4b 0 2024-03-22T02:28:34.033 INFO:tasks.workunit.client.0.smithi103.stdout:7/166: mkdir d1/d39/d21/d3f 0 2024-03-22T02:28:34.034 INFO:tasks.workunit.client.0.smithi103.stdout:7/167: read d1/f2d [6114066,26158] 0 2024-03-22T02:28:34.034 INFO:tasks.workunit.client.0.smithi103.stdout:1/210: symlink db/d10/d21/l4c 0 2024-03-22T02:28:34.039 INFO:tasks.workunit.client.0.smithi103.stdout:1/211: creat db/d10/d18/d2d/f4d x:0 0 0 2024-03-22T02:28:34.042 INFO:tasks.workunit.client.0.smithi103.stdout:1/212: fdatasync db/d10/d18/f40 0 2024-03-22T02:28:34.046 INFO:tasks.workunit.client.0.smithi103.stdout:7/168: dwrite d1/f2d [4194304,4194304] 0 2024-03-22T02:28:34.047 INFO:tasks.workunit.client.0.smithi103.stdout:7/169: chown d1/da/l2b 57 1 2024-03-22T02:28:34.052 INFO:tasks.workunit.client.0.smithi103.stdout:3/260: dread de/d12/d30/f35 [0,4194304] 0 2024-03-22T02:28:34.053 INFO:tasks.workunit.client.0.smithi103.stdout:7/170: mknod d1/d10/d11/d19/d29/d33/c40 0 2024-03-22T02:28:34.053 INFO:tasks.workunit.client.0.smithi103.stdout:1/213: fsync db/d10/d18/d2d/f4d 0 2024-03-22T02:28:34.055 INFO:tasks.workunit.client.0.smithi103.stdout:3/261: mkdir de/d12/d5d 0 2024-03-22T02:28:34.057 INFO:tasks.workunit.client.0.smithi103.stdout:1/214: mknod db/d10/d2e/c4e 0 2024-03-22T02:28:34.058 INFO:tasks.workunit.client.0.smithi103.stdout:3/262: chown de/d12/d19/f28 7135 1 2024-03-22T02:28:34.061 INFO:tasks.workunit.client.0.smithi103.stdout:1/215: mknod db/d10/d28/c4f 0 2024-03-22T02:28:34.063 INFO:tasks.workunit.client.0.smithi103.stdout:1/216: readlink db/d10/d18/l1d 0 2024-03-22T02:28:34.064 INFO:tasks.workunit.client.0.smithi103.stdout:7/171: dwrite d1/d39/f24 [0,4194304] 0 2024-03-22T02:28:34.066 INFO:tasks.workunit.client.0.smithi103.stdout:3/263: truncate de/d1a/f2a 872369 0 2024-03-22T02:28:34.073 INFO:tasks.workunit.client.0.smithi103.stdout:1/217: dwrite db/d10/d18/f3f [0,4194304] 0 2024-03-22T02:28:34.077 INFO:tasks.workunit.client.0.smithi103.stdout:1/218: symlink db/l50 0 2024-03-22T02:28:34.080 INFO:tasks.workunit.client.0.smithi103.stdout:7/172: dwrite d1/d39/f35 [0,4194304] 0 2024-03-22T02:28:34.085 INFO:tasks.workunit.client.0.smithi103.stdout:1/219: link db/d10/d18/f37 db/d10/d2e/f51 0 2024-03-22T02:28:34.099 INFO:tasks.workunit.client.0.smithi103.stdout:1/220: dread db/d10/d18/f40 [0,4194304] 0 2024-03-22T02:28:34.100 INFO:tasks.workunit.client.0.smithi103.stdout:7/173: dread d1/da/fb [0,4194304] 0 2024-03-22T02:28:34.100 INFO:tasks.workunit.client.0.smithi103.stdout:1/221: rmdir db 39 2024-03-22T02:28:34.105 INFO:tasks.workunit.client.0.smithi103.stdout:1/222: rename db/d10/d21/l2a to db/d4a/l52 0 2024-03-22T02:28:34.106 INFO:tasks.workunit.client.0.smithi103.stdout:1/223: chown f5 45 1 2024-03-22T02:28:34.106 INFO:tasks.workunit.client.0.smithi103.stdout:7/174: dread d1/da/fe [0,4194304] 0 2024-03-22T02:28:34.110 INFO:tasks.workunit.client.0.smithi103.stdout:7/175: creat d1/d39/d1d/f41 x:0 0 0 2024-03-22T02:28:34.111 INFO:tasks.workunit.client.0.smithi103.stdout:1/224: getdents db 0 2024-03-22T02:28:34.111 INFO:tasks.workunit.client.0.smithi103.stdout:1/225: fdatasync db/f35 0 2024-03-22T02:28:34.112 INFO:tasks.workunit.client.0.smithi103.stdout:7/176: rmdir d1/d10/d11/d19 39 2024-03-22T02:28:34.115 INFO:tasks.workunit.client.0.smithi103.stdout:1/226: rename f7 to db/d4a/f53 0 2024-03-22T02:28:34.116 INFO:tasks.workunit.client.0.smithi103.stdout:1/227: dread - db/d10/d21/f33 zero size 2024-03-22T02:28:34.116 INFO:tasks.workunit.client.0.smithi103.stdout:7/177: getdents d1/d39/d21/d3f 0 2024-03-22T02:28:34.117 INFO:tasks.workunit.client.0.smithi103.stdout:7/178: readlink d1/da/l2b 0 2024-03-22T02:28:34.118 INFO:tasks.workunit.client.0.smithi103.stdout:1/228: symlink db/l54 0 2024-03-22T02:28:34.123 INFO:tasks.workunit.client.0.smithi103.stdout:7/179: symlink d1/d10/d11/l42 0 2024-03-22T02:28:34.125 INFO:tasks.workunit.client.0.smithi103.stdout:7/180: symlink d1/d39/d21/l43 0 2024-03-22T02:28:34.128 INFO:tasks.workunit.client.0.smithi103.stdout:1/229: write db/d10/f26 [4447006,53113] 0 2024-03-22T02:28:34.129 INFO:tasks.workunit.client.0.smithi103.stdout:1/230: fdatasync db/f35 0 2024-03-22T02:28:34.130 INFO:tasks.workunit.client.0.smithi103.stdout:7/181: symlink d1/d10/d11/d19/d29/d33/l44 0 2024-03-22T02:28:34.135 INFO:tasks.workunit.client.0.smithi103.stdout:1/231: write db/d10/d18/f40 [2046491,50869] 0 2024-03-22T02:28:34.136 INFO:tasks.workunit.client.0.smithi103.stdout:7/182: truncate d1/d39/f24 1473979 0 2024-03-22T02:28:34.137 INFO:tasks.workunit.client.0.smithi103.stdout:1/232: mkdir db/d10/d29/d55 0 2024-03-22T02:28:34.138 INFO:tasks.workunit.client.0.smithi103.stdout:7/183: creat d1/d39/f45 x:0 0 0 2024-03-22T02:28:34.140 INFO:tasks.workunit.client.0.smithi103.stdout:7/184: creat d1/d39/f46 x:0 0 0 2024-03-22T02:28:34.141 INFO:tasks.workunit.client.0.smithi103.stdout:7/185: symlink d1/da/d1c/l47 0 2024-03-22T02:28:34.142 INFO:tasks.workunit.client.0.smithi103.stdout:7/186: dread - d1/da/d2f/f38 zero size 2024-03-22T02:28:34.143 INFO:tasks.workunit.client.0.smithi103.stdout:7/187: symlink d1/da/d2f/d36/l48 0 2024-03-22T02:28:34.146 INFO:tasks.workunit.client.0.smithi103.stdout:7/188: link d1/f2d d1/f49 0 2024-03-22T02:28:34.147 INFO:tasks.workunit.client.0.smithi103.stdout:7/189: read - d1/d39/d21/f32 zero size 2024-03-22T02:28:34.148 INFO:tasks.workunit.client.0.smithi103.stdout:7/190: chown d1/d10/d11/l42 1650954 1 2024-03-22T02:28:34.150 INFO:tasks.workunit.client.0.smithi103.stdout:7/191: unlink d1/f2d 0 2024-03-22T02:28:34.152 INFO:tasks.workunit.client.0.smithi103.stdout:7/192: rmdir d1/d39/d21/d3f 0 2024-03-22T02:28:34.154 INFO:tasks.workunit.client.0.smithi103.stdout:7/193: mkdir d1/d10/d11/d19/d29/d33/d4a 0 2024-03-22T02:28:34.169 INFO:tasks.workunit.client.0.smithi103.stdout:7/194: dread d1/d10/d11/d19/f26 [0,4194304] 0 2024-03-22T02:28:34.171 INFO:tasks.workunit.client.0.smithi103.stdout:7/195: mkdir d1/d10/d11/d19/d29/d33/d4b 0 2024-03-22T02:28:34.172 INFO:tasks.workunit.client.0.smithi103.stdout:7/196: stat d1/d39 0 2024-03-22T02:28:34.176 INFO:tasks.workunit.client.0.smithi103.stdout:7/197: dread d1/f49 [0,4194304] 0 2024-03-22T02:28:34.177 INFO:tasks.workunit.client.0.smithi103.stdout:7/198: write d1/d39/f45 [1004611,19882] 0 2024-03-22T02:28:34.177 INFO:tasks.workunit.client.0.smithi103.stdout:7/199: readlink d1/da/l17 0 2024-03-22T02:28:34.178 INFO:tasks.workunit.client.0.smithi103.stdout:7/200: chown d1/d10/d11/d19 25 1 2024-03-22T02:28:34.178 INFO:tasks.workunit.client.0.smithi103.stdout:7/201: readlink d1/d39/d21/l43 0 2024-03-22T02:28:34.184 INFO:tasks.workunit.client.0.smithi103.stdout:7/202: dwrite d1/f1e [0,4194304] 0 2024-03-22T02:28:34.186 INFO:tasks.workunit.client.0.smithi103.stdout:7/203: mknod d1/d39/d21/c4c 0 2024-03-22T02:28:34.187 INFO:tasks.workunit.client.0.smithi103.stdout:7/204: dread - d1/d39/f46 zero size 2024-03-22T02:28:34.187 INFO:tasks.workunit.client.0.smithi103.stdout:7/205: read d1/da/fb [3264897,82700] 0 2024-03-22T02:28:34.213 INFO:tasks.workunit.client.0.smithi103.stdout:7/206: dwrite d1/d39/f24 [0,4194304] 0 2024-03-22T02:28:34.218 INFO:tasks.workunit.client.0.smithi103.stdout:7/207: symlink d1/d10/d11/d19/d29/d33/d4a/l4d 0 2024-03-22T02:28:34.287 INFO:tasks.workunit.client.0.smithi103.stdout:5/205: sync 2024-03-22T02:28:34.290 INFO:tasks.workunit.client.0.smithi103.stdout:5/206: creat d0/d14/d23/d2a/d33/f4c x:0 0 0 2024-03-22T02:28:34.432 INFO:tasks.workunit.client.0.smithi103.stdout:4/206: sync 2024-03-22T02:28:34.434 INFO:tasks.workunit.client.0.smithi103.stdout:4/207: creat d3/d1e/f50 x:0 0 0 2024-03-22T02:28:34.434 INFO:tasks.workunit.client.0.smithi103.stdout:4/208: write d3/d1e/f36 [976301,46110] 0 2024-03-22T02:28:34.438 INFO:tasks.workunit.client.0.smithi103.stdout:4/209: creat d3/f51 x:0 0 0 2024-03-22T02:28:34.439 INFO:tasks.workunit.client.0.smithi103.stdout:4/210: rmdir d3/d30/d45 39 2024-03-22T02:28:34.441 INFO:tasks.workunit.client.0.smithi103.stdout:4/211: write d3/d1e/f43 [467437,74705] 0 2024-03-22T02:28:34.449 INFO:tasks.workunit.client.0.smithi103.stdout:4/212: truncate d3/d1e/f20 459670 0 2024-03-22T02:28:34.518 INFO:tasks.workunit.client.0.smithi103.stdout:4/213: creat d3/d1e/d25/f52 x:0 0 0 2024-03-22T02:28:34.523 INFO:tasks.workunit.client.0.smithi103.stdout:4/214: write d3/fc [4360364,5309] 0 2024-03-22T02:28:34.525 INFO:tasks.workunit.client.0.smithi103.stdout:4/215: chown d3/d31/f47 544176 1 2024-03-22T02:28:34.527 INFO:tasks.workunit.client.0.smithi103.stdout:4/216: mkdir d3/d1e/d53 0 2024-03-22T02:28:34.570 INFO:tasks.workunit.client.0.smithi103.stdout:8/237: sync 2024-03-22T02:28:34.570 INFO:tasks.workunit.client.0.smithi103.stdout:9/191: sync 2024-03-22T02:28:34.578 INFO:tasks.workunit.client.0.smithi103.stdout:4/217: dwrite d3/d1e/f39 [0,4194304] 0 2024-03-22T02:28:34.578 INFO:tasks.workunit.client.0.smithi103.stdout:9/192: dread - d2/f30 zero size 2024-03-22T02:28:34.586 INFO:tasks.workunit.client.0.smithi103.stdout:9/193: dwrite d2/da/de/f1b [0,4194304] 0 2024-03-22T02:28:34.587 INFO:tasks.workunit.client.0.smithi103.stdout:9/194: dread - d2/da/de/f3e zero size 2024-03-22T02:28:34.588 INFO:tasks.workunit.client.0.smithi103.stdout:9/195: chown d2/d10/f13 0 1 2024-03-22T02:28:34.590 INFO:tasks.workunit.client.0.smithi103.stdout:9/196: mkdir d2/da/d18/d1e/d3f 0 2024-03-22T02:28:34.593 INFO:tasks.workunit.client.0.smithi103.stdout:9/197: symlink d2/da/de/l40 0 2024-03-22T02:28:34.593 INFO:tasks.workunit.client.0.smithi103.stdout:9/198: dread - d2/d5/f3c zero size 2024-03-22T02:28:34.597 INFO:tasks.workunit.client.0.smithi103.stdout:8/238: truncate d4/d19/d1a/f2d 294081 0 2024-03-22T02:28:34.597 INFO:tasks.workunit.client.0.smithi103.stdout:9/199: write d2/d5/f3a [556329,18264] 0 2024-03-22T02:28:34.597 INFO:tasks.workunit.client.0.smithi103.stdout:4/218: dread d3/d31/f47 [0,4194304] 0 2024-03-22T02:28:34.599 INFO:tasks.workunit.client.0.smithi103.stdout:8/239: symlink d4/d19/d1a/l42 0 2024-03-22T02:28:34.600 INFO:tasks.workunit.client.0.smithi103.stdout:4/219: fdatasync d3/d1e/d25/d2c/f46 0 2024-03-22T02:28:34.602 INFO:tasks.workunit.client.0.smithi103.stdout:8/240: mknod d4/d19/c43 0 2024-03-22T02:28:34.603 INFO:tasks.workunit.client.0.smithi103.stdout:9/200: getdents d2/d10 0 2024-03-22T02:28:34.607 INFO:tasks.workunit.client.0.smithi103.stdout:4/220: dwrite d3/d31/f37 [0,4194304] 0 2024-03-22T02:28:34.613 INFO:tasks.workunit.client.0.smithi103.stdout:9/201: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:34.613 INFO:tasks.workunit.client.0.smithi103.stdout:8/241: dwrite d4/d19/f2e [0,4194304] 0 2024-03-22T02:28:34.613 INFO:tasks.workunit.client.0.smithi103.stdout:4/221: rename d3/f14 to d3/d1e/d25/d2c/f54 0 2024-03-22T02:28:34.616 INFO:tasks.workunit.client.0.smithi103.stdout:9/202: creat d2/da/d18/f41 x:0 0 0 2024-03-22T02:28:34.616 INFO:tasks.workunit.client.0.smithi103.stdout:8/242: rmdir d4/d26/d27 39 2024-03-22T02:28:34.622 INFO:tasks.workunit.client.0.smithi103.stdout:8/243: creat d4/d13/d35/f44 x:0 0 0 2024-03-22T02:28:34.623 INFO:tasks.workunit.client.0.smithi103.stdout:4/222: dwrite d3/d1e/d25/f4d [0,4194304] 0 2024-03-22T02:28:34.628 INFO:tasks.workunit.client.0.smithi103.stdout:8/244: creat d4/d14/d39/f45 x:0 0 0 2024-03-22T02:28:34.628 INFO:tasks.workunit.client.0.smithi103.stdout:8/245: write d4/f34 [667969,77756] 0 2024-03-22T02:28:34.634 INFO:tasks.workunit.client.0.smithi103.stdout:8/246: dwrite d4/f34 [0,4194304] 0 2024-03-22T02:28:34.637 INFO:tasks.workunit.client.0.smithi103.stdout:8/247: unlink d4/d13/f18 0 2024-03-22T02:28:34.639 INFO:tasks.workunit.client.0.smithi103.stdout:8/248: mknod d4/d2a/d38/c46 0 2024-03-22T02:28:34.642 INFO:tasks.workunit.client.0.smithi103.stdout:8/249: mknod d4/d13/d40/c47 0 2024-03-22T02:28:34.642 INFO:tasks.workunit.client.0.smithi103.stdout:4/223: dread d3/fb [0,4194304] 0 2024-03-22T02:28:34.649 INFO:tasks.workunit.client.0.smithi103.stdout:4/224: dread d3/d1e/f39 [0,4194304] 0 2024-03-22T02:28:34.652 INFO:tasks.workunit.client.0.smithi103.stdout:9/203: dwrite d2/da/d18/f21 [4194304,4194304] 0 2024-03-22T02:28:34.653 INFO:tasks.workunit.client.0.smithi103.stdout:0/296: sync 2024-03-22T02:28:34.653 INFO:tasks.workunit.client.0.smithi103.stdout:6/141: sync 2024-03-22T02:28:34.654 INFO:tasks.workunit.client.0.smithi103.stdout:6/142: read - d0/d14/d20/f2d zero size 2024-03-22T02:28:34.654 INFO:tasks.workunit.client.0.smithi103.stdout:4/225: link d3/d30/f3a d3/d30/f55 0 2024-03-22T02:28:34.655 INFO:tasks.workunit.client.0.smithi103.stdout:9/204: dread d2/d5/f38 [0,4194304] 0 2024-03-22T02:28:34.655 INFO:tasks.workunit.client.0.smithi103.stdout:4/226: read d3/fb [122477,97776] 0 2024-03-22T02:28:34.655 INFO:tasks.workunit.client.0.smithi103.stdout:6/143: fdatasync d0/d3/f9 0 2024-03-22T02:28:34.656 INFO:tasks.workunit.client.0.smithi103.stdout:9/205: read - d2/da/d18/f41 zero size 2024-03-22T02:28:34.657 INFO:tasks.workunit.client.0.smithi103.stdout:6/144: write d0/d3/d22/f15 [145044,25697] 0 2024-03-22T02:28:34.658 INFO:tasks.workunit.client.0.smithi103.stdout:6/145: chown d0/d14/f29 915502401 1 2024-03-22T02:28:34.658 INFO:tasks.workunit.client.0.smithi103.stdout:8/250: fsync d4/d19/f2e 0 2024-03-22T02:28:34.660 INFO:tasks.workunit.client.0.smithi103.stdout:4/227: creat d3/d1e/d53/f56 x:0 0 0 2024-03-22T02:28:34.661 INFO:tasks.workunit.client.0.smithi103.stdout:8/251: read d4/f15 [3799805,54562] 0 2024-03-22T02:28:34.662 INFO:tasks.workunit.client.0.smithi103.stdout:4/228: truncate d3/d1e/f50 141392 0 2024-03-22T02:28:34.663 INFO:tasks.workunit.client.0.smithi103.stdout:9/206: symlink d2/d10/d17/l42 0 2024-03-22T02:28:34.664 INFO:tasks.workunit.client.0.smithi103.stdout:6/146: mknod d0/d3/d22/c32 0 2024-03-22T02:28:34.665 INFO:tasks.workunit.client.0.smithi103.stdout:6/147: chown d0/d14/c18 46 1 2024-03-22T02:28:34.672 INFO:tasks.workunit.client.0.smithi103.stdout:9/207: rename d2/da/de to d2/d10/d43 0 2024-03-22T02:28:34.679 INFO:tasks.workunit.client.0.smithi103.stdout:6/148: mkdir d0/d3/d33 0 2024-03-22T02:28:34.680 INFO:tasks.workunit.client.0.smithi103.stdout:4/229: write d3/d1e/f2f [111680,32466] 0 2024-03-22T02:28:34.681 INFO:tasks.workunit.client.0.smithi103.stdout:0/297: dwrite fc [0,4194304] 0 2024-03-22T02:28:34.684 INFO:tasks.workunit.client.0.smithi103.stdout:8/252: rename d4/d13 to d4/d48 0 2024-03-22T02:28:34.690 INFO:tasks.workunit.client.0.smithi103.stdout:9/208: dread d2/d10/d17/f1f [0,4194304] 0 2024-03-22T02:28:34.691 INFO:tasks.workunit.client.0.smithi103.stdout:8/253: mkdir d4/d14/d39/d49 0 2024-03-22T02:28:34.691 INFO:tasks.workunit.client.0.smithi103.stdout:8/254: readlink d4/l20 0 2024-03-22T02:28:34.692 INFO:tasks.workunit.client.0.smithi103.stdout:8/255: read - d4/d14/d39/f45 zero size 2024-03-22T02:28:34.693 INFO:tasks.workunit.client.0.smithi103.stdout:4/230: rename d3/f3c to d3/d31/f57 0 2024-03-22T02:28:34.696 INFO:tasks.workunit.client.0.smithi103.stdout:8/256: mkdir d4/d19/d1a/d4a 0 2024-03-22T02:28:34.698 INFO:tasks.workunit.client.0.smithi103.stdout:4/231: creat d3/d30/d45/f58 x:0 0 0 2024-03-22T02:28:34.700 INFO:tasks.workunit.client.0.smithi103.stdout:4/232: stat d3/l13 0 2024-03-22T02:28:34.701 INFO:tasks.workunit.client.0.smithi103.stdout:4/233: dread d3/d1e/f50 [0,4194304] 0 2024-03-22T02:28:34.704 INFO:tasks.workunit.client.0.smithi103.stdout:6/149: dread d0/d3/d22/f1c [0,4194304] 0 2024-03-22T02:28:34.705 INFO:tasks.workunit.client.0.smithi103.stdout:4/234: readlink d3/d1e/d25/l3e 0 2024-03-22T02:28:34.705 INFO:tasks.workunit.client.0.smithi103.stdout:6/150: write d0/f2c [513910,71790] 0 2024-03-22T02:28:34.706 INFO:tasks.workunit.client.0.smithi103.stdout:8/257: getdents d4/d14/d39 0 2024-03-22T02:28:34.713 INFO:tasks.workunit.client.0.smithi103.stdout:4/235: dwrite d3/d1e/d25/f3d [0,4194304] 0 2024-03-22T02:28:34.715 INFO:tasks.workunit.client.0.smithi103.stdout:4/236: dread - d3/f1f zero size 2024-03-22T02:28:34.716 INFO:tasks.workunit.client.0.smithi103.stdout:6/151: truncate d0/d3/d22/f1c 3342697 0 2024-03-22T02:28:34.718 INFO:tasks.workunit.client.0.smithi103.stdout:6/152: write d0/f2e [4323466,110511] 0 2024-03-22T02:28:34.721 INFO:tasks.workunit.client.0.smithi103.stdout:6/153: write d0/d3/d22/f15 [76358,62281] 0 2024-03-22T02:28:34.722 INFO:tasks.workunit.client.0.smithi103.stdout:8/258: dwrite d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:34.726 INFO:tasks.workunit.client.0.smithi103.stdout:6/154: read d0/d3/d22/f15 [141657,129047] 0 2024-03-22T02:28:34.727 INFO:tasks.workunit.client.0.smithi103.stdout:6/155: chown d0/d14/f30 12 1 2024-03-22T02:28:34.727 INFO:tasks.workunit.client.0.smithi103.stdout:4/237: dread d3/d1e/f20 [0,4194304] 0 2024-03-22T02:28:34.732 INFO:tasks.workunit.client.0.smithi103.stdout:4/238: rename d3/d31/l3f to d3/d1e/d53/l59 0 2024-03-22T02:28:34.734 INFO:tasks.workunit.client.0.smithi103.stdout:6/156: getdents d0 0 2024-03-22T02:28:34.735 INFO:tasks.workunit.client.0.smithi103.stdout:8/259: dwrite d4/fe [0,4194304] 0 2024-03-22T02:28:34.736 INFO:tasks.workunit.client.0.smithi103.stdout:8/260: fdatasync d4/d14/d39/f45 0 2024-03-22T02:28:34.737 INFO:tasks.workunit.client.0.smithi103.stdout:6/157: mknod d0/c34 0 2024-03-22T02:28:34.737 INFO:tasks.workunit.client.0.smithi103.stdout:4/239: creat d3/d1e/d25/d2c/f5a x:0 0 0 2024-03-22T02:28:34.740 INFO:tasks.workunit.client.0.smithi103.stdout:6/158: truncate d0/f23 656058 0 2024-03-22T02:28:34.740 INFO:tasks.workunit.client.0.smithi103.stdout:6/159: write d0/d14/f30 [812264,7767] 0 2024-03-22T02:28:34.744 INFO:tasks.workunit.client.0.smithi103.stdout:8/261: creat d4/d2a/d3e/f4b x:0 0 0 2024-03-22T02:28:34.746 INFO:tasks.workunit.client.0.smithi103.stdout:6/160: link d0/c10 d0/d14/d20/c35 0 2024-03-22T02:28:34.747 INFO:tasks.workunit.client.0.smithi103.stdout:6/161: write d0/d3/f1b [363235,44917] 0 2024-03-22T02:28:34.750 INFO:tasks.workunit.client.0.smithi103.stdout:8/262: truncate d4/d48/f1b 1284511 0 2024-03-22T02:28:34.750 INFO:tasks.workunit.client.0.smithi103.stdout:4/240: rmdir d3/d1e 39 2024-03-22T02:28:34.751 INFO:tasks.workunit.client.0.smithi103.stdout:8/263: readlink d4/d48/l36 0 2024-03-22T02:28:34.752 INFO:tasks.workunit.client.0.smithi103.stdout:6/162: dread d0/f2e [0,4194304] 0 2024-03-22T02:28:34.753 INFO:tasks.workunit.client.0.smithi103.stdout:6/163: dread - d0/d14/f29 zero size 2024-03-22T02:28:34.756 INFO:tasks.workunit.client.0.smithi103.stdout:8/264: mknod d4/d14/c4c 0 2024-03-22T02:28:34.756 INFO:tasks.workunit.client.0.smithi103.stdout:8/265: chown f2 1598000 1 2024-03-22T02:28:34.757 INFO:tasks.workunit.client.0.smithi103.stdout:4/241: write d3/f8 [4122633,99642] 0 2024-03-22T02:28:34.759 INFO:tasks.workunit.client.0.smithi103.stdout:4/242: symlink d3/d31/l5b 0 2024-03-22T02:28:34.761 INFO:tasks.workunit.client.0.smithi103.stdout:8/266: getdents d4/d26 0 2024-03-22T02:28:34.762 INFO:tasks.workunit.client.0.smithi103.stdout:8/267: write d4/f34 [2233633,4124] 0 2024-03-22T02:28:34.764 INFO:tasks.workunit.client.0.smithi103.stdout:8/268: write d4/d14/f1e [1013960,60900] 0 2024-03-22T02:28:34.767 INFO:tasks.workunit.client.0.smithi103.stdout:8/269: write d4/ff [5829316,66309] 0 2024-03-22T02:28:34.769 INFO:tasks.workunit.client.0.smithi103.stdout:4/243: dwrite d3/d30/d45/f58 [0,4194304] 0 2024-03-22T02:28:34.770 INFO:tasks.workunit.client.0.smithi103.stdout:8/270: fsync d4/d19/d1a/f37 0 2024-03-22T02:28:34.771 INFO:tasks.workunit.client.0.smithi103.stdout:8/271: fdatasync d4/d48/d35/f44 0 2024-03-22T02:28:34.772 INFO:tasks.workunit.client.0.smithi103.stdout:8/272: write d4/f34 [1408013,80259] 0 2024-03-22T02:28:34.773 INFO:tasks.workunit.client.0.smithi103.stdout:8/273: read - d4/d2a/d3e/f4b zero size 2024-03-22T02:28:34.774 INFO:tasks.workunit.client.0.smithi103.stdout:4/244: mkdir d3/d38/d5c 0 2024-03-22T02:28:34.778 INFO:tasks.workunit.client.0.smithi103.stdout:4/245: creat d3/d30/f5d x:0 0 0 2024-03-22T02:28:34.780 INFO:tasks.workunit.client.0.smithi103.stdout:4/246: write d3/f7 [8065,87945] 0 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:1/233: sync 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:7/208: sync 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:3/264: sync 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:2/230: sync 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:5/207: sync 2024-03-22T02:28:34.785 INFO:tasks.workunit.client.0.smithi103.stdout:4/247: rename d3/d1e/d25/d2c to d3/d1e/d25/d5e 0 2024-03-22T02:28:34.786 INFO:tasks.workunit.client.0.smithi103.stdout:5/208: chown d0/d14/d23/d46 915 1 2024-03-22T02:28:34.788 INFO:tasks.workunit.client.0.smithi103.stdout:3/265: write de/d1a/f31 [4559909,8417] 0 2024-03-22T02:28:34.788 INFO:tasks.workunit.client.0.smithi103.stdout:4/248: unlink d3/d1e/f50 0 2024-03-22T02:28:34.790 INFO:tasks.workunit.client.0.smithi103.stdout:7/209: creat d1/f4e x:0 0 0 2024-03-22T02:28:34.794 INFO:tasks.workunit.client.0.smithi103.stdout:1/234: mkdir db/d10/d18/d2d/d56 0 2024-03-22T02:28:34.794 INFO:tasks.workunit.client.0.smithi103.stdout:4/249: write d3/d1e/f2f [300277,106666] 0 2024-03-22T02:28:34.795 INFO:tasks.workunit.client.0.smithi103.stdout:3/266: write de/d1a/d25/d26/f44 [405706,102255] 0 2024-03-22T02:28:34.795 INFO:tasks.workunit.client.0.smithi103.stdout:2/231: unlink d11/d25/d46/d34/f2a 0 2024-03-22T02:28:34.799 INFO:tasks.workunit.client.0.smithi103.stdout:4/250: dread d3/fb [0,4194304] 0 2024-03-22T02:28:34.799 INFO:tasks.workunit.client.0.smithi103.stdout:2/232: read - d11/d25/f35 zero size 2024-03-22T02:28:34.799 INFO:tasks.workunit.client.0.smithi103.stdout:8/274: dwrite d4/d19/d1a/f2d [0,4194304] 0 2024-03-22T02:28:34.801 INFO:tasks.workunit.client.0.smithi103.stdout:2/233: stat d11/d25/d46/d34/l38 0 2024-03-22T02:28:34.804 INFO:tasks.workunit.client.0.smithi103.stdout:7/210: dread d1/d39/f35 [0,4194304] 0 2024-03-22T02:28:34.806 INFO:tasks.workunit.client.0.smithi103.stdout:5/209: rename d0/d14/d23/d46/f47 to d0/d14/d23/d2a/d2c/f4d 0 2024-03-22T02:28:34.807 INFO:tasks.workunit.client.0.smithi103.stdout:1/235: creat db/d10/d29/d41/f57 x:0 0 0 2024-03-22T02:28:34.810 INFO:tasks.workunit.client.0.smithi103.stdout:4/251: symlink d3/d1e/d53/l5f 0 2024-03-22T02:28:34.811 INFO:tasks.workunit.client.0.smithi103.stdout:8/275: creat d4/d26/d27/f4d x:0 0 0 2024-03-22T02:28:34.815 INFO:tasks.workunit.client.0.smithi103.stdout:3/267: rename de/d12/d19/l57 to de/d12/d47/d4f/l5e 0 2024-03-22T02:28:34.816 INFO:tasks.workunit.client.0.smithi103.stdout:5/210: rmdir d0/d14/d23/d2a/d33 39 2024-03-22T02:28:34.819 INFO:tasks.workunit.client.0.smithi103.stdout:4/252: mknod d3/d1e/d25/d5e/c60 0 2024-03-22T02:28:34.822 INFO:tasks.workunit.client.0.smithi103.stdout:7/211: mknod d1/da/d1c/c4f 0 2024-03-22T02:28:34.823 INFO:tasks.workunit.client.0.smithi103.stdout:7/212: dread - d1/d39/f46 zero size 2024-03-22T02:28:34.824 INFO:tasks.workunit.client.0.smithi103.stdout:7/213: fsync d1/da/f3 0 2024-03-22T02:28:34.828 INFO:tasks.workunit.client.0.smithi103.stdout:4/253: symlink d3/d38/l61 0 2024-03-22T02:28:34.829 INFO:tasks.workunit.client.0.smithi103.stdout:4/254: fdatasync d3/d31/f47 0 2024-03-22T02:28:34.830 INFO:tasks.workunit.client.0.smithi103.stdout:2/234: truncate d11/d25/d46/f37 2576203 0 2024-03-22T02:28:34.830 INFO:tasks.workunit.client.0.smithi103.stdout:2/235: read - d11/f3a zero size 2024-03-22T02:28:34.831 INFO:tasks.workunit.client.0.smithi103.stdout:8/276: rename d4/d48/f1b to d4/d26/f4e 0 2024-03-22T02:28:34.833 INFO:tasks.workunit.client.0.smithi103.stdout:8/277: read d4/fd [3271801,116317] 0 2024-03-22T02:28:34.833 INFO:tasks.workunit.client.0.smithi103.stdout:5/211: write d0/d14/f22 [62695,72429] 0 2024-03-22T02:28:34.838 INFO:tasks.workunit.client.0.smithi103.stdout:2/236: symlink d11/d19/l4a 0 2024-03-22T02:28:34.839 INFO:tasks.workunit.client.0.smithi103.stdout:3/268: creat de/f5f x:0 0 0 2024-03-22T02:28:34.842 INFO:tasks.workunit.client.0.smithi103.stdout:5/212: stat d0/cb 0 2024-03-22T02:28:34.842 INFO:tasks.workunit.client.0.smithi103.stdout:7/214: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:34.843 INFO:tasks.workunit.client.0.smithi103.stdout:5/213: fdatasync d0/f1e 0 2024-03-22T02:28:34.843 INFO:tasks.workunit.client.0.smithi103.stdout:8/278: mkdir d4/d2a/d4f 0 2024-03-22T02:28:34.846 INFO:tasks.workunit.client.0.smithi103.stdout:2/237: unlink d11/d25/d46/d34/l3f 0 2024-03-22T02:28:34.847 INFO:tasks.workunit.client.0.smithi103.stdout:1/236: truncate db/d4a/f53 774981 0 2024-03-22T02:28:34.850 INFO:tasks.workunit.client.0.smithi103.stdout:3/269: mknod de/d12/d30/c60 0 2024-03-22T02:28:34.853 INFO:tasks.workunit.client.0.smithi103.stdout:3/270: write de/d12/d47/d4f/f5b [717481,128779] 0 2024-03-22T02:28:34.853 INFO:tasks.workunit.client.0.smithi103.stdout:5/214: unlink d0/cb 0 2024-03-22T02:28:34.854 INFO:tasks.workunit.client.0.smithi103.stdout:3/271: fsync de/f23 0 2024-03-22T02:28:34.855 INFO:tasks.workunit.client.0.smithi103.stdout:7/215: symlink d1/d10/d11/d19/d29/d33/d4a/l50 0 2024-03-22T02:28:34.856 INFO:tasks.workunit.client.0.smithi103.stdout:2/238: mknod d11/d25/d46/d34/d24/c4b 0 2024-03-22T02:28:34.858 INFO:tasks.workunit.client.0.smithi103.stdout:4/255: dwrite d3/d30/f55 [0,4194304] 0 2024-03-22T02:28:34.859 INFO:tasks.workunit.client.0.smithi103.stdout:1/237: rename db/d10/d18/f3f to db/d10/d29/d55/f58 0 2024-03-22T02:28:34.860 INFO:tasks.workunit.client.0.smithi103.stdout:4/256: chown d3/d1e/d25/d5e/c60 24547133 1 2024-03-22T02:28:34.861 INFO:tasks.workunit.client.0.smithi103.stdout:1/238: write db/d10/d18/f40 [2310717,51814] 0 2024-03-22T02:28:34.863 INFO:tasks.workunit.client.0.smithi103.stdout:3/272: unlink de/d12/d47/f5c 0 2024-03-22T02:28:34.864 INFO:tasks.workunit.client.0.smithi103.stdout:3/273: write de/d1a/d25/d26/f44 [1139921,75375] 0 2024-03-22T02:28:34.868 INFO:tasks.workunit.client.0.smithi103.stdout:2/239: rename d11/f12 to d11/d25/f4c 0 2024-03-22T02:28:34.870 INFO:tasks.workunit.client.0.smithi103.stdout:1/239: truncate db/f2b 37234 0 2024-03-22T02:28:34.871 INFO:tasks.workunit.client.0.smithi103.stdout:7/216: symlink d1/d10/d11/l51 0 2024-03-22T02:28:34.872 INFO:tasks.workunit.client.0.smithi103.stdout:3/274: dwrite de/d1a/d25/d26/f53 [0,4194304] 0 2024-03-22T02:28:34.872 INFO:tasks.workunit.client.0.smithi103.stdout:7/217: readlink d1/d10/d11/l51 0 2024-03-22T02:28:34.872 INFO:tasks.workunit.client.0.smithi103.stdout:1/240: truncate db/d10/f26 5036943 0 2024-03-22T02:28:34.873 INFO:tasks.workunit.client.0.smithi103.stdout:7/218: chown d1/da/d1c/l47 82000211 1 2024-03-22T02:28:34.874 INFO:tasks.workunit.client.0.smithi103.stdout:7/219: stat d1/d39 0 2024-03-22T02:28:34.875 INFO:tasks.workunit.client.0.smithi103.stdout:5/215: creat d0/d14/f4e x:0 0 0 2024-03-22T02:28:34.877 INFO:tasks.workunit.client.0.smithi103.stdout:4/257: rename d3/l19 to d3/d1e/d53/l62 0 2024-03-22T02:28:34.880 INFO:tasks.workunit.client.0.smithi103.stdout:2/240: symlink d11/d25/d46/d48/l4d 0 2024-03-22T02:28:34.884 INFO:tasks.workunit.client.0.smithi103.stdout:1/241: creat db/d10/d21/f59 x:0 0 0 2024-03-22T02:28:34.885 INFO:tasks.workunit.client.0.smithi103.stdout:5/216: symlink d0/d14/d23/d2a/d2c/l4f 0 2024-03-22T02:28:34.887 INFO:tasks.workunit.client.0.smithi103.stdout:7/220: creat d1/d10/d11/d19/d29/f52 x:0 0 0 2024-03-22T02:28:34.889 INFO:tasks.workunit.client.0.smithi103.stdout:4/258: unlink d3/d38/c41 0 2024-03-22T02:28:34.891 INFO:tasks.workunit.client.0.smithi103.stdout:2/241: mkdir d11/d19/d4e 0 2024-03-22T02:28:34.892 INFO:tasks.workunit.client.0.smithi103.stdout:5/217: dwrite d0/f12 [0,4194304] 0 2024-03-22T02:28:34.893 INFO:tasks.workunit.client.0.smithi103.stdout:2/242: chown d11/d25/d46/d34/l38 52841160 1 2024-03-22T02:28:34.896 INFO:tasks.workunit.client.0.smithi103.stdout:7/221: chown d1/l6 30856927 1 2024-03-22T02:28:34.901 INFO:tasks.workunit.client.0.smithi103.stdout:3/275: getdents de/d12 0 2024-03-22T02:28:34.905 INFO:tasks.workunit.client.0.smithi103.stdout:2/243: symlink d11/d19/d4e/l4f 0 2024-03-22T02:28:34.906 INFO:tasks.workunit.client.0.smithi103.stdout:1/242: dwrite db/f17 [0,4194304] 0 2024-03-22T02:28:34.908 INFO:tasks.workunit.client.0.smithi103.stdout:2/244: readlink d11/d25/d46/l26 0 2024-03-22T02:28:34.911 INFO:tasks.workunit.client.0.smithi103.stdout:4/259: dread d3/d31/f32 [0,4194304] 0 2024-03-22T02:28:34.911 INFO:tasks.workunit.client.0.smithi103.stdout:7/222: dwrite d1/da/f8 [0,4194304] 0 2024-03-22T02:28:34.915 INFO:tasks.workunit.client.0.smithi103.stdout:3/276: getdents de/d1a/d25/d3f 0 2024-03-22T02:28:34.919 INFO:tasks.workunit.client.0.smithi103.stdout:7/223: rmdir d1/d10 39 2024-03-22T02:28:34.920 INFO:tasks.workunit.client.0.smithi103.stdout:1/243: rmdir db/d10/d18/d2d/d56 0 2024-03-22T02:28:34.923 INFO:tasks.workunit.client.0.smithi103.stdout:3/277: mkdir de/d1a/d25/d61 0 2024-03-22T02:28:34.924 INFO:tasks.workunit.client.0.smithi103.stdout:1/244: fdatasync db/d10/d29/d41/f57 0 2024-03-22T02:28:34.925 INFO:tasks.workunit.client.0.smithi103.stdout:2/245: dwrite d11/d25/f35 [0,4194304] 0 2024-03-22T02:28:34.926 INFO:tasks.workunit.client.0.smithi103.stdout:7/224: chown d1/da/d2f/d36/l37 1 1 2024-03-22T02:28:34.927 INFO:tasks.workunit.client.0.smithi103.stdout:4/260: dwrite d3/f1f [0,4194304] 0 2024-03-22T02:28:34.928 INFO:tasks.workunit.client.0.smithi103.stdout:1/245: dread - db/d10/d28/f43 zero size 2024-03-22T02:28:34.929 INFO:tasks.workunit.client.0.smithi103.stdout:7/225: fsync d1/d39/d21/f32 0 2024-03-22T02:28:34.931 INFO:tasks.workunit.client.0.smithi103.stdout:7/226: write d1/d39/d1d/f41 [492089,7683] 0 2024-03-22T02:28:34.935 INFO:tasks.workunit.client.0.smithi103.stdout:2/246: write d11/d25/d43/d45/d47/f49 [3658460,87520] 0 2024-03-22T02:28:34.937 INFO:tasks.workunit.client.0.smithi103.stdout:4/261: getdents d3/d1e/d53 0 2024-03-22T02:28:34.938 INFO:tasks.workunit.client.0.smithi103.stdout:2/247: readlink lc 0 2024-03-22T02:28:34.940 INFO:tasks.workunit.client.0.smithi103.stdout:5/218: dread d0/fe [0,4194304] 0 2024-03-22T02:28:34.941 INFO:tasks.workunit.client.0.smithi103.stdout:4/262: rename d3/d1e/f39 to d3/d38/f63 0 2024-03-22T02:28:34.942 INFO:tasks.workunit.client.0.smithi103.stdout:5/219: truncate d0/d14/f22 351494 0 2024-03-22T02:28:34.946 INFO:tasks.workunit.client.0.smithi103.stdout:7/227: getdents d1/d10/d11 0 2024-03-22T02:28:34.947 INFO:tasks.workunit.client.0.smithi103.stdout:7/228: chown d1/d39/c31 289002 1 2024-03-22T02:28:34.948 INFO:tasks.workunit.client.0.smithi103.stdout:5/220: mkdir d0/d14/d23/d36/d50 0 2024-03-22T02:28:34.951 INFO:tasks.workunit.client.0.smithi103.stdout:4/263: symlink d3/d30/l64 0 2024-03-22T02:28:34.951 INFO:tasks.workunit.client.0.smithi103.stdout:1/246: link db/d10/f12 db/d10/f5a 0 2024-03-22T02:28:34.955 INFO:tasks.workunit.client.0.smithi103.stdout:1/247: write db/d10/d18/d2d/f48 [397493,54575] 0 2024-03-22T02:28:34.956 INFO:tasks.workunit.client.0.smithi103.stdout:5/221: creat d0/d14/f51 x:0 0 0 2024-03-22T02:28:34.957 INFO:tasks.workunit.client.0.smithi103.stdout:7/229: dwrite d1/d39/f46 [0,4194304] 0 2024-03-22T02:28:34.958 INFO:tasks.workunit.client.0.smithi103.stdout:4/264: mkdir d3/d1e/d25/d65 0 2024-03-22T02:28:34.959 INFO:tasks.workunit.client.0.smithi103.stdout:4/265: stat d3/d31/f47 0 2024-03-22T02:28:34.959 INFO:tasks.workunit.client.0.smithi103.stdout:1/248: dread - db/d10/d28/f43 zero size 2024-03-22T02:28:34.962 INFO:tasks.workunit.client.0.smithi103.stdout:5/222: write d0/f2d [746806,30510] 0 2024-03-22T02:28:34.964 INFO:tasks.workunit.client.0.smithi103.stdout:1/249: fsync db/d10/d18/f40 0 2024-03-22T02:28:34.966 INFO:tasks.workunit.client.0.smithi103.stdout:4/266: rename d3/d30/l64 to d3/d1e/d25/d65/l66 0 2024-03-22T02:28:34.968 INFO:tasks.workunit.client.0.smithi103.stdout:5/223: unlink d0/f17 0 2024-03-22T02:28:34.969 INFO:tasks.workunit.client.0.smithi103.stdout:5/224: truncate d0/f1e 1213807 0 2024-03-22T02:28:34.972 INFO:tasks.workunit.client.0.smithi103.stdout:7/230: fsync d1/d39/f46 0 2024-03-22T02:28:34.972 INFO:tasks.workunit.client.0.smithi103.stdout:5/225: dread d0/d3e/f30 [0,4194304] 0 2024-03-22T02:28:34.972 INFO:tasks.workunit.client.0.smithi103.stdout:4/267: symlink d3/d1e/d53/l67 0 2024-03-22T02:28:34.972 INFO:tasks.workunit.client.0.smithi103.stdout:7/231: readlink d1/l6 0 2024-03-22T02:28:34.973 INFO:tasks.workunit.client.0.smithi103.stdout:1/250: link db/d10/d29/f39 db/d10/d21/f5b 0 2024-03-22T02:28:34.974 INFO:tasks.workunit.client.0.smithi103.stdout:4/268: mkdir d3/d1e/d68 0 2024-03-22T02:28:34.976 INFO:tasks.workunit.client.0.smithi103.stdout:5/226: mknod d0/d14/d23/d36/c52 0 2024-03-22T02:28:34.976 INFO:tasks.workunit.client.0.smithi103.stdout:7/232: creat d1/d39/d1d/f53 x:0 0 0 2024-03-22T02:28:34.978 INFO:tasks.workunit.client.0.smithi103.stdout:1/251: symlink db/d10/d2e/l5c 0 2024-03-22T02:28:34.984 INFO:tasks.workunit.client.0.smithi103.stdout:4/269: dwrite d3/d1e/d25/f4d [0,4194304] 0 2024-03-22T02:28:34.989 INFO:tasks.workunit.client.0.smithi103.stdout:4/270: write d3/d1e/d25/f4d [624716,53915] 0 2024-03-22T02:28:34.989 INFO:tasks.workunit.client.0.smithi103.stdout:7/233: dwrite d1/da/f8 [0,4194304] 0 2024-03-22T02:28:34.990 INFO:tasks.workunit.client.0.smithi103.stdout:5/227: dwrite d0/d14/f51 [0,4194304] 0 2024-03-22T02:28:34.993 INFO:tasks.workunit.client.0.smithi103.stdout:1/252: truncate db/d10/f1c 293268 0 2024-03-22T02:28:35.000 INFO:tasks.workunit.client.0.smithi103.stdout:5/228: dwrite d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:35.001 INFO:tasks.workunit.client.0.smithi103.stdout:1/253: creat db/d10/d18/d2d/f5d x:0 0 0 2024-03-22T02:28:35.002 INFO:tasks.workunit.client.0.smithi103.stdout:4/271: rename d3/d1e/d25/d5e to d3/d69 0 2024-03-22T02:28:35.008 INFO:tasks.workunit.client.0.smithi103.stdout:1/254: dread db/f17 [0,4194304] 0 2024-03-22T02:28:35.010 INFO:tasks.workunit.client.0.smithi103.stdout:5/229: getdents d0/d14/d23 0 2024-03-22T02:28:35.012 INFO:tasks.workunit.client.1.smithi122.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-22T02:28:35.014 INFO:tasks.workunit.client.0.smithi103.stdout:4/272: dwrite d3/d1e/d25/f3d [0,4194304] 0 2024-03-22T02:28:35.019 INFO:tasks.workunit.client.0.smithi103.stdout:5/230: dwrite d0/f2d [0,4194304] 0 2024-03-22T02:28:35.020 INFO:tasks.workunit.client.0.smithi103.stdout:4/273: rename d3/c10 to d3/d30/d45/c6a 0 2024-03-22T02:28:35.022 INFO:tasks.workunit.client.0.smithi103.stdout:4/274: fdatasync d3/d69/f54 0 2024-03-22T02:28:35.030 INFO:tasks.workunit.client.0.smithi103.stdout:4/275: dread d3/d1e/d25/f4d [0,4194304] 0 2024-03-22T02:28:35.030 INFO:tasks.workunit.client.1.smithi122.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-22T02:28:35.031 INFO:tasks.workunit.client.1.smithi122.stderr:+ make 2024-03-22T02:28:35.031 INFO:tasks.workunit.client.0.smithi103.stdout:5/231: dwrite d0/d14/f41 [4194304,4194304] 0 2024-03-22T02:28:35.031 INFO:tasks.workunit.client.0.smithi103.stdout:4/276: chown d3/d31 308653 1 2024-03-22T02:28:35.032 INFO:tasks.workunit.client.0.smithi103.stdout:5/232: chown d0/d14/f3a 3234 1 2024-03-22T02:28:35.033 INFO:tasks.workunit.client.0.smithi103.stdout:5/233: write d0/d14/f22 [1019137,103701] 0 2024-03-22T02:28:35.034 INFO:tasks.workunit.client.0.smithi103.stdout:4/277: mknod d3/d38/d5c/c6b 0 2024-03-22T02:28:35.035 INFO:tasks.workunit.client.0.smithi103.stdout:4/278: chown d3/d1e/f43 4 1 2024-03-22T02:28:35.037 INFO:tasks.workunit.client.0.smithi103.stdout:5/234: symlink d0/d14/d23/d46/l53 0 2024-03-22T02:28:35.040 INFO:tasks.workunit.client.0.smithi103.stdout:4/279: rename d3/d69/f4f to d3/d1e/d25/d65/f6c 0 2024-03-22T02:28:35.044 INFO:tasks.workunit.client.0.smithi103.stdout:5/235: dwrite d0/d14/f22 [0,4194304] 0 2024-03-22T02:28:35.049 INFO:tasks.workunit.client.0.smithi103.stdout:4/280: dwrite d3/d31/f37 [4194304,4194304] 0 2024-03-22T02:28:35.051 INFO:tasks.workunit.client.0.smithi103.stdout:4/281: fdatasync d3/d38/f4c 0 2024-03-22T02:28:35.054 INFO:tasks.workunit.client.0.smithi103.stdout:4/282: creat d3/d1e/f6d x:0 0 0 2024-03-22T02:28:35.055 INFO:tasks.workunit.client.0.smithi103.stdout:5/236: dwrite d0/d14/d23/d2a/f1a [4194304,4194304] 0 2024-03-22T02:28:35.059 INFO:tasks.workunit.client.0.smithi103.stdout:5/237: creat d0/d14/d23/d36/f54 x:0 0 0 2024-03-22T02:28:35.063 INFO:tasks.workunit.client.0.smithi103.stdout:4/283: dwrite d3/d69/f3b [4194304,4194304] 0 2024-03-22T02:28:35.065 INFO:tasks.workunit.client.0.smithi103.stdout:5/238: fsync d0/d14/d23/d2a/d1f/f32 0 2024-03-22T02:28:35.067 INFO:tasks.workunit.client.0.smithi103.stdout:4/284: mkdir d3/d30/d45/d6e 0 2024-03-22T02:28:35.072 INFO:tasks.workunit.client.0.smithi103.stdout:5/239: rename d0/c1b to d0/d14/d23/d46/c55 0 2024-03-22T02:28:35.074 INFO:tasks.workunit.client.0.smithi103.stdout:5/240: unlink d0/d14/l18 0 2024-03-22T02:28:35.074 INFO:tasks.workunit.client.0.smithi103.stdout:5/241: fdatasync d0/d14/f3a 0 2024-03-22T02:28:35.075 INFO:tasks.workunit.client.0.smithi103.stdout:5/242: chown d0/d14/c42 29 1 2024-03-22T02:28:35.076 INFO:tasks.workunit.client.0.smithi103.stdout:4/285: getdents d3/d31 0 2024-03-22T02:28:35.077 INFO:tasks.workunit.client.0.smithi103.stdout:4/286: chown d3/d1e/f43 493 1 2024-03-22T02:28:35.077 INFO:tasks.workunit.client.0.smithi103.stdout:5/243: symlink d0/d14/d23/d2a/d2c/l56 0 2024-03-22T02:28:35.081 INFO:tasks.workunit.client.0.smithi103.stdout:5/244: mkdir d0/d14/d23/d2a/d33/d57 0 2024-03-22T02:28:35.081 INFO:tasks.workunit.client.0.smithi103.stdout:4/287: dread d3/d38/f63 [0,4194304] 0 2024-03-22T02:28:35.083 INFO:tasks.workunit.client.0.smithi103.stdout:4/288: creat d3/d31/f6f x:0 0 0 2024-03-22T02:28:35.084 INFO:tasks.workunit.client.0.smithi103.stdout:5/245: unlink d0/f7 0 2024-03-22T02:28:35.086 INFO:tasks.workunit.client.0.smithi103.stdout:5/246: mknod d0/d14/d23/d46/c58 0 2024-03-22T02:28:35.089 INFO:tasks.workunit.client.0.smithi103.stdout:4/289: dwrite d3/d1e/f43 [0,4194304] 0 2024-03-22T02:28:35.089 INFO:tasks.workunit.client.0.smithi103.stdout:5/247: dread d0/d14/d23/d2a/f1c [0,4194304] 0 2024-03-22T02:28:35.090 INFO:tasks.workunit.client.0.smithi103.stdout:4/290: dread - d3/d1e/f6d zero size 2024-03-22T02:28:35.091 INFO:tasks.workunit.client.0.smithi103.stdout:5/248: symlink d0/d14/d23/d2a/d1f/l59 0 2024-03-22T02:28:35.092 INFO:tasks.workunit.client.0.smithi103.stdout:4/291: creat d3/d38/f70 x:0 0 0 2024-03-22T02:28:35.094 INFO:tasks.workunit.client.0.smithi103.stdout:5/249: creat d0/d14/d23/d36/d50/f5a x:0 0 0 2024-03-22T02:28:35.095 INFO:tasks.workunit.client.0.smithi103.stdout:5/250: mknod d0/c5b 0 2024-03-22T02:28:35.096 INFO:tasks.workunit.client.0.smithi103.stdout:4/292: creat d3/d1e/d68/f71 x:0 0 0 2024-03-22T02:28:35.098 INFO:tasks.workunit.client.0.smithi103.stdout:4/293: mknod d3/d1e/d25/c72 0 2024-03-22T02:28:35.101 INFO:tasks.workunit.client.0.smithi103.stdout:4/294: chown d3/le 15183998 1 2024-03-22T02:28:35.102 INFO:tasks.workunit.client.0.smithi103.stdout:4/295: readlink d3/d30/d45/l4a 0 2024-03-22T02:28:35.103 INFO:tasks.workunit.client.0.smithi103.stdout:5/251: dwrite d0/d14/f4e [0,4194304] 0 2024-03-22T02:28:35.104 INFO:tasks.workunit.client.0.smithi103.stdout:4/296: symlink d3/d69/l73 0 2024-03-22T02:28:35.106 INFO:tasks.workunit.client.0.smithi103.stdout:5/252: mkdir d0/d14/d23/d5c 0 2024-03-22T02:28:35.107 INFO:tasks.workunit.client.0.smithi103.stdout:4/297: fdatasync d3/d1e/d25/d65/f6c 0 2024-03-22T02:28:35.111 INFO:tasks.workunit.client.0.smithi103.stdout:5/253: creat d0/d14/d23/d2a/d2c/f5d x:0 0 0 2024-03-22T02:28:35.112 INFO:tasks.workunit.client.0.smithi103.stdout:5/254: chown d0/d14/d23/d2a/f1a 546 1 2024-03-22T02:28:35.112 INFO:tasks.workunit.client.0.smithi103.stdout:4/298: write d3/fd [2319064,10333] 0 2024-03-22T02:28:35.115 INFO:tasks.workunit.client.0.smithi103.stdout:5/255: symlink d0/d14/d23/l5e 0 2024-03-22T02:28:35.116 INFO:tasks.workunit.client.0.smithi103.stdout:4/299: read d3/d1e/f36 [3014852,48861] 0 2024-03-22T02:28:35.118 INFO:tasks.workunit.client.0.smithi103.stdout:4/300: symlink d3/d69/l74 0 2024-03-22T02:28:35.121 INFO:tasks.workunit.client.0.smithi103.stdout:0/298: sync 2024-03-22T02:28:35.122 INFO:tasks.workunit.client.0.smithi103.stdout:5/256: dwrite d0/f12 [0,4194304] 0 2024-03-22T02:28:35.122 INFO:tasks.workunit.client.0.smithi103.stdout:0/299: dread - dd/d16/d20/d29/f3e zero size 2024-03-22T02:28:35.123 INFO:tasks.workunit.client.0.smithi103.stdout:0/300: dread - dd/d16/d20/d29/f5a zero size 2024-03-22T02:28:35.123 INFO:tasks.workunit.client.0.smithi103.stdout:5/257: dread d0/d14/d23/d2a/f1c [0,4194304] 0 2024-03-22T02:28:35.126 INFO:tasks.workunit.client.0.smithi103.stdout:0/301: unlink dd/d16/d20/c2d 0 2024-03-22T02:28:35.128 INFO:tasks.workunit.client.0.smithi103.stdout:5/258: dread d0/f12 [0,4194304] 0 2024-03-22T02:28:35.129 INFO:tasks.workunit.client.0.smithi103.stdout:0/302: rename dd/d16/d20/d29/d48/c51 to dd/d16/d20/d31/c5e 0 2024-03-22T02:28:35.158 INFO:tasks.workunit.client.0.smithi103.stdout:0/303: symlink dd/d16/l5f 0 2024-03-22T02:28:35.160 INFO:tasks.workunit.client.0.smithi103.stdout:5/259: rename d0/d14/d23/d2a/d2c/l4f to d0/d14/d23/d2a/l5f 0 2024-03-22T02:28:35.161 INFO:tasks.workunit.client.0.smithi103.stdout:5/260: chown d0/d14/d23/d46 450559721 1 2024-03-22T02:28:35.168 INFO:tasks.workunit.client.0.smithi103.stdout:5/261: dwrite d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:35.171 INFO:tasks.workunit.client.0.smithi103.stdout:0/304: truncate dd/f15 1834786 0 2024-03-22T02:28:35.176 INFO:tasks.workunit.client.0.smithi103.stdout:0/305: link dd/d16/d20/f23 dd/d34/f60 0 2024-03-22T02:28:35.176 INFO:tasks.workunit.client.1.smithi122.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-22T02:28:35.176 INFO:tasks.workunit.client.0.smithi103.stdout:5/262: dwrite d0/f12 [4194304,4194304] 0 2024-03-22T02:28:35.178 INFO:tasks.workunit.client.0.smithi103.stdout:0/306: write dd/d16/d20/d29/f42 [1933118,105713] 0 2024-03-22T02:28:35.179 INFO:tasks.workunit.client.0.smithi103.stdout:5/263: write d0/d14/d23/d36/f54 [64863,57937] 0 2024-03-22T02:28:35.180 INFO:tasks.workunit.client.0.smithi103.stdout:5/264: write d0/d14/f51 [4952376,129747] 0 2024-03-22T02:28:35.182 INFO:tasks.workunit.client.0.smithi103.stdout:0/307: mknod dd/d16/d20/d29/d48/c61 0 2024-03-22T02:28:35.184 INFO:tasks.workunit.client.0.smithi103.stdout:0/308: symlink dd/l62 0 2024-03-22T02:28:35.187 INFO:tasks.workunit.client.0.smithi103.stdout:5/265: dwrite d0/f1e [0,4194304] 0 2024-03-22T02:28:35.192 INFO:tasks.workunit.client.0.smithi103.stdout:5/266: symlink d0/d14/l60 0 2024-03-22T02:28:35.192 INFO:tasks.workunit.client.0.smithi103.stdout:5/267: chown d0 452 1 2024-03-22T02:28:35.193 INFO:tasks.workunit.client.0.smithi103.stdout:0/309: dwrite dd/d16/f55 [0,4194304] 0 2024-03-22T02:28:35.195 INFO:tasks.workunit.client.0.smithi103.stdout:0/310: dread - dd/d16/d20/d31/f58 zero size 2024-03-22T02:28:35.199 INFO:tasks.workunit.client.0.smithi103.stdout:5/268: rename d0/d14/f3a to d0/d14/d23/d46/f61 0 2024-03-22T02:28:35.200 INFO:tasks.workunit.client.0.smithi103.stdout:5/269: fdatasync d0/d14/d23/d2a/d1f/f20 0 2024-03-22T02:28:35.216 INFO:tasks.workunit.client.0.smithi103.stdout:5/270: truncate d0/d14/f41 605109 0 2024-03-22T02:28:35.220 INFO:tasks.workunit.client.0.smithi103.stdout:5/271: dread d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:35.222 INFO:tasks.workunit.client.0.smithi103.stdout:5/272: mknod d0/d14/d23/d46/c62 0 2024-03-22T02:28:35.224 INFO:tasks.workunit.client.0.smithi103.stdout:0/311: dread dd/d16/d20/d31/f32 [0,4194304] 0 2024-03-22T02:28:35.224 INFO:tasks.workunit.client.0.smithi103.stdout:5/273: creat d0/d14/d23/d2a/f63 x:0 0 0 2024-03-22T02:28:35.226 INFO:tasks.workunit.client.0.smithi103.stdout:0/312: symlink dd/d16/d20/d29/d48/l63 0 2024-03-22T02:28:35.227 INFO:tasks.workunit.client.0.smithi103.stdout:5/274: truncate d0/d3e/f30 559115 0 2024-03-22T02:28:35.228 INFO:tasks.workunit.client.0.smithi103.stdout:0/313: mkdir dd/d16/d20/d64 0 2024-03-22T02:28:35.230 INFO:tasks.workunit.client.0.smithi103.stdout:0/314: mknod dd/c65 0 2024-03-22T02:28:35.230 INFO:tasks.workunit.client.0.smithi103.stdout:5/275: dread d0/f1e [0,4194304] 0 2024-03-22T02:28:35.231 INFO:tasks.workunit.client.0.smithi103.stdout:0/315: write dd/d16/d20/f5d [958792,107824] 0 2024-03-22T02:28:35.256 INFO:tasks.workunit.client.0.smithi103.stdout:9/209: sync 2024-03-22T02:28:35.285 INFO:tasks.workunit.client.0.smithi103.stdout:9/210: readlink d2/d5/l36 0 2024-03-22T02:28:35.285 INFO:tasks.workunit.client.0.smithi103.stdout:0/316: dread - dd/d16/d20/d29/f4c zero size 2024-03-22T02:28:35.286 INFO:tasks.workunit.client.0.smithi103.stdout:9/211: fsync d2/da/f24 0 2024-03-22T02:28:35.288 INFO:tasks.workunit.client.0.smithi103.stdout:0/317: unlink dd/d34/f60 0 2024-03-22T02:28:35.291 INFO:tasks.workunit.client.0.smithi103.stdout:0/318: mknod dd/c66 0 2024-03-22T02:28:35.291 INFO:tasks.workunit.client.0.smithi103.stdout:5/276: dwrite d0/d14/d23/d2a/d33/f4c [0,4194304] 0 2024-03-22T02:28:35.298 INFO:tasks.workunit.client.0.smithi103.stdout:9/212: dwrite d2/da/d18/f41 [0,4194304] 0 2024-03-22T02:28:35.300 INFO:tasks.workunit.client.0.smithi103.stdout:0/319: rmdir dd/d16/d20/d29/d48 39 2024-03-22T02:28:35.301 INFO:tasks.workunit.client.0.smithi103.stdout:0/320: dread - dd/d16/d20/d31/f58 zero size 2024-03-22T02:28:35.301 INFO:tasks.workunit.client.0.smithi103.stdout:0/321: readlink dd/d16/d20/d35/l3b 0 2024-03-22T02:28:35.303 INFO:tasks.workunit.client.0.smithi103.stdout:5/277: link d0/d3e/l2b d0/d14/d23/d2a/d1f/l64 0 2024-03-22T02:28:35.304 INFO:tasks.workunit.client.0.smithi103.stdout:5/278: dread - d0/d14/d23/d2a/d1f/f38 zero size 2024-03-22T02:28:35.307 INFO:tasks.workunit.client.0.smithi103.stdout:0/322: rename dd/d16/d20/d35/c45 to dd/d16/d20/d64/c67 0 2024-03-22T02:28:35.308 INFO:tasks.workunit.client.0.smithi103.stdout:5/279: symlink d0/d14/d23/d36/d50/l65 0 2024-03-22T02:28:35.313 INFO:tasks.workunit.client.0.smithi103.stdout:9/213: truncate d2/d5/fb 2757072 0 2024-03-22T02:28:35.315 INFO:tasks.workunit.client.0.smithi103.stdout:5/280: rename d0/d14/d23/d36/c49 to d0/d14/d23/d36/d50/c66 0 2024-03-22T02:28:35.316 INFO:tasks.workunit.client.0.smithi103.stdout:9/214: mknod d2/da/d18/d1e/d34/d3d/c44 0 2024-03-22T02:28:35.318 INFO:tasks.workunit.client.0.smithi103.stdout:9/215: readlink d2/d5/l19 0 2024-03-22T02:28:35.319 INFO:tasks.workunit.client.0.smithi103.stdout:9/216: write d2/da/f25 [3271474,19252] 0 2024-03-22T02:28:35.323 INFO:tasks.workunit.client.0.smithi103.stdout:5/281: dwrite d0/d14/d23/d2a/f1c [0,4194304] 0 2024-03-22T02:28:35.332 INFO:tasks.workunit.client.0.smithi103.stdout:9/217: dwrite d2/f3b [0,4194304] 0 2024-03-22T02:28:35.334 INFO:tasks.workunit.client.0.smithi103.stdout:9/218: chown d2/d5/f3a 21 1 2024-03-22T02:28:35.334 INFO:tasks.workunit.client.0.smithi103.stdout:0/323: dwrite fc [0,4194304] 0 2024-03-22T02:28:35.340 INFO:tasks.workunit.client.0.smithi103.stdout:5/282: fdatasync d0/d14/d23/d2a/f1c 0 2024-03-22T02:28:35.340 INFO:tasks.workunit.client.0.smithi103.stdout:0/324: creat dd/d16/d20/f68 x:0 0 0 2024-03-22T02:28:35.345 INFO:tasks.workunit.client.0.smithi103.stdout:5/283: symlink d0/d14/d23/d2a/l67 0 2024-03-22T02:28:35.346 INFO:tasks.workunit.client.0.smithi103.stdout:5/284: stat d0/fe 0 2024-03-22T02:28:35.346 INFO:tasks.workunit.client.0.smithi103.stdout:0/325: mknod dd/d34/d53/c69 0 2024-03-22T02:28:35.353 INFO:tasks.workunit.client.0.smithi103.stdout:5/285: dwrite d0/d14/d23/d2a/f63 [0,4194304] 0 2024-03-22T02:28:35.353 INFO:tasks.workunit.client.0.smithi103.stdout:0/326: dwrite dd/f13 [4194304,4194304] 0 2024-03-22T02:28:35.354 INFO:tasks.workunit.client.0.smithi103.stdout:0/327: stat dd/d16/d20/d31 0 2024-03-22T02:28:35.357 INFO:tasks.workunit.client.0.smithi103.stdout:5/286: creat d0/d14/d23/d2a/d1f/f68 x:0 0 0 2024-03-22T02:28:35.359 INFO:tasks.workunit.client.0.smithi103.stdout:5/287: symlink d0/d14/d23/d36/d50/l69 0 2024-03-22T02:28:35.363 INFO:tasks.workunit.client.0.smithi103.stdout:5/288: dread d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:35.368 INFO:tasks.workunit.client.0.smithi103.stdout:5/289: rename d0/d14/d23/d2a/f1c to d0/d14/d23/d36/f6a 0 2024-03-22T02:28:35.374 INFO:tasks.workunit.client.0.smithi103.stdout:5/290: dwrite d0/d14/f4e [0,4194304] 0 2024-03-22T02:28:35.376 INFO:tasks.workunit.client.0.smithi103.stdout:5/291: write d0/d14/d23/d2a/d1f/f38 [138584,2425] 0 2024-03-22T02:28:35.377 INFO:tasks.workunit.client.0.smithi103.stdout:6/164: sync 2024-03-22T02:28:35.378 INFO:tasks.workunit.client.0.smithi103.stdout:5/292: chown d0/d14/d23/d2a/d33/l37 1462 1 2024-03-22T02:28:35.378 INFO:tasks.workunit.client.0.smithi103.stdout:6/165: chown d0/c34 264454423 1 2024-03-22T02:28:35.380 INFO:tasks.workunit.client.0.smithi103.stdout:6/166: chown d0/d3/d22/c32 13444 1 2024-03-22T02:28:35.380 INFO:tasks.workunit.client.0.smithi103.stdout:5/293: symlink d0/d3e/l6b 0 2024-03-22T02:28:35.382 INFO:tasks.workunit.client.0.smithi103.stdout:6/167: rmdir d0/d3 39 2024-03-22T02:28:35.385 INFO:tasks.workunit.client.0.smithi103.stdout:9/219: dread d2/d5/fb [0,4194304] 0 2024-03-22T02:28:35.386 INFO:tasks.workunit.client.0.smithi103.stdout:9/220: chown d2/l33 844 1 2024-03-22T02:28:35.390 INFO:tasks.workunit.client.0.smithi103.stdout:9/221: mknod d2/da/d18/d1e/d34/c45 0 2024-03-22T02:28:35.394 INFO:tasks.workunit.client.0.smithi103.stdout:9/222: dread d2/da/d18/f21 [4194304,4194304] 0 2024-03-22T02:28:35.395 INFO:tasks.workunit.client.0.smithi103.stdout:9/223: rmdir d2/da/d18/d1e 39 2024-03-22T02:28:35.396 INFO:tasks.workunit.client.0.smithi103.stdout:0/328: dread dd/d16/d20/d29/f2a [0,4194304] 0 2024-03-22T02:28:35.397 INFO:tasks.workunit.client.0.smithi103.stdout:9/224: chown d2/d10/d43/c2d 108673905 1 2024-03-22T02:28:35.397 INFO:tasks.workunit.client.0.smithi103.stdout:5/294: truncate d0/f12 5499495 0 2024-03-22T02:28:35.398 INFO:tasks.workunit.client.0.smithi103.stdout:9/225: chown d2/d5/f3a 0 1 2024-03-22T02:28:35.399 INFO:tasks.workunit.client.0.smithi103.stdout:0/329: write dd/f12 [4060166,114067] 0 2024-03-22T02:28:35.401 INFO:tasks.workunit.client.0.smithi103.stdout:0/330: mkdir dd/d16/d20/d64/d6a 0 2024-03-22T02:28:35.403 INFO:tasks.workunit.client.0.smithi103.stdout:6/168: dread d0/f1a [0,4194304] 0 2024-03-22T02:28:35.408 INFO:tasks.workunit.client.0.smithi103.stdout:0/331: dwrite f4 [0,4194304] 0 2024-03-22T02:28:35.409 INFO:tasks.workunit.client.0.smithi103.stdout:0/332: write dd/f12 [3302904,78414] 0 2024-03-22T02:28:35.410 INFO:tasks.workunit.client.0.smithi103.stdout:3/278: sync 2024-03-22T02:28:35.410 INFO:tasks.workunit.client.0.smithi103.stdout:1/255: sync 2024-03-22T02:28:35.410 INFO:tasks.workunit.client.0.smithi103.stdout:8/279: sync 2024-03-22T02:28:35.410 INFO:tasks.workunit.client.0.smithi103.stdout:2/248: sync 2024-03-22T02:28:35.410 INFO:tasks.workunit.client.0.smithi103.stdout:7/234: sync 2024-03-22T02:28:35.411 INFO:tasks.workunit.client.0.smithi103.stdout:4/301: sync 2024-03-22T02:28:35.413 INFO:tasks.workunit.client.0.smithi103.stdout:3/279: write de/d12/d19/f43 [130843,26363] 0 2024-03-22T02:28:35.414 INFO:tasks.workunit.client.0.smithi103.stdout:6/169: truncate d0/f19 208739 0 2024-03-22T02:28:35.414 INFO:tasks.workunit.client.0.smithi103.stdout:7/235: write d1/da/f8 [4940135,25635] 0 2024-03-22T02:28:35.416 INFO:tasks.workunit.client.0.smithi103.stdout:5/295: dread d0/d14/d23/d46/f61 [0,4194304] 0 2024-03-22T02:28:35.419 INFO:tasks.workunit.client.0.smithi103.stdout:1/256: readlink db/d4a/l52 0 2024-03-22T02:28:35.421 INFO:tasks.workunit.client.0.smithi103.stdout:1/257: write db/d10/d18/d2d/f5d [713478,7809] 0 2024-03-22T02:28:35.424 INFO:tasks.workunit.client.0.smithi103.stdout:3/280: symlink de/d12/d47/d4f/l62 0 2024-03-22T02:28:35.424 INFO:tasks.workunit.client.0.smithi103.stdout:1/258: chown db/d10/f26 194971 1 2024-03-22T02:28:35.426 INFO:tasks.workunit.client.0.smithi103.stdout:8/280: dwrite f1 [0,4194304] 0 2024-03-22T02:28:35.427 INFO:tasks.workunit.client.0.smithi103.stdout:5/296: write d0/d14/d23/d2a/d2c/f3f [4106717,96558] 0 2024-03-22T02:28:35.431 INFO:tasks.workunit.client.0.smithi103.stdout:5/297: write d0/d14/d23/f44 [4568300,91949] 0 2024-03-22T02:28:35.432 INFO:tasks.workunit.client.0.smithi103.stdout:0/333: creat dd/f6b x:0 0 0 2024-03-22T02:28:35.433 INFO:tasks.workunit.client.0.smithi103.stdout:5/298: stat d0/d14/d23/d2a/d33/d57 0 2024-03-22T02:28:35.433 INFO:tasks.workunit.client.0.smithi103.stdout:3/281: stat de/d1a/d25/c41 0 2024-03-22T02:28:35.435 INFO:tasks.workunit.client.0.smithi103.stdout:5/299: fdatasync d0/d14/f51 0 2024-03-22T02:28:35.435 INFO:tasks.workunit.client.0.smithi103.stdout:1/259: creat db/d4a/f5e x:0 0 0 2024-03-22T02:28:35.441 INFO:tasks.workunit.client.0.smithi103.stdout:0/334: dwrite dd/d16/d20/d31/f58 [0,4194304] 0 2024-03-22T02:28:35.442 INFO:tasks.workunit.client.0.smithi103.stdout:3/282: dread - de/f22 zero size 2024-03-22T02:28:35.442 INFO:tasks.workunit.client.0.smithi103.stdout:0/335: write f4 [3625960,31628] 0 2024-03-22T02:28:35.443 INFO:tasks.workunit.client.0.smithi103.stdout:3/283: chown de/d1a/d25/d3f/c5a 23 1 2024-03-22T02:28:35.443 INFO:tasks.workunit.client.0.smithi103.stdout:0/336: truncate dd/f5b 38380 0 2024-03-22T02:28:35.444 INFO:tasks.workunit.client.0.smithi103.stdout:6/170: link d0/f23 d0/d3/d33/f36 0 2024-03-22T02:28:35.446 INFO:tasks.workunit.client.0.smithi103.stdout:5/300: rmdir d0/d14/d23/d2a 39 2024-03-22T02:28:35.449 INFO:tasks.workunit.client.0.smithi103.stdout:8/281: mkdir d4/d19/d1a/d4a/d50 0 2024-03-22T02:28:35.451 INFO:tasks.workunit.client.0.smithi103.stdout:2/249: link d11/d25/d46/d34/c3c d11/d25/d46/d34/c50 0 2024-03-22T02:28:35.452 INFO:tasks.workunit.client.0.smithi103.stdout:4/302: getdents d3/d1e/d25/d65 0 2024-03-22T02:28:35.453 INFO:tasks.workunit.client.0.smithi103.stdout:4/303: fdatasync d3/d31/f37 0 2024-03-22T02:28:35.454 INFO:tasks.workunit.client.0.smithi103.stdout:7/236: truncate d1/d10/d11/f28 1550055 0 2024-03-22T02:28:35.459 INFO:tasks.workunit.client.0.smithi103.stdout:4/304: dwrite d3/d1e/f43 [0,4194304] 0 2024-03-22T02:28:35.544 INFO:tasks.workunit.client.0.smithi103.stdout:6/171: rmdir d0/d14 39 2024-03-22T02:28:35.545 INFO:tasks.workunit.client.0.smithi103.stdout:0/337: readlink dd/d16/d20/d35/l4f 0 2024-03-22T02:28:35.549 INFO:tasks.workunit.client.0.smithi103.stdout:3/284: rename de/d1a/d25/d26/f44 to de/d12/d30/f63 0 2024-03-22T02:28:35.551 INFO:tasks.workunit.client.0.smithi103.stdout:1/260: fsync db/d4a/f53 0 2024-03-22T02:28:35.552 INFO:tasks.workunit.client.0.smithi103.stdout:1/261: chown db/f17 934 1 2024-03-22T02:28:35.554 INFO:tasks.workunit.client.0.smithi103.stdout:5/301: dread d0/d14/f41 [0,4194304] 0 2024-03-22T02:28:35.557 INFO:tasks.workunit.client.0.smithi103.stdout:7/237: chown d1/d39/d21/c25 0 1 2024-03-22T02:28:35.558 INFO:tasks.workunit.client.0.smithi103.stdout:7/238: write d1/da/f8 [3514722,50091] 0 2024-03-22T02:28:35.559 INFO:tasks.workunit.client.0.smithi103.stdout:6/172: stat d0/d14/d20/f28 0 2024-03-22T02:28:35.566 INFO:tasks.workunit.client.0.smithi103.stdout:5/302: chown d0/d14/d23/d2a/f63 2 1 2024-03-22T02:28:35.568 INFO:tasks.workunit.client.0.smithi103.stdout:5/303: write d0/d14/d23/d2a/d33/f4c [3247777,121547] 0 2024-03-22T02:28:35.569 INFO:tasks.workunit.client.0.smithi103.stdout:7/239: symlink d1/d39/d1d/l54 0 2024-03-22T02:28:35.571 INFO:tasks.workunit.client.0.smithi103.stdout:6/173: unlink d0/d14/f30 0 2024-03-22T02:28:35.572 INFO:tasks.workunit.client.0.smithi103.stdout:6/174: chown d0/f27 4 1 2024-03-22T02:28:35.573 INFO:tasks.workunit.client.0.smithi103.stdout:0/338: symlink dd/d16/d20/d29/l6c 0 2024-03-22T02:28:35.575 INFO:tasks.workunit.client.0.smithi103.stdout:0/339: truncate dd/d16/d20/d29/f3e 579658 0 2024-03-22T02:28:35.578 INFO:tasks.workunit.client.0.smithi103.stdout:6/175: dwrite d0/d3/f1b [0,4194304] 0 2024-03-22T02:28:35.579 INFO:tasks.workunit.client.0.smithi103.stdout:7/240: creat d1/d10/d11/d19/f55 x:0 0 0 2024-03-22T02:28:35.580 INFO:tasks.workunit.client.0.smithi103.stdout:4/305: write d3/d1e/f20 [1162090,43059] 0 2024-03-22T02:28:35.583 INFO:tasks.workunit.client.0.smithi103.stdout:0/340: creat dd/d16/d4b/f6d x:0 0 0 2024-03-22T02:28:35.585 INFO:tasks.workunit.client.0.smithi103.stdout:6/176: rmdir d0/d3/d22 39 2024-03-22T02:28:35.586 INFO:tasks.workunit.client.0.smithi103.stdout:2/250: write f10 [1646320,129938] 0 2024-03-22T02:28:35.588 INFO:tasks.workunit.client.0.smithi103.stdout:7/241: creat d1/d10/d11/d19/f56 x:0 0 0 2024-03-22T02:28:35.589 INFO:tasks.workunit.client.0.smithi103.stdout:1/262: getdents db/d10/d18 0 2024-03-22T02:28:35.591 INFO:tasks.workunit.client.0.smithi103.stdout:5/304: creat d0/d14/d23/d2a/f6c x:0 0 0 2024-03-22T02:28:35.592 INFO:tasks.workunit.client.0.smithi103.stdout:3/285: dwrite de/d1a/d25/f29 [0,4194304] 0 2024-03-22T02:28:35.592 INFO:tasks.workunit.client.0.smithi103.stdout:6/177: dread d0/f23 [0,4194304] 0 2024-03-22T02:28:35.592 INFO:tasks.workunit.client.0.smithi103.stdout:0/341: creat dd/d16/d20/d35/d49/f6e x:0 0 0 2024-03-22T02:28:35.592 INFO:tasks.workunit.client.0.smithi103.stdout:2/251: write d11/d25/f35 [1337959,78921] 0 2024-03-22T02:28:35.593 INFO:tasks.workunit.client.0.smithi103.stdout:4/306: dwrite d3/d30/f49 [0,4194304] 0 2024-03-22T02:28:35.595 INFO:tasks.workunit.client.0.smithi103.stdout:7/242: write d1/d39/d1d/f53 [203226,60351] 0 2024-03-22T02:28:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:35 smithi122 ceph-mon[36403]: pgmap v11: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 106 KiB/s rd, 15 MiB/s wr, 379 op/s 2024-03-22T02:28:35.601 INFO:tasks.workunit.client.0.smithi103.stdout:5/305: readlink d0/d14/d23/d36/d50/l69 0 2024-03-22T02:28:35.603 INFO:tasks.workunit.client.0.smithi103.stdout:7/243: mkdir d1/d10/d11/d19/d29/d57 0 2024-03-22T02:28:35.603 INFO:tasks.workunit.client.0.smithi103.stdout:2/252: fsync d11/d25/f4c 0 2024-03-22T02:28:35.605 INFO:tasks.workunit.client.0.smithi103.stdout:2/253: fdatasync d11/f13 0 2024-03-22T02:28:35.606 INFO:tasks.workunit.client.0.smithi103.stdout:0/342: read dd/f12 [3822215,108346] 0 2024-03-22T02:28:35.606 INFO:tasks.workunit.client.0.smithi103.stdout:7/244: write d1/d39/f45 [470825,123643] 0 2024-03-22T02:28:35.606 INFO:tasks.workunit.client.0.smithi103.stdout:3/286: creat de/d12/d47/d4f/f64 x:0 0 0 2024-03-22T02:28:35.610 INFO:tasks.workunit.client.0.smithi103.stdout:6/178: creat d0/d3/d22/f37 x:0 0 0 2024-03-22T02:28:35.610 INFO:tasks.workunit.client.0.smithi103.stdout:5/306: creat d0/d14/d23/d2a/d2c/f6d x:0 0 0 2024-03-22T02:28:35.612 INFO:tasks.workunit.client.0.smithi103.stdout:6/179: read d0/d3/f1b [1427173,128955] 0 2024-03-22T02:28:35.613 INFO:tasks.workunit.client.0.smithi103.stdout:4/307: dwrite d3/d38/f63 [0,4194304] 0 2024-03-22T02:28:35.616 INFO:tasks.workunit.client.0.smithi103.stdout:0/343: creat dd/d16/d20/d31/f6f x:0 0 0 2024-03-22T02:28:35.617 INFO:tasks.workunit.client.0.smithi103.stdout:4/308: write d3/d1e/f35 [734156,120437] 0 2024-03-22T02:28:35.617 INFO:tasks.workunit.client.0.smithi103.stdout:2/254: dwrite f2 [0,4194304] 0 2024-03-22T02:28:35.617 INFO:tasks.workunit.client.0.smithi103.stdout:3/287: creat de/d12/d47/d4f/f65 x:0 0 0 2024-03-22T02:28:35.623 INFO:tasks.workunit.client.0.smithi103.stdout:3/288: fsync de/d1a/f31 0 2024-03-22T02:28:35.623 INFO:tasks.workunit.client.0.smithi103.stdout:1/263: link db/c13 db/d10/d29/d41/c5f 0 2024-03-22T02:28:35.627 INFO:tasks.workunit.client.0.smithi103.stdout:7/245: rename d1/d10/d11/d19/d29/d33/d4a/l4d to d1/da/l58 0 2024-03-22T02:28:35.627 INFO:tasks.workunit.client.0.smithi103.stdout:2/255: dread f2 [0,4194304] 0 2024-03-22T02:28:35.628 INFO:tasks.workunit.client.0.smithi103.stdout:0/344: creat dd/d16/d20/d31/f70 x:0 0 0 2024-03-22T02:28:35.630 INFO:tasks.workunit.client.0.smithi103.stdout:4/309: creat d3/d38/f75 x:0 0 0 2024-03-22T02:28:35.632 INFO:tasks.workunit.client.0.smithi103.stdout:3/289: dwrite de/d1a/d25/d26/f53 [0,4194304] 0 2024-03-22T02:28:35.632 INFO:tasks.workunit.client.0.smithi103.stdout:6/180: fsync d0/d3/d22/f1c 0 2024-03-22T02:28:35.634 INFO:tasks.workunit.client.0.smithi103.stdout:3/290: stat de/d1a/d25/d26/f54 0 2024-03-22T02:28:35.638 INFO:tasks.workunit.client.0.smithi103.stdout:7/246: rmdir d1/d39 39 2024-03-22T02:28:35.641 INFO:tasks.workunit.client.0.smithi103.stdout:0/345: creat dd/d16/d20/d64/f71 x:0 0 0 2024-03-22T02:28:35.642 INFO:tasks.workunit.client.0.smithi103.stdout:2/256: dread d11/f18 [0,4194304] 0 2024-03-22T02:28:35.642 INFO:tasks.workunit.client.0.smithi103.stdout:4/310: rename d3/d1e/d53/l67 to d3/d1e/d53/l76 0 2024-03-22T02:28:35.643 INFO:tasks.workunit.client.0.smithi103.stdout:2/257: truncate d11/d25/f4c 989922 0 2024-03-22T02:28:35.646 INFO:tasks.workunit.client.0.smithi103.stdout:3/291: symlink de/d12/d30/l66 0 2024-03-22T02:28:35.648 INFO:tasks.workunit.client.0.smithi103.stdout:1/264: rename db/d10/d18/c2f to db/d10/c60 0 2024-03-22T02:28:35.649 INFO:tasks.workunit.client.0.smithi103.stdout:4/311: dwrite d3/d69/f3b [4194304,4194304] 0 2024-03-22T02:28:35.650 INFO:tasks.workunit.client.0.smithi103.stdout:1/265: write db/d10/f26 [2436541,93828] 0 2024-03-22T02:28:35.653 INFO:tasks.workunit.client.0.smithi103.stdout:1/266: truncate db/d10/d2e/f51 328745 0 2024-03-22T02:28:35.653 INFO:tasks.workunit.client.0.smithi103.stdout:0/346: mknod dd/d34/d53/c72 0 2024-03-22T02:28:35.653 INFO:tasks.workunit.client.0.smithi103.stdout:6/181: symlink d0/d3/d33/l38 0 2024-03-22T02:28:35.655 INFO:tasks.workunit.client.0.smithi103.stdout:0/347: chown dd/d16/c1f 2 1 2024-03-22T02:28:35.656 INFO:tasks.workunit.client.0.smithi103.stdout:3/292: mknod de/d12/d19/c67 0 2024-03-22T02:28:35.658 INFO:tasks.workunit.client.0.smithi103.stdout:0/348: dread - dd/f6b zero size 2024-03-22T02:28:35.659 INFO:tasks.workunit.client.0.smithi103.stdout:3/293: dread - de/d12/d47/d4f/f64 zero size 2024-03-22T02:28:35.661 INFO:tasks.workunit.client.0.smithi103.stdout:6/182: rmdir d0/d3 39 2024-03-22T02:28:35.662 INFO:tasks.workunit.client.0.smithi103.stdout:5/307: dwrite d0/d14/d23/d2a/d1f/f20 [0,4194304] 0 2024-03-22T02:28:35.662 INFO:tasks.workunit.client.0.smithi103.stdout:3/294: read de/d1a/d25/d26/f58 [1269152,10347] 0 2024-03-22T02:28:35.663 INFO:tasks.workunit.client.0.smithi103.stdout:1/267: creat db/d10/d29/f61 x:0 0 0 2024-03-22T02:28:35.664 INFO:tasks.workunit.client.0.smithi103.stdout:3/295: fdatasync de/d1a/d25/f42 0 2024-03-22T02:28:35.665 INFO:tasks.workunit.client.0.smithi103.stdout:7/247: link d1/l6 d1/d10/d11/l59 0 2024-03-22T02:28:35.665 INFO:tasks.workunit.client.0.smithi103.stdout:1/268: chown db/d10/d28/c46 10 1 2024-03-22T02:28:35.667 INFO:tasks.workunit.client.0.smithi103.stdout:6/183: write d0/f2e [4192886,57672] 0 2024-03-22T02:28:35.670 INFO:tasks.workunit.client.0.smithi103.stdout:5/308: fsync d0/d14/d23/d2a/d1f/f32 0 2024-03-22T02:28:35.672 INFO:tasks.workunit.client.0.smithi103.stdout:7/248: chown d1/d39/f35 25433 1 2024-03-22T02:28:35.673 INFO:tasks.workunit.client.0.smithi103.stdout:1/269: stat db/l50 0 2024-03-22T02:28:35.673 INFO:tasks.workunit.client.0.smithi103.stdout:1/270: read - db/d4a/f5e zero size 2024-03-22T02:28:35.675 INFO:tasks.workunit.client.0.smithi103.stdout:4/312: link d3/d30/f49 d3/f77 0 2024-03-22T02:28:35.676 INFO:tasks.workunit.client.0.smithi103.stdout:4/313: readlink d3/d1e/d53/l5f 0 2024-03-22T02:28:35.676 INFO:tasks.workunit.client.0.smithi103.stdout:4/314: chown d3/d1e/l2e 595074881 1 2024-03-22T02:28:35.678 INFO:tasks.workunit.client.0.smithi103.stdout:0/349: rename dd/d16/f19 to dd/d16/d20/d29/d48/f73 0 2024-03-22T02:28:35.678 INFO:tasks.workunit.client.0.smithi103.stdout:0/350: readlink dd/d16/l54 0 2024-03-22T02:28:35.679 INFO:tasks.workunit.client.0.smithi103.stdout:0/351: readlink dd/l62 0 2024-03-22T02:28:35.679 INFO:tasks.workunit.client.0.smithi103.stdout:0/352: fdatasync dd/d34/f43 0 2024-03-22T02:28:35.681 INFO:tasks.workunit.client.0.smithi103.stdout:5/309: mknod d0/d14/d23/d2a/d33/c6e 0 2024-03-22T02:28:35.682 INFO:tasks.workunit.client.0.smithi103.stdout:7/249: rmdir d1/d39/d1d 39 2024-03-22T02:28:35.683 INFO:tasks.workunit.client.0.smithi103.stdout:1/271: symlink db/d10/d29/d41/l62 0 2024-03-22T02:28:35.686 INFO:tasks.workunit.client.0.smithi103.stdout:4/315: mkdir d3/d1e/d78 0 2024-03-22T02:28:35.688 INFO:tasks.workunit.client.0.smithi103.stdout:4/316: write d3/fc [1558670,67243] 0 2024-03-22T02:28:35.691 INFO:tasks.workunit.client.0.smithi103.stdout:1/272: dwrite db/f2b [0,4194304] 0 2024-03-22T02:28:35.693 INFO:tasks.workunit.client.0.smithi103.stdout:7/250: dread d1/da/f30 [0,4194304] 0 2024-03-22T02:28:35.694 INFO:tasks.workunit.client.0.smithi103.stdout:0/353: link dd/d16/d20/d29/f2a dd/d16/d20/d64/d6a/f74 0 2024-03-22T02:28:35.696 INFO:tasks.workunit.client.0.smithi103.stdout:5/310: mkdir d0/d14/d23/d6f 0 2024-03-22T02:28:35.701 INFO:tasks.workunit.client.0.smithi103.stdout:6/184: creat d0/d3/f39 x:0 0 0 2024-03-22T02:28:35.701 INFO:tasks.workunit.client.0.smithi103.stdout:6/185: stat d0/d14/c18 0 2024-03-22T02:28:35.703 INFO:tasks.workunit.client.0.smithi103.stdout:7/251: read - d1/d10/d11/d19/f56 zero size 2024-03-22T02:28:35.704 INFO:tasks.workunit.client.0.smithi103.stdout:0/354: symlink dd/l75 0 2024-03-22T02:28:35.705 INFO:tasks.workunit.client.0.smithi103.stdout:0/355: chown dd/d16/l1a 331852550 1 2024-03-22T02:28:35.705 INFO:tasks.workunit.client.0.smithi103.stdout:5/311: mkdir d0/d14/d23/d5c/d70 0 2024-03-22T02:28:35.710 INFO:tasks.workunit.client.0.smithi103.stdout:4/317: link d3/d69/c60 d3/d1e/d25/c79 0 2024-03-22T02:28:35.712 INFO:tasks.workunit.client.0.smithi103.stdout:0/356: rename dd/d16/d20/d29/f42 to dd/d16/d4b/f76 0 2024-03-22T02:28:35.713 INFO:tasks.workunit.client.0.smithi103.stdout:1/273: getdents db/d4a 0 2024-03-22T02:28:35.715 INFO:tasks.workunit.client.0.smithi103.stdout:6/186: rename d0/d14/d20/c25 to d0/d14/d20/c3a 0 2024-03-22T02:28:35.717 INFO:tasks.workunit.client.0.smithi103.stdout:4/318: mkdir d3/d1e/d53/d7a 0 2024-03-22T02:28:35.718 INFO:tasks.workunit.client.0.smithi103.stdout:0/357: write dd/f12 [4264446,70593] 0 2024-03-22T02:28:35.720 INFO:tasks.workunit.client.0.smithi103.stdout:6/187: unlink d0/l2a 0 2024-03-22T02:28:35.722 INFO:tasks.workunit.client.0.smithi103.stdout:5/312: rmdir d0/d14/d23/d2a/d33/d57 0 2024-03-22T02:28:35.723 INFO:tasks.workunit.client.0.smithi103.stdout:7/252: getdents d1/da/d2f/d36 0 2024-03-22T02:28:35.725 INFO:tasks.workunit.client.0.smithi103.stdout:6/188: creat d0/d3/d33/f3b x:0 0 0 2024-03-22T02:28:35.725 INFO:tasks.workunit.client.0.smithi103.stdout:0/358: mkdir dd/d16/d20/d64/d6a/d77 0 2024-03-22T02:28:35.726 INFO:tasks.workunit.client.0.smithi103.stdout:1/274: dread f5 [0,4194304] 0 2024-03-22T02:28:35.727 INFO:tasks.workunit.client.0.smithi103.stdout:0/359: dread - dd/d16/d20/d31/f6f zero size 2024-03-22T02:28:35.727 INFO:tasks.workunit.client.0.smithi103.stdout:6/189: write d0/f1e [4656409,79699] 0 2024-03-22T02:28:35.728 INFO:tasks.workunit.client.0.smithi103.stdout:5/313: dread d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:35.729 INFO:tasks.workunit.client.0.smithi103.stdout:0/360: chown dd/d34/f43 52 1 2024-03-22T02:28:35.729 INFO:tasks.workunit.client.0.smithi103.stdout:6/190: fdatasync d0/d3/d22/f37 0 2024-03-22T02:28:35.730 INFO:tasks.workunit.client.0.smithi103.stdout:7/253: rename d1/da/f8 to d1/d10/d11/f5a 0 2024-03-22T02:28:35.730 INFO:tasks.workunit.client.0.smithi103.stdout:5/314: fdatasync d0/d14/d23/d2a/d1f/f68 0 2024-03-22T02:28:35.731 INFO:tasks.workunit.client.0.smithi103.stdout:1/275: creat db/d10/f63 x:0 0 0 2024-03-22T02:28:35.733 INFO:tasks.workunit.client.0.smithi103.stdout:1/276: chown db/d10/d21/f23 118993415 1 2024-03-22T02:28:35.733 INFO:tasks.workunit.client.0.smithi103.stdout:6/191: rename d0/f2e to d0/d3/d22/f3c 0 2024-03-22T02:28:35.735 INFO:tasks.workunit.client.0.smithi103.stdout:7/254: mknod d1/d39/c5b 0 2024-03-22T02:28:35.736 INFO:tasks.workunit.client.0.smithi103.stdout:1/277: fdatasync db/d10/f5a 0 2024-03-22T02:28:35.737 INFO:tasks.workunit.client.0.smithi103.stdout:7/255: chown d1/da/d1c/c20 16351094 1 2024-03-22T02:28:35.739 INFO:tasks.workunit.client.0.smithi103.stdout:0/361: dwrite dd/d16/d20/f68 [0,4194304] 0 2024-03-22T02:28:35.739 INFO:tasks.workunit.client.0.smithi103.stdout:6/192: dread d0/f2c [0,4194304] 0 2024-03-22T02:28:35.741 INFO:tasks.workunit.client.0.smithi103.stdout:7/256: mkdir d1/d39/d21/d5c 0 2024-03-22T02:28:35.743 INFO:tasks.workunit.client.0.smithi103.stdout:5/315: link d0/d14/d23/f44 d0/f71 0 2024-03-22T02:28:35.745 INFO:tasks.workunit.client.0.smithi103.stdout:1/278: symlink db/d10/d29/d41/l64 0 2024-03-22T02:28:35.748 INFO:tasks.workunit.client.0.smithi103.stdout:6/193: dwrite d0/d14/f29 [0,4194304] 0 2024-03-22T02:28:35.750 INFO:tasks.workunit.client.0.smithi103.stdout:9/226: sync 2024-03-22T02:28:35.757 INFO:tasks.workunit.client.0.smithi103.stdout:6/194: creat d0/d3/d22/f3d x:0 0 0 2024-03-22T02:28:35.758 INFO:tasks.workunit.client.0.smithi103.stdout:5/316: mknod d0/d14/d23/d6f/c72 0 2024-03-22T02:28:35.760 INFO:tasks.workunit.client.0.smithi103.stdout:0/362: link dd/c66 dd/d16/d20/d35/c78 0 2024-03-22T02:28:35.761 INFO:tasks.workunit.client.0.smithi103.stdout:0/363: dread - dd/d16/d20/d29/f5a zero size 2024-03-22T02:28:35.763 INFO:tasks.workunit.client.0.smithi103.stdout:5/317: unlink d0/d3e/c35 0 2024-03-22T02:28:35.767 INFO:tasks.workunit.client.0.smithi103.stdout:6/195: creat d0/f3e x:0 0 0 2024-03-22T02:28:35.770 INFO:tasks.workunit.client.0.smithi103.stdout:6/196: mknod d0/d14/c3f 0 2024-03-22T02:28:35.771 INFO:tasks.workunit.client.0.smithi103.stdout:0/364: rmdir dd/d16/d20/d64/d6a/d77 0 2024-03-22T02:28:35.772 INFO:tasks.workunit.client.0.smithi103.stdout:7/257: truncate d1/da/fb 1750069 0 2024-03-22T02:28:35.773 INFO:tasks.workunit.client.0.smithi103.stdout:1/279: write db/d10/d2e/f51 [179353,37862] 0 2024-03-22T02:28:35.775 INFO:tasks.workunit.client.0.smithi103.stdout:0/365: mkdir dd/d16/d20/d31/d79 0 2024-03-22T02:28:35.776 INFO:tasks.workunit.client.0.smithi103.stdout:0/366: write dd/d16/d20/d31/f70 [413755,84654] 0 2024-03-22T02:28:35.777 INFO:tasks.workunit.client.0.smithi103.stdout:6/197: unlink d0/f19 0 2024-03-22T02:28:35.779 INFO:tasks.workunit.client.0.smithi103.stdout:6/198: dread - d0/d14/d20/f28 zero size 2024-03-22T02:28:35.780 INFO:tasks.workunit.client.0.smithi103.stdout:7/258: mknod d1/c5d 0 2024-03-22T02:28:35.780 INFO:tasks.workunit.client.0.smithi103.stdout:5/318: write d0/f12 [3389197,1186] 0 2024-03-22T02:28:35.784 INFO:tasks.workunit.client.0.smithi103.stdout:6/199: creat d0/d14/d20/f40 x:0 0 0 2024-03-22T02:28:35.784 INFO:tasks.workunit.client.0.smithi103.stdout:7/259: rmdir d1/da/d2f/d36 39 2024-03-22T02:28:35.785 INFO:tasks.workunit.client.0.smithi103.stdout:0/367: dwrite dd/d16/d4b/f6d [0,4194304] 0 2024-03-22T02:28:35.786 INFO:tasks.workunit.client.0.smithi103.stdout:6/200: chown d0/d3/d22/f37 56 1 2024-03-22T02:28:35.789 INFO:tasks.workunit.client.0.smithi103.stdout:6/201: chown d0/d14/d20/f28 360851 1 2024-03-22T02:28:35.790 INFO:tasks.workunit.client.0.smithi103.stdout:6/202: write d0/d14/d20/f28 [771618,129148] 0 2024-03-22T02:28:35.791 INFO:tasks.workunit.client.0.smithi103.stdout:5/319: dread d0/d14/d23/d2a/d33/f4c [0,4194304] 0 2024-03-22T02:28:35.792 INFO:tasks.workunit.client.0.smithi103.stdout:0/368: rename dd/d34/d53/c69 to dd/d16/d20/d35/c7a 0 2024-03-22T02:28:35.796 INFO:tasks.workunit.client.0.smithi103.stdout:7/260: rename d1/d10/d11/d19/d29/d33/d4a to d1/d10/d11/d19/d29/d33/d4b/d5e 0 2024-03-22T02:28:35.798 INFO:tasks.workunit.client.0.smithi103.stdout:0/369: rmdir dd/d16/d20/d29 39 2024-03-22T02:28:35.799 INFO:tasks.workunit.client.0.smithi103.stdout:1/280: dwrite db/d10/f1c [0,4194304] 0 2024-03-22T02:28:35.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/203: creat d0/f41 x:0 0 0 2024-03-22T02:28:35.807 INFO:tasks.workunit.client.0.smithi103.stdout:7/261: read d1/f49 [5155836,103294] 0 2024-03-22T02:28:35.809 INFO:tasks.workunit.client.0.smithi103.stdout:5/320: creat d0/f73 x:0 0 0 2024-03-22T02:28:35.811 INFO:tasks.workunit.client.0.smithi103.stdout:5/321: chown d0/d14/d23/f24 12 1 2024-03-22T02:28:35.813 INFO:tasks.workunit.client.0.smithi103.stdout:0/370: unlink dd/d16/d20/d35/c7a 0 2024-03-22T02:28:35.813 INFO:tasks.workunit.client.0.smithi103.stdout:7/262: dread d1/d39/f46 [0,4194304] 0 2024-03-22T02:28:35.815 INFO:tasks.workunit.client.0.smithi103.stdout:0/371: fsync dd/d16/f55 0 2024-03-22T02:28:35.817 INFO:tasks.workunit.client.0.smithi103.stdout:7/263: rmdir d1/d39/d1d 39 2024-03-22T02:28:35.817 INFO:tasks.workunit.client.0.smithi103.stdout:5/322: truncate d0/d14/d23/d36/d50/f5a 757583 0 2024-03-22T02:28:35.820 INFO:tasks.workunit.client.0.smithi103.stdout:1/281: dwrite db/f17 [0,4194304] 0 2024-03-22T02:28:35.822 INFO:tasks.workunit.client.0.smithi103.stdout:5/323: fsync d0/d14/d23/d2a/f6c 0 2024-03-22T02:28:35.822 INFO:tasks.workunit.client.0.smithi103.stdout:6/204: dwrite d0/f2c [0,4194304] 0 2024-03-22T02:28:35.827 INFO:tasks.workunit.client.0.smithi103.stdout:6/205: write d0/d14/d20/f2d [222052,14669] 0 2024-03-22T02:28:35.827 INFO:tasks.workunit.client.0.smithi103.stdout:1/282: creat db/d4a/f65 x:0 0 0 2024-03-22T02:28:35.829 INFO:tasks.workunit.client.0.smithi103.stdout:1/283: fdatasync db/d10/f1c 0 2024-03-22T02:28:35.830 INFO:tasks.workunit.client.0.smithi103.stdout:7/264: getdents d1/da/d1c 0 2024-03-22T02:28:35.832 INFO:tasks.workunit.client.0.smithi103.stdout:5/324: link d0/d14/d23/l3c d0/d3e/l74 0 2024-03-22T02:28:35.835 INFO:tasks.workunit.client.0.smithi103.stdout:0/372: dwrite dd/d16/f3f [4194304,4194304] 0 2024-03-22T02:28:35.835 INFO:tasks.workunit.client.0.smithi103.stdout:7/265: write d1/d10/d11/d19/f55 [780064,76035] 0 2024-03-22T02:28:35.841 INFO:tasks.workunit.client.0.smithi103.stdout:5/325: symlink d0/d14/d23/d46/l75 0 2024-03-22T02:28:35.843 INFO:tasks.workunit.client.0.smithi103.stdout:7/266: creat d1/d10/d11/d19/d29/d33/f5f x:0 0 0 2024-03-22T02:28:35.844 INFO:tasks.workunit.client.0.smithi103.stdout:0/373: creat dd/d16/d20/d35/d49/f7b x:0 0 0 2024-03-22T02:28:35.844 INFO:tasks.workunit.client.0.smithi103.stdout:0/374: readlink dd/d16/l1a 0 2024-03-22T02:28:35.845 INFO:tasks.workunit.client.0.smithi103.stdout:1/284: rename db/d10/d2e/l5c to db/l66 0 2024-03-22T02:28:35.847 INFO:tasks.workunit.client.0.smithi103.stdout:5/326: mknod d0/d14/d23/d6f/c76 0 2024-03-22T02:28:35.847 INFO:tasks.workunit.client.0.smithi103.stdout:6/206: dread d0/d3/f9 [0,4194304] 0 2024-03-22T02:28:35.849 INFO:tasks.workunit.client.0.smithi103.stdout:7/267: creat d1/da/d1c/f60 x:0 0 0 2024-03-22T02:28:35.850 INFO:tasks.workunit.client.0.smithi103.stdout:5/327: dread d0/d14/d23/d36/d50/f5a [0,4194304] 0 2024-03-22T02:28:35.851 INFO:tasks.workunit.client.0.smithi103.stdout:0/375: symlink dd/l7c 0 2024-03-22T02:28:35.852 INFO:tasks.workunit.client.0.smithi103.stdout:7/268: readlink d1/da/l58 0 2024-03-22T02:28:35.853 INFO:tasks.workunit.client.0.smithi103.stdout:6/207: truncate d0/d3/f1b 3992506 0 2024-03-22T02:28:35.854 INFO:tasks.workunit.client.0.smithi103.stdout:6/208: chown d0/f1a 53808 1 2024-03-22T02:28:35.854 INFO:tasks.workunit.client.0.smithi103.stdout:5/328: mkdir d0/d14/d23/d5c/d77 0 2024-03-22T02:28:35.855 INFO:tasks.workunit.client.0.smithi103.stdout:5/329: readlink d0/d14/d23/d2a/d33/l37 0 2024-03-22T02:28:35.855 INFO:tasks.workunit.client.0.smithi103.stdout:0/376: rmdir dd/d16/d20/d64 39 2024-03-22T02:28:35.857 INFO:tasks.workunit.client.0.smithi103.stdout:0/377: truncate dd/d16/d20/d31/f6f 510866 0 2024-03-22T02:28:35.857 INFO:tasks.workunit.client.0.smithi103.stdout:0/378: chown dd/d16/l5f 814 1 2024-03-22T02:28:35.858 INFO:tasks.workunit.client.0.smithi103.stdout:7/269: rename d1/d39 to d1/d10/d11/d19/d29/d61 0 2024-03-22T02:28:35.858 INFO:tasks.workunit.client.0.smithi103.stdout:0/379: chown dd/d16/d4b/f6d 0 1 2024-03-22T02:28:35.859 INFO:tasks.workunit.client.0.smithi103.stdout:0/380: write dd/d16/d20/f5d [190467,87549] 0 2024-03-22T02:28:35.861 INFO:tasks.workunit.client.0.smithi103.stdout:5/330: creat d0/d14/d23/d2a/d1f/f78 x:0 0 0 2024-03-22T02:28:35.863 INFO:tasks.workunit.client.0.smithi103.stdout:1/285: getdents db 0 2024-03-22T02:28:35.865 INFO:tasks.workunit.client.0.smithi103.stdout:1/286: truncate db/d10/d18/f37 724176 0 2024-03-22T02:28:35.868 INFO:tasks.workunit.client.0.smithi103.stdout:1/287: write db/f17 [4080122,17790] 0 2024-03-22T02:28:35.870 INFO:tasks.workunit.client.0.smithi103.stdout:0/381: write dd/d16/d20/d29/f3e [1369199,125157] 0 2024-03-22T02:28:35.871 INFO:tasks.workunit.client.0.smithi103.stdout:6/209: creat d0/d3/f42 x:0 0 0 2024-03-22T02:28:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:35 smithi103 ceph-mon[29258]: pgmap v11: 65 pgs: 65 active+clean; 287 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 106 KiB/s rd, 15 MiB/s wr, 379 op/s 2024-03-22T02:28:35.874 INFO:tasks.workunit.client.0.smithi103.stdout:7/270: dwrite d1/d10/d11/f18 [0,4194304] 0 2024-03-22T02:28:35.876 INFO:tasks.workunit.client.0.smithi103.stdout:5/331: dwrite d0/d14/d23/d2a/d2c/f3f [4194304,4194304] 0 2024-03-22T02:28:35.877 INFO:tasks.workunit.client.0.smithi103.stdout:0/382: unlink l7 0 2024-03-22T02:28:35.881 INFO:tasks.workunit.client.0.smithi103.stdout:7/271: creat d1/d10/d11/d19/d29/d61/d21/f62 x:0 0 0 2024-03-22T02:28:35.884 INFO:tasks.workunit.client.0.smithi103.stdout:6/210: mknod d0/d3/c43 0 2024-03-22T02:28:35.885 INFO:tasks.workunit.client.0.smithi103.stdout:6/211: dread - d0/f41 zero size 2024-03-22T02:28:35.885 INFO:tasks.workunit.client.0.smithi103.stdout:5/332: symlink d0/d14/d23/d2a/d33/l79 0 2024-03-22T02:28:35.887 INFO:tasks.workunit.client.0.smithi103.stdout:0/383: mknod dd/d16/d20/d35/c7d 0 2024-03-22T02:28:35.890 INFO:tasks.workunit.client.0.smithi103.stdout:5/333: dread d0/d14/d23/d2a/f1a [4194304,4194304] 0 2024-03-22T02:28:35.890 INFO:tasks.workunit.client.0.smithi103.stdout:0/384: chown dd/f24 805873918 1 2024-03-22T02:28:35.892 INFO:tasks.workunit.client.0.smithi103.stdout:0/385: write fa [5291175,6215] 0 2024-03-22T02:28:35.893 INFO:tasks.workunit.client.0.smithi103.stdout:5/334: creat d0/d14/d23/d46/f7a x:0 0 0 2024-03-22T02:28:35.894 INFO:tasks.workunit.client.0.smithi103.stdout:8/282: sync 2024-03-22T02:28:35.896 INFO:tasks.workunit.client.0.smithi103.stdout:6/212: getdents d0 0 2024-03-22T02:28:35.897 INFO:tasks.workunit.client.0.smithi103.stdout:0/386: mknod dd/d16/d20/c7e 0 2024-03-22T02:28:35.900 INFO:tasks.workunit.client.0.smithi103.stdout:5/335: mkdir d0/d14/d23/d2a/d7b 0 2024-03-22T02:28:35.900 INFO:tasks.workunit.client.0.smithi103.stdout:1/288: truncate db/d10/d29/d55/f58 138767 0 2024-03-22T02:28:35.902 INFO:tasks.workunit.client.0.smithi103.stdout:6/213: rmdir d0/d3/d33 39 2024-03-22T02:28:35.902 INFO:tasks.workunit.client.0.smithi103.stdout:6/214: read - d0/f41 zero size 2024-03-22T02:28:35.904 INFO:tasks.workunit.client.0.smithi103.stdout:1/289: dread db/d10/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:35.908 INFO:tasks.workunit.client.0.smithi103.stdout:8/283: mkdir d4/d14/d39/d49/d51 0 2024-03-22T02:28:35.912 INFO:tasks.workunit.client.0.smithi103.stdout:7/272: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:35.914 INFO:tasks.workunit.client.0.smithi103.stdout:1/290: dwrite db/d10/d21/f33 [0,4194304] 0 2024-03-22T02:28:35.915 INFO:tasks.workunit.client.0.smithi103.stdout:7/273: fsync d1/d10/d11/d19/d29/d33/f5f 0 2024-03-22T02:28:35.921 INFO:tasks.workunit.client.0.smithi103.stdout:8/284: rename d4/d48/f1c to d4/d14/f52 0 2024-03-22T02:28:35.923 INFO:tasks.workunit.client.0.smithi103.stdout:1/291: mknod db/d10/d18/c67 0 2024-03-22T02:28:35.924 INFO:tasks.workunit.client.0.smithi103.stdout:7/274: symlink d1/d10/d11/d19/l63 0 2024-03-22T02:28:35.927 INFO:tasks.workunit.client.0.smithi103.stdout:0/387: mknod dd/d16/c7f 0 2024-03-22T02:28:35.927 INFO:tasks.workunit.client.0.smithi103.stdout:0/388: dread - dd/f6b zero size 2024-03-22T02:28:35.928 INFO:tasks.workunit.client.0.smithi103.stdout:8/285: symlink d4/d19/d1a/d4a/l53 0 2024-03-22T02:28:35.930 INFO:tasks.workunit.client.0.smithi103.stdout:6/215: getdents d0/d3 0 2024-03-22T02:28:35.932 INFO:tasks.workunit.client.0.smithi103.stdout:7/275: rename d1/d10/d11/d19/l63 to d1/d10/d11/d19/d29/l64 0 2024-03-22T02:28:35.934 INFO:tasks.workunit.client.0.smithi103.stdout:8/286: mknod d4/d14/d39/d49/c54 0 2024-03-22T02:28:35.935 INFO:tasks.workunit.client.0.smithi103.stdout:6/216: fsync d0/d3/d22/f15 0 2024-03-22T02:28:35.943 INFO:tasks.workunit.client.0.smithi103.stdout:6/217: rename d0/f27 to d0/d14/f44 0 2024-03-22T02:28:35.946 INFO:tasks.workunit.client.0.smithi103.stdout:8/287: symlink d4/d26/l55 0 2024-03-22T02:28:35.948 INFO:tasks.workunit.client.0.smithi103.stdout:8/288: chown d4/c10 0 1 2024-03-22T02:28:35.948 INFO:tasks.workunit.client.0.smithi103.stdout:7/276: rmdir d1 39 2024-03-22T02:28:35.949 INFO:tasks.workunit.client.0.smithi103.stdout:8/289: chown d4/d14/l17 4 1 2024-03-22T02:28:35.951 INFO:tasks.workunit.client.0.smithi103.stdout:1/292: dwrite db/d10/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:35.951 INFO:tasks.workunit.client.0.smithi103.stdout:1/293: chown db/f45 477 1 2024-03-22T02:28:35.952 INFO:tasks.workunit.client.0.smithi103.stdout:6/218: rename d0/d14/d20 to d0/d3/d45 0 2024-03-22T02:28:35.955 INFO:tasks.workunit.client.0.smithi103.stdout:8/290: fsync d4/f15 0 2024-03-22T02:28:35.956 INFO:tasks.workunit.client.0.smithi103.stdout:1/294: mkdir db/d10/d29/d41/d68 0 2024-03-22T02:28:35.960 INFO:tasks.workunit.client.0.smithi103.stdout:0/389: creat dd/d16/d20/d29/d48/d5c/f80 x:0 0 0 2024-03-22T02:28:35.961 INFO:tasks.workunit.client.0.smithi103.stdout:7/277: dwrite d1/d10/d11/d19/d29/f52 [0,4194304] 0 2024-03-22T02:28:35.964 INFO:tasks.workunit.client.0.smithi103.stdout:7/278: mknod d1/d10/d11/d19/d29/d33/d4b/c65 0 2024-03-22T02:28:35.966 INFO:tasks.workunit.client.0.smithi103.stdout:8/291: truncate d4/d26/f4e 1962562 0 2024-03-22T02:28:35.969 INFO:tasks.workunit.client.0.smithi103.stdout:7/279: mknod d1/da/d1c/c66 0 2024-03-22T02:28:35.969 INFO:tasks.workunit.client.0.smithi103.stdout:1/295: dread db/d10/f5a [0,4194304] 0 2024-03-22T02:28:35.971 INFO:tasks.workunit.client.0.smithi103.stdout:1/296: read - db/d10/d29/d41/f57 zero size 2024-03-22T02:28:35.972 INFO:tasks.workunit.client.0.smithi103.stdout:1/297: chown db/d10/d21/f23 18430029 1 2024-03-22T02:28:35.972 INFO:tasks.workunit.client.0.smithi103.stdout:6/219: truncate d0/d3/d33/f3b 1023317 0 2024-03-22T02:28:35.973 INFO:tasks.workunit.client.0.smithi103.stdout:7/280: chown d1/d10/d11/d19/d29/d61/d1d/l54 94935315 1 2024-03-22T02:28:35.974 INFO:tasks.workunit.client.0.smithi103.stdout:7/281: stat d1/da/d1c/c1f 0 2024-03-22T02:28:35.975 INFO:tasks.workunit.client.0.smithi103.stdout:1/298: creat db/d10/d18/f69 x:0 0 0 2024-03-22T02:28:35.977 INFO:tasks.workunit.client.0.smithi103.stdout:8/292: link d4/d19/d1a/c25 d4/c56 0 2024-03-22T02:28:35.980 INFO:tasks.workunit.client.0.smithi103.stdout:7/282: rename d1/d10/d11/d19/d29/d61/d1d/l54 to d1/d10/d11/d19/d29/l67 0 2024-03-22T02:28:35.983 INFO:tasks.workunit.client.0.smithi103.stdout:7/283: dread - d1/d10/d11/d19/d29/d61/d21/f32 zero size 2024-03-22T02:28:35.984 INFO:tasks.workunit.client.0.smithi103.stdout:8/293: dwrite d4/d19/d1a/f37 [0,4194304] 0 2024-03-22T02:28:35.985 INFO:tasks.workunit.client.0.smithi103.stdout:7/284: dread - d1/d10/d11/d19/d29/d61/d21/f32 zero size 2024-03-22T02:28:35.986 INFO:tasks.workunit.client.0.smithi103.stdout:1/299: link db/d10/d18/d2d/l47 db/d4a/l6a 0 2024-03-22T02:28:35.987 INFO:tasks.workunit.client.0.smithi103.stdout:1/300: fdatasync db/d10/d21/f33 0 2024-03-22T02:28:35.988 INFO:tasks.workunit.client.0.smithi103.stdout:6/220: truncate d0/d3/d22/f11 1311175 0 2024-03-22T02:28:35.991 INFO:tasks.workunit.client.0.smithi103.stdout:6/221: truncate d0/d3/d45/f40 278207 0 2024-03-22T02:28:35.991 INFO:tasks.workunit.client.0.smithi103.stdout:8/294: getdents d4/d2a/d4f 0 2024-03-22T02:28:35.994 INFO:tasks.workunit.client.0.smithi103.stdout:7/285: rename d1/d10/d11/d19/d29/d33/c40 to d1/d10/d11/d19/d29/d33/d4b/c68 0 2024-03-22T02:28:35.996 INFO:tasks.workunit.client.0.smithi103.stdout:1/301: mkdir db/d10/d6b 0 2024-03-22T02:28:35.996 INFO:tasks.workunit.client.0.smithi103.stdout:7/286: fdatasync d1/d10/d11/d19/d29/d33/f5f 0 2024-03-22T02:28:35.996 INFO:tasks.workunit.client.0.smithi103.stdout:7/287: chown d1/d10/d11/d19/d29/d61/d21 1770 1 2024-03-22T02:28:35.998 INFO:tasks.workunit.client.0.smithi103.stdout:7/288: truncate d1/d10/d11/d19/f56 824821 0 2024-03-22T02:28:35.998 INFO:tasks.workunit.client.0.smithi103.stdout:8/295: symlink d4/d14/d39/d49/d51/l57 0 2024-03-22T02:28:36.000 INFO:tasks.workunit.client.0.smithi103.stdout:8/296: write d4/d26/d27/f2c [5006066,108301] 0 2024-03-22T02:28:36.003 INFO:tasks.workunit.client.0.smithi103.stdout:7/289: rename d1/d10/d11/d19/d29/d61/d21 to d1/d10/d11/d19/d29/d61/d69 0 2024-03-22T02:28:36.005 INFO:tasks.workunit.client.0.smithi103.stdout:8/297: rmdir d4/d19/d1a/d4a/d50 0 2024-03-22T02:28:36.005 INFO:tasks.workunit.client.0.smithi103.stdout:7/290: dread d1/d10/d11/d19/d29/d61/d1d/f53 [0,4194304] 0 2024-03-22T02:28:36.006 INFO:tasks.workunit.client.0.smithi103.stdout:8/298: readlink d4/d26/l3a 0 2024-03-22T02:28:36.007 INFO:tasks.workunit.client.0.smithi103.stdout:7/291: write d1/da/f3 [551089,61199] 0 2024-03-22T02:28:36.008 INFO:tasks.workunit.client.0.smithi103.stdout:8/299: symlink d4/d2a/l58 0 2024-03-22T02:28:36.008 INFO:tasks.workunit.client.0.smithi103.stdout:7/292: rmdir d1/da/d2f 39 2024-03-22T02:28:36.009 INFO:tasks.workunit.client.0.smithi103.stdout:8/300: write d4/d14/f1e [1486847,63571] 0 2024-03-22T02:28:36.009 INFO:tasks.workunit.client.0.smithi103.stdout:8/301: chown d4/d2a/d38 108 1 2024-03-22T02:28:36.015 INFO:tasks.workunit.client.0.smithi103.stdout:8/302: dread d4/d26/f4e [0,4194304] 0 2024-03-22T02:28:36.015 INFO:tasks.workunit.client.0.smithi103.stdout:7/293: dwrite d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:36.045 INFO:tasks.workunit.client.0.smithi103.stdout:2/258: sync 2024-03-22T02:28:36.067 INFO:tasks.workunit.client.0.smithi103.stdout:7/294: dread - d1/f4e zero size 2024-03-22T02:28:36.068 INFO:tasks.workunit.client.0.smithi103.stdout:1/302: write db/d10/f5a [992974,66906] 0 2024-03-22T02:28:36.069 INFO:tasks.workunit.client.0.smithi103.stdout:2/259: write d11/f20 [690280,39024] 0 2024-03-22T02:28:36.071 INFO:tasks.workunit.client.0.smithi103.stdout:7/295: read - d1/d10/d11/d19/d29/d33/f5f zero size 2024-03-22T02:28:36.075 INFO:tasks.workunit.client.0.smithi103.stdout:6/222: dwrite d0/d3/d22/f15 [0,4194304] 0 2024-03-22T02:28:36.079 INFO:tasks.workunit.client.0.smithi103.stdout:1/303: dwrite db/d10/d29/f61 [0,4194304] 0 2024-03-22T02:28:36.079 INFO:tasks.workunit.client.0.smithi103.stdout:2/260: dwrite d11/d19/f1f [0,4194304] 0 2024-03-22T02:28:36.081 INFO:tasks.workunit.client.0.smithi103.stdout:1/304: fsync db/d10/d18/f37 0 2024-03-22T02:28:36.082 INFO:tasks.workunit.client.0.smithi103.stdout:1/305: chown db/l50 21630217 1 2024-03-22T02:28:36.090 INFO:tasks.workunit.client.0.smithi103.stdout:7/296: unlink d1/da/l2b 0 2024-03-22T02:28:36.092 INFO:tasks.workunit.client.0.smithi103.stdout:2/261: creat d11/d25/f51 x:0 0 0 2024-03-22T02:28:36.094 INFO:tasks.workunit.client.0.smithi103.stdout:1/306: mkdir db/d10/d29/d6c 0 2024-03-22T02:28:36.097 INFO:tasks.workunit.client.0.smithi103.stdout:8/303: getdents d4/d14/d39/d49 0 2024-03-22T02:28:36.099 INFO:tasks.workunit.client.0.smithi103.stdout:6/223: creat d0/d3/f46 x:0 0 0 2024-03-22T02:28:36.099 INFO:tasks.workunit.client.0.smithi103.stdout:8/304: chown d4/d2a/d3e 199340790 1 2024-03-22T02:28:36.099 INFO:tasks.workunit.client.0.smithi103.stdout:1/307: creat db/d10/d29/d55/f6d x:0 0 0 2024-03-22T02:28:36.101 INFO:tasks.workunit.client.0.smithi103.stdout:7/297: dwrite d1/d10/d11/d19/f55 [0,4194304] 0 2024-03-22T02:28:36.103 INFO:tasks.workunit.client.0.smithi103.stdout:1/308: write db/f35 [4602813,115997] 0 2024-03-22T02:28:36.105 INFO:tasks.workunit.client.0.smithi103.stdout:7/298: dread - d1/d10/d11/d19/d29/d61/d69/f32 zero size 2024-03-22T02:28:36.106 INFO:tasks.workunit.client.0.smithi103.stdout:2/262: dwrite d11/d25/f35 [0,4194304] 0 2024-03-22T02:28:36.106 INFO:tasks.workunit.client.0.smithi103.stdout:8/305: rename d4/d14/l17 to d4/d26/d27/l59 0 2024-03-22T02:28:36.107 INFO:tasks.workunit.client.0.smithi103.stdout:7/299: chown d1/da/d1c/c66 3093 1 2024-03-22T02:28:36.107 INFO:tasks.workunit.client.0.smithi103.stdout:8/306: fdatasync d4/fd 0 2024-03-22T02:28:36.108 INFO:tasks.workunit.client.0.smithi103.stdout:6/224: creat d0/d3/d22/f47 x:0 0 0 2024-03-22T02:28:36.109 INFO:tasks.workunit.client.0.smithi103.stdout:1/309: creat db/d10/d29/f6e x:0 0 0 2024-03-22T02:28:36.111 INFO:tasks.workunit.client.0.smithi103.stdout:1/310: truncate db/d10/d29/d41/f57 822712 0 2024-03-22T02:28:36.111 INFO:tasks.workunit.client.0.smithi103.stdout:2/263: creat d11/d25/d46/d34/f52 x:0 0 0 2024-03-22T02:28:36.112 INFO:tasks.workunit.client.0.smithi103.stdout:2/264: chown d11/l3d 944992594 1 2024-03-22T02:28:36.114 INFO:tasks.workunit.client.0.smithi103.stdout:8/307: creat d4/d14/f5a x:0 0 0 2024-03-22T02:28:36.116 INFO:tasks.workunit.client.0.smithi103.stdout:1/311: creat db/d10/d29/f6f x:0 0 0 2024-03-22T02:28:36.118 INFO:tasks.workunit.client.0.smithi103.stdout:2/265: creat d11/d19/d4e/f53 x:0 0 0 2024-03-22T02:28:36.120 INFO:tasks.workunit.client.0.smithi103.stdout:2/266: write d11/f13 [3958758,128615] 0 2024-03-22T02:28:36.120 INFO:tasks.workunit.client.0.smithi103.stdout:6/225: creat d0/d3/d33/f48 x:0 0 0 2024-03-22T02:28:36.121 INFO:tasks.workunit.client.0.smithi103.stdout:8/308: symlink d4/d19/d1a/l5b 0 2024-03-22T02:28:36.121 INFO:tasks.workunit.client.0.smithi103.stdout:1/312: mknod db/d10/d29/d41/c70 0 2024-03-22T02:28:36.122 INFO:tasks.workunit.client.0.smithi103.stdout:1/313: write db/d10/d18/f20 [4349662,87255] 0 2024-03-22T02:28:36.124 INFO:tasks.workunit.client.0.smithi103.stdout:2/267: write d11/d25/d46/d40/f41 [755286,78430] 0 2024-03-22T02:28:36.125 INFO:tasks.workunit.client.0.smithi103.stdout:8/309: fsync d4/d26/f4e 0 2024-03-22T02:28:36.126 INFO:tasks.workunit.client.0.smithi103.stdout:2/268: write d11/f16 [311309,111115] 0 2024-03-22T02:28:36.128 INFO:tasks.workunit.client.0.smithi103.stdout:2/269: read - d11/d19/d4e/f53 zero size 2024-03-22T02:28:36.128 INFO:tasks.workunit.client.0.smithi103.stdout:2/270: read - d11/f3a zero size 2024-03-22T02:28:36.129 INFO:tasks.workunit.client.0.smithi103.stdout:6/226: creat d0/d3/f49 x:0 0 0 2024-03-22T02:28:36.129 INFO:tasks.workunit.client.0.smithi103.stdout:1/314: creat db/d10/d29/d6c/f71 x:0 0 0 2024-03-22T02:28:36.132 INFO:tasks.workunit.client.0.smithi103.stdout:1/315: creat db/d10/d18/f72 x:0 0 0 2024-03-22T02:28:36.133 INFO:tasks.workunit.client.0.smithi103.stdout:2/271: symlink d11/d25/d43/d45/d47/l54 0 2024-03-22T02:28:36.133 INFO:tasks.workunit.client.0.smithi103.stdout:8/310: dwrite f2 [4194304,4194304] 0 2024-03-22T02:28:36.134 INFO:tasks.workunit.client.0.smithi103.stdout:2/272: write d11/f3a [353871,87475] 0 2024-03-22T02:28:36.135 INFO:tasks.workunit.client.0.smithi103.stdout:6/227: rename d0/d3/d33/f3b to d0/d3/d33/f4a 0 2024-03-22T02:28:36.138 INFO:tasks.workunit.client.0.smithi103.stdout:1/316: creat db/d4a/f73 x:0 0 0 2024-03-22T02:28:36.139 INFO:tasks.workunit.client.0.smithi103.stdout:1/317: chown db/d10/d21/f23 5461 1 2024-03-22T02:28:36.139 INFO:tasks.workunit.client.0.smithi103.stdout:1/318: chown db/d10/d28/f43 2730 1 2024-03-22T02:28:36.140 INFO:tasks.workunit.client.0.smithi103.stdout:2/273: mknod d11/d25/d43/d45/c55 0 2024-03-22T02:28:36.141 INFO:tasks.workunit.client.0.smithi103.stdout:1/319: dread - db/d10/d28/f43 zero size 2024-03-22T02:28:36.144 INFO:tasks.workunit.client.0.smithi103.stdout:2/274: unlink d11/d25/d46/d34/l38 0 2024-03-22T02:28:36.144 INFO:tasks.workunit.client.0.smithi103.stdout:1/320: stat db/d10/d18/d2d/l47 0 2024-03-22T02:28:36.145 INFO:tasks.workunit.client.0.smithi103.stdout:2/275: mknod d11/d19/c56 0 2024-03-22T02:28:36.146 INFO:tasks.workunit.client.0.smithi103.stdout:1/321: chown db/d10/d29/d55/f58 19392208 1 2024-03-22T02:28:36.150 INFO:tasks.workunit.client.0.smithi103.stdout:2/276: link d11/d25/d46/l2d d11/d25/d46/d34/d24/l57 0 2024-03-22T02:28:36.153 INFO:tasks.workunit.client.0.smithi103.stdout:1/322: dwrite db/f25 [0,4194304] 0 2024-03-22T02:28:36.153 INFO:tasks.workunit.client.0.smithi103.stdout:2/277: creat d11/f58 x:0 0 0 2024-03-22T02:28:36.154 INFO:tasks.workunit.client.0.smithi103.stdout:1/323: write db/d4a/f5e [282161,3753] 0 2024-03-22T02:28:36.154 INFO:tasks.workunit.client.0.smithi103.stdout:2/278: fdatasync d11/d25/d46/f29 0 2024-03-22T02:28:36.157 INFO:tasks.workunit.client.0.smithi103.stdout:1/324: write db/d10/d21/f59 [669704,72603] 0 2024-03-22T02:28:36.159 INFO:tasks.workunit.client.0.smithi103.stdout:2/279: rename d11/l23 to d11/d25/d43/l59 0 2024-03-22T02:28:36.162 INFO:tasks.workunit.client.0.smithi103.stdout:1/325: mkdir db/d10/d29/d41/d68/d74 0 2024-03-22T02:28:36.163 INFO:tasks.workunit.client.0.smithi103.stdout:1/326: chown db/d10/d2e/f51 13047 1 2024-03-22T02:28:36.164 INFO:tasks.workunit.client.0.smithi103.stdout:1/327: mknod db/d10/c75 0 2024-03-22T02:28:36.167 INFO:tasks.workunit.client.0.smithi103.stdout:1/328: creat db/d10/d29/d41/d68/f76 x:0 0 0 2024-03-22T02:28:36.169 INFO:tasks.workunit.client.0.smithi103.stdout:1/329: mknod db/d10/d18/d2d/c77 0 2024-03-22T02:28:36.173 INFO:tasks.workunit.client.0.smithi103.stdout:1/330: mkdir db/d4a/d78 0 2024-03-22T02:28:36.173 INFO:tasks.workunit.client.0.smithi103.stdout:7/300: dwrite d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:36.176 INFO:tasks.workunit.client.0.smithi103.stdout:7/301: dread - d1/d10/d11/d19/d29/d61/d69/f32 zero size 2024-03-22T02:28:36.180 INFO:tasks.workunit.client.0.smithi103.stdout:2/280: dread f7 [0,4194304] 0 2024-03-22T02:28:36.180 INFO:tasks.workunit.client.0.smithi103.stdout:7/302: write d1/d10/d11/d19/d29/d61/d69/f62 [3789881,722] 0 2024-03-22T02:28:36.185 INFO:tasks.workunit.client.0.smithi103.stdout:7/303: rmdir d1/d10/d11 39 2024-03-22T02:28:36.185 INFO:tasks.workunit.client.0.smithi103.stdout:1/331: dwrite db/d10/d18/d2d/f48 [0,4194304] 0 2024-03-22T02:28:36.187 INFO:tasks.workunit.client.0.smithi103.stdout:2/281: creat d11/f5a x:0 0 0 2024-03-22T02:28:36.192 INFO:tasks.workunit.client.0.smithi103.stdout:8/311: dwrite d4/d26/f4e [0,4194304] 0 2024-03-22T02:28:36.194 INFO:tasks.workunit.client.0.smithi103.stdout:6/228: dwrite d0/d3/d22/f11 [0,4194304] 0 2024-03-22T02:28:36.195 INFO:tasks.workunit.client.0.smithi103.stdout:1/332: write db/d10/f63 [549333,60819] 0 2024-03-22T02:28:36.196 INFO:tasks.workunit.client.0.smithi103.stdout:2/282: mknod d11/d25/d43/c5b 0 2024-03-22T02:28:36.203 INFO:tasks.workunit.client.0.smithi103.stdout:7/304: unlink d1/d10/d11/d19/d29/d61/d1d/f53 0 2024-03-22T02:28:36.206 INFO:tasks.workunit.client.0.smithi103.stdout:8/312: creat d4/d19/f5c x:0 0 0 2024-03-22T02:28:36.209 INFO:tasks.workunit.client.0.smithi103.stdout:6/229: link d0/c34 d0/d14/c4b 0 2024-03-22T02:28:36.209 INFO:tasks.workunit.client.0.smithi103.stdout:1/333: dwrite db/d10/f1c [4194304,4194304] 0 2024-03-22T02:28:36.212 INFO:tasks.workunit.client.0.smithi103.stdout:2/283: dwrite d11/f30 [8388608,4194304] 0 2024-03-22T02:28:36.214 INFO:tasks.workunit.client.0.smithi103.stdout:6/230: write d0/d3/d22/f15 [466183,37717] 0 2024-03-22T02:28:36.215 INFO:tasks.workunit.client.0.smithi103.stdout:1/334: read - db/d10/d29/d55/f6d zero size 2024-03-22T02:28:36.217 INFO:tasks.workunit.client.0.smithi103.stdout:2/284: symlink d11/d25/d43/l5c 0 2024-03-22T02:28:36.218 INFO:tasks.workunit.client.0.smithi103.stdout:6/231: dread - d0/f41 zero size 2024-03-22T02:28:36.222 INFO:tasks.workunit.client.0.smithi103.stdout:3/296: sync 2024-03-22T02:28:36.222 INFO:tasks.workunit.client.0.smithi103.stdout:6/232: fsync d0/d3/d22/f11 0 2024-03-22T02:28:36.222 INFO:tasks.workunit.client.0.smithi103.stdout:2/285: readlink d11/d19/l4a 0 2024-03-22T02:28:36.223 INFO:tasks.workunit.client.0.smithi103.stdout:7/305: dwrite d1/d10/d11/d19/d29/d61/d69/f32 [0,4194304] 0 2024-03-22T02:28:36.224 INFO:tasks.workunit.client.0.smithi103.stdout:1/335: dread db/d10/f12 [0,4194304] 0 2024-03-22T02:28:36.229 INFO:tasks.workunit.client.0.smithi103.stdout:7/306: chown d1/d10/d11/d19/d29/d61/d69/c27 11619791 1 2024-03-22T02:28:36.230 INFO:tasks.workunit.client.0.smithi103.stdout:7/307: chown d1/d10/d11/d19/d29/d61/d69/f32 205151 1 2024-03-22T02:28:36.230 INFO:tasks.workunit.client.0.smithi103.stdout:2/286: unlink d11/d25/d46/l1b 0 2024-03-22T02:28:36.234 INFO:tasks.workunit.client.0.smithi103.stdout:6/233: creat d0/d14/f4c x:0 0 0 2024-03-22T02:28:36.234 INFO:tasks.workunit.client.0.smithi103.stdout:7/308: chown d1/d10/d11/d19/d29/d33 2 1 2024-03-22T02:28:36.235 INFO:tasks.workunit.client.0.smithi103.stdout:2/287: rmdir d11/d25/d43/d45/d47 39 2024-03-22T02:28:36.236 INFO:tasks.workunit.client.0.smithi103.stdout:1/336: dread db/d10/d2e/f51 [0,4194304] 0 2024-03-22T02:28:36.237 INFO:tasks.workunit.client.0.smithi103.stdout:6/234: write d0/d14/f29 [4176539,64278] 0 2024-03-22T02:28:36.237 INFO:tasks.workunit.client.0.smithi103.stdout:2/288: fdatasync d11/f3a 0 2024-03-22T02:28:36.238 INFO:tasks.workunit.client.0.smithi103.stdout:1/337: readlink db/l50 0 2024-03-22T02:28:36.238 INFO:tasks.workunit.client.0.smithi103.stdout:2/289: readlink d11/d19/l22 0 2024-03-22T02:28:36.239 INFO:tasks.workunit.client.0.smithi103.stdout:3/297: link de/d12/d19/f28 de/d1a/d25/d3f/f68 0 2024-03-22T02:28:36.241 INFO:tasks.workunit.client.0.smithi103.stdout:3/298: chown de/f5f 242 1 2024-03-22T02:28:36.241 INFO:tasks.workunit.client.0.smithi103.stdout:8/313: dread d4/d19/d1a/f2d [0,4194304] 0 2024-03-22T02:28:36.242 INFO:tasks.workunit.client.0.smithi103.stdout:6/235: readlink d0/d3/l12 0 2024-03-22T02:28:36.243 INFO:tasks.workunit.client.0.smithi103.stdout:6/236: stat d0/f1a 0 2024-03-22T02:28:36.244 INFO:tasks.workunit.client.0.smithi103.stdout:8/314: read d4/d26/f4e [3874476,47260] 0 2024-03-22T02:28:36.244 INFO:tasks.workunit.client.0.smithi103.stdout:2/290: mkdir d11/d19/d5d 0 2024-03-22T02:28:36.247 INFO:tasks.workunit.client.0.smithi103.stdout:7/309: dread d1/da/fe [0,4194304] 0 2024-03-22T02:28:36.248 INFO:tasks.workunit.client.0.smithi103.stdout:1/338: write db/d10/f5a [1932371,13087] 0 2024-03-22T02:28:36.248 INFO:tasks.workunit.client.0.smithi103.stdout:8/315: symlink d4/d14/d39/l5d 0 2024-03-22T02:28:36.250 INFO:tasks.workunit.client.0.smithi103.stdout:1/339: write db/f17 [3058371,21434] 0 2024-03-22T02:28:36.255 INFO:tasks.workunit.client.0.smithi103.stdout:6/237: rename d0/d3/d33/l38 to d0/d3/d33/l4d 0 2024-03-22T02:28:36.256 INFO:tasks.workunit.client.0.smithi103.stdout:1/340: dread db/d4a/f53 [0,4194304] 0 2024-03-22T02:28:36.256 INFO:tasks.workunit.client.0.smithi103.stdout:7/310: dwrite d1/f4e [0,4194304] 0 2024-03-22T02:28:36.256 INFO:tasks.workunit.client.0.smithi103.stdout:8/316: mkdir d4/d48/d40/d5e 0 2024-03-22T02:28:36.261 INFO:tasks.workunit.client.0.smithi103.stdout:1/341: dread db/d10/d21/f23 [0,4194304] 0 2024-03-22T02:28:36.267 INFO:tasks.workunit.client.0.smithi103.stdout:3/299: dwrite de/d12/f1c [0,4194304] 0 2024-03-22T02:28:36.274 INFO:tasks.workunit.client.0.smithi103.stdout:8/317: mkdir d4/d14/d39/d49/d5f 0 2024-03-22T02:28:36.279 INFO:tasks.workunit.client.0.smithi103.stdout:3/300: creat de/d1a/d25/d26/f69 x:0 0 0 2024-03-22T02:28:36.279 INFO:tasks.workunit.client.0.smithi103.stdout:6/238: symlink d0/d3/d45/l4e 0 2024-03-22T02:28:36.282 INFO:tasks.workunit.client.0.smithi103.stdout:1/342: link db/d10/d18/f72 db/d10/f79 0 2024-03-22T02:28:36.286 INFO:tasks.workunit.client.0.smithi103.stdout:8/318: creat d4/d14/d39/d49/d5f/f60 x:0 0 0 2024-03-22T02:28:36.287 INFO:tasks.workunit.client.0.smithi103.stdout:6/239: dwrite d0/d3/d45/f40 [0,4194304] 0 2024-03-22T02:28:36.287 INFO:tasks.workunit.client.0.smithi103.stdout:7/311: symlink d1/da/d2f/d36/l6a 0 2024-03-22T02:28:36.287 INFO:tasks.workunit.client.0.smithi103.stdout:3/301: dread de/d1a/f2a [0,4194304] 0 2024-03-22T02:28:36.292 INFO:tasks.workunit.client.0.smithi103.stdout:3/302: creat de/d1a/d25/f6a x:0 0 0 2024-03-22T02:28:36.293 INFO:tasks.workunit.client.0.smithi103.stdout:1/343: getdents db/d10/d18/d2d 0 2024-03-22T02:28:36.294 INFO:tasks.workunit.client.0.smithi103.stdout:6/240: mkdir d0/d3/d4f 0 2024-03-22T02:28:36.295 INFO:tasks.workunit.client.0.smithi103.stdout:7/312: mkdir d1/d10/d11/d19/d29/d61/d69/d6b 0 2024-03-22T02:28:36.301 INFO:tasks.workunit.client.0.smithi103.stdout:1/344: dwrite db/f45 [0,4194304] 0 2024-03-22T02:28:36.302 INFO:tasks.workunit.client.0.smithi103.stdout:3/303: mkdir de/d6b 0 2024-03-22T02:28:36.304 INFO:tasks.workunit.client.0.smithi103.stdout:6/241: dwrite d0/d3/d33/f48 [0,4194304] 0 2024-03-22T02:28:36.311 INFO:tasks.workunit.client.0.smithi103.stdout:3/304: creat de/d1a/d25/f6c x:0 0 0 2024-03-22T02:28:36.312 INFO:tasks.workunit.client.0.smithi103.stdout:8/319: truncate d4/d19/d1a/f37 530901 0 2024-03-22T02:28:36.313 INFO:tasks.workunit.client.0.smithi103.stdout:7/313: getdents d1/d10/d11/d19/d29/d61 0 2024-03-22T02:28:36.314 INFO:tasks.workunit.client.0.smithi103.stdout:1/345: dwrite db/d10/d18/f3b [4194304,4194304] 0 2024-03-22T02:28:36.316 INFO:tasks.workunit.client.0.smithi103.stdout:3/305: write de/d1a/d25/d26/f53 [695367,67927] 0 2024-03-22T02:28:36.316 INFO:tasks.workunit.client.0.smithi103.stdout:6/242: creat d0/d3/d22/f50 x:0 0 0 2024-03-22T02:28:36.317 INFO:tasks.workunit.client.0.smithi103.stdout:7/314: write d1/f4e [3157473,12425] 0 2024-03-22T02:28:36.318 INFO:tasks.workunit.client.0.smithi103.stdout:1/346: chown db/d10/d18/f40 116 1 2024-03-22T02:28:36.318 INFO:tasks.workunit.client.0.smithi103.stdout:8/320: chown d4/d19/d1a/d4a/l53 0 1 2024-03-22T02:28:36.319 INFO:tasks.workunit.client.0.smithi103.stdout:8/321: stat d4/d14/c16 0 2024-03-22T02:28:36.320 INFO:tasks.workunit.client.0.smithi103.stdout:3/306: unlink de/d1a/f2a 0 2024-03-22T02:28:36.321 INFO:tasks.workunit.client.0.smithi103.stdout:3/307: dread - de/d1a/d25/f6a zero size 2024-03-22T02:28:36.322 INFO:tasks.workunit.client.0.smithi103.stdout:8/322: symlink d4/d19/d1a/d4a/l61 0 2024-03-22T02:28:36.324 INFO:tasks.workunit.client.0.smithi103.stdout:8/323: chown d4/c56 148967099 1 2024-03-22T02:28:36.325 INFO:tasks.workunit.client.0.smithi103.stdout:7/315: truncate d1/d10/d11/d19/d29/d61/f46 2388744 0 2024-03-22T02:28:36.326 INFO:tasks.workunit.client.0.smithi103.stdout:8/324: read - d4/d26/d27/f4d zero size 2024-03-22T02:28:36.329 INFO:tasks.workunit.client.0.smithi103.stdout:7/316: rmdir d1/da 39 2024-03-22T02:28:36.330 INFO:tasks.workunit.client.0.smithi103.stdout:8/325: truncate d4/d26/f4e 1484204 0 2024-03-22T02:28:36.331 INFO:tasks.workunit.client.0.smithi103.stdout:3/308: link de/cf de/d1a/d25/d3f/c6d 0 2024-03-22T02:28:36.338 INFO:tasks.workunit.client.0.smithi103.stdout:3/309: dwrite de/f5f [0,4194304] 0 2024-03-22T02:28:36.339 INFO:tasks.workunit.client.0.smithi103.stdout:8/326: creat d4/d19/f62 x:0 0 0 2024-03-22T02:28:36.346 INFO:tasks.workunit.client.0.smithi103.stdout:3/310: dwrite de/f5f [0,4194304] 0 2024-03-22T02:28:36.348 INFO:tasks.workunit.client.0.smithi103.stdout:7/317: write d1/da/fe [1681209,125330] 0 2024-03-22T02:28:36.350 INFO:tasks.workunit.client.0.smithi103.stdout:3/311: creat de/d12/d30/f6e x:0 0 0 2024-03-22T02:28:36.350 INFO:tasks.workunit.client.0.smithi103.stdout:7/318: write d1/d10/d11/d19/d29/d61/d69/f32 [1732199,3397] 0 2024-03-22T02:28:36.352 INFO:tasks.workunit.client.0.smithi103.stdout:3/312: write de/f5f [5130981,50442] 0 2024-03-22T02:28:36.353 INFO:tasks.workunit.client.0.smithi103.stdout:1/347: dread db/d10/d18/f20 [0,4194304] 0 2024-03-22T02:28:36.354 INFO:tasks.workunit.client.0.smithi103.stdout:7/319: readlink d1/d10/d11/l12 0 2024-03-22T02:28:36.355 INFO:tasks.workunit.client.0.smithi103.stdout:3/313: creat de/d1a/d25/d26/f6f x:0 0 0 2024-03-22T02:28:36.356 INFO:tasks.workunit.client.0.smithi103.stdout:1/348: symlink db/d10/l7a 0 2024-03-22T02:28:36.359 INFO:tasks.workunit.client.0.smithi103.stdout:3/314: mkdir de/d12/d30/d70 0 2024-03-22T02:28:36.359 INFO:tasks.workunit.client.0.smithi103.stdout:3/315: readlink de/d12/l18 0 2024-03-22T02:28:36.363 INFO:tasks.workunit.client.0.smithi103.stdout:7/320: getdents d1/d10/d11/d19/d29/d33/d4b 0 2024-03-22T02:28:36.363 INFO:tasks.workunit.client.0.smithi103.stdout:1/349: getdents db 0 2024-03-22T02:28:36.365 INFO:tasks.workunit.client.0.smithi103.stdout:1/350: chown db/d10/d29/c44 1234 1 2024-03-22T02:28:36.366 INFO:tasks.workunit.client.0.smithi103.stdout:1/351: stat c6 0 2024-03-22T02:28:36.367 INFO:tasks.workunit.client.0.smithi103.stdout:3/316: dwrite de/d1a/d25/d26/f54 [0,4194304] 0 2024-03-22T02:28:36.369 INFO:tasks.workunit.client.0.smithi103.stdout:7/321: dread d1/f4e [0,4194304] 0 2024-03-22T02:28:36.371 INFO:tasks.workunit.client.0.smithi103.stdout:1/352: truncate db/d4a/f53 1877272 0 2024-03-22T02:28:36.374 INFO:tasks.workunit.client.0.smithi103.stdout:7/322: fsync d1/d10/d11/d19/d29/d61/f46 0 2024-03-22T02:28:36.376 INFO:tasks.workunit.client.0.smithi103.stdout:1/353: unlink db/d10/d29/d55/f58 0 2024-03-22T02:28:36.376 INFO:tasks.workunit.client.0.smithi103.stdout:0/390: sync 2024-03-22T02:28:36.376 INFO:tasks.workunit.client.0.smithi103.stdout:4/319: sync 2024-03-22T02:28:36.376 INFO:tasks.workunit.client.0.smithi103.stdout:9/227: sync 2024-03-22T02:28:36.377 INFO:tasks.workunit.client.0.smithi103.stdout:5/336: sync 2024-03-22T02:28:36.379 INFO:tasks.workunit.client.0.smithi103.stdout:3/317: dwrite de/d12/d47/d4f/f65 [0,4194304] 0 2024-03-22T02:28:36.381 INFO:tasks.workunit.client.0.smithi103.stdout:5/337: write d0/f2d [2233021,125725] 0 2024-03-22T02:28:36.381 INFO:tasks.workunit.client.0.smithi103.stdout:9/228: read d2/da/fd [1330569,36388] 0 2024-03-22T02:28:36.381 INFO:tasks.workunit.client.0.smithi103.stdout:0/391: dread - dd/d16/d20/d35/d49/f7b zero size 2024-03-22T02:28:36.382 INFO:tasks.workunit.client.0.smithi103.stdout:7/323: mknod d1/d10/d11/d19/d29/c6c 0 2024-03-22T02:28:36.382 INFO:tasks.workunit.client.0.smithi103.stdout:1/354: creat db/d10/d29/d6c/f7b x:0 0 0 2024-03-22T02:28:36.383 INFO:tasks.workunit.client.0.smithi103.stdout:7/324: chown d1/d10 0 1 2024-03-22T02:28:36.385 INFO:tasks.workunit.client.0.smithi103.stdout:4/320: dread d3/d31/f32 [0,4194304] 0 2024-03-22T02:28:36.385 INFO:tasks.workunit.client.0.smithi103.stdout:5/338: chown d0/d14/d23/d46/l53 14 1 2024-03-22T02:28:36.385 INFO:tasks.workunit.client.0.smithi103.stdout:9/229: dread - d2/f2c zero size 2024-03-22T02:28:36.386 INFO:tasks.workunit.client.0.smithi103.stdout:1/355: dread - db/d10/d29/d41/d68/f76 zero size 2024-03-22T02:28:36.387 INFO:tasks.workunit.client.0.smithi103.stdout:3/318: write de/d1a/f31 [11609578,38101] 0 2024-03-22T02:28:36.391 INFO:tasks.workunit.client.0.smithi103.stdout:3/319: chown de/d1a/d25/d3f/c5a 9079374 1 2024-03-22T02:28:36.391 INFO:tasks.workunit.client.0.smithi103.stdout:0/392: dread dd/d16/d20/d31/f70 [0,4194304] 0 2024-03-22T02:28:36.393 INFO:tasks.workunit.client.0.smithi103.stdout:1/356: creat db/d10/d29/d55/f7c x:0 0 0 2024-03-22T02:28:36.395 INFO:tasks.workunit.client.0.smithi103.stdout:3/320: creat de/d12/d19/f71 x:0 0 0 2024-03-22T02:28:36.395 INFO:tasks.workunit.client.0.smithi103.stdout:1/357: fdatasync db/d10/f26 0 2024-03-22T02:28:36.395 INFO:tasks.workunit.client.0.smithi103.stdout:7/325: read d1/d10/d11/d19/d29/d61/f35 [2295976,119513] 0 2024-03-22T02:28:36.396 INFO:tasks.workunit.client.0.smithi103.stdout:4/321: dwrite d3/fd [0,4194304] 0 2024-03-22T02:28:36.397 INFO:tasks.workunit.client.0.smithi103.stdout:1/358: readlink db/d10/d29/l38 0 2024-03-22T02:28:36.398 INFO:tasks.workunit.client.0.smithi103.stdout:7/326: read d1/f4e [3756914,121350] 0 2024-03-22T02:28:36.398 INFO:tasks.workunit.client.0.smithi103.stdout:3/321: chown de/d1a/d25/l38 6261736 1 2024-03-22T02:28:36.399 INFO:tasks.workunit.client.0.smithi103.stdout:9/230: mkdir d2/da/d18/d1e/d34/d3d/d46 0 2024-03-22T02:28:36.401 INFO:tasks.workunit.client.0.smithi103.stdout:0/393: unlink dd/d16/d4b/l56 0 2024-03-22T02:28:36.402 INFO:tasks.workunit.client.0.smithi103.stdout:1/359: dread f5 [0,4194304] 0 2024-03-22T02:28:36.403 INFO:tasks.workunit.client.0.smithi103.stdout:4/322: symlink d3/d31/l7b 0 2024-03-22T02:28:36.404 INFO:tasks.workunit.client.0.smithi103.stdout:9/231: unlink d2/d5/f3a 0 2024-03-22T02:28:36.405 INFO:tasks.workunit.client.0.smithi103.stdout:7/327: write d1/d10/d11/d19/f26 [1792133,92606] 0 2024-03-22T02:28:36.406 INFO:tasks.workunit.client.0.smithi103.stdout:3/322: truncate de/d1a/d25/d26/f58 5127628 0 2024-03-22T02:28:36.406 INFO:tasks.workunit.client.0.smithi103.stdout:0/394: mknod dd/d16/d20/d29/d48/d5c/c81 0 2024-03-22T02:28:36.412 INFO:tasks.workunit.client.0.smithi103.stdout:5/339: dread d0/d14/d23/f44 [0,4194304] 0 2024-03-22T02:28:36.414 INFO:tasks.workunit.client.0.smithi103.stdout:0/395: rename dd/d16/d20/d35/d49 to dd/d16/d20/d35/d82 0 2024-03-22T02:28:36.416 INFO:tasks.workunit.client.0.smithi103.stdout:3/323: dwrite de/f2d [0,4194304] 0 2024-03-22T02:28:36.417 INFO:tasks.workunit.client.0.smithi103.stdout:7/328: rename d1/d10/d11/d19/d29/d61/f35 to d1/d10/d11/d19/d29/d57/f6d 0 2024-03-22T02:28:36.420 INFO:tasks.workunit.client.0.smithi103.stdout:5/340: read d0/d14/d23/d2a/d1f/f32 [3125772,85035] 0 2024-03-22T02:28:36.421 INFO:tasks.workunit.client.0.smithi103.stdout:9/232: rmdir d2/da/d18/d1e/d3f 0 2024-03-22T02:28:36.424 INFO:tasks.workunit.client.0.smithi103.stdout:3/324: unlink de/d1a/d25/c32 0 2024-03-22T02:28:36.426 INFO:tasks.workunit.client.0.smithi103.stdout:4/323: link d3/d69/l73 d3/d1e/d25/l7c 0 2024-03-22T02:28:36.430 INFO:tasks.workunit.client.0.smithi103.stdout:3/325: read de/d1a/d25/d26/f53 [1468980,33768] 0 2024-03-22T02:28:36.431 INFO:tasks.workunit.client.0.smithi103.stdout:3/326: write de/d12/d19/f71 [762944,22713] 0 2024-03-22T02:28:36.436 INFO:tasks.workunit.client.0.smithi103.stdout:7/329: unlink d1/c9 0 2024-03-22T02:28:36.439 INFO:tasks.workunit.client.0.smithi103.stdout:5/341: unlink d0/d3e/l2b 0 2024-03-22T02:28:36.439 INFO:tasks.workunit.client.0.smithi103.stdout:1/360: dwrite db/d10/d18/f72 [0,4194304] 0 2024-03-22T02:28:36.440 INFO:tasks.workunit.client.0.smithi103.stdout:5/342: fsync d0/d14/d23/d36/f54 0 2024-03-22T02:28:36.442 INFO:tasks.workunit.client.0.smithi103.stdout:5/343: truncate d0/d14/d23/d2a/d1f/f78 431458 0 2024-03-22T02:28:36.444 INFO:tasks.workunit.client.0.smithi103.stdout:0/396: link dd/d16/d20/d29/f4d dd/d16/d20/d29/f83 0 2024-03-22T02:28:36.444 INFO:tasks.workunit.client.0.smithi103.stdout:9/233: rmdir d2/da 39 2024-03-22T02:28:36.448 INFO:tasks.workunit.client.0.smithi103.stdout:5/344: stat d0/d14/d23/d2a/d1f/f78 0 2024-03-22T02:28:36.448 INFO:tasks.workunit.client.0.smithi103.stdout:9/234: write d2/f30 [635798,122968] 0 2024-03-22T02:28:36.449 INFO:tasks.workunit.client.0.smithi103.stdout:0/397: truncate dd/d16/d4b/f50 263283 0 2024-03-22T02:28:36.449 INFO:tasks.workunit.client.0.smithi103.stdout:5/345: truncate d0/f71 5424776 0 2024-03-22T02:28:36.449 INFO:tasks.workunit.client.0.smithi103.stdout:4/324: creat d3/d1e/d53/d7a/f7d x:0 0 0 2024-03-22T02:28:36.451 INFO:tasks.workunit.client.0.smithi103.stdout:7/330: fdatasync d1/d10/d11/d19/d29/d61/f46 0 2024-03-22T02:28:36.454 INFO:tasks.workunit.client.0.smithi103.stdout:9/235: mknod d2/d10/d43/c47 0 2024-03-22T02:28:36.456 INFO:tasks.workunit.client.0.smithi103.stdout:4/325: creat d3/d1e/f7e x:0 0 0 2024-03-22T02:28:36.459 INFO:tasks.workunit.client.0.smithi103.stdout:7/331: symlink d1/d10/d11/d19/d29/d61/l6e 0 2024-03-22T02:28:36.459 INFO:tasks.workunit.client.0.smithi103.stdout:4/326: write d3/d30/f55 [3191130,115478] 0 2024-03-22T02:28:36.460 INFO:tasks.workunit.client.0.smithi103.stdout:4/327: chown d3/d1e/d53/d7a 86627 1 2024-03-22T02:28:36.460 INFO:tasks.workunit.client.0.smithi103.stdout:9/236: write d2/d10/d17/f26 [4445434,23104] 0 2024-03-22T02:28:36.465 INFO:tasks.workunit.client.0.smithi103.stdout:0/398: creat dd/d16/f84 x:0 0 0 2024-03-22T02:28:36.467 INFO:tasks.workunit.client.0.smithi103.stdout:1/361: dread db/f2b [0,4194304] 0 2024-03-22T02:28:36.467 INFO:tasks.workunit.client.0.smithi103.stdout:5/346: dread d0/d14/d23/d2a/f63 [0,4194304] 0 2024-03-22T02:28:36.468 INFO:tasks.workunit.client.0.smithi103.stdout:4/328: fsync d3/d31/f32 0 2024-03-22T02:28:36.473 INFO:tasks.workunit.client.0.smithi103.stdout:7/332: dwrite d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:36.474 INFO:tasks.workunit.client.0.smithi103.stdout:7/333: fdatasync d1/da/fe 0 2024-03-22T02:28:36.476 INFO:tasks.workunit.client.0.smithi103.stdout:7/334: write d1/d10/d11/d19/f56 [944903,47028] 0 2024-03-22T02:28:36.477 INFO:tasks.workunit.client.0.smithi103.stdout:7/335: fsync d1/d10/d11/d19/d29/d61/d69/f32 0 2024-03-22T02:28:36.481 INFO:tasks.workunit.client.0.smithi103.stdout:4/329: rename d3/d30/f3a to d3/d1e/f7f 0 2024-03-22T02:28:36.482 INFO:tasks.workunit.client.0.smithi103.stdout:1/362: getdents db/d10/d6b 0 2024-03-22T02:28:36.483 INFO:tasks.workunit.client.0.smithi103.stdout:4/330: write d3/d1e/f43 [190179,96752] 0 2024-03-22T02:28:36.483 INFO:tasks.workunit.client.0.smithi103.stdout:1/363: stat db/d10/f26 0 2024-03-22T02:28:36.485 INFO:tasks.workunit.client.0.smithi103.stdout:1/364: chown db/d10/d28/c46 68565 1 2024-03-22T02:28:36.486 INFO:tasks.workunit.client.0.smithi103.stdout:5/347: mknod d0/d14/d23/d2a/d7b/c7c 0 2024-03-22T02:28:36.494 INFO:tasks.workunit.client.0.smithi103.stdout:5/348: creat d0/d14/d23/d36/d50/f7d x:0 0 0 2024-03-22T02:28:36.494 INFO:tasks.workunit.client.0.smithi103.stdout:7/336: mkdir d1/d6f 0 2024-03-22T02:28:36.497 INFO:tasks.workunit.client.0.smithi103.stdout:9/237: getdents d2/da 0 2024-03-22T02:28:36.499 INFO:tasks.workunit.client.0.smithi103.stdout:0/399: rename dd/d16/d20/d35/d82/f6e to dd/d16/d20/d29/f85 0 2024-03-22T02:28:36.500 INFO:tasks.workunit.client.0.smithi103.stdout:9/238: write d2/da/f25 [17412,119310] 0 2024-03-22T02:28:36.500 INFO:tasks.workunit.client.0.smithi103.stdout:1/365: dwrite db/d10/d18/d2d/f16 [4194304,4194304] 0 2024-03-22T02:28:36.502 INFO:tasks.workunit.client.0.smithi103.stdout:4/331: link d3/d1e/d53/l5f d3/d69/l80 0 2024-03-22T02:28:36.503 INFO:tasks.workunit.client.0.smithi103.stdout:4/332: read - d3/d1e/d25/f52 zero size 2024-03-22T02:28:36.505 INFO:tasks.workunit.client.0.smithi103.stdout:7/337: mkdir d1/d10/d11/d70 0 2024-03-22T02:28:36.506 INFO:tasks.workunit.client.0.smithi103.stdout:1/366: mknod db/d10/d29/c7d 0 2024-03-22T02:28:36.506 INFO:tasks.workunit.client.0.smithi103.stdout:5/349: dread d0/d14/d23/d2a/d1f/f38 [0,4194304] 0 2024-03-22T02:28:36.507 INFO:tasks.workunit.client.0.smithi103.stdout:7/338: stat d1/d10/d11/d19/d29/d61/d1d/f41 0 2024-03-22T02:28:36.507 INFO:tasks.workunit.client.0.smithi103.stdout:5/350: fdatasync d0/d14/d23/d46/f7a 0 2024-03-22T02:28:36.507 INFO:tasks.workunit.client.0.smithi103.stdout:1/367: chown db/d10/d18/d2d/c77 46 1 2024-03-22T02:28:36.507 INFO:tasks.workunit.client.0.smithi103.stdout:4/333: read d3/d1e/f7f [2609797,112856] 0 2024-03-22T02:28:36.508 INFO:tasks.workunit.client.0.smithi103.stdout:1/368: chown f5 32644 1 2024-03-22T02:28:36.508 INFO:tasks.workunit.client.0.smithi103.stdout:4/334: dread - d3/d1e/f6d zero size 2024-03-22T02:28:36.511 INFO:tasks.workunit.client.0.smithi103.stdout:1/369: dread db/d10/d18/d2d/f48 [0,4194304] 0 2024-03-22T02:28:36.601 INFO:tasks.workunit.client.0.smithi103.stdout:2/291: sync 2024-03-22T02:28:36.604 INFO:tasks.workunit.client.0.smithi103.stdout:0/400: mknod dd/d16/d20/d29/c86 0 2024-03-22T02:28:36.607 INFO:tasks.workunit.client.0.smithi103.stdout:1/370: symlink db/d10/d29/d6c/l7e 0 2024-03-22T02:28:36.612 INFO:tasks.workunit.client.0.smithi103.stdout:4/335: dwrite d3/d1e/f36 [4194304,4194304] 0 2024-03-22T02:28:36.613 INFO:tasks.workunit.client.0.smithi103.stdout:1/371: dread db/f45 [0,4194304] 0 2024-03-22T02:28:36.622 INFO:tasks.workunit.client.0.smithi103.stdout:1/372: dwrite db/d10/d29/f6e [0,4194304] 0 2024-03-22T02:28:36.630 INFO:tasks.workunit.client.0.smithi103.stdout:0/401: dwrite dd/d16/d20/d64/f71 [0,4194304] 0 2024-03-22T02:28:36.630 INFO:tasks.workunit.client.0.smithi103.stdout:4/336: link d3/d31/f6f d3/d38/f81 0 2024-03-22T02:28:36.634 INFO:tasks.workunit.client.0.smithi103.stdout:2/292: write d11/d25/d43/d45/d47/f49 [1674602,117542] 0 2024-03-22T02:28:36.634 INFO:tasks.workunit.client.0.smithi103.stdout:1/373: creat db/d10/d6b/f7f x:0 0 0 2024-03-22T02:28:36.638 INFO:tasks.workunit.client.0.smithi103.stdout:2/293: write d11/d19/f1f [3953356,66712] 0 2024-03-22T02:28:36.638 INFO:tasks.workunit.client.0.smithi103.stdout:7/339: dwrite d1/d10/d11/d19/d29/d57/f6d [0,4194304] 0 2024-03-22T02:28:36.638 INFO:tasks.workunit.client.0.smithi103.stdout:1/374: read db/d4a/f53 [347202,27771] 0 2024-03-22T02:28:36.640 INFO:tasks.workunit.client.0.smithi103.stdout:7/340: readlink d1/da/ld 0 2024-03-22T02:28:36.642 INFO:tasks.workunit.client.0.smithi103.stdout:1/375: dread db/d4a/f53 [0,4194304] 0 2024-03-22T02:28:36.642 INFO:tasks.workunit.client.0.smithi103.stdout:0/402: rename dd/d16/d20/d35/l4f to dd/d16/d20/l87 0 2024-03-22T02:28:36.645 INFO:tasks.workunit.client.0.smithi103.stdout:1/376: symlink db/d10/d21/l80 0 2024-03-22T02:28:36.647 INFO:tasks.workunit.client.0.smithi103.stdout:7/341: dread d1/da/fb [0,4194304] 0 2024-03-22T02:28:36.649 INFO:tasks.workunit.client.0.smithi103.stdout:2/294: dread d11/f18 [8388608,4194304] 0 2024-03-22T02:28:36.651 INFO:tasks.workunit.client.0.smithi103.stdout:1/377: rename db/d10/d21/f5b to db/d10/d29/d55/f81 0 2024-03-22T02:28:36.653 INFO:tasks.workunit.client.0.smithi103.stdout:1/378: fdatasync db/d10/d18/f72 0 2024-03-22T02:28:36.656 INFO:tasks.workunit.client.0.smithi103.stdout:2/295: symlink d11/d25/d43/d45/l5e 0 2024-03-22T02:28:36.658 INFO:tasks.workunit.client.0.smithi103.stdout:7/342: dwrite d1/d10/d11/f18 [0,4194304] 0 2024-03-22T02:28:36.659 INFO:tasks.workunit.client.0.smithi103.stdout:4/337: truncate d3/d1e/f20 285637 0 2024-03-22T02:28:36.662 INFO:tasks.workunit.client.0.smithi103.stdout:0/403: dwrite dd/d16/d20/d31/f32 [4194304,4194304] 0 2024-03-22T02:28:36.662 INFO:tasks.workunit.client.0.smithi103.stdout:1/379: rename db/d10/d18/f40 to db/d10/d29/d41/d68/d74/f82 0 2024-03-22T02:28:36.667 INFO:tasks.workunit.client.0.smithi103.stdout:0/404: write dd/d16/d20/f39 [344873,50938] 0 2024-03-22T02:28:36.670 INFO:tasks.workunit.client.0.smithi103.stdout:2/296: dwrite d11/d25/f35 [0,4194304] 0 2024-03-22T02:28:36.672 INFO:tasks.workunit.client.0.smithi103.stdout:2/297: chown lb 650155 1 2024-03-22T02:28:36.673 INFO:tasks.workunit.client.0.smithi103.stdout:1/380: write db/d10/d18/f20 [1481938,94701] 0 2024-03-22T02:28:36.674 INFO:tasks.workunit.client.0.smithi103.stdout:1/381: stat db/d10/d2e/c4e 0 2024-03-22T02:28:36.677 INFO:tasks.workunit.client.0.smithi103.stdout:2/298: symlink d11/d25/d46/d34/l5f 0 2024-03-22T02:28:36.677 INFO:tasks.workunit.client.0.smithi103.stdout:0/405: dwrite dd/f15 [0,4194304] 0 2024-03-22T02:28:36.677 INFO:tasks.workunit.client.0.smithi103.stdout:1/382: fsync db/d10/d29/d41/d68/f76 0 2024-03-22T02:28:36.679 INFO:tasks.workunit.client.0.smithi103.stdout:0/406: dread - dd/d16/d20/d29/f85 zero size 2024-03-22T02:28:36.680 INFO:tasks.workunit.client.0.smithi103.stdout:1/383: write db/f17 [335800,53151] 0 2024-03-22T02:28:36.681 INFO:tasks.workunit.client.0.smithi103.stdout:1/384: readlink db/d4a/l6a 0 2024-03-22T02:28:36.685 INFO:tasks.workunit.client.0.smithi103.stdout:4/338: write d3/d30/f49 [1164470,66952] 0 2024-03-22T02:28:36.686 INFO:tasks.workunit.client.0.smithi103.stdout:0/407: mknod dd/d34/d53/c88 0 2024-03-22T02:28:36.687 INFO:tasks.workunit.client.0.smithi103.stdout:1/385: stat db/f45 0 2024-03-22T02:28:36.688 INFO:tasks.workunit.client.0.smithi103.stdout:1/386: truncate db/d10/d29/d55/f7c 63661 0 2024-03-22T02:28:36.690 INFO:tasks.workunit.client.0.smithi103.stdout:7/343: truncate d1/d10/d11/d19/f55 1608031 0 2024-03-22T02:28:36.693 INFO:tasks.workunit.client.0.smithi103.stdout:2/299: getdents d11/d25/d46/d34 0 2024-03-22T02:28:36.694 INFO:tasks.workunit.client.0.smithi103.stdout:1/387: creat db/d10/f83 x:0 0 0 2024-03-22T02:28:36.697 INFO:tasks.workunit.client.0.smithi103.stdout:1/388: write db/d10/d29/d55/f6d [494841,26214] 0 2024-03-22T02:28:36.697 INFO:tasks.workunit.client.0.smithi103.stdout:7/344: dwrite d1/d10/d11/d19/f26 [0,4194304] 0 2024-03-22T02:28:36.697 INFO:tasks.workunit.client.0.smithi103.stdout:2/300: write d11/f58 [270810,128755] 0 2024-03-22T02:28:36.699 INFO:tasks.workunit.client.0.smithi103.stdout:2/301: stat d11/d25/d46/d34/f52 0 2024-03-22T02:28:36.701 INFO:tasks.workunit.client.0.smithi103.stdout:2/302: stat d11/f20 0 2024-03-22T02:28:36.702 INFO:tasks.workunit.client.0.smithi103.stdout:7/345: mknod d1/d10/d11/d19/d29/d33/d4b/d5e/c71 0 2024-03-22T02:28:36.703 INFO:tasks.workunit.client.0.smithi103.stdout:2/303: fdatasync d11/d25/d43/d45/d47/f49 0 2024-03-22T02:28:36.706 INFO:tasks.workunit.client.0.smithi103.stdout:7/346: write d1/d10/d11/d19/d29/d33/f5f [467210,106702] 0 2024-03-22T02:28:36.707 INFO:tasks.workunit.client.0.smithi103.stdout:1/389: dwrite db/d10/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:36.707 INFO:tasks.workunit.client.0.smithi103.stdout:4/339: read d3/d30/f49 [1516536,38270] 0 2024-03-22T02:28:36.710 INFO:tasks.workunit.client.0.smithi103.stdout:2/304: creat d11/d25/d46/d34/d24/f60 x:0 0 0 2024-03-22T02:28:36.710 INFO:tasks.workunit.client.0.smithi103.stdout:0/408: write dd/f13 [8829467,44625] 0 2024-03-22T02:28:36.711 INFO:tasks.workunit.client.0.smithi103.stdout:7/347: symlink d1/da/d1c/l72 0 2024-03-22T02:28:36.713 INFO:tasks.workunit.client.0.smithi103.stdout:4/340: creat d3/d1e/d25/d65/f82 x:0 0 0 2024-03-22T02:28:36.713 INFO:tasks.workunit.client.0.smithi103.stdout:0/409: dread - dd/d16/d20/d29/f4d zero size 2024-03-22T02:28:36.715 INFO:tasks.workunit.client.0.smithi103.stdout:1/390: dread db/d10/f5a [0,4194304] 0 2024-03-22T02:28:36.716 INFO:tasks.workunit.client.0.smithi103.stdout:1/391: chown db/d4a/f73 90 1 2024-03-22T02:28:36.717 INFO:tasks.workunit.client.0.smithi103.stdout:7/348: creat d1/d10/d11/d19/d29/f73 x:0 0 0 2024-03-22T02:28:36.718 INFO:tasks.workunit.client.0.smithi103.stdout:2/305: mknod d11/d19/d5d/c61 0 2024-03-22T02:28:36.719 INFO:tasks.workunit.client.0.smithi103.stdout:7/349: write d1/d10/d11/d19/d29/d61/d69/f32 [2094066,44446] 0 2024-03-22T02:28:36.720 INFO:tasks.workunit.client.0.smithi103.stdout:1/392: read db/f17 [3402893,102982] 0 2024-03-22T02:28:36.720 INFO:tasks.workunit.client.0.smithi103.stdout:4/341: rmdir d3/d38 39 2024-03-22T02:28:36.722 INFO:tasks.workunit.client.0.smithi103.stdout:0/410: write dd/f24 [1010142,116989] 0 2024-03-22T02:28:36.724 INFO:tasks.workunit.client.0.smithi103.stdout:1/393: fdatasync db/d4a/f53 0 2024-03-22T02:28:36.725 INFO:tasks.workunit.client.0.smithi103.stdout:4/342: mkdir d3/d1e/d25/d65/d83 0 2024-03-22T02:28:36.728 INFO:tasks.workunit.client.0.smithi103.stdout:1/394: mkdir db/d10/d18/d84 0 2024-03-22T02:28:36.728 INFO:tasks.workunit.client.0.smithi103.stdout:4/343: readlink d3/d38/l61 0 2024-03-22T02:28:36.729 INFO:tasks.workunit.client.0.smithi103.stdout:2/306: creat d11/d25/d43/d45/f62 x:0 0 0 2024-03-22T02:28:36.730 INFO:tasks.workunit.client.0.smithi103.stdout:7/350: creat d1/da/f74 x:0 0 0 2024-03-22T02:28:36.731 INFO:tasks.workunit.client.0.smithi103.stdout:1/395: symlink db/d10/l85 0 2024-03-22T02:28:36.731 INFO:tasks.workunit.client.0.smithi103.stdout:7/351: write d1/da/f74 [762206,80428] 0 2024-03-22T02:28:36.732 INFO:tasks.workunit.client.0.smithi103.stdout:4/344: write d3/f24 [5469019,115729] 0 2024-03-22T02:28:36.733 INFO:tasks.workunit.client.0.smithi103.stdout:4/345: chown d3/le 22389608 1 2024-03-22T02:28:36.734 INFO:tasks.workunit.client.0.smithi103.stdout:4/346: fdatasync d3/f8 0 2024-03-22T02:28:36.735 INFO:tasks.workunit.client.0.smithi103.stdout:4/347: chown d3/f24 3256660 1 2024-03-22T02:28:36.780 INFO:tasks.workunit.client.0.smithi103.stdout:6/243: sync 2024-03-22T02:28:36.821 INFO:tasks.workunit.client.0.smithi103.stdout:2/307: rename d11/d25/d43 to d11/d25/d63 0 2024-03-22T02:28:36.821 INFO:tasks.workunit.client.0.smithi103.stdout:6/244: readlink d0/d3/l12 0 2024-03-22T02:28:36.822 INFO:tasks.workunit.client.0.smithi103.stdout:7/352: creat d1/da/f75 x:0 0 0 2024-03-22T02:28:36.824 INFO:tasks.workunit.client.0.smithi103.stdout:7/353: write d1/da/fe [512532,8853] 0 2024-03-22T02:28:36.828 INFO:tasks.workunit.client.0.smithi103.stdout:6/245: link d0/d3/d22/c32 d0/d14/c51 0 2024-03-22T02:28:36.828 INFO:tasks.workunit.client.0.smithi103.stdout:1/396: dwrite db/d10/f5a [0,4194304] 0 2024-03-22T02:28:36.830 INFO:tasks.workunit.client.0.smithi103.stdout:1/397: chown c3 628 1 2024-03-22T02:28:36.831 INFO:tasks.workunit.client.0.smithi103.stdout:6/246: truncate d0/f3e 994216 0 2024-03-22T02:28:36.831 INFO:tasks.workunit.client.0.smithi103.stdout:7/354: creat d1/f76 x:0 0 0 2024-03-22T02:28:36.831 INFO:tasks.workunit.client.0.smithi103.stdout:4/348: dread d3/d1e/f43 [0,4194304] 0 2024-03-22T02:28:36.832 INFO:tasks.workunit.client.0.smithi103.stdout:2/308: dwrite d11/d25/d46/d34/f36 [0,4194304] 0 2024-03-22T02:28:36.832 INFO:tasks.workunit.client.0.smithi103.stdout:6/247: stat d0/d3 0 2024-03-22T02:28:36.833 INFO:tasks.workunit.client.0.smithi103.stdout:2/309: fsync d11/d25/f4c 0 2024-03-22T02:28:36.835 INFO:tasks.workunit.client.0.smithi103.stdout:4/349: write d3/f1f [619953,119857] 0 2024-03-22T02:28:36.837 INFO:tasks.workunit.client.0.smithi103.stdout:2/310: read - d11/d19/d4e/f53 zero size 2024-03-22T02:28:36.837 INFO:tasks.workunit.client.0.smithi103.stdout:6/248: chown d0/c34 789312 1 2024-03-22T02:28:36.838 INFO:tasks.workunit.client.0.smithi103.stdout:4/350: stat d3/d1e/l2e 0 2024-03-22T02:28:36.842 INFO:tasks.workunit.client.0.smithi103.stdout:2/311: symlink d11/d19/l64 0 2024-03-22T02:28:36.845 INFO:tasks.workunit.client.0.smithi103.stdout:1/398: dwrite db/d4a/f73 [0,4194304] 0 2024-03-22T02:28:36.847 INFO:tasks.workunit.client.0.smithi103.stdout:6/249: mknod d0/d3/c52 0 2024-03-22T02:28:36.850 INFO:tasks.workunit.client.0.smithi103.stdout:7/355: dwrite d1/d10/d11/d19/d29/d61/d1d/f41 [0,4194304] 0 2024-03-22T02:28:36.852 INFO:tasks.workunit.client.0.smithi103.stdout:4/351: dwrite d3/d69/f46 [0,4194304] 0 2024-03-22T02:28:36.854 INFO:tasks.workunit.client.0.smithi103.stdout:7/356: fdatasync d1/d10/d11/d19/d29/d57/f6d 0 2024-03-22T02:28:36.855 INFO:tasks.workunit.client.0.smithi103.stdout:1/399: creat db/d10/d29/d41/d68/f86 x:0 0 0 2024-03-22T02:28:36.855 INFO:tasks.workunit.client.0.smithi103.stdout:1/400: chown db/l50 165314683 1 2024-03-22T02:28:36.856 INFO:tasks.workunit.client.0.smithi103.stdout:6/250: creat d0/d3/f53 x:0 0 0 2024-03-22T02:28:36.858 INFO:tasks.workunit.client.0.smithi103.stdout:4/352: creat d3/d30/d45/f84 x:0 0 0 2024-03-22T02:28:36.864 INFO:tasks.workunit.client.0.smithi103.stdout:1/401: link db/f35 db/d10/d29/d6c/f87 0 2024-03-22T02:28:36.866 INFO:tasks.workunit.client.0.smithi103.stdout:2/312: truncate f7 1871994 0 2024-03-22T02:28:36.866 INFO:tasks.workunit.client.0.smithi103.stdout:6/251: dwrite d0/d3/d22/f3d [0,4194304] 0 2024-03-22T02:28:36.870 INFO:tasks.workunit.client.0.smithi103.stdout:6/252: readlink d0/d3/d33/l4d 0 2024-03-22T02:28:36.871 INFO:tasks.workunit.client.0.smithi103.stdout:6/253: chown d0/d14/c18 8284031 1 2024-03-22T02:28:36.871 INFO:tasks.workunit.client.0.smithi103.stdout:6/254: fdatasync d0/d3/d22/f50 0 2024-03-22T02:28:36.873 INFO:tasks.workunit.client.0.smithi103.stdout:2/313: creat d11/d25/d63/f65 x:0 0 0 2024-03-22T02:28:36.875 INFO:tasks.workunit.client.0.smithi103.stdout:7/357: getdents d1/d10/d11 0 2024-03-22T02:28:36.878 INFO:tasks.workunit.client.0.smithi103.stdout:7/358: write d1/d10/d11/f18 [882194,41054] 0 2024-03-22T02:28:36.878 INFO:tasks.workunit.client.0.smithi103.stdout:1/402: link db/d10/d18/d2d/f48 db/d10/d18/d2d/f88 0 2024-03-22T02:28:36.883 INFO:tasks.workunit.client.0.smithi103.stdout:6/255: dwrite d0/d3/d45/f2d [0,4194304] 0 2024-03-22T02:28:36.885 INFO:tasks.workunit.client.0.smithi103.stdout:4/353: truncate d3/d1e/f36 7639686 0 2024-03-22T02:28:36.886 INFO:tasks.workunit.client.0.smithi103.stdout:2/314: dwrite d11/d19/f1f [4194304,4194304] 0 2024-03-22T02:28:36.886 INFO:tasks.workunit.client.0.smithi103.stdout:1/403: write db/d10/d21/f23 [987021,22464] 0 2024-03-22T02:28:36.890 INFO:tasks.workunit.client.0.smithi103.stdout:6/256: fdatasync d0/d3/f39 0 2024-03-22T02:28:36.891 INFO:tasks.workunit.client.0.smithi103.stdout:4/354: creat d3/d38/f85 x:0 0 0 2024-03-22T02:28:36.893 INFO:tasks.workunit.client.0.smithi103.stdout:7/359: dread d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:36.896 INFO:tasks.workunit.client.0.smithi103.stdout:6/257: truncate d0/d3/f9 4969817 0 2024-03-22T02:28:36.896 INFO:tasks.workunit.client.0.smithi103.stdout:4/355: mkdir d3/d38/d5c/d86 0 2024-03-22T02:28:36.897 INFO:tasks.workunit.client.0.smithi103.stdout:6/258: chown d0/d14/f29 0 1 2024-03-22T02:28:36.898 INFO:tasks.workunit.client.0.smithi103.stdout:4/356: stat d3/d69/l74 0 2024-03-22T02:28:36.899 INFO:tasks.workunit.client.0.smithi103.stdout:4/357: stat d3/d30 0 2024-03-22T02:28:36.899 INFO:tasks.workunit.client.0.smithi103.stdout:2/315: link d11/f58 d11/d19/d5d/f66 0 2024-03-22T02:28:36.899 INFO:tasks.workunit.client.0.smithi103.stdout:1/404: dwrite f5 [0,4194304] 0 2024-03-22T02:28:36.900 INFO:tasks.workunit.client.0.smithi103.stdout:6/259: fsync d0/d14/f4c 0 2024-03-22T02:28:36.901 INFO:tasks.workunit.client.0.smithi103.stdout:4/358: write d3/d38/f63 [2663535,87955] 0 2024-03-22T02:28:36.907 INFO:tasks.workunit.client.0.smithi103.stdout:8/327: sync 2024-03-22T02:28:36.910 INFO:tasks.workunit.client.0.smithi103.stdout:4/359: unlink d3/d31/f6f 0 2024-03-22T02:28:36.912 INFO:tasks.workunit.client.0.smithi103.stdout:8/328: creat d4/d26/d27/f63 x:0 0 0 2024-03-22T02:28:36.913 INFO:tasks.workunit.client.0.smithi103.stdout:4/360: truncate d3/d1e/d53/d7a/f7d 451687 0 2024-03-22T02:28:36.915 INFO:tasks.workunit.client.0.smithi103.stdout:1/405: link db/d10/d29/d55/f6d db/d10/d29/d41/f89 0 2024-03-22T02:28:36.920 INFO:tasks.workunit.client.0.smithi103.stdout:2/316: dwrite d11/f58 [0,4194304] 0 2024-03-22T02:28:36.922 INFO:tasks.workunit.client.0.smithi103.stdout:6/260: getdents d0/d3/d45 0 2024-03-22T02:28:36.925 INFO:tasks.workunit.client.0.smithi103.stdout:1/406: dread db/d10/d29/d55/f6d [0,4194304] 0 2024-03-22T02:28:36.925 INFO:tasks.workunit.client.0.smithi103.stdout:8/329: dwrite d4/d48/d35/f44 [0,4194304] 0 2024-03-22T02:28:36.927 INFO:tasks.workunit.client.0.smithi103.stdout:7/360: dwrite d1/f49 [0,4194304] 0 2024-03-22T02:28:36.930 INFO:tasks.workunit.client.0.smithi103.stdout:8/330: fsync d4/d19/f5c 0 2024-03-22T02:28:36.930 INFO:tasks.workunit.client.0.smithi103.stdout:2/317: truncate d11/d25/d46/d34/f52 146617 0 2024-03-22T02:28:36.935 INFO:tasks.workunit.client.0.smithi103.stdout:6/261: truncate d0/d3/d22/f1c 1720865 0 2024-03-22T02:28:36.936 INFO:tasks.workunit.client.0.smithi103.stdout:2/318: creat d11/d25/d46/d40/f67 x:0 0 0 2024-03-22T02:28:36.937 INFO:tasks.workunit.client.0.smithi103.stdout:6/262: dread - d0/d3/d22/f37 zero size 2024-03-22T02:28:36.938 INFO:tasks.workunit.client.0.smithi103.stdout:8/331: rmdir d4/d14/d39/d49 39 2024-03-22T02:28:36.940 INFO:tasks.workunit.client.0.smithi103.stdout:1/407: dwrite db/d10/d29/d41/d68/f76 [0,4194304] 0 2024-03-22T02:28:36.940 INFO:tasks.workunit.client.0.smithi103.stdout:4/361: getdents d3/d69 0 2024-03-22T02:28:36.941 INFO:tasks.workunit.client.0.smithi103.stdout:8/332: chown d4/d14/d39/f45 0 1 2024-03-22T02:28:36.945 INFO:tasks.workunit.client.0.smithi103.stdout:4/362: truncate d3/d1e/d53/f56 132898 0 2024-03-22T02:28:36.945 INFO:tasks.workunit.client.0.smithi103.stdout:4/363: readlink d3/l13 0 2024-03-22T02:28:36.947 INFO:tasks.workunit.client.0.smithi103.stdout:6/263: write d0/d3/f42 [94807,116822] 0 2024-03-22T02:28:36.947 INFO:tasks.workunit.client.0.smithi103.stdout:6/264: readlink d0/d3/l12 0 2024-03-22T02:28:36.947 INFO:tasks.workunit.client.0.smithi103.stdout:1/408: creat db/d10/d2e/f8a x:0 0 0 2024-03-22T02:28:36.949 INFO:tasks.workunit.client.0.smithi103.stdout:7/361: rename d1/da/d2f/d36/l6a to d1/l77 0 2024-03-22T02:28:36.951 INFO:tasks.workunit.client.0.smithi103.stdout:4/364: unlink d3/d1e/f35 0 2024-03-22T02:28:36.953 INFO:tasks.workunit.client.0.smithi103.stdout:1/409: mknod db/c8b 0 2024-03-22T02:28:36.954 INFO:tasks.workunit.client.0.smithi103.stdout:7/362: write d1/d10/d11/d19/d29/d33/f5f [834730,124306] 0 2024-03-22T02:28:36.957 INFO:tasks.workunit.client.0.smithi103.stdout:6/265: dwrite d0/f3e [0,4194304] 0 2024-03-22T02:28:36.957 INFO:tasks.workunit.client.0.smithi103.stdout:7/363: dread - d1/da/f75 zero size 2024-03-22T02:28:36.958 INFO:tasks.workunit.client.0.smithi103.stdout:2/319: getdents d11/d25 0 2024-03-22T02:28:36.958 INFO:tasks.workunit.client.0.smithi103.stdout:8/333: dread d4/fd [0,4194304] 0 2024-03-22T02:28:36.963 INFO:tasks.workunit.client.0.smithi103.stdout:6/266: mknod d0/d14/c54 0 2024-03-22T02:28:36.965 INFO:tasks.workunit.client.0.smithi103.stdout:3/327: sync 2024-03-22T02:28:36.965 INFO:tasks.workunit.client.0.smithi103.stdout:0/411: sync 2024-03-22T02:28:36.966 INFO:tasks.workunit.client.0.smithi103.stdout:9/239: sync 2024-03-22T02:28:36.966 INFO:tasks.workunit.client.0.smithi103.stdout:5/351: sync 2024-03-22T02:28:36.967 INFO:tasks.workunit.client.0.smithi103.stdout:7/364: read d1/d10/d11/d19/d29/d61/f45 [1015831,56601] 0 2024-03-22T02:28:36.970 INFO:tasks.workunit.client.0.smithi103.stdout:6/267: read d0/d3/f42 [154504,83289] 0 2024-03-22T02:28:36.973 INFO:tasks.workunit.client.0.smithi103.stdout:5/352: write d0/d14/f51 [5215519,24919] 0 2024-03-22T02:28:36.974 INFO:tasks.workunit.client.0.smithi103.stdout:9/240: write d2/da/f25 [801078,9538] 0 2024-03-22T02:28:36.976 INFO:tasks.workunit.client.0.smithi103.stdout:7/365: fsync d1/d10/d11/f5a 0 2024-03-22T02:28:36.977 INFO:tasks.workunit.client.0.smithi103.stdout:0/412: chown dd/d16/f38 31060 1 2024-03-22T02:28:36.980 INFO:tasks.workunit.client.0.smithi103.stdout:4/365: dwrite d3/d31/f57 [0,4194304] 0 2024-03-22T02:28:36.983 INFO:tasks.workunit.client.0.smithi103.stdout:9/241: stat d2/da/fd 0 2024-03-22T02:28:36.985 INFO:tasks.workunit.client.0.smithi103.stdout:3/328: dread de/d12/d19/f48 [0,4194304] 0 2024-03-22T02:28:36.985 INFO:tasks.workunit.client.0.smithi103.stdout:6/268: creat d0/d3/d45/f55 x:0 0 0 2024-03-22T02:28:36.985 INFO:tasks.workunit.client.0.smithi103.stdout:2/320: dwrite d11/f58 [0,4194304] 0 2024-03-22T02:28:36.988 INFO:tasks.workunit.client.0.smithi103.stdout:4/366: mknod d3/d1e/d53/c87 0 2024-03-22T02:28:36.990 INFO:tasks.workunit.client.0.smithi103.stdout:8/334: dwrite d4/fe [4194304,4194304] 0 2024-03-22T02:28:36.991 INFO:tasks.workunit.client.0.smithi103.stdout:9/242: read d2/da/f24 [1305634,19074] 0 2024-03-22T02:28:36.991 INFO:tasks.workunit.client.0.smithi103.stdout:5/353: unlink d0/d14/d23/d2a/l67 0 2024-03-22T02:28:36.994 INFO:tasks.workunit.client.0.smithi103.stdout:7/366: dread d1/da/fe [0,4194304] 0 2024-03-22T02:28:36.996 INFO:tasks.workunit.client.0.smithi103.stdout:6/269: creat d0/d14/f56 x:0 0 0 2024-03-22T02:28:36.997 INFO:tasks.workunit.client.0.smithi103.stdout:1/410: dread db/f25 [0,4194304] 0 2024-03-22T02:28:36.998 INFO:tasks.workunit.client.0.smithi103.stdout:3/329: symlink de/d12/d4d/l72 0 2024-03-22T02:28:37.001 INFO:tasks.workunit.client.0.smithi103.stdout:5/354: unlink d0/d14/c21 0 2024-03-22T02:28:37.005 INFO:tasks.workunit.client.0.smithi103.stdout:4/367: dwrite d3/d1e/d25/f3d [0,4194304] 0 2024-03-22T02:28:37.005 INFO:tasks.workunit.client.0.smithi103.stdout:6/270: rename d0/d3/d4f to d0/d14/d57 0 2024-03-22T02:28:37.008 INFO:tasks.workunit.client.0.smithi103.stdout:7/367: symlink d1/da/d2f/l78 0 2024-03-22T02:28:37.009 INFO:tasks.workunit.client.0.smithi103.stdout:2/321: dread d11/d25/d46/d40/f41 [0,4194304] 0 2024-03-22T02:28:37.012 INFO:tasks.workunit.client.0.smithi103.stdout:3/330: dwrite fa [0,4194304] 0 2024-03-22T02:28:37.013 INFO:tasks.workunit.client.0.smithi103.stdout:5/355: fdatasync d0/d14/d23/d36/d50/f5a 0 2024-03-22T02:28:37.013 INFO:tasks.workunit.client.0.smithi103.stdout:7/368: chown d1/da/l17 9 1 2024-03-22T02:28:37.014 INFO:tasks.workunit.client.0.smithi103.stdout:0/413: getdents dd/d16/d4b 0 2024-03-22T02:28:37.014 INFO:tasks.workunit.client.0.smithi103.stdout:6/271: mkdir d0/d3/d22/d58 0 2024-03-22T02:28:37.016 INFO:tasks.workunit.client.0.smithi103.stdout:2/322: write f10 [747610,2546] 0 2024-03-22T02:28:37.018 INFO:tasks.workunit.client.0.smithi103.stdout:3/331: write de/d12/d30/f6e [209130,55354] 0 2024-03-22T02:28:37.019 INFO:tasks.workunit.client.0.smithi103.stdout:4/368: mknod d3/d30/d45/d6e/c88 0 2024-03-22T02:28:37.020 INFO:tasks.workunit.client.0.smithi103.stdout:1/411: link db/d10/d18/c24 db/d10/d18/d84/c8c 0 2024-03-22T02:28:37.021 INFO:tasks.workunit.client.0.smithi103.stdout:0/414: write dd/d16/d20/d31/f6f [628555,75116] 0 2024-03-22T02:28:37.021 INFO:tasks.workunit.client.0.smithi103.stdout:5/356: mknod d0/d14/d23/d2a/d2c/c7e 0 2024-03-22T02:28:37.021 INFO:tasks.workunit.client.0.smithi103.stdout:6/272: truncate d0/d3/d22/f3d 4674320 0 2024-03-22T02:28:37.023 INFO:tasks.workunit.client.0.smithi103.stdout:3/332: fsync de/d12/d30/f3d 0 2024-03-22T02:28:37.026 INFO:tasks.workunit.client.0.smithi103.stdout:6/273: chown d0/d3/f39 14206311 1 2024-03-22T02:28:37.028 INFO:tasks.workunit.client.0.smithi103.stdout:2/323: creat d11/d19/d5d/f68 x:0 0 0 2024-03-22T02:28:37.031 INFO:tasks.workunit.client.0.smithi103.stdout:8/335: dread d4/f9 [0,4194304] 0 2024-03-22T02:28:37.031 INFO:tasks.workunit.client.0.smithi103.stdout:6/274: dread d0/f23 [0,4194304] 0 2024-03-22T02:28:37.031 INFO:tasks.workunit.client.0.smithi103.stdout:0/415: truncate dd/f24 1873020 0 2024-03-22T02:28:37.032 INFO:tasks.workunit.client.0.smithi103.stdout:1/412: creat db/d10/d6b/f8d x:0 0 0 2024-03-22T02:28:37.036 INFO:tasks.workunit.client.0.smithi103.stdout:9/243: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:37.036 INFO:tasks.workunit.client.0.smithi103.stdout:5/357: dwrite d0/d14/f22 [0,4194304] 0 2024-03-22T02:28:37.037 INFO:tasks.workunit.client.0.smithi103.stdout:6/275: symlink d0/d14/d57/l59 0 2024-03-22T02:28:37.037 INFO:tasks.workunit.client.0.smithi103.stdout:0/416: fdatasync dd/d16/d20/d64/d6a/f74 0 2024-03-22T02:28:37.043 INFO:tasks.workunit.client.0.smithi103.stdout:8/336: creat d4/d19/d1a/d4a/f64 x:0 0 0 2024-03-22T02:28:37.044 INFO:tasks.workunit.client.0.smithi103.stdout:3/333: dread de/d1a/d25/d26/f54 [0,4194304] 0 2024-03-22T02:28:37.047 INFO:tasks.workunit.client.0.smithi103.stdout:1/413: symlink db/d10/d18/d84/l8e 0 2024-03-22T02:28:37.048 INFO:tasks.workunit.client.0.smithi103.stdout:5/358: write d0/fe [2989450,65938] 0 2024-03-22T02:28:37.049 INFO:tasks.workunit.client.0.smithi103.stdout:7/369: write d1/d10/d11/d19/d29/d61/f45 [972977,16736] 0 2024-03-22T02:28:37.050 INFO:tasks.workunit.client.0.smithi103.stdout:6/276: symlink d0/d3/d33/l5a 0 2024-03-22T02:28:37.051 INFO:tasks.workunit.client.0.smithi103.stdout:5/359: chown d0/d14/c42 1 1 2024-03-22T02:28:37.052 INFO:tasks.workunit.client.0.smithi103.stdout:6/277: fsync d0/d3/d45/f2d 0 2024-03-22T02:28:37.052 INFO:tasks.workunit.client.0.smithi103.stdout:8/337: chown d4/d14/c30 31016 1 2024-03-22T02:28:37.052 INFO:tasks.workunit.client.0.smithi103.stdout:4/369: link d3/d1e/d53/l76 d3/d1e/d53/d7a/l89 0 2024-03-22T02:28:37.054 INFO:tasks.workunit.client.0.smithi103.stdout:6/278: chown d0/d14/d57 67591322 1 2024-03-22T02:28:37.055 INFO:tasks.workunit.client.0.smithi103.stdout:6/279: readlink d0/d3/l12 0 2024-03-22T02:28:37.055 INFO:tasks.workunit.client.0.smithi103.stdout:3/334: mkdir de/d12/d47/d73 0 2024-03-22T02:28:37.057 INFO:tasks.workunit.client.0.smithi103.stdout:1/414: creat db/d10/d29/d55/f8f x:0 0 0 2024-03-22T02:28:37.060 INFO:tasks.workunit.client.0.smithi103.stdout:8/338: symlink d4/d48/d40/l65 0 2024-03-22T02:28:37.062 INFO:tasks.workunit.client.0.smithi103.stdout:5/360: mknod d0/d14/d23/d5c/d77/c7f 0 2024-03-22T02:28:37.062 INFO:tasks.workunit.client.0.smithi103.stdout:1/415: dread - db/d10/d6b/f8d zero size 2024-03-22T02:28:37.064 INFO:tasks.workunit.client.0.smithi103.stdout:6/280: truncate d0/d14/f4c 587459 0 2024-03-22T02:28:37.065 INFO:tasks.workunit.client.0.smithi103.stdout:7/370: dread d1/da/f3 [0,4194304] 0 2024-03-22T02:28:37.065 INFO:tasks.workunit.client.0.smithi103.stdout:2/324: dread d11/f30 [4194304,4194304] 0 2024-03-22T02:28:37.068 INFO:tasks.workunit.client.0.smithi103.stdout:9/244: dread d2/d10/d43/f1b [0,4194304] 0 2024-03-22T02:28:37.068 INFO:tasks.workunit.client.0.smithi103.stdout:4/370: dwrite d3/f8 [0,4194304] 0 2024-03-22T02:28:37.070 INFO:tasks.workunit.client.0.smithi103.stdout:7/371: readlink d1/da/d2f/d36/l37 0 2024-03-22T02:28:37.070 INFO:tasks.workunit.client.0.smithi103.stdout:0/417: dread dd/d16/d20/d29/d48/f73 [0,4194304] 0 2024-03-22T02:28:37.073 INFO:tasks.workunit.client.0.smithi103.stdout:6/281: truncate d0/f1a 907328 0 2024-03-22T02:28:37.076 INFO:tasks.workunit.client.0.smithi103.stdout:4/371: mkdir d3/d1e/d53/d8a 0 2024-03-22T02:28:37.077 INFO:tasks.workunit.client.0.smithi103.stdout:2/325: write d11/d25/d63/d45/d47/f49 [2352332,114541] 0 2024-03-22T02:28:37.077 INFO:tasks.workunit.client.0.smithi103.stdout:1/416: dread db/d10/d2e/f51 [0,4194304] 0 2024-03-22T02:28:37.078 INFO:tasks.workunit.client.0.smithi103.stdout:6/282: creat d0/f5b x:0 0 0 2024-03-22T02:28:37.080 INFO:tasks.workunit.client.0.smithi103.stdout:8/339: dread d4/fd [0,4194304] 0 2024-03-22T02:28:37.080 INFO:tasks.workunit.client.0.smithi103.stdout:3/335: dwrite de/d12/d19/f51 [0,4194304] 0 2024-03-22T02:28:37.080 INFO:tasks.workunit.client.0.smithi103.stdout:0/418: chown dd/d16/d20/d31/c5e 1111 1 2024-03-22T02:28:37.083 INFO:tasks.workunit.client.0.smithi103.stdout:1/417: write db/d10/d29/f6f [607908,110916] 0 2024-03-22T02:28:37.086 INFO:tasks.workunit.client.0.smithi103.stdout:4/372: mknod d3/d1e/d53/d7a/c8b 0 2024-03-22T02:28:37.087 INFO:tasks.workunit.client.0.smithi103.stdout:9/245: write d2/d10/d43/f1b [3457893,118199] 0 2024-03-22T02:28:37.087 INFO:tasks.workunit.client.0.smithi103.stdout:7/372: symlink d1/d10/d11/d19/d29/d33/d4b/l79 0 2024-03-22T02:28:37.089 INFO:tasks.workunit.client.0.smithi103.stdout:2/326: symlink d11/d25/d46/d40/l69 0 2024-03-22T02:28:37.091 INFO:tasks.workunit.client.0.smithi103.stdout:5/361: link d0/d14/d23/d2a/d1f/l2f d0/l80 0 2024-03-22T02:28:37.093 INFO:tasks.workunit.client.0.smithi103.stdout:9/246: readlink d2/da/l15 0 2024-03-22T02:28:37.096 INFO:tasks.workunit.client.0.smithi103.stdout:6/283: link d0/d3/d22/f3c d0/d3/d22/f5c 0 2024-03-22T02:28:37.097 INFO:tasks.workunit.client.0.smithi103.stdout:7/373: write d1/d10/d11/d19/d29/d33/f5f [538013,52903] 0 2024-03-22T02:28:37.098 INFO:tasks.workunit.client.0.smithi103.stdout:2/327: dread - d11/d25/d46/d40/f67 zero size 2024-03-22T02:28:37.098 INFO:tasks.workunit.client.0.smithi103.stdout:3/336: truncate de/d1a/d25/d3f/f68 1187198 0 2024-03-22T02:28:37.100 INFO:tasks.workunit.client.0.smithi103.stdout:0/419: dwrite dd/f6b [0,4194304] 0 2024-03-22T02:28:37.102 INFO:tasks.workunit.client.0.smithi103.stdout:8/340: dwrite d4/d19/d1a/d4a/f64 [0,4194304] 0 2024-03-22T02:28:37.103 INFO:tasks.workunit.client.0.smithi103.stdout:6/284: fsync d0/d3/d33/f48 0 2024-03-22T02:28:37.103 INFO:tasks.workunit.client.0.smithi103.stdout:5/362: fsync d0/d14/d23/d2a/d1f/f68 0 2024-03-22T02:28:37.105 INFO:tasks.workunit.client.0.smithi103.stdout:6/285: read - d0/d3/f53 zero size 2024-03-22T02:28:37.108 INFO:tasks.workunit.client.0.smithi103.stdout:8/341: chown d4/d19/d1a/l42 36 1 2024-03-22T02:28:37.110 INFO:tasks.workunit.client.0.smithi103.stdout:0/420: write dd/f13 [9409172,73909] 0 2024-03-22T02:28:37.111 INFO:tasks.workunit.client.0.smithi103.stdout:7/374: truncate d1/d10/d11/d19/d29/f73 17869 0 2024-03-22T02:28:37.112 INFO:tasks.workunit.client.0.smithi103.stdout:1/418: getdents db/d10/d29/d6c 0 2024-03-22T02:28:37.113 INFO:tasks.workunit.client.0.smithi103.stdout:2/328: mknod d11/d19/d5d/c6a 0 2024-03-22T02:28:37.114 INFO:tasks.workunit.client.0.smithi103.stdout:6/286: rename d0/d3/d22/f11 to d0/d3/d33/f5d 0 2024-03-22T02:28:37.120 INFO:tasks.workunit.client.0.smithi103.stdout:8/342: dread d4/d19/d1a/d4a/f64 [0,4194304] 0 2024-03-22T02:28:37.120 INFO:tasks.workunit.client.0.smithi103.stdout:7/375: write d1/f4e [3045787,55] 0 2024-03-22T02:28:37.121 INFO:tasks.workunit.client.0.smithi103.stdout:0/421: mkdir dd/d16/d20/d31/d89 0 2024-03-22T02:28:37.122 INFO:tasks.workunit.client.0.smithi103.stdout:1/419: creat db/d10/d29/d41/f90 x:0 0 0 2024-03-22T02:28:37.127 INFO:tasks.workunit.client.0.smithi103.stdout:3/337: dwrite de/d12/f2b [0,4194304] 0 2024-03-22T02:28:37.128 INFO:tasks.workunit.client.0.smithi103.stdout:2/329: link d11/d25/d46/d40/l42 d11/d25/d46/d34/d24/l6b 0 2024-03-22T02:28:37.129 INFO:tasks.workunit.client.0.smithi103.stdout:7/376: stat d1/d10/d11/d19/f55 0 2024-03-22T02:28:37.132 INFO:tasks.workunit.client.0.smithi103.stdout:8/343: rename d4/d19/f62 to d4/d14/d39/d49/d5f/f66 0 2024-03-22T02:28:37.133 INFO:tasks.workunit.client.0.smithi103.stdout:3/338: chown de/d12/d19/f48 15093 1 2024-03-22T02:28:37.134 INFO:tasks.workunit.client.0.smithi103.stdout:4/373: dread d3/d38/f63 [0,4194304] 0 2024-03-22T02:28:37.137 INFO:tasks.workunit.client.0.smithi103.stdout:6/287: getdents d0/d3/d22 0 2024-03-22T02:28:37.139 INFO:tasks.workunit.client.0.smithi103.stdout:0/422: creat dd/f8a x:0 0 0 2024-03-22T02:28:37.142 INFO:tasks.workunit.client.0.smithi103.stdout:5/363: dread d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:37.143 INFO:tasks.workunit.client.0.smithi103.stdout:7/377: mkdir d1/d10/d11/d19/d29/d61/d69/d6b/d7a 0 2024-03-22T02:28:37.145 INFO:tasks.workunit.client.0.smithi103.stdout:3/339: write de/d1a/d25/d26/f54 [267696,24954] 0 2024-03-22T02:28:37.146 INFO:tasks.workunit.client.0.smithi103.stdout:1/420: dwrite db/f2b [0,4194304] 0 2024-03-22T02:28:37.147 INFO:tasks.workunit.client.0.smithi103.stdout:6/288: unlink d0/d3/d33/f48 0 2024-03-22T02:28:37.149 INFO:tasks.workunit.client.0.smithi103.stdout:7/378: fsync d1/d10/d11/d19/d29/f73 0 2024-03-22T02:28:37.149 INFO:tasks.workunit.client.0.smithi103.stdout:2/330: dwrite d11/f58 [0,4194304] 0 2024-03-22T02:28:37.152 INFO:tasks.workunit.client.0.smithi103.stdout:8/344: getdents d4/d26 0 2024-03-22T02:28:37.152 INFO:tasks.workunit.client.0.smithi103.stdout:5/364: chown d0/c26 106315324 1 2024-03-22T02:28:37.156 INFO:tasks.workunit.client.0.smithi103.stdout:6/289: write d0/d3/d45/f40 [1158051,43161] 0 2024-03-22T02:28:37.157 INFO:tasks.workunit.client.0.smithi103.stdout:0/423: unlink dd/d16/f3f 0 2024-03-22T02:28:37.159 INFO:tasks.workunit.client.0.smithi103.stdout:3/340: dread de/f1f [0,4194304] 0 2024-03-22T02:28:37.160 INFO:tasks.workunit.client.0.smithi103.stdout:4/374: dwrite d3/d38/f70 [0,4194304] 0 2024-03-22T02:28:37.161 INFO:tasks.workunit.client.0.smithi103.stdout:2/331: truncate d11/d25/d46/d34/d24/f60 903489 0 2024-03-22T02:28:37.162 INFO:tasks.workunit.client.0.smithi103.stdout:8/345: write d4/d2a/d3e/f4b [227762,49803] 0 2024-03-22T02:28:37.162 INFO:tasks.workunit.client.0.smithi103.stdout:7/379: rename d1/f49 to d1/d10/d11/d19/d29/d61/f7b 0 2024-03-22T02:28:37.163 INFO:tasks.workunit.client.0.smithi103.stdout:3/341: write de/f23 [713638,26698] 0 2024-03-22T02:28:37.166 INFO:tasks.workunit.client.0.smithi103.stdout:7/380: chown d1/d10/d11/l34 344314 1 2024-03-22T02:28:37.166 INFO:tasks.workunit.client.0.smithi103.stdout:5/365: mknod d0/c81 0 2024-03-22T02:28:37.167 INFO:tasks.workunit.client.0.smithi103.stdout:2/332: mknod d11/d25/d46/d34/c6c 0 2024-03-22T02:28:37.169 INFO:tasks.workunit.client.0.smithi103.stdout:5/366: readlink d0/d14/d23/d46/l75 0 2024-03-22T02:28:37.169 INFO:tasks.workunit.client.0.smithi103.stdout:8/346: mkdir d4/d14/d67 0 2024-03-22T02:28:37.170 INFO:tasks.workunit.client.0.smithi103.stdout:3/342: mknod de/d1a/d25/c74 0 2024-03-22T02:28:37.173 INFO:tasks.workunit.client.0.smithi103.stdout:2/333: creat d11/d25/d46/d48/f6d x:0 0 0 2024-03-22T02:28:37.175 INFO:tasks.workunit.client.0.smithi103.stdout:5/367: write d0/fe [1277981,13458] 0 2024-03-22T02:28:37.176 INFO:tasks.workunit.client.0.smithi103.stdout:0/424: dwrite dd/f24 [0,4194304] 0 2024-03-22T02:28:37.179 INFO:tasks.workunit.client.0.smithi103.stdout:3/343: unlink f8 0 2024-03-22T02:28:37.182 INFO:tasks.workunit.client.0.smithi103.stdout:6/290: getdents d0/d14 0 2024-03-22T02:28:37.182 INFO:tasks.workunit.client.0.smithi103.stdout:0/425: chown dd/d16/d20/d35/l47 419075 1 2024-03-22T02:28:37.182 INFO:tasks.workunit.client.0.smithi103.stdout:7/381: dwrite d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:37.183 INFO:tasks.workunit.client.0.smithi103.stdout:6/291: chown d0/d14/d57 0 1 2024-03-22T02:28:37.184 INFO:tasks.workunit.client.0.smithi103.stdout:8/347: dread d4/d19/d1a/d4a/f64 [0,4194304] 0 2024-03-22T02:28:37.184 INFO:tasks.workunit.client.0.smithi103.stdout:3/344: write de/f5f [644369,127758] 0 2024-03-22T02:28:37.186 INFO:tasks.workunit.client.0.smithi103.stdout:0/426: dread dd/f5b [0,4194304] 0 2024-03-22T02:28:37.188 INFO:tasks.workunit.client.0.smithi103.stdout:3/345: read de/f23 [1772938,61996] 0 2024-03-22T02:28:37.194 INFO:tasks.workunit.client.0.smithi103.stdout:7/382: rename d1/d10/d11/l3c to d1/d10/d11/d19/d29/d57/l7c 0 2024-03-22T02:28:37.196 INFO:tasks.workunit.client.0.smithi103.stdout:3/346: dread de/d12/d19/f48 [0,4194304] 0 2024-03-22T02:28:37.197 INFO:tasks.workunit.client.0.smithi103.stdout:6/292: dwrite d0/d3/d22/f37 [0,4194304] 0 2024-03-22T02:28:37.199 INFO:tasks.workunit.client.0.smithi103.stdout:8/348: dwrite d4/d14/f52 [0,4194304] 0 2024-03-22T02:28:37.206 INFO:tasks.workunit.client.0.smithi103.stdout:3/347: creat de/d12/d5d/f75 x:0 0 0 2024-03-22T02:28:37.207 INFO:tasks.workunit.client.0.smithi103.stdout:6/293: mknod d0/d3/d22/d58/c5e 0 2024-03-22T02:28:37.211 INFO:tasks.workunit.client.0.smithi103.stdout:8/349: dwrite d4/d2a/d3e/f4b [0,4194304] 0 2024-03-22T02:28:37.212 INFO:tasks.workunit.client.0.smithi103.stdout:8/350: fdatasync d4/ff 0 2024-03-22T02:28:37.214 INFO:tasks.workunit.client.0.smithi103.stdout:6/294: symlink d0/d14/l5f 0 2024-03-22T02:28:37.215 INFO:tasks.workunit.client.0.smithi103.stdout:8/351: mkdir d4/d48/d40/d68 0 2024-03-22T02:28:37.217 INFO:tasks.workunit.client.0.smithi103.stdout:7/383: rmdir d1/d10/d11/d19/d29/d61/d1d 39 2024-03-22T02:28:37.220 INFO:tasks.workunit.client.0.smithi103.stdout:6/295: rename d0/d14/f29 to d0/d3/d33/f60 0 2024-03-22T02:28:37.222 INFO:tasks.workunit.client.0.smithi103.stdout:6/296: chown d0/f3e 2732 1 2024-03-22T02:28:37.222 INFO:tasks.workunit.client.0.smithi103.stdout:8/352: dwrite d4/d19/f2e [0,4194304] 0 2024-03-22T02:28:37.226 INFO:tasks.workunit.client.0.smithi103.stdout:6/297: unlink d0/d14/c18 0 2024-03-22T02:28:37.226 INFO:tasks.workunit.client.0.smithi103.stdout:5/368: dread d0/d14/d23/d2a/d33/f4c [0,4194304] 0 2024-03-22T02:28:37.228 INFO:tasks.workunit.client.0.smithi103.stdout:7/384: getdents d1/d10/d11/d19/d29 0 2024-03-22T02:28:37.229 INFO:tasks.workunit.client.0.smithi103.stdout:8/353: mknod d4/c69 0 2024-03-22T02:28:37.230 INFO:tasks.workunit.client.0.smithi103.stdout:5/369: creat d0/d14/d23/d36/f82 x:0 0 0 2024-03-22T02:28:37.231 INFO:tasks.workunit.client.0.smithi103.stdout:5/370: stat d0/d14/d23/d2a/f6c 0 2024-03-22T02:28:37.232 INFO:tasks.workunit.client.0.smithi103.stdout:7/385: symlink d1/d10/d11/d19/d29/d33/d4b/l7d 0 2024-03-22T02:28:37.233 INFO:tasks.workunit.client.0.smithi103.stdout:8/354: creat d4/d2a/d38/f6a x:0 0 0 2024-03-22T02:28:37.234 INFO:tasks.workunit.client.0.smithi103.stdout:8/355: chown d4/d48/d40/l65 4072247 1 2024-03-22T02:28:37.235 INFO:tasks.workunit.client.0.smithi103.stdout:5/371: mkdir d0/d14/d23/d6f/d83 0 2024-03-22T02:28:37.235 INFO:tasks.workunit.client.0.smithi103.stdout:3/348: write de/d1a/d25/d26/f53 [282487,105690] 0 2024-03-22T02:28:37.236 INFO:tasks.workunit.client.0.smithi103.stdout:3/349: chown de/d12/f2b 288 1 2024-03-22T02:28:37.238 INFO:tasks.workunit.client.0.smithi103.stdout:8/356: unlink d4/d14/c30 0 2024-03-22T02:28:37.240 INFO:tasks.workunit.client.0.smithi103.stdout:8/357: chown d4/d26/d27/f63 849 1 2024-03-22T02:28:37.242 INFO:tasks.workunit.client.0.smithi103.stdout:3/350: mknod de/d1a/d25/d26/c76 0 2024-03-22T02:28:37.242 INFO:tasks.workunit.client.0.smithi103.stdout:7/386: dwrite d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:37.243 INFO:tasks.workunit.client.0.smithi103.stdout:5/372: symlink d0/l84 0 2024-03-22T02:28:37.246 INFO:tasks.workunit.client.0.smithi103.stdout:3/351: chown de/c34 17367728 1 2024-03-22T02:28:37.247 INFO:tasks.workunit.client.0.smithi103.stdout:7/387: write d1/d10/d11/d19/d29/d61/d69/f32 [2407862,90201] 0 2024-03-22T02:28:37.249 INFO:tasks.workunit.client.0.smithi103.stdout:3/352: creat de/d12/d47/d4f/f77 x:0 0 0 2024-03-22T02:28:37.249 INFO:tasks.workunit.client.0.smithi103.stdout:8/358: dwrite d4/d14/f1e [0,4194304] 0 2024-03-22T02:28:37.251 INFO:tasks.workunit.client.0.smithi103.stdout:8/359: fdatasync d4/d19/f5c 0 2024-03-22T02:28:37.253 INFO:tasks.workunit.client.0.smithi103.stdout:0/427: dread dd/d16/f55 [0,4194304] 0 2024-03-22T02:28:37.255 INFO:tasks.workunit.client.0.smithi103.stdout:7/388: write d1/d10/d11/d19/d29/d61/f7b [8498511,88290] 0 2024-03-22T02:28:37.255 INFO:tasks.workunit.client.0.smithi103.stdout:8/360: write d4/fd [2147289,58945] 0 2024-03-22T02:28:37.255 INFO:tasks.workunit.client.0.smithi103.stdout:0/428: write f4 [1921523,116281] 0 2024-03-22T02:28:37.257 INFO:tasks.workunit.client.0.smithi103.stdout:8/361: dread - d4/d19/f5c zero size 2024-03-22T02:28:37.259 INFO:tasks.workunit.client.0.smithi103.stdout:7/389: readlink d1/l77 0 2024-03-22T02:28:37.261 INFO:tasks.workunit.client.0.smithi103.stdout:7/390: chown d1/d10/d11/d19/d29/d61/f7b 0 1 2024-03-22T02:28:37.262 INFO:tasks.workunit.client.0.smithi103.stdout:0/429: creat dd/d16/d20/d31/d89/f8b x:0 0 0 2024-03-22T02:28:37.266 INFO:tasks.workunit.client.0.smithi103.stdout:0/430: mkdir dd/d34/d53/d8c 0 2024-03-22T02:28:37.267 INFO:tasks.workunit.client.0.smithi103.stdout:3/353: write de/d1a/f31 [12410274,49577] 0 2024-03-22T02:28:37.270 INFO:tasks.workunit.client.0.smithi103.stdout:8/362: dread d4/d19/d1a/f2d [0,4194304] 0 2024-03-22T02:28:37.271 INFO:tasks.workunit.client.0.smithi103.stdout:0/431: mkdir dd/d16/d20/d64/d6a/d8d 0 2024-03-22T02:28:37.273 INFO:tasks.workunit.client.0.smithi103.stdout:3/354: creat de/d12/d30/d70/f78 x:0 0 0 2024-03-22T02:28:37.275 INFO:tasks.workunit.client.0.smithi103.stdout:7/391: dwrite d1/da/fb [0,4194304] 0 2024-03-22T02:28:37.275 INFO:tasks.workunit.client.0.smithi103.stdout:8/363: dread - d4/d26/d27/f63 zero size 2024-03-22T02:28:37.277 INFO:tasks.workunit.client.0.smithi103.stdout:5/373: dwrite d0/d14/d23/d2a/d1f/f38 [0,4194304] 0 2024-03-22T02:28:37.279 INFO:tasks.workunit.client.0.smithi103.stdout:5/374: chown d0/d14/d23/d2a/d1f/f68 0 1 2024-03-22T02:28:37.280 INFO:tasks.workunit.client.0.smithi103.stdout:3/355: creat de/d1a/d25/f79 x:0 0 0 2024-03-22T02:28:37.284 INFO:tasks.workunit.client.0.smithi103.stdout:3/356: chown de/d12/d30/c60 7 1 2024-03-22T02:28:37.284 INFO:tasks.workunit.client.0.smithi103.stdout:8/364: creat d4/d2a/d3e/f6b x:0 0 0 2024-03-22T02:28:37.286 INFO:tasks.workunit.client.0.smithi103.stdout:0/432: dread dd/d16/f38 [0,4194304] 0 2024-03-22T02:28:37.288 INFO:tasks.workunit.client.0.smithi103.stdout:5/375: unlink d0/l80 0 2024-03-22T02:28:37.290 INFO:tasks.workunit.client.0.smithi103.stdout:0/433: read fc [3206617,112798] 0 2024-03-22T02:28:37.291 INFO:tasks.workunit.client.0.smithi103.stdout:3/357: read de/d12/d30/f3d [165688,71840] 0 2024-03-22T02:28:37.294 INFO:tasks.workunit.client.0.smithi103.stdout:7/392: dwrite d1/d10/d11/f18 [0,4194304] 0 2024-03-22T02:28:37.295 INFO:tasks.workunit.client.0.smithi103.stdout:0/434: chown dd/d16/d20/d35/c37 58 1 2024-03-22T02:28:37.299 INFO:tasks.workunit.client.0.smithi103.stdout:8/365: dwrite d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:37.300 INFO:tasks.workunit.client.0.smithi103.stdout:5/376: dwrite d0/d14/d23/f40 [0,4194304] 0 2024-03-22T02:28:37.301 INFO:tasks.workunit.client.0.smithi103.stdout:0/435: truncate dd/d16/d20/d29/f2a 9990097 0 2024-03-22T02:28:37.306 INFO:tasks.workunit.client.0.smithi103.stdout:5/377: fdatasync d0/d14/d23/d36/d50/f5a 0 2024-03-22T02:28:37.309 INFO:tasks.workunit.client.0.smithi103.stdout:8/366: link d4/d48/d35/f44 d4/d2a/f6c 0 2024-03-22T02:28:37.310 INFO:tasks.workunit.client.0.smithi103.stdout:5/378: read d0/d14/d23/f24 [3577268,55790] 0 2024-03-22T02:28:37.311 INFO:tasks.workunit.client.0.smithi103.stdout:5/379: dread - d0/d14/d23/d2a/d1f/f68 zero size 2024-03-22T02:28:37.313 INFO:tasks.workunit.client.0.smithi103.stdout:8/367: symlink d4/d14/d67/l6d 0 2024-03-22T02:28:37.318 INFO:tasks.workunit.client.0.smithi103.stdout:3/358: dwrite de/f1f [0,4194304] 0 2024-03-22T02:28:37.332 INFO:tasks.workunit.client.0.smithi103.stdout:3/359: dread de/d12/f1c [0,4194304] 0 2024-03-22T02:28:37.340 INFO:tasks.workunit.client.0.smithi103.stdout:3/360: dread de/d12/d30/f35 [0,4194304] 0 2024-03-22T02:28:37.350 INFO:tasks.workunit.client.0.smithi103.stdout:5/380: creat d0/d14/d23/d36/f85 x:0 0 0 2024-03-22T02:28:37.351 INFO:tasks.workunit.client.0.smithi103.stdout:7/393: truncate d1/d10/d11/f28 2085386 0 2024-03-22T02:28:37.352 INFO:tasks.workunit.client.0.smithi103.stdout:0/436: write dd/d16/d20/d29/f83 [31103,127954] 0 2024-03-22T02:28:37.355 INFO:tasks.workunit.client.0.smithi103.stdout:3/361: creat de/d12/d47/f7a x:0 0 0 2024-03-22T02:28:37.356 INFO:tasks.workunit.client.0.smithi103.stdout:3/362: dread de/d12/d19/f48 [0,4194304] 0 2024-03-22T02:28:37.357 INFO:tasks.workunit.client.0.smithi103.stdout:5/381: write d0/d14/d23/d2a/f1a [2670546,108667] 0 2024-03-22T02:28:37.357 INFO:tasks.workunit.client.0.smithi103.stdout:3/363: stat de/d1a/d25/d3f/c4b 0 2024-03-22T02:28:37.359 INFO:tasks.workunit.client.0.smithi103.stdout:3/364: write de/f2d [8417118,9856] 0 2024-03-22T02:28:37.360 INFO:tasks.workunit.client.0.smithi103.stdout:8/368: creat d4/f6e x:0 0 0 2024-03-22T02:28:37.363 INFO:tasks.workunit.client.0.smithi103.stdout:5/382: creat d0/d14/d23/d6f/f86 x:0 0 0 2024-03-22T02:28:37.364 INFO:tasks.workunit.client.0.smithi103.stdout:5/383: fdatasync d0/d14/d23/d2a/d2c/f6d 0 2024-03-22T02:28:37.364 INFO:tasks.workunit.client.0.smithi103.stdout:5/384: stat d0/d14/d23/l5e 0 2024-03-22T02:28:37.365 INFO:tasks.workunit.client.0.smithi103.stdout:5/385: fsync d0/d14/f22 0 2024-03-22T02:28:37.365 INFO:tasks.workunit.client.0.smithi103.stdout:7/394: creat d1/f7e x:0 0 0 2024-03-22T02:28:37.367 INFO:tasks.workunit.client.0.smithi103.stdout:8/369: mknod d4/d19/d1a/d4a/c6f 0 2024-03-22T02:28:37.367 INFO:tasks.workunit.client.0.smithi103.stdout:5/386: dread d0/d14/d23/d36/d50/f5a [0,4194304] 0 2024-03-22T02:28:37.368 INFO:tasks.workunit.client.0.smithi103.stdout:3/365: symlink de/d6b/l7b 0 2024-03-22T02:28:37.370 INFO:tasks.workunit.client.0.smithi103.stdout:8/370: rmdir d4/d14/d39/d49 39 2024-03-22T02:28:37.371 INFO:tasks.workunit.client.0.smithi103.stdout:3/366: creat de/d12/d47/f7c x:0 0 0 2024-03-22T02:28:37.372 INFO:tasks.workunit.client.0.smithi103.stdout:7/395: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f x:0 0 0 2024-03-22T02:28:37.372 INFO:tasks.workunit.client.0.smithi103.stdout:7/396: dread - d1/f7e zero size 2024-03-22T02:28:37.373 INFO:tasks.workunit.client.0.smithi103.stdout:8/371: mkdir d4/d26/d27/d70 0 2024-03-22T02:28:37.374 INFO:tasks.workunit.client.0.smithi103.stdout:3/367: mknod de/d12/d47/c7d 0 2024-03-22T02:28:37.376 INFO:tasks.workunit.client.0.smithi103.stdout:5/387: link d0/d14/d23/d2a/d2c/f6d d0/d14/d23/d5c/d70/f87 0 2024-03-22T02:28:37.378 INFO:tasks.workunit.client.0.smithi103.stdout:8/372: unlink d4/d14/d39/l5d 0 2024-03-22T02:28:37.380 INFO:tasks.workunit.client.0.smithi103.stdout:7/397: mkdir d1/d10/d80 0 2024-03-22T02:28:37.380 INFO:tasks.workunit.client.0.smithi103.stdout:3/368: dwrite de/d1a/d25/d26/f6f [0,4194304] 0 2024-03-22T02:28:37.382 INFO:tasks.workunit.client.0.smithi103.stdout:3/369: chown de/f23 28068 1 2024-03-22T02:28:37.383 INFO:tasks.workunit.client.0.smithi103.stdout:8/373: creat d4/d26/d27/f71 x:0 0 0 2024-03-22T02:28:37.389 INFO:tasks.workunit.client.0.smithi103.stdout:7/398: rename d1/d10/d11/l59 to d1/d10/d11/d19/d29/d61/d69/d5c/l81 0 2024-03-22T02:28:37.390 INFO:tasks.workunit.client.0.smithi103.stdout:8/374: symlink d4/d26/d27/d70/l72 0 2024-03-22T02:28:37.395 INFO:tasks.workunit.client.0.smithi103.stdout:8/375: unlink d4/d19/d1a/f37 0 2024-03-22T02:28:37.395 INFO:tasks.workunit.client.0.smithi103.stdout:8/376: dread - d4/d26/d27/f71 zero size 2024-03-22T02:28:37.399 INFO:tasks.workunit.client.0.smithi103.stdout:8/377: rename d4/d14/f5a to d4/d48/f73 0 2024-03-22T02:28:37.402 INFO:tasks.workunit.client.0.smithi103.stdout:8/378: creat d4/d14/d39/d49/f74 x:0 0 0 2024-03-22T02:28:37.405 INFO:tasks.workunit.client.0.smithi103.stdout:3/370: dread de/f23 [0,4194304] 0 2024-03-22T02:28:37.406 INFO:tasks.workunit.client.0.smithi103.stdout:7/399: chown d1/d10/d11/d19/d29/d33 103218061 1 2024-03-22T02:28:37.407 INFO:tasks.workunit.client.0.smithi103.stdout:8/379: unlink d4/d19/d1a/d4a/f64 0 2024-03-22T02:28:37.409 INFO:tasks.workunit.client.0.smithi103.stdout:5/388: dwrite d0/d14/d23/d36/d50/f5a [0,4194304] 0 2024-03-22T02:28:37.411 INFO:tasks.workunit.client.0.smithi103.stdout:7/400: write d1/d10/d11/d19/d29/d61/d1d/f41 [407806,9383] 0 2024-03-22T02:28:37.415 INFO:tasks.workunit.client.0.smithi103.stdout:5/389: mknod d0/d3e/c88 0 2024-03-22T02:28:37.416 INFO:tasks.workunit.client.0.smithi103.stdout:7/401: mknod d1/da/d2f/d36/c82 0 2024-03-22T02:28:37.418 INFO:tasks.workunit.client.0.smithi103.stdout:5/390: dread - d0/d14/d23/d2a/d1f/f68 zero size 2024-03-22T02:28:37.419 INFO:tasks.workunit.client.0.smithi103.stdout:8/380: dwrite d4/f6e [0,4194304] 0 2024-03-22T02:28:37.421 INFO:tasks.workunit.client.0.smithi103.stdout:8/381: write f1 [1175936,12367] 0 2024-03-22T02:28:37.421 INFO:tasks.workunit.client.0.smithi103.stdout:7/402: creat d1/d10/d11/d19/d29/d33/d4b/f83 x:0 0 0 2024-03-22T02:28:37.422 INFO:tasks.workunit.client.0.smithi103.stdout:8/382: chown d4/l21 2535 1 2024-03-22T02:28:37.423 INFO:tasks.workunit.client.0.smithi103.stdout:7/403: write d1/da/d2f/f38 [908119,94287] 0 2024-03-22T02:28:37.423 INFO:tasks.workunit.client.0.smithi103.stdout:5/391: symlink d0/d14/d23/d6f/l89 0 2024-03-22T02:28:37.428 INFO:tasks.workunit.client.0.smithi103.stdout:5/392: stat d0/d14/d23/d36/d50/f5a 0 2024-03-22T02:28:37.429 INFO:tasks.workunit.client.0.smithi103.stdout:8/383: write d4/d19/d1a/f2d [2144306,25910] 0 2024-03-22T02:28:37.430 INFO:tasks.workunit.client.0.smithi103.stdout:5/393: readlink d0/d14/d23/d36/d50/l65 0 2024-03-22T02:28:37.432 INFO:tasks.workunit.client.0.smithi103.stdout:8/384: rmdir d4/d2a/d38 39 2024-03-22T02:28:37.434 INFO:tasks.workunit.client.0.smithi103.stdout:7/404: dwrite d1/d10/d11/d19/d29/f52 [0,4194304] 0 2024-03-22T02:28:37.435 INFO:tasks.workunit.client.0.smithi103.stdout:3/371: dread de/f1e [0,4194304] 0 2024-03-22T02:28:37.438 INFO:tasks.workunit.client.0.smithi103.stdout:7/405: chown d1/d10/d11/l51 472554357 1 2024-03-22T02:28:37.439 INFO:tasks.workunit.client.0.smithi103.stdout:8/385: mkdir d4/d48/d40/d68/d75 0 2024-03-22T02:28:37.442 INFO:tasks.workunit.client.0.smithi103.stdout:8/386: chown d4/d26/d27/f63 106717374 1 2024-03-22T02:28:37.443 INFO:tasks.workunit.client.0.smithi103.stdout:7/406: creat d1/d10/d11/d19/d29/d61/d69/f84 x:0 0 0 2024-03-22T02:28:37.446 INFO:tasks.workunit.client.0.smithi103.stdout:3/372: dwrite de/f21 [0,4194304] 0 2024-03-22T02:28:37.450 INFO:tasks.workunit.client.0.smithi103.stdout:8/387: rmdir d4/d19/d1a 39 2024-03-22T02:28:37.452 INFO:tasks.workunit.client.0.smithi103.stdout:7/407: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f85 x:0 0 0 2024-03-22T02:28:37.454 INFO:tasks.workunit.client.0.smithi103.stdout:7/408: mkdir d1/d10/d11/d19/d86 0 2024-03-22T02:28:37.456 INFO:tasks.workunit.client.0.smithi103.stdout:7/409: chown d1/d10/d11/d19/d29/d33/d4b/d5e 54815 1 2024-03-22T02:28:37.456 INFO:tasks.workunit.client.0.smithi103.stdout:8/388: read d4/ff [3204737,50076] 0 2024-03-22T02:28:37.456 INFO:tasks.workunit.client.0.smithi103.stdout:7/410: chown d1/da/fe 216 1 2024-03-22T02:28:37.458 INFO:tasks.workunit.client.0.smithi103.stdout:7/411: readlink d1/d10/d11/d19/d29/l64 0 2024-03-22T02:28:37.459 INFO:tasks.workunit.client.0.smithi103.stdout:8/389: mknod d4/d26/d27/d70/c76 0 2024-03-22T02:28:37.465 INFO:tasks.workunit.client.0.smithi103.stdout:3/373: dwrite de/f22 [0,4194304] 0 2024-03-22T02:28:37.466 INFO:tasks.workunit.client.0.smithi103.stdout:7/412: dread d1/da/f74 [0,4194304] 0 2024-03-22T02:28:37.468 INFO:tasks.workunit.client.0.smithi103.stdout:8/390: dwrite d4/fe [4194304,4194304] 0 2024-03-22T02:28:37.470 INFO:tasks.workunit.client.0.smithi103.stdout:3/374: symlink de/d6b/l7e 0 2024-03-22T02:28:37.473 INFO:tasks.workunit.client.0.smithi103.stdout:3/375: fsync de/d12/f16 0 2024-03-22T02:28:37.475 INFO:tasks.workunit.client.0.smithi103.stdout:3/376: chown de/d12/d47/f7c 42 1 2024-03-22T02:28:37.477 INFO:tasks.workunit.client.0.smithi103.stdout:7/413: getdents d1/da/d2f 0 2024-03-22T02:28:37.481 INFO:tasks.workunit.client.0.smithi103.stdout:3/377: rename de/d12/d30/c60 to de/d12/d19/c7f 0 2024-03-22T02:28:37.481 INFO:tasks.workunit.client.0.smithi103.stdout:7/414: fdatasync d1/d10/d11/d19/d29/d61/d69/f32 0 2024-03-22T02:28:37.481 INFO:tasks.workunit.client.0.smithi103.stdout:8/391: dwrite d4/d48/f73 [0,4194304] 0 2024-03-22T02:28:37.483 INFO:tasks.workunit.client.0.smithi103.stdout:7/415: truncate d1/f4e 4907518 0 2024-03-22T02:28:37.487 INFO:tasks.workunit.client.0.smithi103.stdout:7/416: symlink d1/l87 0 2024-03-22T02:28:37.489 INFO:tasks.workunit.client.0.smithi103.stdout:3/378: truncate de/d1a/d25/d26/f58 3876349 0 2024-03-22T02:28:37.489 INFO:tasks.workunit.client.0.smithi103.stdout:8/392: dwrite d4/d2a/d3e/f4b [0,4194304] 0 2024-03-22T02:28:37.586 INFO:tasks.workunit.client.0.smithi103.stdout:7/417: mkdir d1/d10/d11/d70/d88 0 2024-03-22T02:28:37.587 INFO:tasks.workunit.client.0.smithi103.stdout:7/418: fsync d1/f4e 0 2024-03-22T02:28:37.589 INFO:tasks.workunit.client.0.smithi103.stdout:8/393: dread d4/fd [0,4194304] 0 2024-03-22T02:28:37.590 INFO:tasks.workunit.client.0.smithi103.stdout:7/419: rmdir d1/d10/d11/d19/d29/d61/d69/d6b/d7a 39 2024-03-22T02:28:37.591 INFO:tasks.workunit.client.0.smithi103.stdout:7/420: chown d1/d10/d11/d19 105926094 1 2024-03-22T02:28:37.591 INFO:tasks.workunit.client.0.smithi103.stdout:8/394: chown d4/d19/d1a/d4a/l61 683 1 2024-03-22T02:28:37.595 INFO:tasks.workunit.client.0.smithi103.stdout:7/421: fdatasync d1/d10/d11/d19/d29/d33/d4b/f83 0 2024-03-22T02:28:37.596 INFO:tasks.workunit.client.0.smithi103.stdout:7/422: chown d1/da/d1c/l72 246663004 1 2024-03-22T02:28:37.598 INFO:tasks.workunit.client.0.smithi103.stdout:8/395: dread d4/d19/f2e [0,4194304] 0 2024-03-22T02:28:37.601 INFO:tasks.workunit.client.0.smithi103.stdout:3/379: dread fd [0,4194304] 0 2024-03-22T02:28:37.602 INFO:tasks.workunit.client.0.smithi103.stdout:3/380: write de/d12/d19/f51 [4535632,120971] 0 2024-03-22T02:28:37.603 INFO:tasks.workunit.client.0.smithi103.stdout:3/381: dread - de/d12/d47/f7a zero size 2024-03-22T02:28:37.608 INFO:tasks.workunit.client.0.smithi103.stdout:3/382: rename de/d12/d47 to de/d6b/d80 0 2024-03-22T02:28:37.613 INFO:tasks.workunit.client.0.smithi103.stdout:3/383: fdatasync de/d12/d30/f6e 0 2024-03-22T02:28:37.615 INFO:tasks.workunit.client.0.smithi103.stdout:8/396: dread d4/ff [0,4194304] 0 2024-03-22T02:28:37.615 INFO:tasks.workunit.client.0.smithi103.stdout:3/384: symlink de/d1a/l81 0 2024-03-22T02:28:37.621 INFO:tasks.workunit.client.0.smithi103.stdout:3/385: truncate de/d6b/d80/d4f/f65 1648517 0 2024-03-22T02:28:37.623 INFO:tasks.workunit.client.0.smithi103.stdout:3/386: mknod de/d12/c82 0 2024-03-22T02:28:37.624 INFO:tasks.workunit.client.0.smithi103.stdout:3/387: mknod de/d1a/d25/d3f/c83 0 2024-03-22T02:28:37.625 INFO:tasks.workunit.client.0.smithi103.stdout:3/388: chown de/d12/d4d 1301175388 1 2024-03-22T02:28:37.628 INFO:tasks.workunit.client.0.smithi103.stdout:3/389: read de/d12/f16 [607928,97197] 0 2024-03-22T02:28:37.629 INFO:tasks.workunit.client.0.smithi103.stdout:8/397: dread f1 [0,4194304] 0 2024-03-22T02:28:37.631 INFO:tasks.workunit.client.0.smithi103.stdout:8/398: fdatasync d4/d2a/f6c 0 2024-03-22T02:28:37.635 INFO:tasks.workunit.client.0.smithi103.stdout:3/390: dwrite de/d6b/d80/d4f/f77 [0,4194304] 0 2024-03-22T02:28:37.638 INFO:tasks.workunit.client.0.smithi103.stdout:8/399: dread d4/d14/f52 [0,4194304] 0 2024-03-22T02:28:37.638 INFO:tasks.workunit.client.0.smithi103.stdout:8/400: chown d4/d2a/c32 1737 1 2024-03-22T02:28:37.639 INFO:tasks.workunit.client.0.smithi103.stdout:8/401: stat d4/d14/d39/d49/f74 0 2024-03-22T02:28:37.639 INFO:tasks.workunit.client.0.smithi103.stdout:8/402: readlink d4/d26/d27/d70/l72 0 2024-03-22T02:28:37.640 INFO:tasks.workunit.client.0.smithi103.stdout:8/403: symlink d4/d48/d40/l77 0 2024-03-22T02:28:37.643 INFO:tasks.workunit.client.0.smithi103.stdout:8/404: link d4/f9 d4/d14/d39/d49/d5f/f78 0 2024-03-22T02:28:37.647 INFO:tasks.workunit.client.0.smithi103.stdout:8/405: getdents d4/d2a/d38 0 2024-03-22T02:28:37.647 INFO:tasks.workunit.client.0.smithi103.stdout:8/406: stat d4/d48/d40/d68 0 2024-03-22T02:28:37.648 INFO:tasks.workunit.client.0.smithi103.stdout:8/407: write d4/d26/d27/f71 [718193,116239] 0 2024-03-22T02:28:37.650 INFO:tasks.workunit.client.0.smithi103.stdout:8/408: mkdir d4/d14/d39/d49/d79 0 2024-03-22T02:28:37.651 INFO:tasks.workunit.client.0.smithi103.stdout:8/409: symlink d4/d48/d35/l7a 0 2024-03-22T02:28:37.653 INFO:tasks.workunit.client.0.smithi103.stdout:8/410: unlink d4/d26/l3a 0 2024-03-22T02:28:37.653 INFO:tasks.workunit.client.0.smithi103.stdout:8/411: dread - d4/d14/d39/d49/d5f/f66 zero size 2024-03-22T02:28:37.654 INFO:tasks.workunit.client.0.smithi103.stdout:8/412: chown d4/d19/d1a 357 1 2024-03-22T02:28:37.655 INFO:tasks.workunit.client.0.smithi103.stdout:8/413: rmdir d4/d2a/d38 39 2024-03-22T02:28:37.658 INFO:tasks.workunit.client.0.smithi103.stdout:8/414: rename d4/d48/d40/d5e to d4/d14/d39/d7b 0 2024-03-22T02:28:37.658 INFO:tasks.workunit.client.0.smithi103.stdout:8/415: fdatasync d4/d2a/d3e/f4b 0 2024-03-22T02:28:37.660 INFO:tasks.workunit.client.0.smithi103.stdout:8/416: unlink d4/d2a/d38/c46 0 2024-03-22T02:28:37.683 INFO:tasks.workunit.client.0.smithi103.stdout:8/417: creat d4/d26/f7c x:0 0 0 2024-03-22T02:28:37.686 INFO:tasks.workunit.client.0.smithi103.stdout:8/418: dread d4/d2a/d3e/f4b [0,4194304] 0 2024-03-22T02:28:37.697 INFO:tasks.workunit.client.0.smithi103.stdout:8/419: dread d4/fe [0,4194304] 0 2024-03-22T02:28:37.699 INFO:tasks.workunit.client.0.smithi103.stdout:8/420: creat d4/d2a/f7d x:0 0 0 2024-03-22T02:28:37.702 INFO:tasks.workunit.client.0.smithi103.stdout:8/421: rmdir d4/d48/d40/d68/d75 0 2024-03-22T02:28:37.714 INFO:tasks.workunit.client.0.smithi103.stdout:8/422: dread d4/d26/d27/f2c [4194304,4194304] 0 2024-03-22T02:28:37.719 INFO:tasks.workunit.client.0.smithi103.stdout:8/423: rename d4/d14 to d4/d26/d27/d7e 0 2024-03-22T02:28:37.719 INFO:tasks.workunit.client.0.smithi103.stdout:8/424: chown d4/d48/d40/d68 3211 1 2024-03-22T02:28:37.720 INFO:tasks.workunit.client.0.smithi103.stdout:8/425: stat d4/d26/d27/d7e/f1e 0 2024-03-22T02:28:37.721 INFO:tasks.workunit.client.0.smithi103.stdout:8/426: read - d4/d26/d27/f4d zero size 2024-03-22T02:28:37.722 INFO:tasks.workunit.client.0.smithi103.stdout:8/427: write f1 [1981674,24023] 0 2024-03-22T02:28:37.723 INFO:tasks.workunit.client.0.smithi103.stdout:8/428: write d4/d26/d27/d7e/d39/f45 [432457,17951] 0 2024-03-22T02:28:37.731 INFO:tasks.workunit.client.0.smithi103.stdout:8/429: write d4/d26/f4e [1297234,54354] 0 2024-03-22T02:28:37.732 INFO:tasks.workunit.client.0.smithi103.stdout:8/430: write d4/f34 [2579977,112287] 0 2024-03-22T02:28:37.734 INFO:tasks.workunit.client.0.smithi103.stdout:8/431: write d4/fe [1103338,100387] 0 2024-03-22T02:28:37.736 INFO:tasks.workunit.client.0.smithi103.stdout:8/432: symlink d4/d26/d27/l7f 0 2024-03-22T02:28:37.737 INFO:tasks.workunit.client.0.smithi103.stdout:8/433: symlink d4/d48/d40/d68/l80 0 2024-03-22T02:28:37.738 INFO:tasks.workunit.client.0.smithi103.stdout:8/434: stat d4/d19/d1a/f2d 0 2024-03-22T02:28:37.740 INFO:tasks.workunit.client.0.smithi103.stdout:8/435: dread d4/fd [0,4194304] 0 2024-03-22T02:28:37.741 INFO:tasks.workunit.client.0.smithi103.stdout:8/436: read d4/d26/d27/d7e/f1e [2112514,110088] 0 2024-03-22T02:28:37.743 INFO:tasks.workunit.client.0.smithi103.stdout:8/437: creat d4/d26/f81 x:0 0 0 2024-03-22T02:28:37.746 INFO:tasks.workunit.client.0.smithi103.stdout:8/438: dread d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:37.751 INFO:tasks.workunit.client.0.smithi103.stdout:8/439: truncate d4/fe 7483777 0 2024-03-22T02:28:37.753 INFO:tasks.workunit.client.0.smithi103.stdout:8/440: mknod d4/d48/d40/c82 0 2024-03-22T02:28:37.758 INFO:tasks.workunit.client.0.smithi103.stdout:8/441: write d4/d26/d27/d7e/f1e [1568221,41761] 0 2024-03-22T02:28:37.762 INFO:tasks.workunit.client.0.smithi103.stdout:8/442: dread d4/d26/d27/d7e/f52 [0,4194304] 0 2024-03-22T02:28:37.765 INFO:tasks.workunit.client.0.smithi103.stdout:8/443: rename d4/d26/d27/d70/l72 to d4/d2a/l83 0 2024-03-22T02:28:37.766 INFO:tasks.workunit.client.0.smithi103.stdout:8/444: write d4/ff [5157353,21640] 0 2024-03-22T02:28:37.769 INFO:tasks.workunit.client.0.smithi103.stdout:8/445: getdents d4/d26/d27/d7e/d39/d49 0 2024-03-22T02:28:37.774 INFO:tasks.workunit.client.0.smithi103.stdout:8/446: dwrite d4/d26/d27/d7e/d39/d49/d5f/f60 [0,4194304] 0 2024-03-22T02:28:37.776 INFO:tasks.workunit.client.0.smithi103.stdout:8/447: write d4/d26/f7c [6074,50062] 0 2024-03-22T02:28:37.779 INFO:tasks.workunit.client.0.smithi103.stdout:8/448: mknod d4/d26/d27/d7e/d39/d49/d79/c84 0 2024-03-22T02:28:37.782 INFO:tasks.workunit.client.0.smithi103.stdout:8/449: dread d4/d26/d27/d7e/f52 [0,4194304] 0 2024-03-22T02:28:37.787 INFO:tasks.workunit.client.0.smithi103.stdout:8/450: dwrite d4/f6e [0,4194304] 0 2024-03-22T02:28:37.791 INFO:tasks.workunit.client.1.smithi122.stderr:++ readlink -f fsstress 2024-03-22T02:28:37.793 INFO:tasks.workunit.client.1.smithi122.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-22T02:28:37.793 INFO:tasks.workunit.client.1.smithi122.stderr:+ popd 2024-03-22T02:28:37.795 INFO:tasks.workunit.client.1.smithi122.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-22T02:28:37.795 INFO:tasks.workunit.client.1.smithi122.stderr:+ popd 2024-03-22T02:28:37.796 INFO:tasks.workunit.client.1.smithi122.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-22T02:28:37.796 INFO:tasks.workunit.client.1.smithi122.stderr:++ mktemp -d -p . 2024-03-22T02:28:37.799 INFO:tasks.workunit.client.1.smithi122.stderr:+ T=./tmp.3l0pIfzLJO 2024-03-22T02:28:37.799 INFO:tasks.workunit.client.1.smithi122.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.3l0pIfzLJO -l 1 -n 1000 -p 10 -v 2024-03-22T02:28:37.802 INFO:tasks.workunit.client.1.smithi122.stdout:seed = 1710271647 2024-03-22T02:28:37.805 INFO:tasks.workunit.client.1.smithi122.stdout:0/0: rmdir - no directory 2024-03-22T02:28:37.805 INFO:tasks.workunit.client.1.smithi122.stdout:0/1: write - no filename 2024-03-22T02:28:37.806 INFO:tasks.workunit.client.1.smithi122.stdout:1/0: write - no filename 2024-03-22T02:28:37.806 INFO:tasks.workunit.client.1.smithi122.stdout:1/1: write - no filename 2024-03-22T02:28:37.807 INFO:tasks.workunit.client.1.smithi122.stdout:0/2: symlink l0 0 2024-03-22T02:28:37.808 INFO:tasks.workunit.client.1.smithi122.stdout:2/0: stat - no entries 2024-03-22T02:28:37.808 INFO:tasks.workunit.client.1.smithi122.stdout:2/1: fsync - no filename 2024-03-22T02:28:37.808 INFO:tasks.workunit.client.1.smithi122.stdout:1/2: creat f0 x:0 0 0 2024-03-22T02:28:37.809 INFO:tasks.workunit.client.1.smithi122.stdout:3/0: rename - no filename 2024-03-22T02:28:37.810 INFO:tasks.workunit.client.1.smithi122.stdout:1/3: mknod c1 0 2024-03-22T02:28:37.811 INFO:tasks.workunit.client.1.smithi122.stdout:4/0: write - no filename 2024-03-22T02:28:37.811 INFO:tasks.workunit.client.1.smithi122.stdout:4/1: dwrite - no filename 2024-03-22T02:28:37.811 INFO:tasks.workunit.client.1.smithi122.stdout:4/2: truncate - no filename 2024-03-22T02:28:37.811 INFO:tasks.workunit.client.1.smithi122.stdout:4/3: write - no filename 2024-03-22T02:28:37.811 INFO:tasks.workunit.client.1.smithi122.stdout:3/1: symlink l0 0 2024-03-22T02:28:37.812 INFO:tasks.workunit.client.1.smithi122.stdout:3/2: chown l0 133969759 1 2024-03-22T02:28:37.812 INFO:tasks.workunit.client.1.smithi122.stdout:5/0: link - no file 2024-03-22T02:28:37.812 INFO:tasks.workunit.client.1.smithi122.stdout:5/1: dread - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/2: dread - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/3: rename - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/4: write - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/5: dread - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/6: rename - no filename 2024-03-22T02:28:37.813 INFO:tasks.workunit.client.1.smithi122.stdout:4/4: mkdir d0 0 2024-03-22T02:28:37.815 INFO:tasks.workunit.client.1.smithi122.stdout:5/7: getdents . 0 2024-03-22T02:28:37.815 INFO:tasks.workunit.client.1.smithi122.stdout:5/8: rename - no filename 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:4/5: symlink d0/l1 0 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:4/6: read - no filename 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:7/0: dwrite - no filename 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:7/1: chown . 0 1 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:7/2: rmdir - no directory 2024-03-22T02:28:37.817 INFO:tasks.workunit.client.1.smithi122.stdout:7/3: fdatasync - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/4: fdatasync - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/5: stat - no entries 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/6: readlink - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/7: rename - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/8: write - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/9: dwrite - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:7/10: dwrite - no filename 2024-03-22T02:28:37.818 INFO:tasks.workunit.client.1.smithi122.stdout:8/0: dread - no filename 2024-03-22T02:28:37.819 INFO:tasks.workunit.client.1.smithi122.stdout:6/0: symlink l0 0 2024-03-22T02:28:37.819 INFO:tasks.workunit.client.1.smithi122.stdout:5/9: creat f0 x:0 0 0 2024-03-22T02:28:37.820 INFO:tasks.workunit.client.1.smithi122.stdout:4/7: unlink d0/l1 0 2024-03-22T02:28:37.822 INFO:tasks.workunit.client.1.smithi122.stdout:6/1: rename l0 to l1 0 2024-03-22T02:28:37.823 INFO:tasks.workunit.client.1.smithi122.stdout:5/10: mknod c1 0 2024-03-22T02:28:37.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/11: mkdir d0 0 2024-03-22T02:28:37.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/12: rename d0 to d0/d1 22 2024-03-22T02:28:37.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/13: dwrite - no filename 2024-03-22T02:28:37.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/14: truncate - no filename 2024-03-22T02:28:37.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/15: dread - no filename 2024-03-22T02:28:37.825 INFO:tasks.workunit.client.1.smithi122.stdout:4/8: mkdir d0/d2 0 2024-03-22T02:28:37.825 INFO:tasks.workunit.client.1.smithi122.stdout:8/1: mknod c0 0 2024-03-22T02:28:37.825 INFO:tasks.workunit.client.1.smithi122.stdout:4/9: chown d0/d2 16352880 1 2024-03-22T02:28:37.826 INFO:tasks.workunit.client.1.smithi122.stdout:4/10: write - no filename 2024-03-22T02:28:37.826 INFO:tasks.workunit.client.1.smithi122.stdout:4/11: write - no filename 2024-03-22T02:28:37.827 INFO:tasks.workunit.client.1.smithi122.stdout:6/2: creat f2 x:0 0 0 2024-03-22T02:28:37.828 INFO:tasks.workunit.client.1.smithi122.stdout:9/0: creat f0 x:0 0 0 2024-03-22T02:28:37.828 INFO:tasks.workunit.client.1.smithi122.stdout:8/2: mkdir d1 0 2024-03-22T02:28:37.828 INFO:tasks.workunit.client.1.smithi122.stdout:8/3: chown d1 0 1 2024-03-22T02:28:37.829 INFO:tasks.workunit.client.1.smithi122.stdout:4/12: creat d0/f3 x:0 0 0 2024-03-22T02:28:37.830 INFO:tasks.workunit.client.1.smithi122.stdout:7/16: creat d0/f2 x:0 0 0 2024-03-22T02:28:37.830 INFO:tasks.workunit.client.1.smithi122.stdout:8/4: rename d1 to d1/d2 22 2024-03-22T02:28:37.831 INFO:tasks.workunit.client.1.smithi122.stdout:6/3: creat f3 x:0 0 0 2024-03-22T02:28:37.831 INFO:tasks.workunit.client.1.smithi122.stdout:9/1: creat f1 x:0 0 0 2024-03-22T02:28:37.831 INFO:tasks.workunit.client.1.smithi122.stdout:4/13: dread - d0/f3 zero size 2024-03-22T02:28:37.833 INFO:tasks.workunit.client.1.smithi122.stdout:9/2: fdatasync f0 0 2024-03-22T02:28:37.833 INFO:tasks.workunit.client.1.smithi122.stdout:9/3: rmdir - no directory 2024-03-22T02:28:37.835 INFO:tasks.workunit.client.1.smithi122.stdout:6/4: rename f2 to f4 0 2024-03-22T02:28:37.835 INFO:tasks.workunit.client.1.smithi122.stdout:5/11: dwrite f0 [0,4194304] 0 2024-03-22T02:28:37.837 INFO:tasks.workunit.client.1.smithi122.stdout:8/5: rename c0 to d1/c3 0 2024-03-22T02:28:37.837 INFO:tasks.workunit.client.1.smithi122.stdout:8/6: readlink - no filename 2024-03-22T02:28:37.837 INFO:tasks.workunit.client.1.smithi122.stdout:8/7: dwrite - no filename 2024-03-22T02:28:37.838 INFO:tasks.workunit.client.1.smithi122.stdout:9/4: unlink f0 0 2024-03-22T02:28:37.842 INFO:tasks.workunit.client.1.smithi122.stdout:4/14: creat d0/d2/f4 x:0 0 0 2024-03-22T02:28:37.844 INFO:tasks.workunit.client.1.smithi122.stdout:9/5: unlink f1 0 2024-03-22T02:28:37.844 INFO:tasks.workunit.client.1.smithi122.stdout:9/6: unlink - no file 2024-03-22T02:28:37.844 INFO:tasks.workunit.client.1.smithi122.stdout:9/7: chown . 7107 1 2024-03-22T02:28:37.845 INFO:tasks.workunit.client.1.smithi122.stdout:4/15: mkdir d0/d5 0 2024-03-22T02:28:37.845 INFO:tasks.workunit.client.1.smithi122.stdout:7/17: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:37.847 INFO:tasks.workunit.client.1.smithi122.stdout:9/8: creat f2 x:0 0 0 2024-03-22T02:28:37.847 INFO:tasks.workunit.client.1.smithi122.stdout:5/12: dread f0 [0,4194304] 0 2024-03-22T02:28:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:37 smithi122 ceph-mon[36403]: pgmap v12: 65 pgs: 65 active+clean; 735 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 11 MiB/s rd, 86 MiB/s wr, 537 op/s 2024-03-22T02:28:37.848 INFO:tasks.workunit.client.1.smithi122.stdout:9/9: fdatasync f2 0 2024-03-22T02:28:37.849 INFO:tasks.workunit.client.1.smithi122.stdout:7/18: mknod d0/c3 0 2024-03-22T02:28:37.851 INFO:tasks.workunit.client.1.smithi122.stdout:5/13: write f0 [4003472,671] 0 2024-03-22T02:28:37.856 INFO:tasks.workunit.client.1.smithi122.stdout:5/14: mknod c2 0 2024-03-22T02:28:37.872 INFO:tasks.workunit.client.0.smithi103.stdout:9/247: sync 2024-03-22T02:28:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:37 smithi103 ceph-mon[29258]: pgmap v12: 65 pgs: 65 active+clean; 735 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 11 MiB/s rd, 86 MiB/s wr, 537 op/s 2024-03-22T02:28:37.875 INFO:tasks.workunit.client.0.smithi103.stdout:9/248: getdents d2/da/d18/d1e/d34/d3d 0 2024-03-22T02:28:37.876 INFO:tasks.workunit.client.0.smithi103.stdout:9/249: symlink d2/l48 0 2024-03-22T02:28:37.877 INFO:tasks.workunit.client.0.smithi103.stdout:9/250: dread - d2/d10/d27/f37 zero size 2024-03-22T02:28:38.035 INFO:tasks.workunit.client.0.smithi103.stdout:1/421: sync 2024-03-22T02:28:38.035 INFO:tasks.workunit.client.0.smithi103.stdout:2/334: sync 2024-03-22T02:28:38.035 INFO:tasks.workunit.client.0.smithi103.stdout:4/375: sync 2024-03-22T02:28:38.037 INFO:tasks.workunit.client.0.smithi103.stdout:4/376: rmdir d3/d1e/d25 39 2024-03-22T02:28:38.037 INFO:tasks.workunit.client.0.smithi103.stdout:1/422: truncate db/d10/d21/f59 1524380 0 2024-03-22T02:28:38.038 INFO:tasks.workunit.client.0.smithi103.stdout:2/335: unlink d11/f3a 0 2024-03-22T02:28:38.040 INFO:tasks.workunit.client.1.smithi122.stdout:4/16: dwrite d0/f3 [0,4194304] 0 2024-03-22T02:28:38.041 INFO:tasks.workunit.client.0.smithi103.stdout:9/251: truncate d2/da/d18/f21 6450971 0 2024-03-22T02:28:38.042 INFO:tasks.workunit.client.1.smithi122.stdout:9/10: dwrite f2 [0,4194304] 0 2024-03-22T02:28:38.043 INFO:tasks.workunit.client.0.smithi103.stdout:1/423: mknod db/d10/d29/d6c/c91 0 2024-03-22T02:28:38.044 INFO:tasks.workunit.client.1.smithi122.stdout:7/19: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:38.045 INFO:tasks.workunit.client.1.smithi122.stdout:7/20: write d0/f2 [1941272,100869] 0 2024-03-22T02:28:38.046 INFO:tasks.workunit.client.1.smithi122.stdout:5/15: dread f0 [0,4194304] 0 2024-03-22T02:28:38.047 INFO:tasks.workunit.client.0.smithi103.stdout:2/336: dwrite d11/d25/d63/d45/f62 [0,4194304] 0 2024-03-22T02:28:38.047 INFO:tasks.workunit.client.0.smithi103.stdout:9/252: creat d2/da/d18/d1e/d34/d3d/d46/f49 x:0 0 0 2024-03-22T02:28:38.048 INFO:tasks.workunit.client.0.smithi103.stdout:2/337: write f10 [669297,6482] 0 2024-03-22T02:28:38.052 INFO:tasks.workunit.client.0.smithi103.stdout:9/253: mknod d2/d10/d17/c4a 0 2024-03-22T02:28:38.055 INFO:tasks.workunit.client.0.smithi103.stdout:2/338: creat d11/d25/d46/d48/f6e x:0 0 0 2024-03-22T02:28:38.058 INFO:tasks.workunit.client.0.smithi103.stdout:1/424: mkdir db/d4a/d78/d92 0 2024-03-22T02:28:38.059 INFO:tasks.workunit.client.0.smithi103.stdout:9/254: dwrite d2/d10/f13 [0,4194304] 0 2024-03-22T02:28:38.062 INFO:tasks.workunit.client.0.smithi103.stdout:1/425: unlink db/d10/d18/d84/l8e 0 2024-03-22T02:28:38.063 INFO:tasks.workunit.client.0.smithi103.stdout:9/255: fsync d2/da/fd 0 2024-03-22T02:28:38.066 INFO:tasks.workunit.client.0.smithi103.stdout:2/339: dwrite d11/d25/d46/d40/f67 [0,4194304] 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:9/256: mkdir d2/d10/d43/d4b 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:1/426: mknod db/d10/d29/d41/d68/c93 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:9/257: read d2/d10/d43/f1b [468435,73605] 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:2/340: creat d11/d25/d46/d40/f6f x:0 0 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:2/341: fsync d11/d25/d46/d34/d24/f60 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:9/258: mkdir d2/d5/d4c 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:2/342: stat d11/d25/d63/c5b 0 2024-03-22T02:28:38.071 INFO:tasks.workunit.client.0.smithi103.stdout:9/259: write d2/d10/d43/f3e [481174,83742] 0 2024-03-22T02:28:38.074 INFO:tasks.workunit.client.0.smithi103.stdout:9/260: stat d2/da/d18/d1e/d34 0 2024-03-22T02:28:38.075 INFO:tasks.workunit.client.0.smithi103.stdout:2/343: creat d11/d25/d46/f70 x:0 0 0 2024-03-22T02:28:38.076 INFO:tasks.workunit.client.0.smithi103.stdout:9/261: write d2/da/d18/f41 [276786,106247] 0 2024-03-22T02:28:38.078 INFO:tasks.workunit.client.0.smithi103.stdout:1/427: dwrite db/d10/d29/d55/f6d [0,4194304] 0 2024-03-22T02:28:38.083 INFO:tasks.workunit.client.0.smithi103.stdout:9/262: mkdir d2/da/d18/d1e/d34/d3d/d4d 0 2024-03-22T02:28:38.084 INFO:tasks.workunit.client.0.smithi103.stdout:1/428: dread db/f2b [0,4194304] 0 2024-03-22T02:28:38.085 INFO:tasks.workunit.client.1.smithi122.stdout:4/17: symlink d0/d2/l6 0 2024-03-22T02:28:38.085 INFO:tasks.workunit.client.0.smithi103.stdout:2/344: getdents d11/d25/d46/d34 0 2024-03-22T02:28:38.086 INFO:tasks.workunit.client.0.smithi103.stdout:2/345: chown d11/d19/f1f 923216 1 2024-03-22T02:28:38.086 INFO:tasks.workunit.client.0.smithi103.stdout:2/346: chown d11/d25/d46/d34 24652 1 2024-03-22T02:28:38.087 INFO:tasks.workunit.client.0.smithi103.stdout:1/429: symlink db/d10/d18/d2d/l94 0 2024-03-22T02:28:38.088 INFO:tasks.workunit.client.0.smithi103.stdout:2/347: creat d11/d19/d5d/f71 x:0 0 0 2024-03-22T02:28:38.089 INFO:tasks.workunit.client.0.smithi103.stdout:2/348: chown d11/d25/f35 1850364 1 2024-03-22T02:28:38.090 INFO:tasks.workunit.client.1.smithi122.stdout:7/21: mkdir d0/d4 0 2024-03-22T02:28:38.091 INFO:tasks.workunit.client.0.smithi103.stdout:2/349: write d11/d25/d46/d48/f6e [692270,96256] 0 2024-03-22T02:28:38.093 INFO:tasks.workunit.client.1.smithi122.stdout:4/18: mkdir d0/d2/d7 0 2024-03-22T02:28:38.093 INFO:tasks.workunit.client.0.smithi103.stdout:2/350: readlink d11/d19/l64 0 2024-03-22T02:28:38.095 INFO:tasks.workunit.client.0.smithi103.stdout:1/430: dwrite db/d10/f12 [0,4194304] 0 2024-03-22T02:28:38.096 INFO:tasks.workunit.client.0.smithi103.stdout:0/437: sync 2024-03-22T02:28:38.096 INFO:tasks.workunit.client.0.smithi103.stdout:6/298: sync 2024-03-22T02:28:38.099 INFO:tasks.workunit.client.0.smithi103.stdout:1/431: unlink db/d4a/f53 0 2024-03-22T02:28:38.103 INFO:tasks.workunit.client.0.smithi103.stdout:0/438: link dd/d16/d20/f3c dd/d16/d20/d31/f8e 0 2024-03-22T02:28:38.104 INFO:tasks.workunit.client.1.smithi122.stdout:4/19: dwrite d0/f3 [0,4194304] 0 2024-03-22T02:28:38.105 INFO:tasks.workunit.client.0.smithi103.stdout:6/299: rename d0/d3/d33/l5a to d0/d3/l61 0 2024-03-22T02:28:38.105 INFO:tasks.workunit.client.0.smithi103.stdout:1/432: mknod db/d4a/d78/d92/c95 0 2024-03-22T02:28:38.106 INFO:tasks.workunit.client.1.smithi122.stdout:4/20: write d0/d2/f4 [906323,121202] 0 2024-03-22T02:28:38.107 INFO:tasks.workunit.client.1.smithi122.stdout:4/21: chown d0/d5 220933656 1 2024-03-22T02:28:38.107 INFO:tasks.workunit.client.0.smithi103.stdout:6/300: creat d0/d3/d45/f62 x:0 0 0 2024-03-22T02:28:38.108 INFO:tasks.workunit.client.0.smithi103.stdout:0/439: rename dd/l10 to dd/d16/d20/d31/d79/l8f 0 2024-03-22T02:28:38.109 INFO:tasks.workunit.client.1.smithi122.stdout:4/22: unlink d0/d2/l6 0 2024-03-22T02:28:38.110 INFO:tasks.workunit.client.0.smithi103.stdout:1/433: creat db/d10/d18/d84/f96 x:0 0 0 2024-03-22T02:28:38.111 INFO:tasks.workunit.client.0.smithi103.stdout:6/301: write d0/d3/d33/f60 [765411,91380] 0 2024-03-22T02:28:38.112 INFO:tasks.workunit.client.0.smithi103.stdout:1/434: mkdir db/d10/d28/d97 0 2024-03-22T02:28:38.113 INFO:tasks.workunit.client.0.smithi103.stdout:0/440: mknod dd/d16/d20/d29/c90 0 2024-03-22T02:28:38.115 INFO:tasks.workunit.client.0.smithi103.stdout:6/302: chown d0/d3/d45/c3a 0 1 2024-03-22T02:28:38.116 INFO:tasks.workunit.client.0.smithi103.stdout:1/435: fsync db/d10/d29/d41/d68/d74/f82 0 2024-03-22T02:28:38.118 INFO:tasks.workunit.client.0.smithi103.stdout:0/441: mkdir dd/d16/d20/d64/d91 0 2024-03-22T02:28:38.118 INFO:tasks.workunit.client.0.smithi103.stdout:1/436: write db/d10/d29/d55/f7c [896814,49779] 0 2024-03-22T02:28:38.125 INFO:tasks.workunit.client.0.smithi103.stdout:1/437: mkdir db/d10/d18/d98 0 2024-03-22T02:28:38.125 INFO:tasks.workunit.client.0.smithi103.stdout:1/438: readlink db/d4a/l52 0 2024-03-22T02:28:38.126 INFO:tasks.workunit.client.0.smithi103.stdout:0/442: symlink dd/l92 0 2024-03-22T02:28:38.127 INFO:tasks.workunit.client.0.smithi103.stdout:1/439: creat db/d10/d29/d41/f99 x:0 0 0 2024-03-22T02:28:38.128 INFO:tasks.workunit.client.0.smithi103.stdout:1/440: chown db/d10/d29/f61 430075 1 2024-03-22T02:28:38.130 INFO:tasks.workunit.client.0.smithi103.stdout:1/441: mknod db/d10/d28/c9a 0 2024-03-22T02:28:38.132 INFO:tasks.workunit.client.0.smithi103.stdout:1/442: mknod db/d10/d29/d55/c9b 0 2024-03-22T02:28:38.135 INFO:tasks.workunit.client.0.smithi103.stdout:9/263: truncate d2/f3 3575418 0 2024-03-22T02:28:38.137 INFO:tasks.workunit.client.0.smithi103.stdout:9/264: mknod d2/da/d18/d1e/c4e 0 2024-03-22T02:28:38.142 INFO:tasks.workunit.client.0.smithi103.stdout:0/443: dread dd/d16/d20/d31/f32 [4194304,4194304] 0 2024-03-22T02:28:38.143 INFO:tasks.workunit.client.1.smithi122.stdout:4/23: fdatasync d0/d2/f4 0 2024-03-22T02:28:38.144 INFO:tasks.workunit.client.1.smithi122.stdout:4/24: chown d0 55325943 1 2024-03-22T02:28:38.144 INFO:tasks.workunit.client.0.smithi103.stdout:2/351: fsync d11/d25/d46/d40/f67 0 2024-03-22T02:28:38.144 INFO:tasks.workunit.client.0.smithi103.stdout:8/451: sync 2024-03-22T02:28:38.145 INFO:tasks.workunit.client.0.smithi103.stdout:7/423: sync 2024-03-22T02:28:38.145 INFO:tasks.workunit.client.0.smithi103.stdout:3/391: sync 2024-03-22T02:28:38.145 INFO:tasks.workunit.client.0.smithi103.stdout:5/394: sync 2024-03-22T02:28:38.145 INFO:tasks.workunit.client.0.smithi103.stdout:4/377: sync 2024-03-22T02:28:38.146 INFO:tasks.workunit.client.1.smithi122.stdout:4/25: creat d0/d2/f8 x:0 0 0 2024-03-22T02:28:38.147 INFO:tasks.workunit.client.1.smithi122.stdout:4/26: write d0/d2/f4 [1486424,29111] 0 2024-03-22T02:28:38.147 INFO:tasks.workunit.client.1.smithi122.stdout:4/27: chown d0/d2 55296835 1 2024-03-22T02:28:38.148 INFO:tasks.workunit.client.0.smithi103.stdout:9/265: dwrite d2/d5/fb [0,4194304] 0 2024-03-22T02:28:38.149 INFO:tasks.workunit.client.0.smithi103.stdout:7/424: dread - d1/da/f75 zero size 2024-03-22T02:28:38.149 INFO:tasks.workunit.client.0.smithi103.stdout:3/392: stat de/d6b/d80/d4f/f5b 0 2024-03-22T02:28:38.151 INFO:tasks.workunit.client.1.smithi122.stdout:4/28: rename d0/d2/f4 to d0/f9 0 2024-03-22T02:28:38.151 INFO:tasks.workunit.client.0.smithi103.stdout:1/443: dread db/d10/d18/d2d/f88 [0,4194304] 0 2024-03-22T02:28:38.153 INFO:tasks.workunit.client.0.smithi103.stdout:4/378: rmdir d3/d69 39 2024-03-22T02:28:38.154 INFO:tasks.workunit.client.0.smithi103.stdout:7/425: write d1/da/fb [2042276,80357] 0 2024-03-22T02:28:38.157 INFO:tasks.workunit.client.0.smithi103.stdout:0/444: creat dd/d16/d20/d31/d79/f93 x:0 0 0 2024-03-22T02:28:38.159 INFO:tasks.workunit.client.0.smithi103.stdout:8/452: symlink d4/d26/d27/d7e/d67/l85 0 2024-03-22T02:28:38.160 INFO:tasks.workunit.client.1.smithi122.stdout:4/29: dwrite d0/d2/f8 [0,4194304] 0 2024-03-22T02:28:38.161 INFO:tasks.workunit.client.0.smithi103.stdout:9/266: dwrite d2/f30 [0,4194304] 0 2024-03-22T02:28:38.162 INFO:tasks.workunit.client.1.smithi122.stdout:4/30: creat d0/d2/fa x:0 0 0 2024-03-22T02:28:38.162 INFO:tasks.workunit.client.0.smithi103.stdout:2/352: rename d11/d25/f51 to d11/d25/d63/d45/d47/f72 0 2024-03-22T02:28:38.162 INFO:tasks.workunit.client.0.smithi103.stdout:3/393: dread de/f23 [0,4194304] 0 2024-03-22T02:28:38.165 INFO:tasks.workunit.client.0.smithi103.stdout:9/267: truncate d2/d10/d27/f37 1012791 0 2024-03-22T02:28:38.167 INFO:tasks.workunit.client.1.smithi122.stdout:4/31: dread d0/d2/f8 [0,4194304] 0 2024-03-22T02:28:38.169 INFO:tasks.workunit.client.0.smithi103.stdout:4/379: fdatasync d3/d30/f49 0 2024-03-22T02:28:38.169 INFO:tasks.workunit.client.0.smithi103.stdout:7/426: creat d1/d10/d11/d19/d29/d33/d4b/f89 x:0 0 0 2024-03-22T02:28:38.170 INFO:tasks.workunit.client.0.smithi103.stdout:0/445: mkdir dd/d16/d20/d29/d48/d94 0 2024-03-22T02:28:38.173 INFO:tasks.workunit.client.0.smithi103.stdout:8/453: symlink d4/d26/d27/d7e/d39/d49/d79/l86 0 2024-03-22T02:28:38.175 INFO:tasks.workunit.client.0.smithi103.stdout:2/353: write d11/f16 [3881409,32861] 0 2024-03-22T02:28:38.175 INFO:tasks.workunit.client.0.smithi103.stdout:3/394: mkdir de/d6b/d80/d4f/d84 0 2024-03-22T02:28:38.176 INFO:tasks.workunit.client.0.smithi103.stdout:8/454: fdatasync d4/d26/d27/f4d 0 2024-03-22T02:28:38.177 INFO:tasks.workunit.client.0.smithi103.stdout:6/303: dwrite d0/f23 [0,4194304] 0 2024-03-22T02:28:38.179 INFO:tasks.workunit.client.0.smithi103.stdout:3/395: chown de/d1a/d25/d26/f69 0 1 2024-03-22T02:28:38.182 INFO:tasks.workunit.client.0.smithi103.stdout:7/427: mkdir d1/d10/d11/d70/d8a 0 2024-03-22T02:28:38.187 INFO:tasks.workunit.client.0.smithi103.stdout:2/354: write f2 [3499318,49658] 0 2024-03-22T02:28:38.188 INFO:tasks.workunit.client.0.smithi103.stdout:2/355: write d11/d19/d5d/f66 [2097576,18977] 0 2024-03-22T02:28:38.190 INFO:tasks.workunit.client.0.smithi103.stdout:0/446: rename dd/d16/d4b/f6d to dd/d16/d20/d64/d6a/d8d/f95 0 2024-03-22T02:28:38.193 INFO:tasks.workunit.client.0.smithi103.stdout:6/304: creat d0/d14/f63 x:0 0 0 2024-03-22T02:28:38.194 INFO:tasks.workunit.client.0.smithi103.stdout:3/396: dread - de/d12/d30/d70/f78 zero size 2024-03-22T02:28:38.195 INFO:tasks.workunit.client.0.smithi103.stdout:4/380: mknod d3/d1e/d25/d65/c8c 0 2024-03-22T02:28:38.196 INFO:tasks.workunit.client.0.smithi103.stdout:9/268: dread d2/da/d18/f41 [0,4194304] 0 2024-03-22T02:28:38.197 INFO:tasks.workunit.client.0.smithi103.stdout:0/447: dread dd/d16/d20/d31/f6f [0,4194304] 0 2024-03-22T02:28:38.197 INFO:tasks.workunit.client.0.smithi103.stdout:7/428: creat d1/da/d1c/f8b x:0 0 0 2024-03-22T02:28:38.202 INFO:tasks.workunit.client.0.smithi103.stdout:1/444: dread db/d10/d2e/f51 [0,4194304] 0 2024-03-22T02:28:38.202 INFO:tasks.workunit.client.0.smithi103.stdout:5/395: truncate d0/d14/d23/d2a/f63 2575143 0 2024-03-22T02:28:38.205 INFO:tasks.workunit.client.0.smithi103.stdout:2/356: mknod d11/d19/d4e/c73 0 2024-03-22T02:28:38.205 INFO:tasks.workunit.client.0.smithi103.stdout:1/445: write db/d4a/f73 [3277215,64683] 0 2024-03-22T02:28:38.206 INFO:tasks.workunit.client.0.smithi103.stdout:9/269: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:38.215 INFO:tasks.workunit.client.0.smithi103.stdout:5/396: dwrite d0/d14/d23/d6f/f86 [0,4194304] 0 2024-03-22T02:28:38.218 INFO:tasks.workunit.client.0.smithi103.stdout:2/357: dwrite d11/d25/d46/f70 [0,4194304] 0 2024-03-22T02:28:38.221 INFO:tasks.workunit.client.0.smithi103.stdout:4/381: dread d3/fc [0,4194304] 0 2024-03-22T02:28:38.221 INFO:tasks.workunit.client.0.smithi103.stdout:0/448: rename dd/d16/d20 to dd/d16/d4b/d96 0 2024-03-22T02:28:38.223 INFO:tasks.workunit.client.0.smithi103.stdout:4/382: chown d3/d1e/d25/d65/c8c 13 1 2024-03-22T02:28:38.230 INFO:tasks.workunit.client.0.smithi103.stdout:1/446: dread - db/d10/d2e/f8a zero size 2024-03-22T02:28:38.230 INFO:tasks.workunit.client.0.smithi103.stdout:8/455: truncate d4/f34 4029571 0 2024-03-22T02:28:38.233 INFO:tasks.workunit.client.0.smithi103.stdout:7/429: creat d1/d6f/f8c x:0 0 0 2024-03-22T02:28:38.235 INFO:tasks.workunit.client.0.smithi103.stdout:5/397: write d0/d14/d23/d2a/d1f/f78 [1127960,106225] 0 2024-03-22T02:28:38.236 INFO:tasks.workunit.client.0.smithi103.stdout:4/383: rename d3/d1e/d25/l27 to d3/d1e/d25/d65/l8d 0 2024-03-22T02:28:38.239 INFO:tasks.workunit.client.0.smithi103.stdout:8/456: creat d4/d26/d27/d7e/f87 x:0 0 0 2024-03-22T02:28:38.240 INFO:tasks.workunit.client.0.smithi103.stdout:7/430: readlink d1/d10/d11/d19/d29/d61/d69/d5c/l81 0 2024-03-22T02:28:38.241 INFO:tasks.workunit.client.0.smithi103.stdout:8/457: chown d4/d26/d27/d7e/d39/d7b 0 1 2024-03-22T02:28:38.241 INFO:tasks.workunit.client.0.smithi103.stdout:2/358: mknod d11/c74 0 2024-03-22T02:28:38.242 INFO:tasks.workunit.client.0.smithi103.stdout:8/458: readlink d4/d26/d27/d7e/d67/l85 0 2024-03-22T02:28:38.243 INFO:tasks.workunit.client.0.smithi103.stdout:0/449: mknod dd/d16/d4b/d96/d29/d48/d94/c97 0 2024-03-22T02:28:38.244 INFO:tasks.workunit.client.0.smithi103.stdout:8/459: chown d4/d26/d27/d7e/d39/f45 59 1 2024-03-22T02:28:38.245 INFO:tasks.workunit.client.0.smithi103.stdout:5/398: unlink d0/d14/d23/d2a/f1a 0 2024-03-22T02:28:38.246 INFO:tasks.workunit.client.0.smithi103.stdout:2/359: truncate d11/d25/d63/d45/d47/f72 612695 0 2024-03-22T02:28:38.249 INFO:tasks.workunit.client.0.smithi103.stdout:7/431: dwrite d1/da/d1c/f60 [0,4194304] 0 2024-03-22T02:28:38.252 INFO:tasks.workunit.client.0.smithi103.stdout:5/399: chown d0/c10 16 1 2024-03-22T02:28:38.252 INFO:tasks.workunit.client.0.smithi103.stdout:0/450: mknod dd/d16/d4b/d96/d64/d6a/c98 0 2024-03-22T02:28:38.253 INFO:tasks.workunit.client.0.smithi103.stdout:0/451: chown dd/f8a 8798 1 2024-03-22T02:28:38.257 INFO:tasks.workunit.client.0.smithi103.stdout:8/460: unlink d4/d26/d27/d7e/c4c 0 2024-03-22T02:28:38.257 INFO:tasks.workunit.client.0.smithi103.stdout:6/305: truncate d0/d3/d22/f37 3185093 0 2024-03-22T02:28:38.260 INFO:tasks.workunit.client.0.smithi103.stdout:0/452: dwrite dd/d16/f84 [0,4194304] 0 2024-03-22T02:28:38.262 INFO:tasks.workunit.client.0.smithi103.stdout:2/360: creat d11/d19/d5d/f75 x:0 0 0 2024-03-22T02:28:38.264 INFO:tasks.workunit.client.0.smithi103.stdout:2/361: write d11/d25/d46/d40/f6f [320303,14848] 0 2024-03-22T02:28:38.265 INFO:tasks.workunit.client.0.smithi103.stdout:7/432: read d1/d10/d11/f18 [1348094,124265] 0 2024-03-22T02:28:38.267 INFO:tasks.workunit.client.0.smithi103.stdout:9/270: truncate d2/f30 2918095 0 2024-03-22T02:28:38.267 INFO:tasks.workunit.client.0.smithi103.stdout:7/433: truncate d1/da/d1c/f60 5048785 0 2024-03-22T02:28:38.269 INFO:tasks.workunit.client.0.smithi103.stdout:8/461: creat d4/d26/d27/d7e/d67/f88 x:0 0 0 2024-03-22T02:28:38.271 INFO:tasks.workunit.client.0.smithi103.stdout:8/462: dread d4/d26/f7c [0,4194304] 0 2024-03-22T02:28:38.273 INFO:tasks.workunit.client.0.smithi103.stdout:0/453: rename dd/d16/d4b/d96/d29/f4c to dd/d16/d4b/d96/d64/d91/f99 0 2024-03-22T02:28:38.278 INFO:tasks.workunit.client.0.smithi103.stdout:2/362: creat d11/d25/d46/d34/d24/f76 x:0 0 0 2024-03-22T02:28:38.278 INFO:tasks.workunit.client.0.smithi103.stdout:1/447: write db/d10/d29/d41/d68/d74/f82 [814154,97334] 0 2024-03-22T02:28:38.279 INFO:tasks.workunit.client.0.smithi103.stdout:2/363: chown d11/d25/d46/d48/f6e 311 1 2024-03-22T02:28:38.279 INFO:tasks.workunit.client.0.smithi103.stdout:5/400: mknod d0/d14/d23/d2a/d1f/c8a 0 2024-03-22T02:28:38.279 INFO:tasks.workunit.client.0.smithi103.stdout:9/271: rmdir d2/d5 39 2024-03-22T02:28:38.287 INFO:tasks.workunit.client.0.smithi103.stdout:0/454: mkdir dd/d16/d4b/d96/d64/d91/d9a 0 2024-03-22T02:28:38.289 INFO:tasks.workunit.client.0.smithi103.stdout:5/401: symlink d0/d14/d23/d36/l8b 0 2024-03-22T02:28:38.289 INFO:tasks.workunit.client.0.smithi103.stdout:8/463: dwrite d4/fd [0,4194304] 0 2024-03-22T02:28:38.291 INFO:tasks.workunit.client.0.smithi103.stdout:9/272: truncate d2/d5/f38 387113 0 2024-03-22T02:28:38.293 INFO:tasks.workunit.client.0.smithi103.stdout:9/273: write d2/d10/d43/f3e [138636,122239] 0 2024-03-22T02:28:38.294 INFO:tasks.workunit.client.0.smithi103.stdout:9/274: chown d2/f2c 1 1 2024-03-22T02:28:38.297 INFO:tasks.workunit.client.0.smithi103.stdout:6/306: truncate d0/d14/f44 3880384 0 2024-03-22T02:28:38.298 INFO:tasks.workunit.client.0.smithi103.stdout:9/275: dread d2/d10/d27/f37 [0,4194304] 0 2024-03-22T02:28:38.299 INFO:tasks.workunit.client.0.smithi103.stdout:6/307: write d0/f5b [436803,95906] 0 2024-03-22T02:28:38.299 INFO:tasks.workunit.client.0.smithi103.stdout:8/464: write d4/d19/f2e [4615521,98250] 0 2024-03-22T02:28:38.300 INFO:tasks.workunit.client.0.smithi103.stdout:4/384: link d3/d69/c60 d3/d1e/d25/c8e 0 2024-03-22T02:28:38.302 INFO:tasks.workunit.client.0.smithi103.stdout:2/364: creat d11/d25/d63/d45/f77 x:0 0 0 2024-03-22T02:28:38.303 INFO:tasks.workunit.client.0.smithi103.stdout:2/365: fdatasync d11/d19/d5d/f71 0 2024-03-22T02:28:38.303 INFO:tasks.workunit.client.0.smithi103.stdout:7/434: truncate d1/da/fe 931867 0 2024-03-22T02:28:38.306 INFO:tasks.workunit.client.0.smithi103.stdout:5/402: mkdir d0/d14/d23/d8c 0 2024-03-22T02:28:38.308 INFO:tasks.workunit.client.0.smithi103.stdout:9/276: creat d2/d10/d27/f4f x:0 0 0 2024-03-22T02:28:38.308 INFO:tasks.workunit.client.0.smithi103.stdout:9/277: stat d2/da/d18/f41 0 2024-03-22T02:28:38.309 INFO:tasks.workunit.client.0.smithi103.stdout:6/308: creat d0/f64 x:0 0 0 2024-03-22T02:28:38.312 INFO:tasks.workunit.client.0.smithi103.stdout:2/366: rmdir d11/d25/d46/d48 39 2024-03-22T02:28:38.314 INFO:tasks.workunit.client.0.smithi103.stdout:2/367: read d11/f18 [4614763,44554] 0 2024-03-22T02:28:38.319 INFO:tasks.workunit.client.0.smithi103.stdout:6/309: mknod d0/d3/d22/c65 0 2024-03-22T02:28:38.321 INFO:tasks.workunit.client.0.smithi103.stdout:2/368: mkdir d11/d25/d46/d34/d24/d78 0 2024-03-22T02:28:38.323 INFO:tasks.workunit.client.0.smithi103.stdout:8/465: creat d4/f89 x:0 0 0 2024-03-22T02:28:38.323 INFO:tasks.workunit.client.0.smithi103.stdout:0/455: dwrite dd/d16/f2f [0,4194304] 0 2024-03-22T02:28:38.324 INFO:tasks.workunit.client.0.smithi103.stdout:9/278: mkdir d2/da/d18/d1e/d34/d3d/d4d/d50 0 2024-03-22T02:28:38.327 INFO:tasks.workunit.client.0.smithi103.stdout:5/403: creat d0/d14/f8d x:0 0 0 2024-03-22T02:28:38.331 INFO:tasks.workunit.client.0.smithi103.stdout:0/456: write dd/d16/d4b/d96/d64/d91/f99 [406916,123720] 0 2024-03-22T02:28:38.332 INFO:tasks.workunit.client.0.smithi103.stdout:4/385: rename d3/d1e/f20 to d3/d30/f8f 0 2024-03-22T02:28:38.332 INFO:tasks.workunit.client.0.smithi103.stdout:9/279: dwrite d2/da/fd [0,4194304] 0 2024-03-22T02:28:38.336 INFO:tasks.workunit.client.0.smithi103.stdout:5/404: chown d0/d14/d23/d2a/d1f/l2f 12 1 2024-03-22T02:28:38.340 INFO:tasks.workunit.client.0.smithi103.stdout:2/369: link d11/f30 d11/d25/d63/d45/f79 0 2024-03-22T02:28:38.341 INFO:tasks.workunit.client.0.smithi103.stdout:9/280: write d2/d10/d17/f1f [4398823,101080] 0 2024-03-22T02:28:38.343 INFO:tasks.workunit.client.0.smithi103.stdout:0/457: mkdir dd/d16/d4b/d96/d29/d48/d5c/d9b 0 2024-03-22T02:28:38.343 INFO:tasks.workunit.client.0.smithi103.stdout:9/281: write d2/da/f23 [51683,125485] 0 2024-03-22T02:28:38.344 INFO:tasks.workunit.client.0.smithi103.stdout:2/370: read d11/d25/d46/f37 [1237436,99882] 0 2024-03-22T02:28:38.345 INFO:tasks.workunit.client.0.smithi103.stdout:4/386: fdatasync d3/d30/f55 0 2024-03-22T02:28:38.345 INFO:tasks.workunit.client.0.smithi103.stdout:5/405: rmdir d0/d14/d23/d2a/d33 39 2024-03-22T02:28:38.346 INFO:tasks.workunit.client.0.smithi103.stdout:4/387: readlink d3/d31/l7b 0 2024-03-22T02:28:38.347 INFO:tasks.workunit.client.0.smithi103.stdout:9/282: write d2/d10/d27/f4f [937457,22517] 0 2024-03-22T02:28:38.350 INFO:tasks.workunit.client.0.smithi103.stdout:5/406: mknod d0/d14/d23/d36/d50/c8e 0 2024-03-22T02:28:38.351 INFO:tasks.workunit.client.0.smithi103.stdout:8/466: dwrite d4/d26/f7c [0,4194304] 0 2024-03-22T02:28:38.352 INFO:tasks.workunit.client.0.smithi103.stdout:8/467: chown d4/d19/d1a 521 1 2024-03-22T02:28:38.353 INFO:tasks.workunit.client.0.smithi103.stdout:8/468: dread - d4/f89 zero size 2024-03-22T02:28:38.355 INFO:tasks.workunit.client.0.smithi103.stdout:8/469: truncate d4/d26/d27/d7e/d39/d49/d5f/f66 492264 0 2024-03-22T02:28:38.358 INFO:tasks.workunit.client.0.smithi103.stdout:4/388: mkdir d3/d38/d5c/d90 0 2024-03-22T02:28:38.362 INFO:tasks.workunit.client.0.smithi103.stdout:8/470: symlink d4/d19/d1a/d4a/l8a 0 2024-03-22T02:28:38.362 INFO:tasks.workunit.client.0.smithi103.stdout:0/458: link dd/d16/c7f dd/c9c 0 2024-03-22T02:28:38.363 INFO:tasks.workunit.client.0.smithi103.stdout:0/459: chown dd/l75 2 1 2024-03-22T02:28:38.365 INFO:tasks.workunit.client.0.smithi103.stdout:4/389: symlink d3/d30/d45/d6e/l91 0 2024-03-22T02:28:38.365 INFO:tasks.workunit.client.0.smithi103.stdout:2/371: getdents d11/d25/d63/d45/d47 0 2024-03-22T02:28:38.368 INFO:tasks.workunit.client.0.smithi103.stdout:4/390: unlink d3/d1e/d25/f4d 0 2024-03-22T02:28:38.369 INFO:tasks.workunit.client.0.smithi103.stdout:8/471: mkdir d4/d2a/d8b 0 2024-03-22T02:28:38.371 INFO:tasks.workunit.client.0.smithi103.stdout:2/372: readlink d11/d25/d46/d34/d24/l6b 0 2024-03-22T02:28:38.372 INFO:tasks.workunit.client.0.smithi103.stdout:2/373: chown d11/d25/d63/d45/f77 6731 1 2024-03-22T02:28:38.375 INFO:tasks.workunit.client.0.smithi103.stdout:4/391: mkdir d3/d1e/d78/d92 0 2024-03-22T02:28:38.375 INFO:tasks.workunit.client.0.smithi103.stdout:9/283: write d2/f30 [2806141,72006] 0 2024-03-22T02:28:38.376 INFO:tasks.workunit.client.0.smithi103.stdout:0/460: getdents dd/d16/d4b/d96/d31/d79 0 2024-03-22T02:28:38.377 INFO:tasks.workunit.client.0.smithi103.stdout:8/472: dwrite d4/f89 [0,4194304] 0 2024-03-22T02:28:38.378 INFO:tasks.workunit.client.0.smithi103.stdout:5/407: write d0/d14/d23/d2a/f63 [1592914,77564] 0 2024-03-22T02:28:38.380 INFO:tasks.workunit.client.0.smithi103.stdout:4/392: chown d3/d30/d45/d6e/l91 45 1 2024-03-22T02:28:38.383 INFO:tasks.workunit.client.0.smithi103.stdout:9/284: creat d2/d10/d27/f51 x:0 0 0 2024-03-22T02:28:38.386 INFO:tasks.workunit.client.0.smithi103.stdout:2/374: dwrite f4 [0,4194304] 0 2024-03-22T02:28:38.393 INFO:tasks.workunit.client.0.smithi103.stdout:9/285: rename d2/d5/f3c to d2/da/f52 0 2024-03-22T02:28:38.395 INFO:tasks.workunit.client.0.smithi103.stdout:4/393: rmdir d3/d1e/d78/d92 0 2024-03-22T02:28:38.397 INFO:tasks.workunit.client.0.smithi103.stdout:2/375: dwrite d11/d25/d63/d45/d47/f49 [0,4194304] 0 2024-03-22T02:28:38.398 INFO:tasks.workunit.client.0.smithi103.stdout:5/408: link d0/d14/f22 d0/d14/d23/d5c/d70/f8f 0 2024-03-22T02:28:38.399 INFO:tasks.workunit.client.0.smithi103.stdout:5/409: chown d0/d14/d23/d8c 1319692568 1 2024-03-22T02:28:38.401 INFO:tasks.workunit.client.0.smithi103.stdout:9/286: fdatasync d2/d10/d27/f37 0 2024-03-22T02:28:38.404 INFO:tasks.workunit.client.0.smithi103.stdout:8/473: dwrite d4/d26/d27/d7e/f52 [0,4194304] 0 2024-03-22T02:28:38.404 INFO:tasks.workunit.client.0.smithi103.stdout:4/394: chown d3/d1e/d25/l7c 2403 1 2024-03-22T02:28:38.405 INFO:tasks.workunit.client.0.smithi103.stdout:0/461: dread dd/f12 [0,4194304] 0 2024-03-22T02:28:38.405 INFO:tasks.workunit.client.0.smithi103.stdout:9/287: truncate d2/d10/d27/f4f 1869044 0 2024-03-22T02:28:38.408 INFO:tasks.workunit.client.0.smithi103.stdout:4/395: read d3/d1e/f7f [5803,119142] 0 2024-03-22T02:28:38.411 INFO:tasks.workunit.client.0.smithi103.stdout:2/376: unlink d11/d19/c2e 0 2024-03-22T02:28:38.413 INFO:tasks.workunit.client.0.smithi103.stdout:9/288: fdatasync d2/d10/d43/f1b 0 2024-03-22T02:28:38.418 INFO:tasks.workunit.client.0.smithi103.stdout:5/410: dread d0/d14/d23/d2a/d33/f4c [0,4194304] 0 2024-03-22T02:28:38.418 INFO:tasks.workunit.client.0.smithi103.stdout:9/289: rmdir d2/d10/d43 39 2024-03-22T02:28:38.419 INFO:tasks.workunit.client.0.smithi103.stdout:5/411: chown d0/d14/d23/d36/f85 51937 1 2024-03-22T02:28:38.419 INFO:tasks.workunit.client.0.smithi103.stdout:4/396: mknod d3/d1e/c93 0 2024-03-22T02:28:38.420 INFO:tasks.workunit.client.0.smithi103.stdout:2/377: creat d11/d25/d46/d48/f7a x:0 0 0 2024-03-22T02:28:38.421 INFO:tasks.workunit.client.0.smithi103.stdout:8/474: getdents d4/d2a/d38 0 2024-03-22T02:28:38.423 INFO:tasks.workunit.client.0.smithi103.stdout:2/378: chown d11/d19/d5d/f68 7 1 2024-03-22T02:28:38.423 INFO:tasks.workunit.client.0.smithi103.stdout:5/412: symlink d0/d14/d23/d2a/d2c/l90 0 2024-03-22T02:28:38.427 INFO:tasks.workunit.client.0.smithi103.stdout:0/462: write dd/f5b [596259,102352] 0 2024-03-22T02:28:38.428 INFO:tasks.workunit.client.0.smithi103.stdout:4/397: truncate d3/d1e/f28 30680 0 2024-03-22T02:28:38.428 INFO:tasks.workunit.client.0.smithi103.stdout:9/290: dwrite d2/d10/d17/f26 [0,4194304] 0 2024-03-22T02:28:38.430 INFO:tasks.workunit.client.0.smithi103.stdout:2/379: unlink d11/d19/f1f 0 2024-03-22T02:28:38.434 INFO:tasks.workunit.client.0.smithi103.stdout:0/463: write dd/d16/d4b/d96/d29/f4d [763105,92995] 0 2024-03-22T02:28:38.439 INFO:tasks.workunit.client.0.smithi103.stdout:8/475: dwrite d4/d26/d27/d7e/d39/d49/f74 [0,4194304] 0 2024-03-22T02:28:38.440 INFO:tasks.workunit.client.0.smithi103.stdout:2/380: write d11/d25/d46/f70 [3634399,100306] 0 2024-03-22T02:28:38.441 INFO:tasks.workunit.client.0.smithi103.stdout:5/413: dwrite d0/d14/d23/d36/d50/f5a [0,4194304] 0 2024-03-22T02:28:38.443 INFO:tasks.workunit.client.0.smithi103.stdout:8/476: dread - d4/d26/d27/f4d zero size 2024-03-22T02:28:38.444 INFO:tasks.workunit.client.0.smithi103.stdout:8/477: chown f2 100618515 1 2024-03-22T02:28:38.447 INFO:tasks.workunit.client.0.smithi103.stdout:4/398: symlink d3/d1e/d53/d8a/l94 0 2024-03-22T02:28:38.447 INFO:tasks.workunit.client.0.smithi103.stdout:0/464: rmdir dd/d16/d4b/d96/d64/d6a/d8d 39 2024-03-22T02:28:38.452 INFO:tasks.workunit.client.0.smithi103.stdout:2/381: symlink d11/d25/d46/d40/l7b 0 2024-03-22T02:28:38.455 INFO:tasks.workunit.client.0.smithi103.stdout:0/465: dread dd/d16/f84 [0,4194304] 0 2024-03-22T02:28:38.456 INFO:tasks.workunit.client.0.smithi103.stdout:4/399: mkdir d3/d30/d45/d95 0 2024-03-22T02:28:38.457 INFO:tasks.workunit.client.0.smithi103.stdout:0/466: write dd/d16/f55 [526236,59286] 0 2024-03-22T02:28:38.458 INFO:tasks.workunit.client.0.smithi103.stdout:8/478: mkdir d4/d19/d8c 0 2024-03-22T02:28:38.459 INFO:tasks.workunit.client.0.smithi103.stdout:2/382: creat d11/d25/f7c x:0 0 0 2024-03-22T02:28:38.459 INFO:tasks.workunit.client.0.smithi103.stdout:8/479: stat d4/d26/d27/f63 0 2024-03-22T02:28:38.460 INFO:tasks.workunit.client.0.smithi103.stdout:9/291: dwrite d2/da/d18/f21 [4194304,4194304] 0 2024-03-22T02:28:38.465 INFO:tasks.workunit.client.0.smithi103.stdout:8/480: unlink d4/d26/d27/d70/c76 0 2024-03-22T02:28:38.470 INFO:tasks.workunit.client.0.smithi103.stdout:4/400: dwrite d3/d1e/d25/f2b [4194304,4194304] 0 2024-03-22T02:28:38.471 INFO:tasks.workunit.client.0.smithi103.stdout:4/401: truncate d3/d69/f5a 163928 0 2024-03-22T02:28:38.472 INFO:tasks.workunit.client.0.smithi103.stdout:0/467: mkdir dd/d16/d4b/d96/d64/d91/d9a/d9d 0 2024-03-22T02:28:38.473 INFO:tasks.workunit.client.0.smithi103.stdout:4/402: truncate d3/f24 5904829 0 2024-03-22T02:28:38.478 INFO:tasks.workunit.client.0.smithi103.stdout:8/481: mkdir d4/d26/d27/d7e/d39/d8d 0 2024-03-22T02:28:38.482 INFO:tasks.workunit.client.0.smithi103.stdout:8/482: readlink d4/d2a/l58 0 2024-03-22T02:28:38.482 INFO:tasks.workunit.client.0.smithi103.stdout:4/403: readlink d3/d69/l42 0 2024-03-22T02:28:38.482 INFO:tasks.workunit.client.0.smithi103.stdout:9/292: link d2/da/d18/d1e/d34/c45 d2/da/d18/c53 0 2024-03-22T02:28:38.482 INFO:tasks.workunit.client.0.smithi103.stdout:4/404: dread - d3/d69/f54 zero size 2024-03-22T02:28:38.483 INFO:tasks.workunit.client.0.smithi103.stdout:0/468: dwrite dd/d16/d4b/d96/f21 [0,4194304] 0 2024-03-22T02:28:38.484 INFO:tasks.workunit.client.0.smithi103.stdout:9/293: symlink d2/da/d18/d1e/d34/l54 0 2024-03-22T02:28:38.488 INFO:tasks.workunit.client.0.smithi103.stdout:5/414: dread d0/d14/f22 [0,4194304] 0 2024-03-22T02:28:38.488 INFO:tasks.workunit.client.0.smithi103.stdout:4/405: unlink d3/d38/f63 0 2024-03-22T02:28:38.490 INFO:tasks.workunit.client.0.smithi103.stdout:0/469: symlink dd/d16/d4b/d96/d29/d48/d94/l9e 0 2024-03-22T02:28:38.493 INFO:tasks.workunit.client.0.smithi103.stdout:8/483: rename d4/d26/d27/d7e/d39/d49/d5f/f78 to d4/d26/f8e 0 2024-03-22T02:28:38.493 INFO:tasks.workunit.client.0.smithi103.stdout:2/383: dread f10 [0,4194304] 0 2024-03-22T02:28:38.494 INFO:tasks.workunit.client.0.smithi103.stdout:4/406: truncate d3/f7 993867 0 2024-03-22T02:28:38.494 INFO:tasks.workunit.client.0.smithi103.stdout:0/470: mknod dd/d16/d4b/d96/d64/d91/d9a/c9f 0 2024-03-22T02:28:38.495 INFO:tasks.workunit.client.0.smithi103.stdout:2/384: chown d11/d25/d46/l26 250 1 2024-03-22T02:28:38.496 INFO:tasks.workunit.client.0.smithi103.stdout:8/484: stat d4/c24 0 2024-03-22T02:28:38.496 INFO:tasks.workunit.client.0.smithi103.stdout:2/385: write f2 [3081484,54814] 0 2024-03-22T02:28:38.498 INFO:tasks.workunit.client.0.smithi103.stdout:9/294: rename d2/d5/l19 to d2/d5/d4c/l55 0 2024-03-22T02:28:38.498 INFO:tasks.workunit.client.0.smithi103.stdout:0/471: fsync dd/d16/d4b/d96/f39 0 2024-03-22T02:28:38.498 INFO:tasks.workunit.client.0.smithi103.stdout:5/415: dwrite d0/d14/d23/d6f/f86 [0,4194304] 0 2024-03-22T02:28:38.503 INFO:tasks.workunit.client.0.smithi103.stdout:2/386: creat d11/d19/d4e/f7d x:0 0 0 2024-03-22T02:28:38.506 INFO:tasks.workunit.client.0.smithi103.stdout:5/416: unlink d0/c10 0 2024-03-22T02:28:38.508 INFO:tasks.workunit.client.0.smithi103.stdout:9/295: dwrite d2/d5/fb [0,4194304] 0 2024-03-22T02:28:38.511 INFO:tasks.workunit.client.0.smithi103.stdout:5/417: write d0/d14/d23/d2a/d33/f4c [3470569,101914] 0 2024-03-22T02:28:38.514 INFO:tasks.workunit.client.0.smithi103.stdout:9/296: mkdir d2/d5/d4c/d56 0 2024-03-22T02:28:38.515 INFO:tasks.workunit.client.0.smithi103.stdout:9/297: fdatasync d2/f3b 0 2024-03-22T02:28:38.518 INFO:tasks.workunit.client.0.smithi103.stdout:5/418: unlink d0/d14/d23/d2a/d2c/f6d 0 2024-03-22T02:28:38.519 INFO:tasks.workunit.client.0.smithi103.stdout:5/419: stat d0/d14/d23/d36/c52 0 2024-03-22T02:28:38.520 INFO:tasks.workunit.client.0.smithi103.stdout:5/420: fdatasync d0/d14/d23/d2a/d1f/f78 0 2024-03-22T02:28:38.521 INFO:tasks.workunit.client.1.smithi122.stdout:6/5: fsync f4 0 2024-03-22T02:28:38.522 INFO:tasks.workunit.client.0.smithi103.stdout:8/485: getdents d4/d26/d27/d7e/d39/d49 0 2024-03-22T02:28:38.524 INFO:tasks.workunit.client.0.smithi103.stdout:9/298: symlink d2/d5/d4c/d56/l57 0 2024-03-22T02:28:38.525 INFO:tasks.workunit.client.0.smithi103.stdout:5/421: write d0/d14/d23/d36/f6a [1747777,12097] 0 2024-03-22T02:28:38.528 INFO:tasks.workunit.client.0.smithi103.stdout:9/299: unlink d2/d10/d27/f37 0 2024-03-22T02:28:38.531 INFO:tasks.workunit.client.0.smithi103.stdout:8/486: creat d4/d19/d8c/f8f x:0 0 0 2024-03-22T02:28:38.532 INFO:tasks.workunit.client.0.smithi103.stdout:5/422: dwrite d0/d14/d23/d2a/f63 [0,4194304] 0 2024-03-22T02:28:38.534 INFO:tasks.workunit.client.0.smithi103.stdout:9/300: creat d2/da/d18/d1e/d34/d3d/d4d/f58 x:0 0 0 2024-03-22T02:28:38.537 INFO:tasks.workunit.client.0.smithi103.stdout:2/387: write d11/f18 [7664463,49922] 0 2024-03-22T02:28:38.537 INFO:tasks.workunit.client.0.smithi103.stdout:9/301: truncate d2/da/d18/d1e/d34/d3d/d4d/f58 742591 0 2024-03-22T02:28:38.539 INFO:tasks.workunit.client.0.smithi103.stdout:0/472: truncate dd/d16/f84 1255876 0 2024-03-22T02:28:38.541 INFO:tasks.workunit.client.0.smithi103.stdout:8/487: rmdir d4/d26/d27/d70 0 2024-03-22T02:28:38.543 INFO:tasks.workunit.client.0.smithi103.stdout:8/488: dread - d4/d2a/d38/f6a zero size 2024-03-22T02:28:38.547 INFO:tasks.workunit.client.0.smithi103.stdout:5/423: dwrite d0/d14/d23/d46/f61 [4194304,4194304] 0 2024-03-22T02:28:38.547 INFO:tasks.workunit.client.0.smithi103.stdout:0/473: fsync dd/f12 0 2024-03-22T02:28:38.553 INFO:tasks.workunit.client.0.smithi103.stdout:2/388: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:38.556 INFO:tasks.workunit.client.0.smithi103.stdout:0/474: rmdir dd/d16/d4b/d96 39 2024-03-22T02:28:38.557 INFO:tasks.workunit.client.1.smithi122.stdout:9/11: write f2 [4625450,103705] 0 2024-03-22T02:28:38.557 INFO:tasks.workunit.client.1.smithi122.stdout:7/22: truncate d0/f2 2154563 0 2024-03-22T02:28:38.558 INFO:tasks.workunit.client.0.smithi103.stdout:8/489: dwrite d4/d26/d27/f71 [0,4194304] 0 2024-03-22T02:28:38.558 INFO:tasks.workunit.client.1.smithi122.stdout:9/12: chown f2 387 1 2024-03-22T02:28:38.558 INFO:tasks.workunit.client.1.smithi122.stdout:4/32: fsync d0/d2/fa 0 2024-03-22T02:28:38.558 INFO:tasks.workunit.client.1.smithi122.stdout:8/8: chown d1/c3 3262206 1 2024-03-22T02:28:38.559 INFO:tasks.workunit.client.1.smithi122.stdout:8/9: fsync - no filename 2024-03-22T02:28:38.559 INFO:tasks.workunit.client.0.smithi103.stdout:5/424: rename d0/d14/l15 to d0/d14/d23/d6f/d83/l91 0 2024-03-22T02:28:38.561 INFO:tasks.workunit.client.0.smithi103.stdout:2/389: mkdir d11/d19/d7e 0 2024-03-22T02:28:38.562 INFO:tasks.workunit.client.0.smithi103.stdout:2/390: readlink d11/d25/d46/d48/l4d 0 2024-03-22T02:28:38.562 INFO:tasks.workunit.client.1.smithi122.stdout:9/13: rename f2 to f3 0 2024-03-22T02:28:38.563 INFO:tasks.workunit.client.1.smithi122.stdout:8/10: symlink d1/l4 0 2024-03-22T02:28:38.563 INFO:tasks.workunit.client.1.smithi122.stdout:8/11: rename d1 to d1/d5 22 2024-03-22T02:28:38.567 INFO:tasks.workunit.client.1.smithi122.stdout:9/14: mknod c4 0 2024-03-22T02:28:38.568 INFO:tasks.workunit.client.1.smithi122.stdout:4/33: creat d0/d2/d7/fb x:0 0 0 2024-03-22T02:28:38.569 INFO:tasks.workunit.client.0.smithi103.stdout:2/391: mknod d11/d25/d46/d34/c7f 0 2024-03-22T02:28:38.569 INFO:tasks.workunit.client.0.smithi103.stdout:5/425: dread d0/d14/d23/d2a/d1f/f32 [0,4194304] 0 2024-03-22T02:28:38.571 INFO:tasks.workunit.client.0.smithi103.stdout:9/302: dread d2/d5/ff [0,4194304] 0 2024-03-22T02:28:38.571 INFO:tasks.workunit.client.1.smithi122.stdout:8/12: rename d1/l4 to d1/l6 0 2024-03-22T02:28:38.571 INFO:tasks.workunit.client.1.smithi122.stdout:8/13: dread - no filename 2024-03-22T02:28:38.571 INFO:tasks.workunit.client.1.smithi122.stdout:8/14: fdatasync - no filename 2024-03-22T02:28:38.571 INFO:tasks.workunit.client.1.smithi122.stdout:9/15: creat f5 x:0 0 0 2024-03-22T02:28:38.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/15: chown d1 27 1 2024-03-22T02:28:38.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/16: dwrite - no filename 2024-03-22T02:28:38.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/17: read - no filename 2024-03-22T02:28:38.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/18: truncate - no filename 2024-03-22T02:28:38.573 INFO:tasks.workunit.client.1.smithi122.stdout:4/34: write d0/f9 [139587,74027] 0 2024-03-22T02:28:38.573 INFO:tasks.workunit.client.1.smithi122.stdout:9/16: fdatasync f3 0 2024-03-22T02:28:38.573 INFO:tasks.workunit.client.0.smithi103.stdout:5/426: write d0/d14/d23/d36/f54 [310876,47515] 0 2024-03-22T02:28:38.573 INFO:tasks.workunit.client.0.smithi103.stdout:2/392: write d11/d25/d63/d45/f77 [328770,56238] 0 2024-03-22T02:28:38.573 INFO:tasks.workunit.client.0.smithi103.stdout:0/475: creat dd/d34/d53/d8c/fa0 x:0 0 0 2024-03-22T02:28:38.574 INFO:tasks.workunit.client.1.smithi122.stdout:8/19: symlink d1/l7 0 2024-03-22T02:28:38.574 INFO:tasks.workunit.client.1.smithi122.stdout:8/20: dread - no filename 2024-03-22T02:28:38.574 INFO:tasks.workunit.client.0.smithi103.stdout:8/490: mknod d4/d26/d27/d7e/d39/d8d/c90 0 2024-03-22T02:28:38.575 INFO:tasks.workunit.client.0.smithi103.stdout:5/427: write d0/d14/d23/d2a/d1f/f68 [622391,14497] 0 2024-03-22T02:28:38.575 INFO:tasks.workunit.client.1.smithi122.stdout:4/35: rename d0/d5 to d0/dc 0 2024-03-22T02:28:38.579 INFO:tasks.workunit.client.0.smithi103.stdout:0/476: creat dd/d34/d53/fa1 x:0 0 0 2024-03-22T02:28:38.581 INFO:tasks.workunit.client.0.smithi103.stdout:5/428: creat d0/d14/d23/d5c/f92 x:0 0 0 2024-03-22T02:28:38.582 INFO:tasks.workunit.client.0.smithi103.stdout:8/491: unlink d4/d26/d27/d7e/d67/l6d 0 2024-03-22T02:28:38.583 INFO:tasks.workunit.client.0.smithi103.stdout:9/303: truncate d2/d10/d43/f1b 3954000 0 2024-03-22T02:28:38.583 INFO:tasks.workunit.client.0.smithi103.stdout:2/393: creat d11/d19/d7e/f80 x:0 0 0 2024-03-22T02:28:38.583 INFO:tasks.workunit.client.0.smithi103.stdout:9/304: write d2/f30 [3539192,53199] 0 2024-03-22T02:28:38.584 INFO:tasks.workunit.client.0.smithi103.stdout:2/394: chown d11/d25 451 1 2024-03-22T02:28:38.584 INFO:tasks.workunit.client.0.smithi103.stdout:5/429: chown d0/d14/d23/d2a/l5f 1 1 2024-03-22T02:28:38.585 INFO:tasks.workunit.client.0.smithi103.stdout:8/492: creat d4/d19/d1a/f91 x:0 0 0 2024-03-22T02:28:38.586 INFO:tasks.workunit.client.0.smithi103.stdout:8/493: chown d4/d2a/d4f 256696975 1 2024-03-22T02:28:38.586 INFO:tasks.workunit.client.0.smithi103.stdout:3/397: sync 2024-03-22T02:28:38.603 INFO:tasks.workunit.client.0.smithi103.stdout:9/305: mkdir d2/d10/d59 0 2024-03-22T02:28:38.634 INFO:tasks.workunit.client.0.smithi103.stdout:2/395: dwrite d11/d25/d63/d45/f62 [0,4194304] 0 2024-03-22T02:28:38.635 INFO:tasks.workunit.client.0.smithi103.stdout:5/430: dwrite d0/d14/d23/d36/f6a [0,4194304] 0 2024-03-22T02:28:38.638 INFO:tasks.workunit.client.0.smithi103.stdout:3/398: dwrite de/d1a/d25/f42 [0,4194304] 0 2024-03-22T02:28:38.642 INFO:tasks.workunit.client.0.smithi103.stdout:9/306: dread d2/da/d18/f21 [4194304,4194304] 0 2024-03-22T02:28:38.642 INFO:tasks.workunit.client.0.smithi103.stdout:0/477: link dd/d16/d4b/d96/d35/c78 dd/d16/d4b/d96/d64/d6a/ca2 0 2024-03-22T02:28:38.646 INFO:tasks.workunit.client.0.smithi103.stdout:5/431: dread - d0/d14/d23/d5c/d70/f87 zero size 2024-03-22T02:28:38.648 INFO:tasks.workunit.client.0.smithi103.stdout:2/396: dwrite d11/d19/d5d/f71 [0,4194304] 0 2024-03-22T02:28:38.649 INFO:tasks.workunit.client.0.smithi103.stdout:3/399: rmdir de/d1a/d25/d61 0 2024-03-22T02:28:38.650 INFO:tasks.workunit.client.0.smithi103.stdout:0/478: creat dd/d16/d4b/d96/d64/fa3 x:0 0 0 2024-03-22T02:28:38.652 INFO:tasks.workunit.client.0.smithi103.stdout:0/479: write dd/d16/d4b/d96/d64/d91/f99 [356462,103921] 0 2024-03-22T02:28:38.652 INFO:tasks.workunit.client.0.smithi103.stdout:3/400: creat de/d1a/d25/d3f/f85 x:0 0 0 2024-03-22T02:28:38.654 INFO:tasks.workunit.client.0.smithi103.stdout:2/397: mknod d11/d25/d63/d45/d47/c81 0 2024-03-22T02:28:38.656 INFO:tasks.workunit.client.0.smithi103.stdout:5/432: creat d0/d14/d23/d8c/f93 x:0 0 0 2024-03-22T02:28:38.659 INFO:tasks.workunit.client.0.smithi103.stdout:3/401: dwrite de/f5f [0,4194304] 0 2024-03-22T02:28:38.661 INFO:tasks.workunit.client.0.smithi103.stdout:2/398: dread d11/d25/d63/d45/f77 [0,4194304] 0 2024-03-22T02:28:38.662 INFO:tasks.workunit.client.0.smithi103.stdout:2/399: read - d11/d19/d4e/f53 zero size 2024-03-22T02:28:38.664 INFO:tasks.workunit.client.1.smithi122.stdout:4/36: rename d0/f3 to d0/dc/fd 0 2024-03-22T02:28:38.666 INFO:tasks.workunit.client.1.smithi122.stdout:9/17: dwrite f5 [0,4194304] 0 2024-03-22T02:28:38.682 INFO:tasks.workunit.client.0.smithi103.stdout:0/480: chown dd/c27 0 1 2024-03-22T02:28:38.690 INFO:tasks.workunit.client.0.smithi103.stdout:0/481: rmdir dd/d34/d53 39 2024-03-22T02:28:38.695 INFO:tasks.workunit.client.0.smithi103.stdout:0/482: symlink dd/d16/d4b/la4 0 2024-03-22T02:28:38.698 INFO:tasks.workunit.client.0.smithi103.stdout:8/494: truncate d4/fd 1562371 0 2024-03-22T02:28:38.699 INFO:tasks.workunit.client.0.smithi103.stdout:0/483: read dd/f24 [3623227,28967] 0 2024-03-22T02:28:38.700 INFO:tasks.workunit.client.0.smithi103.stdout:8/495: dread - d4/d2a/d3e/f6b zero size 2024-03-22T02:28:38.701 INFO:tasks.workunit.client.0.smithi103.stdout:8/496: write d4/d26/f7c [4921381,23132] 0 2024-03-22T02:28:38.703 INFO:tasks.workunit.client.0.smithi103.stdout:0/484: symlink dd/d16/d4b/d96/d29/d48/d5c/la5 0 2024-03-22T02:28:38.711 INFO:tasks.workunit.client.1.smithi122.stdout:9/18: fdatasync f5 0 2024-03-22T02:28:38.719 INFO:tasks.workunit.client.0.smithi103.stdout:3/402: dread de/d12/d19/f28 [0,4194304] 0 2024-03-22T02:28:38.720 INFO:tasks.workunit.client.0.smithi103.stdout:3/403: write de/d1a/d25/f6a [438480,122817] 0 2024-03-22T02:28:38.721 INFO:tasks.workunit.client.0.smithi103.stdout:9/307: truncate d2/da/f23 289615 0 2024-03-22T02:28:38.724 INFO:tasks.workunit.client.0.smithi103.stdout:9/308: chown d2/da/d18/c53 30312 1 2024-03-22T02:28:38.725 INFO:tasks.workunit.client.0.smithi103.stdout:3/404: read de/d1a/d25/d26/f6f [2362504,92081] 0 2024-03-22T02:28:38.725 INFO:tasks.workunit.client.0.smithi103.stdout:9/309: fsync d2/da/d18/d1e/d34/d3d/d4d/f58 0 2024-03-22T02:28:38.729 INFO:tasks.workunit.client.0.smithi103.stdout:9/310: creat d2/d10/d17/f5a x:0 0 0 2024-03-22T02:28:38.730 INFO:tasks.workunit.client.0.smithi103.stdout:9/311: write d2/da/f25 [489967,126261] 0 2024-03-22T02:28:38.731 INFO:tasks.workunit.client.0.smithi103.stdout:9/312: stat d2/d10/d17/c4a 0 2024-03-22T02:28:38.731 INFO:tasks.workunit.client.0.smithi103.stdout:3/405: creat de/d12/f86 x:0 0 0 2024-03-22T02:28:38.732 INFO:tasks.workunit.client.0.smithi103.stdout:3/406: chown de/d6b/d80/d4f/l62 17224 1 2024-03-22T02:28:38.733 INFO:tasks.workunit.client.0.smithi103.stdout:5/433: unlink d0/d14/d23/d2a/d1f/f20 0 2024-03-22T02:28:38.736 INFO:tasks.workunit.client.0.smithi103.stdout:9/313: rename d2/d10/d43/c20 to d2/da/d18/d1e/d34/d3d/d4d/c5b 0 2024-03-22T02:28:38.737 INFO:tasks.workunit.client.0.smithi103.stdout:9/314: write d2/d10/d27/f51 [698615,48566] 0 2024-03-22T02:28:38.738 INFO:tasks.workunit.client.0.smithi103.stdout:5/434: mknod d0/d14/d23/d6f/c94 0 2024-03-22T02:28:38.745 INFO:tasks.workunit.client.0.smithi103.stdout:5/435: dwrite d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:38.753 INFO:tasks.workunit.client.0.smithi103.stdout:5/436: getdents d0/d14/d23/d5c 0 2024-03-22T02:28:38.756 INFO:tasks.workunit.client.0.smithi103.stdout:5/437: rename d0/d14/d23/d36/d50 to d0/d14/d23/d36/d95 0 2024-03-22T02:28:38.763 INFO:tasks.workunit.client.0.smithi103.stdout:5/438: dwrite d0/d14/d23/d36/f85 [0,4194304] 0 2024-03-22T02:28:38.767 INFO:tasks.workunit.client.0.smithi103.stdout:5/439: mkdir d0/d14/d23/d36/d95/d96 0 2024-03-22T02:28:38.768 INFO:tasks.workunit.client.0.smithi103.stdout:5/440: readlink d0/d14/l43 0 2024-03-22T02:28:38.777 INFO:tasks.workunit.client.0.smithi103.stdout:5/441: dwrite d0/d14/d23/d2a/d2c/f4d [4194304,4194304] 0 2024-03-22T02:28:38.780 INFO:tasks.workunit.client.1.smithi122.stdout:7/23: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:38.781 INFO:tasks.workunit.client.1.smithi122.stdout:4/37: getdents d0/d2/d7 0 2024-03-22T02:28:38.782 INFO:tasks.workunit.client.1.smithi122.stdout:4/38: stat d0/dc/fd 0 2024-03-22T02:28:38.783 INFO:tasks.workunit.client.1.smithi122.stdout:7/24: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:38.784 INFO:tasks.workunit.client.0.smithi103.stdout:5/442: dread d0/d14/d23/d36/f6a [0,4194304] 0 2024-03-22T02:28:38.785 INFO:tasks.workunit.client.1.smithi122.stdout:8/21: chown d1/l6 64198 1 2024-03-22T02:28:38.785 INFO:tasks.workunit.client.1.smithi122.stdout:8/22: dwrite - no filename 2024-03-22T02:28:38.786 INFO:tasks.workunit.client.0.smithi103.stdout:2/400: symlink d11/d25/d46/l82 0 2024-03-22T02:28:38.786 INFO:tasks.workunit.client.1.smithi122.stdout:7/25: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:38.787 INFO:tasks.workunit.client.1.smithi122.stdout:7/26: read d0/f2 [958455,16661] 0 2024-03-22T02:28:38.788 INFO:tasks.workunit.client.1.smithi122.stdout:4/39: rename d0/dc to d0/d2/d7/de 0 2024-03-22T02:28:38.788 INFO:tasks.workunit.client.1.smithi122.stdout:7/27: stat d0/c3 0 2024-03-22T02:28:38.788 INFO:tasks.workunit.client.1.smithi122.stdout:4/40: rename d0/d2 to d0/d2/d7/df 22 2024-03-22T02:28:38.789 INFO:tasks.workunit.client.1.smithi122.stdout:8/23: mkdir d1/d8 0 2024-03-22T02:28:38.792 INFO:tasks.workunit.client.0.smithi103.stdout:9/315: dwrite d2/d10/f13 [4194304,4194304] 0 2024-03-22T02:28:38.792 INFO:tasks.workunit.client.1.smithi122.stdout:7/28: creat d0/d4/f5 x:0 0 0 2024-03-22T02:28:38.793 INFO:tasks.workunit.client.0.smithi103.stdout:2/401: rename d11/d25/d46/d34/d24 to d11/d25/d63/d45/d47/d83 0 2024-03-22T02:28:38.793 INFO:tasks.workunit.client.1.smithi122.stdout:8/24: mkdir d1/d9 0 2024-03-22T02:28:38.793 INFO:tasks.workunit.client.1.smithi122.stdout:8/25: dread - no filename 2024-03-22T02:28:38.794 INFO:tasks.workunit.client.1.smithi122.stdout:7/29: mknod d0/d4/c6 0 2024-03-22T02:28:38.798 INFO:tasks.workunit.client.1.smithi122.stdout:8/26: creat d1/d9/fa x:0 0 0 2024-03-22T02:28:38.801 INFO:tasks.workunit.client.0.smithi103.stdout:0/485: unlink dd/l7c 0 2024-03-22T02:28:38.803 INFO:tasks.workunit.client.1.smithi122.stdout:8/27: rename d1/d9/fa to d1/d9/fb 0 2024-03-22T02:28:38.803 INFO:tasks.workunit.client.1.smithi122.stdout:7/30: dwrite d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:38.804 INFO:tasks.workunit.client.0.smithi103.stdout:3/407: rmdir de/d1a/d25/d3f 39 2024-03-22T02:28:38.805 INFO:tasks.workunit.client.0.smithi103.stdout:0/486: creat dd/d16/d4b/d96/d64/d91/d9a/fa6 x:0 0 0 2024-03-22T02:28:38.807 INFO:tasks.workunit.client.1.smithi122.stdout:7/31: getdents d0 0 2024-03-22T02:28:38.807 INFO:tasks.workunit.client.0.smithi103.stdout:0/487: chown dd/l44 0 1 2024-03-22T02:28:38.808 INFO:tasks.workunit.client.1.smithi122.stdout:7/32: stat d0/d4/c6 0 2024-03-22T02:28:38.809 INFO:tasks.workunit.client.1.smithi122.stdout:7/33: write d0/f2 [2298280,95429] 0 2024-03-22T02:28:38.810 INFO:tasks.workunit.client.1.smithi122.stdout:7/34: chown d0/c3 33445704 1 2024-03-22T02:28:38.812 INFO:tasks.workunit.client.1.smithi122.stdout:7/35: symlink d0/d4/l7 0 2024-03-22T02:28:38.812 INFO:tasks.workunit.client.1.smithi122.stdout:8/28: fsync d1/d9/fb 0 2024-03-22T02:28:38.813 INFO:tasks.workunit.client.0.smithi103.stdout:3/408: unlink de/d1a/d25/d3f/c83 0 2024-03-22T02:28:38.813 INFO:tasks.workunit.client.0.smithi103.stdout:0/488: rmdir dd/d16/d4b/d96/d31/d79 39 2024-03-22T02:28:38.813 INFO:tasks.workunit.client.1.smithi122.stdout:7/36: write d0/f2 [1990258,75554] 0 2024-03-22T02:28:38.813 INFO:tasks.workunit.client.0.smithi103.stdout:3/409: symlink de/d12/d4d/l87 0 2024-03-22T02:28:38.814 INFO:tasks.workunit.client.0.smithi103.stdout:0/489: write dd/d16/d4b/d96/f5d [1583502,82885] 0 2024-03-22T02:28:38.815 INFO:tasks.workunit.client.0.smithi103.stdout:5/443: truncate d0/d14/d23/d2a/d1f/f68 195156 0 2024-03-22T02:28:38.816 INFO:tasks.workunit.client.0.smithi103.stdout:5/444: chown d0/d14/d23/d2a/d1f/l2f 4466 1 2024-03-22T02:28:38.816 INFO:tasks.workunit.client.1.smithi122.stdout:8/29: symlink d1/d8/lc 0 2024-03-22T02:28:38.817 INFO:tasks.workunit.client.0.smithi103.stdout:0/490: unlink dd/d16/d4b/d96/d64/d91/d9a/fa6 0 2024-03-22T02:28:38.820 INFO:tasks.workunit.client.1.smithi122.stdout:7/37: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:38.823 INFO:tasks.workunit.client.1.smithi122.stdout:8/30: dwrite d1/d9/fb [0,4194304] 0 2024-03-22T02:28:38.825 INFO:tasks.workunit.client.1.smithi122.stdout:8/31: rmdir d1/d9 39 2024-03-22T02:28:38.827 INFO:tasks.workunit.client.1.smithi122.stdout:8/32: write d1/d9/fb [2467646,121] 0 2024-03-22T02:28:38.828 INFO:tasks.workunit.client.1.smithi122.stdout:8/33: truncate d1/d9/fb 4816675 0 2024-03-22T02:28:38.831 INFO:tasks.workunit.client.1.smithi122.stdout:8/34: getdents d1/d9 0 2024-03-22T02:28:38.835 INFO:tasks.workunit.client.1.smithi122.stdout:8/35: dread d1/d9/fb [0,4194304] 0 2024-03-22T02:28:38.835 INFO:tasks.workunit.client.1.smithi122.stdout:8/36: stat d1/l6 0 2024-03-22T02:28:38.835 INFO:tasks.workunit.client.1.smithi122.stdout:8/37: chown d1 1743492 1 2024-03-22T02:28:38.836 INFO:tasks.workunit.client.1.smithi122.stdout:8/38: rename d1 to d1/d8/dd 22 2024-03-22T02:28:38.838 INFO:tasks.workunit.client.1.smithi122.stdout:8/39: mknod d1/d9/ce 0 2024-03-22T02:28:38.839 INFO:tasks.workunit.client.1.smithi122.stdout:8/40: mknod d1/d8/cf 0 2024-03-22T02:28:38.841 INFO:tasks.workunit.client.1.smithi122.stdout:8/41: creat d1/f10 x:0 0 0 2024-03-22T02:28:38.843 INFO:tasks.workunit.client.1.smithi122.stdout:8/42: creat d1/d9/f11 x:0 0 0 2024-03-22T02:28:38.847 INFO:tasks.workunit.client.1.smithi122.stdout:8/43: link d1/l6 d1/l12 0 2024-03-22T02:28:38.848 INFO:tasks.workunit.client.0.smithi103.stdout:0/491: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 x:0 0 0 2024-03-22T02:28:38.851 INFO:tasks.workunit.client.0.smithi103.stdout:9/316: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:38.851 INFO:tasks.workunit.client.0.smithi103.stdout:0/492: read - dd/d16/d4b/d96/d31/f8e zero size 2024-03-22T02:28:38.853 INFO:tasks.workunit.client.0.smithi103.stdout:5/445: dwrite d0/d14/f22 [4194304,4194304] 0 2024-03-22T02:28:38.853 INFO:tasks.workunit.client.1.smithi122.stdout:8/44: dwrite d1/f10 [0,4194304] 0 2024-03-22T02:28:38.857 INFO:tasks.workunit.client.0.smithi103.stdout:5/446: stat d0/d14/d23/d2a/l5f 0 2024-03-22T02:28:38.860 INFO:tasks.workunit.client.0.smithi103.stdout:9/317: rename d2/f30 to d2/d5/d4c/f5c 0 2024-03-22T02:28:38.862 INFO:tasks.workunit.client.0.smithi103.stdout:0/493: creat dd/d16/d4b/d96/d29/d48/d5c/fa8 x:0 0 0 2024-03-22T02:28:38.868 INFO:tasks.workunit.client.0.smithi103.stdout:0/494: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/fa9 x:0 0 0 2024-03-22T02:28:38.872 INFO:tasks.workunit.client.0.smithi103.stdout:9/318: dwrite d2/da/d18/d1e/d34/d3d/d4d/f58 [0,4194304] 0 2024-03-22T02:28:38.873 INFO:tasks.workunit.client.0.smithi103.stdout:0/495: dread - dd/d16/d4b/d96/d31/d79/f93 zero size 2024-03-22T02:28:38.874 INFO:tasks.workunit.client.0.smithi103.stdout:5/447: dwrite d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:38.874 INFO:tasks.workunit.client.0.smithi103.stdout:3/410: dread de/d12/d30/f3d [0,4194304] 0 2024-03-22T02:28:38.876 INFO:tasks.workunit.client.0.smithi103.stdout:5/448: chown d0/d14/d23/d8c/f93 6 1 2024-03-22T02:28:38.877 INFO:tasks.workunit.client.1.smithi122.stdout:8/45: fsync d1/f10 0 2024-03-22T02:28:38.878 INFO:tasks.workunit.client.0.smithi103.stdout:3/411: write de/d1a/d25/d26/f54 [4279641,79589] 0 2024-03-22T02:28:38.878 INFO:tasks.workunit.client.0.smithi103.stdout:5/449: stat d0/d14/d23/d2a/d1f/f78 0 2024-03-22T02:28:38.879 INFO:tasks.workunit.client.0.smithi103.stdout:0/496: read - dd/d16/d4b/d96/d29/d48/d5c/f80 zero size 2024-03-22T02:28:38.879 INFO:tasks.workunit.client.1.smithi122.stdout:8/46: dread - d1/d9/f11 zero size 2024-03-22T02:28:38.879 INFO:tasks.workunit.client.1.smithi122.stdout:8/47: readlink d1/l7 0 2024-03-22T02:28:38.880 INFO:tasks.workunit.client.0.smithi103.stdout:3/412: readlink de/d6b/d80/d4f/l62 0 2024-03-22T02:28:38.880 INFO:tasks.workunit.client.0.smithi103.stdout:0/497: readlink dd/d16/d4b/la4 0 2024-03-22T02:28:38.880 INFO:tasks.workunit.client.1.smithi122.stdout:8/48: write d1/d9/f11 [959310,27953] 0 2024-03-22T02:28:38.881 INFO:tasks.workunit.client.0.smithi103.stdout:5/450: write d0/d14/d23/f44 [3754948,121629] 0 2024-03-22T02:28:38.882 INFO:tasks.workunit.client.1.smithi122.stdout:8/49: readlink d1/l12 0 2024-03-22T02:28:38.883 INFO:tasks.workunit.client.1.smithi122.stdout:8/50: stat d1/d9/fb 0 2024-03-22T02:28:38.886 INFO:tasks.workunit.client.1.smithi122.stdout:8/51: getdents d1/d8 0 2024-03-22T02:28:38.888 INFO:tasks.workunit.client.0.smithi103.stdout:3/413: creat de/d6b/d80/d4f/d84/f88 x:0 0 0 2024-03-22T02:28:38.888 INFO:tasks.workunit.client.1.smithi122.stdout:8/52: rmdir d1/d8 39 2024-03-22T02:28:38.890 INFO:tasks.workunit.client.0.smithi103.stdout:9/319: write d2/d5/ff [4992482,9860] 0 2024-03-22T02:28:38.890 INFO:tasks.workunit.client.0.smithi103.stdout:0/498: dwrite dd/d16/d4b/d96/d29/f3e [0,4194304] 0 2024-03-22T02:28:38.890 INFO:tasks.workunit.client.0.smithi103.stdout:5/451: creat d0/f97 x:0 0 0 2024-03-22T02:28:38.891 INFO:tasks.workunit.client.1.smithi122.stdout:8/53: dread d1/f10 [0,4194304] 0 2024-03-22T02:28:38.894 INFO:tasks.workunit.client.0.smithi103.stdout:9/320: write d2/da/d18/d1e/d34/d3d/d4d/f58 [5003742,38261] 0 2024-03-22T02:28:38.894 INFO:tasks.workunit.client.1.smithi122.stdout:8/54: unlink d1/l6 0 2024-03-22T02:28:38.894 INFO:tasks.workunit.client.1.smithi122.stdout:8/55: creat d1/d9/f13 x:0 0 0 2024-03-22T02:28:38.896 INFO:tasks.workunit.client.0.smithi103.stdout:9/321: mknod d2/da/d18/d1e/c5d 0 2024-03-22T02:28:38.896 INFO:tasks.workunit.client.1.smithi122.stdout:8/56: symlink d1/d8/l14 0 2024-03-22T02:28:38.897 INFO:tasks.workunit.client.0.smithi103.stdout:9/322: stat d2/da/d18/d1e/d34/d3d/c44 0 2024-03-22T02:28:38.898 INFO:tasks.workunit.client.0.smithi103.stdout:5/452: link d0/fe d0/d3e/f98 0 2024-03-22T02:28:38.899 INFO:tasks.workunit.client.0.smithi103.stdout:9/323: write d2/da/fd [877941,63576] 0 2024-03-22T02:28:38.900 INFO:tasks.workunit.client.1.smithi122.stdout:8/57: dread d1/d9/f11 [0,4194304] 0 2024-03-22T02:28:38.901 INFO:tasks.workunit.client.0.smithi103.stdout:5/453: symlink d0/d14/l99 0 2024-03-22T02:28:38.901 INFO:tasks.workunit.client.1.smithi122.stdout:8/58: mknod d1/d9/c15 0 2024-03-22T02:28:38.902 INFO:tasks.workunit.client.0.smithi103.stdout:9/324: rename d2/da/d18/f21 to d2/d10/d59/f5e 0 2024-03-22T02:28:38.903 INFO:tasks.workunit.client.1.smithi122.stdout:8/59: fsync d1/d9/fb 0 2024-03-22T02:28:38.905 INFO:tasks.workunit.client.1.smithi122.stdout:8/60: dread d1/d9/f11 [0,4194304] 0 2024-03-22T02:28:38.905 INFO:tasks.workunit.client.0.smithi103.stdout:9/325: symlink d2/da/d18/d1e/d34/l5f 0 2024-03-22T02:28:38.906 INFO:tasks.workunit.client.0.smithi103.stdout:5/454: creat d0/f9a x:0 0 0 2024-03-22T02:28:38.907 INFO:tasks.workunit.client.0.smithi103.stdout:9/326: mknod d2/da/d18/d1e/c60 0 2024-03-22T02:28:38.908 INFO:tasks.workunit.client.0.smithi103.stdout:5/455: symlink d0/d14/d23/d5c/d70/l9b 0 2024-03-22T02:28:38.910 INFO:tasks.workunit.client.0.smithi103.stdout:9/327: mkdir d2/da/d18/d1e/d34/d3d/d61 0 2024-03-22T02:28:38.912 INFO:tasks.workunit.client.1.smithi122.stdout:8/61: getdents d1 0 2024-03-22T02:28:38.912 INFO:tasks.workunit.client.1.smithi122.stdout:8/62: readlink d1/d8/l14 0 2024-03-22T02:28:38.915 INFO:tasks.workunit.client.0.smithi103.stdout:9/328: rmdir d2/da/d18/d1e/d34/d3d/d4d/d50 0 2024-03-22T02:28:38.916 INFO:tasks.workunit.client.1.smithi122.stdout:8/63: dread - d1/d9/f13 zero size 2024-03-22T02:28:38.917 INFO:tasks.workunit.client.0.smithi103.stdout:3/414: dread de/d1a/d25/d26/f58 [0,4194304] 0 2024-03-22T02:28:38.917 INFO:tasks.workunit.client.0.smithi103.stdout:5/456: dread d0/f12 [4194304,4194304] 0 2024-03-22T02:28:38.918 INFO:tasks.workunit.client.0.smithi103.stdout:3/415: chown de/d6b/d80/d4f/l62 5956 1 2024-03-22T02:28:38.918 INFO:tasks.workunit.client.0.smithi103.stdout:9/329: rename d2/d10/d43/c47 to d2/d10/d43/c62 0 2024-03-22T02:28:38.919 INFO:tasks.workunit.client.0.smithi103.stdout:9/330: chown d2/da/d18/d1e/d34 11476207 1 2024-03-22T02:28:38.920 INFO:tasks.workunit.client.1.smithi122.stdout:8/64: symlink d1/d9/l16 0 2024-03-22T02:28:38.922 INFO:tasks.workunit.client.0.smithi103.stdout:5/457: dread d0/f71 [4194304,4194304] 0 2024-03-22T02:28:38.922 INFO:tasks.workunit.client.0.smithi103.stdout:9/331: creat d2/d5/f63 x:0 0 0 2024-03-22T02:28:38.923 INFO:tasks.workunit.client.0.smithi103.stdout:9/332: write d2/da/f24 [2277879,96872] 0 2024-03-22T02:28:38.924 INFO:tasks.workunit.client.0.smithi103.stdout:9/333: fsync d2/d5/ff 0 2024-03-22T02:28:38.929 INFO:tasks.workunit.client.0.smithi103.stdout:5/458: getdents d0/d14/d23/d5c/d77 0 2024-03-22T02:28:38.930 INFO:tasks.workunit.client.0.smithi103.stdout:9/334: dwrite d2/da/fd [0,4194304] 0 2024-03-22T02:28:38.931 INFO:tasks.workunit.client.0.smithi103.stdout:5/459: truncate d0/d14/d23/d36/d95/f7d 762693 0 2024-03-22T02:28:38.935 INFO:tasks.workunit.client.0.smithi103.stdout:3/416: write de/d1a/d25/f29 [3479161,85786] 0 2024-03-22T02:28:38.936 INFO:tasks.workunit.client.0.smithi103.stdout:9/335: truncate d2/f3b 5155 0 2024-03-22T02:28:38.939 INFO:tasks.workunit.client.0.smithi103.stdout:9/336: symlink d2/da/d18/d1e/d34/d3d/d4d/l64 0 2024-03-22T02:28:38.939 INFO:tasks.workunit.client.0.smithi103.stdout:3/417: link de/d6b/l7e de/d6b/l89 0 2024-03-22T02:28:38.940 INFO:tasks.workunit.client.0.smithi103.stdout:3/418: rmdir de/d1a/d25 39 2024-03-22T02:28:38.941 INFO:tasks.workunit.client.0.smithi103.stdout:9/337: mkdir d2/d10/d43/d4b/d65 0 2024-03-22T02:28:38.945 INFO:tasks.workunit.client.0.smithi103.stdout:9/338: dread d2/d10/d17/f26 [0,4194304] 0 2024-03-22T02:28:38.948 INFO:tasks.workunit.client.0.smithi103.stdout:9/339: dread d2/d10/d27/f51 [0,4194304] 0 2024-03-22T02:28:38.952 INFO:tasks.workunit.client.0.smithi103.stdout:9/340: creat d2/d10/d43/f66 x:0 0 0 2024-03-22T02:28:38.952 INFO:tasks.workunit.client.0.smithi103.stdout:3/419: write de/d12/d30/f35 [3121944,118241] 0 2024-03-22T02:28:38.954 INFO:tasks.workunit.client.0.smithi103.stdout:3/420: mkdir de/d12/d8a 0 2024-03-22T02:28:38.960 INFO:tasks.workunit.client.0.smithi103.stdout:3/421: dwrite de/d1a/d25/d26/f6f [0,4194304] 0 2024-03-22T02:28:38.961 INFO:tasks.workunit.client.0.smithi103.stdout:3/422: fsync de/d12/d19/f71 0 2024-03-22T02:28:38.962 INFO:tasks.workunit.client.0.smithi103.stdout:3/423: chown de/d12/d30 843 1 2024-03-22T02:28:38.963 INFO:tasks.workunit.client.0.smithi103.stdout:3/424: write de/d12/f86 [300246,45567] 0 2024-03-22T02:28:38.966 INFO:tasks.workunit.client.0.smithi103.stdout:3/425: creat de/d1a/d25/f8b x:0 0 0 2024-03-22T02:28:38.967 INFO:tasks.workunit.client.0.smithi103.stdout:3/426: truncate f1 4927332 0 2024-03-22T02:28:38.967 INFO:tasks.workunit.client.0.smithi103.stdout:3/427: chown de/d1a/d25/f42 1585689 1 2024-03-22T02:28:38.970 INFO:tasks.workunit.client.0.smithi103.stdout:3/428: mknod de/d12/d5d/c8c 0 2024-03-22T02:28:38.970 INFO:tasks.workunit.client.0.smithi103.stdout:3/429: readlink de/l3b 0 2024-03-22T02:28:38.975 INFO:tasks.workunit.client.0.smithi103.stdout:3/430: dwrite de/d12/d19/f51 [0,4194304] 0 2024-03-22T02:28:38.978 INFO:tasks.workunit.client.0.smithi103.stdout:3/431: symlink de/d1a/l8d 0 2024-03-22T02:28:39.021 INFO:tasks.workunit.client.0.smithi103.stdout:1/448: sync 2024-03-22T02:28:39.022 INFO:tasks.workunit.client.0.smithi103.stdout:1/449: creat db/d4a/f9c x:0 0 0 2024-03-22T02:28:39.023 INFO:tasks.workunit.client.0.smithi103.stdout:1/450: write db/d10/d29/d41/f57 [107571,68819] 0 2024-03-22T02:28:39.024 INFO:tasks.workunit.client.0.smithi103.stdout:1/451: symlink db/d10/d29/l9d 0 2024-03-22T02:28:39.026 INFO:tasks.workunit.client.0.smithi103.stdout:1/452: rmdir db/d10/d21 39 2024-03-22T02:28:39.055 INFO:tasks.workunit.client.0.smithi103.stdout:1/453: creat db/f9e x:0 0 0 2024-03-22T02:28:39.059 INFO:tasks.workunit.client.0.smithi103.stdout:1/454: link db/d10/d29/c7d db/d10/d18/d98/c9f 0 2024-03-22T02:28:39.060 INFO:tasks.workunit.client.0.smithi103.stdout:1/455: creat db/d4a/fa0 x:0 0 0 2024-03-22T02:28:39.063 INFO:tasks.workunit.client.0.smithi103.stdout:1/456: unlink db/d10/f79 0 2024-03-22T02:28:39.069 INFO:tasks.workunit.client.0.smithi103.stdout:1/457: dwrite db/d10/d21/f33 [4194304,4194304] 0 2024-03-22T02:28:39.070 INFO:tasks.workunit.client.0.smithi103.stdout:1/458: chown db/d10/d18/d2d/c77 1954 1 2024-03-22T02:28:39.073 INFO:tasks.workunit.client.0.smithi103.stdout:1/459: creat db/d10/d29/d41/d68/fa1 x:0 0 0 2024-03-22T02:28:39.077 INFO:tasks.workunit.client.0.smithi103.stdout:1/460: chown db/d10/d18/f72 18363 1 2024-03-22T02:28:39.082 INFO:tasks.workunit.client.0.smithi103.stdout:1/461: dwrite db/d10/d18/d2d/f4d [0,4194304] 0 2024-03-22T02:28:39.086 INFO:tasks.workunit.client.0.smithi103.stdout:1/462: symlink db/d10/d28/la2 0 2024-03-22T02:28:39.087 INFO:tasks.workunit.client.0.smithi103.stdout:1/463: dread - db/d10/d29/d41/d68/f86 zero size 2024-03-22T02:28:39.089 INFO:tasks.workunit.client.0.smithi103.stdout:1/464: rmdir db/d10/d6b 39 2024-03-22T02:28:39.089 INFO:tasks.workunit.client.0.smithi103.stdout:1/465: chown db/f9e 6827989 1 2024-03-22T02:28:39.091 INFO:tasks.workunit.client.0.smithi103.stdout:1/466: read - db/d10/d6b/f8d zero size 2024-03-22T02:28:39.097 INFO:tasks.workunit.client.0.smithi103.stdout:1/467: dwrite db/d10/d29/f61 [0,4194304] 0 2024-03-22T02:28:39.100 INFO:tasks.workunit.client.0.smithi103.stdout:1/468: creat db/d10/d29/d41/d68/fa3 x:0 0 0 2024-03-22T02:28:39.104 INFO:tasks.workunit.client.0.smithi103.stdout:1/469: creat db/d10/d18/fa4 x:0 0 0 2024-03-22T02:28:39.105 INFO:tasks.workunit.client.0.smithi103.stdout:1/470: stat db/d10/d18/c42 0 2024-03-22T02:28:39.105 INFO:tasks.workunit.client.0.smithi103.stdout:1/471: stat db 0 2024-03-22T02:28:39.107 INFO:tasks.workunit.client.0.smithi103.stdout:1/472: dread db/d10/d18/d2d/f4d [0,4194304] 0 2024-03-22T02:28:39.180 INFO:tasks.workunit.client.0.smithi103.stdout:7/435: sync 2024-03-22T02:28:39.181 INFO:tasks.workunit.client.0.smithi103.stdout:7/436: write d1/da/f75 [949997,116107] 0 2024-03-22T02:28:39.187 INFO:tasks.workunit.client.0.smithi103.stdout:7/437: dwrite d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:39.190 INFO:tasks.workunit.client.0.smithi103.stdout:7/438: dread - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f zero size 2024-03-22T02:28:39.192 INFO:tasks.workunit.client.0.smithi103.stdout:7/439: symlink d1/da/d1c/l8d 0 2024-03-22T02:28:39.198 INFO:tasks.workunit.client.0.smithi103.stdout:7/440: dwrite d1/d10/d11/d19/d29/f52 [0,4194304] 0 2024-03-22T02:28:39.202 INFO:tasks.workunit.client.0.smithi103.stdout:7/441: mknod d1/da/d2f/d36/c8e 0 2024-03-22T02:28:39.205 INFO:tasks.workunit.client.0.smithi103.stdout:7/442: symlink d1/d10/d11/d19/d29/d61/l8f 0 2024-03-22T02:28:39.209 INFO:tasks.workunit.client.0.smithi103.stdout:7/443: rename d1/da/d1c/c1f to d1/d10/d80/c90 0 2024-03-22T02:28:39.211 INFO:tasks.workunit.client.0.smithi103.stdout:7/444: rmdir d1/d10/d11/d19/d29/d61/d1d 39 2024-03-22T02:28:39.317 INFO:tasks.workunit.client.0.smithi103.stdout:6/310: sync 2024-03-22T02:28:39.317 INFO:tasks.workunit.client.0.smithi103.stdout:6/311: dread - d0/d3/d22/f50 zero size 2024-03-22T02:28:39.324 INFO:tasks.workunit.client.0.smithi103.stdout:6/312: write d0/d3/d22/f5c [4636512,53428] 0 2024-03-22T02:28:39.330 INFO:tasks.workunit.client.0.smithi103.stdout:6/313: dwrite d0/d3/d22/f47 [0,4194304] 0 2024-03-22T02:28:39.354 INFO:tasks.workunit.client.0.smithi103.stdout:6/314: dread d0/d3/f1b [0,4194304] 0 2024-03-22T02:28:39.356 INFO:tasks.workunit.client.0.smithi103.stdout:6/315: getdents d0/d3/d22 0 2024-03-22T02:28:39.357 INFO:tasks.workunit.client.0.smithi103.stdout:6/316: chown d0/f64 1485731 1 2024-03-22T02:28:39.357 INFO:tasks.workunit.client.0.smithi103.stdout:6/317: fsync d0/d3/d22/f15 0 2024-03-22T02:28:39.359 INFO:tasks.workunit.client.0.smithi103.stdout:6/318: unlink d0/d14/c54 0 2024-03-22T02:28:39.360 INFO:tasks.workunit.client.0.smithi103.stdout:6/319: mknod d0/d14/c66 0 2024-03-22T02:28:39.361 INFO:tasks.workunit.client.0.smithi103.stdout:6/320: chown d0/d3/d22 847480290 1 2024-03-22T02:28:39.361 INFO:tasks.workunit.client.0.smithi103.stdout:6/321: chown d0/d3/l12 413387 1 2024-03-22T02:28:39.362 INFO:tasks.workunit.client.0.smithi103.stdout:6/322: rmdir d0/d3/d22/d58 39 2024-03-22T02:28:39.363 INFO:tasks.workunit.client.0.smithi103.stdout:6/323: mknod d0/c67 0 2024-03-22T02:28:39.366 INFO:tasks.workunit.client.0.smithi103.stdout:6/324: rename d0/d3/d45 to d0/d14/d57/d68 0 2024-03-22T02:28:39.367 INFO:tasks.workunit.client.0.smithi103.stdout:6/325: creat d0/f69 x:0 0 0 2024-03-22T02:28:39.373 INFO:tasks.workunit.client.0.smithi103.stdout:6/326: getdents d0 0 2024-03-22T02:28:39.373 INFO:tasks.workunit.client.0.smithi103.stdout:6/327: chown d0/c34 5652321 1 2024-03-22T02:28:39.381 INFO:tasks.workunit.client.0.smithi103.stdout:6/328: dwrite d0/f1a [0,4194304] 0 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:8/497: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:0/499: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:5/460: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:9/341: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:1/473: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:7/445: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:4/407: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:3/432: sync 2024-03-22T02:28:39.389 INFO:tasks.workunit.client.0.smithi103.stdout:2/402: sync 2024-03-22T02:28:39.390 INFO:tasks.workunit.client.0.smithi103.stdout:6/329: dwrite d0/d3/d33/f5d [0,4194304] 0 2024-03-22T02:28:39.392 INFO:tasks.workunit.client.0.smithi103.stdout:4/408: readlink d3/d69/l74 0 2024-03-22T02:28:39.393 INFO:tasks.workunit.client.0.smithi103.stdout:2/403: rename d11 to d11/d25/d63/d45/d47/d84 22 2024-03-22T02:28:39.394 INFO:tasks.workunit.client.0.smithi103.stdout:9/342: read d2/d5/fb [1156222,36923] 0 2024-03-22T02:28:39.399 INFO:tasks.workunit.client.0.smithi103.stdout:7/446: rmdir d1/d10/d11/d19/d29/d33/d4b/d5e 39 2024-03-22T02:28:39.399 INFO:tasks.workunit.client.0.smithi103.stdout:8/498: symlink d4/d19/d1a/d4a/l92 0 2024-03-22T02:28:39.400 INFO:tasks.workunit.client.0.smithi103.stdout:0/500: mknod dd/d16/d4b/d96/d35/d82/caa 0 2024-03-22T02:28:39.402 INFO:tasks.workunit.client.0.smithi103.stdout:9/343: stat d2/d10/d17/f1f 0 2024-03-22T02:28:39.405 INFO:tasks.workunit.client.0.smithi103.stdout:1/474: link db/d10/d18/f3b db/d10/d29/fa5 0 2024-03-22T02:28:39.406 INFO:tasks.workunit.client.0.smithi103.stdout:4/409: creat d3/d1e/d25/d65/f96 x:0 0 0 2024-03-22T02:28:39.408 INFO:tasks.workunit.client.0.smithi103.stdout:8/499: creat d4/d26/d27/f93 x:0 0 0 2024-03-22T02:28:39.409 INFO:tasks.workunit.client.0.smithi103.stdout:5/461: link d0/d14/d23/d2a/d7b/c7c d0/d14/d23/c9c 0 2024-03-22T02:28:39.409 INFO:tasks.workunit.client.0.smithi103.stdout:0/501: mkdir dd/d16/d4b/d96/d64/dab 0 2024-03-22T02:28:39.410 INFO:tasks.workunit.client.0.smithi103.stdout:3/433: dread de/d1a/d25/f29 [0,4194304] 0 2024-03-22T02:28:39.411 INFO:tasks.workunit.client.0.smithi103.stdout:0/502: readlink dd/l44 0 2024-03-22T02:28:39.411 INFO:tasks.workunit.client.0.smithi103.stdout:8/500: chown d4/c56 6 1 2024-03-22T02:28:39.412 INFO:tasks.workunit.client.0.smithi103.stdout:7/447: read d1/d10/d11/d19/d29/d57/f6d [1076432,122604] 0 2024-03-22T02:28:39.414 INFO:tasks.workunit.client.0.smithi103.stdout:6/330: dwrite d0/f1e [4194304,4194304] 0 2024-03-22T02:28:39.414 INFO:tasks.workunit.client.0.smithi103.stdout:5/462: dread - d0/f73 zero size 2024-03-22T02:28:39.414 INFO:tasks.workunit.client.0.smithi103.stdout:1/475: unlink db/d10/d29/d41/l64 0 2024-03-22T02:28:39.414 INFO:tasks.workunit.client.0.smithi103.stdout:2/404: dwrite d11/d25/d46/f29 [0,4194304] 0 2024-03-22T02:28:39.415 INFO:tasks.workunit.client.0.smithi103.stdout:4/410: symlink d3/d1e/d25/d65/l97 0 2024-03-22T02:28:39.419 INFO:tasks.workunit.client.0.smithi103.stdout:3/434: chown de/d1a/d25/d26/c49 6 1 2024-03-22T02:28:39.419 INFO:tasks.workunit.client.0.smithi103.stdout:1/476: dread - db/f9e zero size 2024-03-22T02:28:39.420 INFO:tasks.workunit.client.0.smithi103.stdout:0/503: fsync dd/d16/d4b/d96/d31/d89/f8b 0 2024-03-22T02:28:39.422 INFO:tasks.workunit.client.0.smithi103.stdout:2/405: write d11/d25/d46/f70 [2743754,55224] 0 2024-03-22T02:28:39.425 INFO:tasks.workunit.client.0.smithi103.stdout:8/501: write d4/d26/f81 [530750,72479] 0 2024-03-22T02:28:39.426 INFO:tasks.workunit.client.0.smithi103.stdout:6/331: creat d0/f6a x:0 0 0 2024-03-22T02:28:39.427 INFO:tasks.workunit.client.0.smithi103.stdout:9/344: dwrite d2/d10/d43/f66 [0,4194304] 0 2024-03-22T02:28:39.428 INFO:tasks.workunit.client.0.smithi103.stdout:1/477: dread - db/d10/d29/d55/f8f zero size 2024-03-22T02:28:39.428 INFO:tasks.workunit.client.0.smithi103.stdout:4/411: write d3/d1e/d25/f52 [151089,54220] 0 2024-03-22T02:28:39.429 INFO:tasks.workunit.client.0.smithi103.stdout:7/448: rmdir d1/d10/d11/d70/d88 0 2024-03-22T02:28:39.429 INFO:tasks.workunit.client.0.smithi103.stdout:9/345: dread - d2/d5/f63 zero size 2024-03-22T02:28:39.429 INFO:tasks.workunit.client.0.smithi103.stdout:6/332: chown d0/d3/d22 9 1 2024-03-22T02:28:39.430 INFO:tasks.workunit.client.0.smithi103.stdout:2/406: chown d11/d25/d46/l26 15 1 2024-03-22T02:28:39.434 INFO:tasks.workunit.client.0.smithi103.stdout:4/412: fsync d3/d30/f49 0 2024-03-22T02:28:39.437 INFO:tasks.workunit.client.0.smithi103.stdout:8/502: creat d4/d2a/f94 x:0 0 0 2024-03-22T02:28:39.439 INFO:tasks.workunit.client.0.smithi103.stdout:7/449: truncate d1/f4e 5657197 0 2024-03-22T02:28:39.440 INFO:tasks.workunit.client.0.smithi103.stdout:1/478: symlink db/d10/d18/d2d/la6 0 2024-03-22T02:28:39.445 INFO:tasks.workunit.client.0.smithi103.stdout:8/503: dread - d4/d19/d1a/f91 zero size 2024-03-22T02:28:39.448 INFO:tasks.workunit.client.0.smithi103.stdout:9/346: getdents d2/d5/d4c 0 2024-03-22T02:28:39.449 INFO:tasks.workunit.client.0.smithi103.stdout:7/450: chown d1/d10/d11/d19/d29/d61/d69 427170370 1 2024-03-22T02:28:39.450 INFO:tasks.workunit.client.0.smithi103.stdout:2/407: dread d11/d25/f4c [0,4194304] 0 2024-03-22T02:28:39.453 INFO:tasks.workunit.client.0.smithi103.stdout:3/435: dwrite de/d12/d19/f43 [0,4194304] 0 2024-03-22T02:28:39.459 INFO:tasks.workunit.client.0.smithi103.stdout:0/504: dwrite dd/d16/d4b/d96/d29/f85 [0,4194304] 0 2024-03-22T02:28:39.462 INFO:tasks.workunit.client.0.smithi103.stdout:6/333: dwrite d0/d3/f39 [0,4194304] 0 2024-03-22T02:28:39.466 INFO:tasks.workunit.client.0.smithi103.stdout:1/479: dwrite db/d10/f1c [0,4194304] 0 2024-03-22T02:28:39.468 INFO:tasks.workunit.client.0.smithi103.stdout:9/347: chown d2/da/d18/d1e/d34/d3d/c44 2 1 2024-03-22T02:28:39.468 INFO:tasks.workunit.client.0.smithi103.stdout:2/408: write d11/d19/d5d/f71 [2631915,127448] 0 2024-03-22T02:28:39.474 INFO:tasks.workunit.client.0.smithi103.stdout:5/463: dread d0/d14/d23/f40 [0,4194304] 0 2024-03-22T02:28:39.475 INFO:tasks.workunit.client.0.smithi103.stdout:3/436: rename de/d12/l4a to de/d6b/l8e 0 2024-03-22T02:28:39.476 INFO:tasks.workunit.client.0.smithi103.stdout:2/409: write d11/f13 [4715525,124381] 0 2024-03-22T02:28:39.476 INFO:tasks.workunit.client.0.smithi103.stdout:4/413: dread d3/f24 [4194304,4194304] 0 2024-03-22T02:28:39.479 INFO:tasks.workunit.client.0.smithi103.stdout:6/334: creat d0/d14/d57/d68/f6b x:0 0 0 2024-03-22T02:28:39.480 INFO:tasks.workunit.client.0.smithi103.stdout:0/505: creat dd/d34/fac x:0 0 0 2024-03-22T02:28:39.481 INFO:tasks.workunit.client.0.smithi103.stdout:5/464: rmdir d0/d14/d23 39 2024-03-22T02:28:39.482 INFO:tasks.workunit.client.0.smithi103.stdout:8/504: dwrite d4/f15 [0,4194304] 0 2024-03-22T02:28:39.482 INFO:tasks.workunit.client.0.smithi103.stdout:3/437: write de/f2d [4195986,67639] 0 2024-03-22T02:28:39.483 INFO:tasks.workunit.client.0.smithi103.stdout:7/451: rename d1/d10/d11/d19/d29/f73 to d1/da/d2f/d36/f91 0 2024-03-22T02:28:39.483 INFO:tasks.workunit.client.0.smithi103.stdout:2/410: stat d11/d25/d63/d45 0 2024-03-22T02:28:39.486 INFO:tasks.workunit.client.0.smithi103.stdout:9/348: write d2/da/d18/d1e/d34/d3d/d4d/f58 [477499,39585] 0 2024-03-22T02:28:39.487 INFO:tasks.workunit.client.0.smithi103.stdout:6/335: unlink d0/f23 0 2024-03-22T02:28:39.492 INFO:tasks.workunit.client.0.smithi103.stdout:6/336: read - d0/d14/f63 zero size 2024-03-22T02:28:39.494 INFO:tasks.workunit.client.0.smithi103.stdout:1/480: dwrite db/d10/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:39.494 INFO:tasks.workunit.client.0.smithi103.stdout:9/349: chown d2/d10/d17 7015 1 2024-03-22T02:28:39.494 INFO:tasks.workunit.client.0.smithi103.stdout:8/505: write d4/d26/d27/d7e/d39/d49/d5f/f60 [2687991,30747] 0 2024-03-22T02:28:39.495 INFO:tasks.workunit.client.0.smithi103.stdout:4/414: rename d3/d1e/c93 to d3/d30/d45/c98 0 2024-03-22T02:28:39.498 INFO:tasks.workunit.client.0.smithi103.stdout:5/465: read d0/d14/d23/d2a/d1f/f38 [3907661,3051] 0 2024-03-22T02:28:39.505 INFO:tasks.workunit.client.0.smithi103.stdout:4/415: chown d3/d38/f70 92104299 1 2024-03-22T02:28:39.508 INFO:tasks.workunit.client.0.smithi103.stdout:8/506: truncate d4/d26/f4e 1610234 0 2024-03-22T02:28:39.510 INFO:tasks.workunit.client.0.smithi103.stdout:1/481: chown db/d10/l85 62 1 2024-03-22T02:28:39.510 INFO:tasks.workunit.client.0.smithi103.stdout:5/466: symlink d0/d14/d23/d6f/l9d 0 2024-03-22T02:28:39.512 INFO:tasks.workunit.client.0.smithi103.stdout:0/506: dwrite dd/d34/fac [0,4194304] 0 2024-03-22T02:28:39.513 INFO:tasks.workunit.client.0.smithi103.stdout:4/416: unlink d3/d1e/d68/f71 0 2024-03-22T02:28:39.517 INFO:tasks.workunit.client.0.smithi103.stdout:2/411: dread d11/f30 [4194304,4194304] 0 2024-03-22T02:28:39.518 INFO:tasks.workunit.client.0.smithi103.stdout:0/507: readlink dd/d16/d4b/d96/d35/l47 0 2024-03-22T02:28:39.520 INFO:tasks.workunit.client.0.smithi103.stdout:5/467: creat d0/d3e/f9e x:0 0 0 2024-03-22T02:28:39.521 INFO:tasks.workunit.client.0.smithi103.stdout:9/350: dwrite d2/f2c [4194304,4194304] 0 2024-03-22T02:28:39.521 INFO:tasks.workunit.client.0.smithi103.stdout:7/452: dwrite d1/da/f3 [0,4194304] 0 2024-03-22T02:28:39.523 INFO:tasks.workunit.client.0.smithi103.stdout:8/507: creat d4/d48/d40/d68/f95 x:0 0 0 2024-03-22T02:28:39.524 INFO:tasks.workunit.client.0.smithi103.stdout:9/351: chown d2/d10/d27 23 1 2024-03-22T02:28:39.527 INFO:tasks.workunit.client.0.smithi103.stdout:8/508: stat d4/d48/d35/f44 0 2024-03-22T02:28:39.531 INFO:tasks.workunit.client.0.smithi103.stdout:6/337: fsync d0/d3/d33/f36 0 2024-03-22T02:28:39.538 INFO:tasks.workunit.client.0.smithi103.stdout:3/438: dread de/d12/f16 [0,4194304] 0 2024-03-22T02:28:39.538 INFO:tasks.workunit.client.0.smithi103.stdout:2/412: creat d11/d25/d63/d45/d47/f85 x:0 0 0 2024-03-22T02:28:39.540 INFO:tasks.workunit.client.0.smithi103.stdout:2/413: readlink d11/d19/d4e/l4f 0 2024-03-22T02:28:39.540 INFO:tasks.workunit.client.0.smithi103.stdout:6/338: dwrite d0/f6a [0,4194304] 0 2024-03-22T02:28:39.541 INFO:tasks.workunit.client.0.smithi103.stdout:3/439: write de/d1a/d25/d3f/f85 [510086,111243] 0 2024-03-22T02:28:39.544 INFO:tasks.workunit.client.0.smithi103.stdout:2/414: dread - d11/d19/d4e/f7d zero size 2024-03-22T02:28:39.545 INFO:tasks.workunit.client.0.smithi103.stdout:7/453: mkdir d1/d10/d11/d19/d86/d92 0 2024-03-22T02:28:39.549 INFO:tasks.workunit.client.0.smithi103.stdout:6/339: write d0/d3/f49 [319288,48076] 0 2024-03-22T02:28:39.552 INFO:tasks.workunit.client.0.smithi103.stdout:7/454: rmdir d1/d6f 39 2024-03-22T02:28:39.553 INFO:tasks.workunit.client.0.smithi103.stdout:3/440: mkdir de/d12/d8f 0 2024-03-22T02:28:39.555 INFO:tasks.workunit.client.0.smithi103.stdout:2/415: symlink d11/d25/d46/d48/l86 0 2024-03-22T02:28:39.556 INFO:tasks.workunit.client.0.smithi103.stdout:9/352: dread d2/d10/d59/f5e [0,4194304] 0 2024-03-22T02:28:39.558 INFO:tasks.workunit.client.0.smithi103.stdout:4/417: dwrite d3/d38/f81 [0,4194304] 0 2024-03-22T02:28:39.559 INFO:tasks.workunit.client.0.smithi103.stdout:4/418: write d3/d69/f54 [584707,28977] 0 2024-03-22T02:28:39.562 INFO:tasks.workunit.client.0.smithi103.stdout:7/455: symlink d1/d10/d11/d70/l93 0 2024-03-22T02:28:39.563 INFO:tasks.workunit.client.0.smithi103.stdout:8/509: dread d4/d26/d27/d7e/f1e [0,4194304] 0 2024-03-22T02:28:39.563 INFO:tasks.workunit.client.0.smithi103.stdout:3/441: symlink de/d1a/d25/d3f/l90 0 2024-03-22T02:28:39.564 INFO:tasks.workunit.client.0.smithi103.stdout:0/508: truncate dd/d16/d4b/f76 2032531 0 2024-03-22T02:28:39.567 INFO:tasks.workunit.client.0.smithi103.stdout:9/353: symlink d2/d10/d27/l67 0 2024-03-22T02:28:39.567 INFO:tasks.workunit.client.0.smithi103.stdout:7/456: dread d1/da/d2f/d36/f91 [0,4194304] 0 2024-03-22T02:28:39.569 INFO:tasks.workunit.client.0.smithi103.stdout:2/416: rename ce to d11/d25/d63/d45/d47/d83/c87 0 2024-03-22T02:28:39.569 INFO:tasks.workunit.client.0.smithi103.stdout:0/509: dread - dd/d16/d4b/d96/d29/d48/d5c/f80 zero size 2024-03-22T02:28:39.570 INFO:tasks.workunit.client.0.smithi103.stdout:9/354: dread - d2/d5/f63 zero size 2024-03-22T02:28:39.571 INFO:tasks.workunit.client.0.smithi103.stdout:4/419: dwrite d3/d31/f4e [4194304,4194304] 0 2024-03-22T02:28:39.572 INFO:tasks.workunit.client.0.smithi103.stdout:8/510: mknod d4/d19/d8c/c96 0 2024-03-22T02:28:39.573 INFO:tasks.workunit.client.0.smithi103.stdout:7/457: read d1/d10/d11/d19/f26 [1854526,17227] 0 2024-03-22T02:28:39.575 INFO:tasks.workunit.client.0.smithi103.stdout:9/355: write d2/d10/d17/f1f [5538069,48951] 0 2024-03-22T02:28:39.575 INFO:tasks.workunit.client.0.smithi103.stdout:3/442: dread de/d1a/d25/d3f/f85 [0,4194304] 0 2024-03-22T02:28:39.578 INFO:tasks.workunit.client.0.smithi103.stdout:9/356: stat d2/d10 0 2024-03-22T02:28:39.578 INFO:tasks.workunit.client.0.smithi103.stdout:2/417: truncate d11/d25/d46/d34/f52 462361 0 2024-03-22T02:28:39.578 INFO:tasks.workunit.client.0.smithi103.stdout:8/511: write d4/d48/f73 [3511248,119054] 0 2024-03-22T02:28:39.580 INFO:tasks.workunit.client.0.smithi103.stdout:9/357: stat d2/d10/d59 0 2024-03-22T02:28:39.582 INFO:tasks.workunit.client.0.smithi103.stdout:0/510: getdents dd/d16/d4b/d96/d31/d89 0 2024-03-22T02:28:39.583 INFO:tasks.workunit.client.0.smithi103.stdout:0/511: fdatasync dd/d16/d4b/d96/f5d 0 2024-03-22T02:28:39.584 INFO:tasks.workunit.client.0.smithi103.stdout:0/512: chown dd/l44 27 1 2024-03-22T02:28:39.585 INFO:tasks.workunit.client.0.smithi103.stdout:0/513: chown dd/d16/d4b/d96/d64/d6a/c98 2222937 1 2024-03-22T02:28:39.586 INFO:tasks.workunit.client.0.smithi103.stdout:3/443: unlink de/d6b/d80/d4f/l62 0 2024-03-22T02:28:39.587 INFO:tasks.workunit.client.0.smithi103.stdout:4/420: getdents d3/d38/d5c/d86 0 2024-03-22T02:28:39.587 INFO:tasks.workunit.client.0.smithi103.stdout:0/514: chown dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 0 1 2024-03-22T02:28:39.587 INFO:tasks.workunit.client.0.smithi103.stdout:7/458: read d1/d10/d11/d19/d29/d61/f24 [2873536,130552] 0 2024-03-22T02:28:39.588 INFO:tasks.workunit.client.0.smithi103.stdout:6/340: truncate d0/f3e 886592 0 2024-03-22T02:28:39.589 INFO:tasks.workunit.client.0.smithi103.stdout:4/421: write d3/d69/f5a [1114207,67995] 0 2024-03-22T02:28:39.589 INFO:tasks.workunit.client.0.smithi103.stdout:2/418: creat d11/d19/d4e/f88 x:0 0 0 2024-03-22T02:28:39.589 INFO:tasks.workunit.client.0.smithi103.stdout:3/444: truncate de/d12/d19/f71 1225312 0 2024-03-22T02:28:39.590 INFO:tasks.workunit.client.0.smithi103.stdout:9/358: rmdir d2/d10 39 2024-03-22T02:28:39.591 INFO:tasks.workunit.client.0.smithi103.stdout:8/512: symlink d4/d26/d27/d7e/d67/l97 0 2024-03-22T02:28:39.595 INFO:tasks.workunit.client.0.smithi103.stdout:0/515: unlink dd/f8a 0 2024-03-22T02:28:39.595 INFO:tasks.workunit.client.0.smithi103.stdout:4/422: dwrite d3/fd [0,4194304] 0 2024-03-22T02:28:39.597 INFO:tasks.workunit.client.0.smithi103.stdout:0/516: chown dd/d16/d4b/d96/d29/d48/d94/l9e 1700441 1 2024-03-22T02:28:39.597 INFO:tasks.workunit.client.0.smithi103.stdout:6/341: creat d0/d14/f6c x:0 0 0 2024-03-22T02:28:39.599 INFO:tasks.workunit.client.0.smithi103.stdout:6/342: chown d0/d3/d22/c65 222 1 2024-03-22T02:28:39.599 INFO:tasks.workunit.client.0.smithi103.stdout:2/419: symlink d11/d25/d46/d40/l89 0 2024-03-22T02:28:39.601 INFO:tasks.workunit.client.0.smithi103.stdout:3/445: creat de/d6b/d80/d4f/f91 x:0 0 0 2024-03-22T02:28:39.601 INFO:tasks.workunit.client.0.smithi103.stdout:8/513: rmdir d4/d26/d27/d7e/d39/d49 39 2024-03-22T02:28:39.602 INFO:tasks.workunit.client.0.smithi103.stdout:3/446: chown de/d6b/d80/d4f/f91 5496 1 2024-03-22T02:28:39.605 INFO:tasks.workunit.client.0.smithi103.stdout:9/359: mknod d2/d10/d43/d4b/c68 0 2024-03-22T02:28:39.606 INFO:tasks.workunit.client.0.smithi103.stdout:9/360: chown d2/f2c 7215389 1 2024-03-22T02:28:39.608 INFO:tasks.workunit.client.0.smithi103.stdout:9/361: write d2/da/d18/d1e/d34/d3d/d4d/f58 [400406,126758] 0 2024-03-22T02:28:39.608 INFO:tasks.workunit.client.0.smithi103.stdout:2/420: unlink d11/d25/d46/f70 0 2024-03-22T02:28:39.609 INFO:tasks.workunit.client.0.smithi103.stdout:3/447: mkdir de/d12/d5d/d92 0 2024-03-22T02:28:39.610 INFO:tasks.workunit.client.0.smithi103.stdout:2/421: dread - d11/d19/d5d/f68 zero size 2024-03-22T02:28:39.611 INFO:tasks.workunit.client.0.smithi103.stdout:4/423: dread d3/d69/f3b [0,4194304] 0 2024-03-22T02:28:39.613 INFO:tasks.workunit.client.0.smithi103.stdout:4/424: write d3/d1e/d53/f56 [337542,107040] 0 2024-03-22T02:28:39.614 INFO:tasks.workunit.client.0.smithi103.stdout:0/517: creat dd/d34/fad x:0 0 0 2024-03-22T02:28:39.615 INFO:tasks.workunit.client.0.smithi103.stdout:6/343: creat d0/d3/f6d x:0 0 0 2024-03-22T02:28:39.619 INFO:tasks.workunit.client.0.smithi103.stdout:6/344: truncate d0/f64 108707 0 2024-03-22T02:28:39.619 INFO:tasks.workunit.client.0.smithi103.stdout:2/422: creat d11/d25/d46/d40/f8a x:0 0 0 2024-03-22T02:28:39.620 INFO:tasks.workunit.client.0.smithi103.stdout:3/448: mkdir de/d12/d8a/d93 0 2024-03-22T02:28:39.621 INFO:tasks.workunit.client.0.smithi103.stdout:2/423: stat d11/f13 0 2024-03-22T02:28:39.621 INFO:tasks.workunit.client.0.smithi103.stdout:9/362: dwrite d2/da/d18/d1e/d34/d3d/d4d/f58 [4194304,4194304] 0 2024-03-22T02:28:39.622 INFO:tasks.workunit.client.0.smithi103.stdout:3/449: chown de/c36 5 1 2024-03-22T02:28:39.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:39 smithi103 ceph-mon[29258]: pgmap v13: 65 pgs: 65 active+clean; 735 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 11 MiB/s rd, 84 MiB/s wr, 348 op/s 2024-03-22T02:28:39.624 INFO:tasks.workunit.client.0.smithi103.stdout:2/424: chown d11/d25/d46/d48/f6d 3249 1 2024-03-22T02:28:39.627 INFO:tasks.workunit.client.0.smithi103.stdout:4/425: rename d3/f51 to d3/d1e/d25/d65/f99 0 2024-03-22T02:28:39.629 INFO:tasks.workunit.client.0.smithi103.stdout:8/514: getdents d4/d26/d27/d7e/d67 0 2024-03-22T02:28:39.630 INFO:tasks.workunit.client.0.smithi103.stdout:9/363: mknod d2/d5/c69 0 2024-03-22T02:28:39.631 INFO:tasks.workunit.client.0.smithi103.stdout:0/518: dwrite dd/d16/d4b/d96/d29/f83 [0,4194304] 0 2024-03-22T02:28:39.635 INFO:tasks.workunit.client.0.smithi103.stdout:3/450: unlink de/d6b/d80/c7d 0 2024-03-22T02:28:39.635 INFO:tasks.workunit.client.0.smithi103.stdout:9/364: chown d2/da/d18/c53 0 1 2024-03-22T02:28:39.635 INFO:tasks.workunit.client.0.smithi103.stdout:6/345: fdatasync d0/d14/f44 0 2024-03-22T02:28:39.637 INFO:tasks.workunit.client.0.smithi103.stdout:9/365: write d2/f2c [8841693,88893] 0 2024-03-22T02:28:39.637 INFO:tasks.workunit.client.0.smithi103.stdout:8/515: symlink d4/d48/d35/l98 0 2024-03-22T02:28:39.640 INFO:tasks.workunit.client.0.smithi103.stdout:0/519: chown dd/d16/l54 47 1 2024-03-22T02:28:39.640 INFO:tasks.workunit.client.0.smithi103.stdout:9/366: read d2/d10/d27/f4f [1771026,93560] 0 2024-03-22T02:28:39.643 INFO:tasks.workunit.client.0.smithi103.stdout:3/451: dwrite de/d1a/d25/f8b [0,4194304] 0 2024-03-22T02:28:39.646 INFO:tasks.workunit.client.0.smithi103.stdout:0/520: read f4 [2179163,75290] 0 2024-03-22T02:28:39.697 INFO:tasks.workunit.client.0.smithi103.stdout:8/516: unlink d4/d26/f4e 0 2024-03-22T02:28:39.698 INFO:tasks.workunit.client.0.smithi103.stdout:8/517: write d4/d26/d27/f93 [561582,25577] 0 2024-03-22T02:28:39.700 INFO:tasks.workunit.client.0.smithi103.stdout:9/367: creat d2/da/d18/d1e/d34/d3d/d46/f6a x:0 0 0 2024-03-22T02:28:39.702 INFO:tasks.workunit.client.0.smithi103.stdout:3/452: unlink de/d6b/d80/d4f/f91 0 2024-03-22T02:28:39.705 INFO:tasks.workunit.client.0.smithi103.stdout:2/425: creat d11/d25/d63/d45/f8b x:0 0 0 2024-03-22T02:28:39.715 INFO:tasks.workunit.client.0.smithi103.stdout:3/453: symlink de/d12/d30/l94 0 2024-03-22T02:28:39.716 INFO:tasks.workunit.client.0.smithi103.stdout:0/521: creat dd/d34/d53/d8c/fae x:0 0 0 2024-03-22T02:28:39.716 INFO:tasks.workunit.client.0.smithi103.stdout:4/426: getdents d3/d1e/d78 0 2024-03-22T02:28:39.717 INFO:tasks.workunit.client.0.smithi103.stdout:6/346: dwrite d0/d3/f1b [0,4194304] 0 2024-03-22T02:28:39.717 INFO:tasks.workunit.client.0.smithi103.stdout:9/368: fdatasync d2/f3b 0 2024-03-22T02:28:39.718 INFO:tasks.workunit.client.0.smithi103.stdout:3/454: mknod de/d12/d5d/c95 0 2024-03-22T02:28:39.722 INFO:tasks.workunit.client.0.smithi103.stdout:0/522: unlink dd/d16/f55 0 2024-03-22T02:28:39.722 INFO:tasks.workunit.client.0.smithi103.stdout:0/523: chown dd/l75 1 1 2024-03-22T02:28:39.722 INFO:tasks.workunit.client.0.smithi103.stdout:8/518: link d4/d48/d40/l65 d4/d2a/d8b/l99 0 2024-03-22T02:28:39.725 INFO:tasks.workunit.client.0.smithi103.stdout:6/347: mknod d0/d14/d57/d68/c6e 0 2024-03-22T02:28:39.725 INFO:tasks.workunit.client.0.smithi103.stdout:6/348: chown d0/d3/l12 35228946 1 2024-03-22T02:28:39.726 INFO:tasks.workunit.client.0.smithi103.stdout:9/369: write d2/d10/d59/f5e [4386618,76746] 0 2024-03-22T02:28:39.728 INFO:tasks.workunit.client.0.smithi103.stdout:8/519: rmdir d4/d26 39 2024-03-22T02:28:39.730 INFO:tasks.workunit.client.0.smithi103.stdout:4/427: rename d3/d30/f8f to d3/d1e/d53/d8a/f9a 0 2024-03-22T02:28:39.732 INFO:tasks.workunit.client.0.smithi103.stdout:9/370: creat d2/da/f6b x:0 0 0 2024-03-22T02:28:39.733 INFO:tasks.workunit.client.0.smithi103.stdout:3/455: creat de/f96 x:0 0 0 2024-03-22T02:28:39.734 INFO:tasks.workunit.client.0.smithi103.stdout:6/349: mkdir d0/d3/d22/d58/d6f 0 2024-03-22T02:28:39.737 INFO:tasks.workunit.client.0.smithi103.stdout:0/524: rename dd/d16/d4b/d96/d29/d48/d94/c97 to dd/caf 0 2024-03-22T02:28:39.738 INFO:tasks.workunit.client.0.smithi103.stdout:4/428: creat d3/f9b x:0 0 0 2024-03-22T02:28:39.739 INFO:tasks.workunit.client.0.smithi103.stdout:4/429: fsync d3/d30/f49 0 2024-03-22T02:28:39.739 INFO:tasks.workunit.client.0.smithi103.stdout:4/430: chown d3/d38/d5c/d86 27857715 1 2024-03-22T02:28:39.740 INFO:tasks.workunit.client.0.smithi103.stdout:9/371: creat d2/da/d18/f6c x:0 0 0 2024-03-22T02:28:39.741 INFO:tasks.workunit.client.0.smithi103.stdout:4/431: truncate d3/d1e/d25/f3d 5045566 0 2024-03-22T02:28:39.741 INFO:tasks.workunit.client.0.smithi103.stdout:8/520: creat d4/d26/d27/d7e/d39/f9a x:0 0 0 2024-03-22T02:28:39.742 INFO:tasks.workunit.client.0.smithi103.stdout:4/432: fsync d3/d30/d45/f58 0 2024-03-22T02:28:39.743 INFO:tasks.workunit.client.0.smithi103.stdout:9/372: write d2/d5/d4c/f5c [574910,92359] 0 2024-03-22T02:28:39.743 INFO:tasks.workunit.client.0.smithi103.stdout:4/433: write d3/d1e/d25/d65/f96 [401863,60016] 0 2024-03-22T02:28:39.744 INFO:tasks.workunit.client.0.smithi103.stdout:3/456: getdents de/d12/d4d 0 2024-03-22T02:28:39.746 INFO:tasks.workunit.client.0.smithi103.stdout:0/525: getdents dd/d34 0 2024-03-22T02:28:39.746 INFO:tasks.workunit.client.0.smithi103.stdout:9/373: write d2/da/d18/d1e/d34/d3d/d46/f6a [942783,27184] 0 2024-03-22T02:28:39.747 INFO:tasks.workunit.client.0.smithi103.stdout:4/434: creat d3/d1e/d53/d8a/f9c x:0 0 0 2024-03-22T02:28:39.747 INFO:tasks.workunit.client.0.smithi103.stdout:9/374: fsync d2/d10/f13 0 2024-03-22T02:28:39.749 INFO:tasks.workunit.client.0.smithi103.stdout:4/435: chown d3/d38/f4c 811459 1 2024-03-22T02:28:39.750 INFO:tasks.workunit.client.0.smithi103.stdout:3/457: mkdir de/d12/d97 0 2024-03-22T02:28:39.752 INFO:tasks.workunit.client.0.smithi103.stdout:8/521: creat d4/d19/f9b x:0 0 0 2024-03-22T02:28:39.752 INFO:tasks.workunit.client.0.smithi103.stdout:4/436: dread d3/d69/f54 [0,4194304] 0 2024-03-22T02:28:39.753 INFO:tasks.workunit.client.0.smithi103.stdout:9/375: creat d2/d10/d59/f6d x:0 0 0 2024-03-22T02:28:39.754 INFO:tasks.workunit.client.0.smithi103.stdout:0/526: symlink dd/d16/d4b/d96/d64/dab/lb0 0 2024-03-22T02:28:39.754 INFO:tasks.workunit.client.0.smithi103.stdout:9/376: write d2/d10/d17/f1f [3542707,67323] 0 2024-03-22T02:28:39.756 INFO:tasks.workunit.client.0.smithi103.stdout:4/437: fdatasync d3/d31/f32 0 2024-03-22T02:28:39.757 INFO:tasks.workunit.client.0.smithi103.stdout:0/527: read dd/d16/d4b/d96/d29/d48/f73 [2426611,91257] 0 2024-03-22T02:28:39.758 INFO:tasks.workunit.client.0.smithi103.stdout:4/438: dread - d3/d1e/d25/d65/f6c zero size 2024-03-22T02:28:39.759 INFO:tasks.workunit.client.0.smithi103.stdout:9/377: rename d2/d10/d27/c31 to d2/d10/d59/c6e 0 2024-03-22T02:28:39.761 INFO:tasks.workunit.client.0.smithi103.stdout:9/378: read d2/f3b [2229,23699] 0 2024-03-22T02:28:39.761 INFO:tasks.workunit.client.0.smithi103.stdout:0/528: creat dd/d34/d53/fb1 x:0 0 0 2024-03-22T02:28:39.762 INFO:tasks.workunit.client.0.smithi103.stdout:4/439: mknod d3/d1e/d25/d65/c9d 0 2024-03-22T02:28:39.763 INFO:tasks.workunit.client.0.smithi103.stdout:4/440: readlink d3/d31/l7b 0 2024-03-22T02:28:39.764 INFO:tasks.workunit.client.0.smithi103.stdout:4/441: truncate d3/f8 5110600 0 2024-03-22T02:28:39.765 INFO:tasks.workunit.client.0.smithi103.stdout:9/379: dread d2/d10/d43/f3e [0,4194304] 0 2024-03-22T02:28:39.768 INFO:tasks.workunit.client.0.smithi103.stdout:8/522: dread d4/fe [0,4194304] 0 2024-03-22T02:28:39.769 INFO:tasks.workunit.client.0.smithi103.stdout:9/380: mkdir d2/d5/d4c/d56/d6f 0 2024-03-22T02:28:39.769 INFO:tasks.workunit.client.0.smithi103.stdout:4/442: getdents d3/d30/d45/d6e 0 2024-03-22T02:28:39.771 INFO:tasks.workunit.client.0.smithi103.stdout:4/443: write d3/d1e/d25/d65/f82 [650549,82421] 0 2024-03-22T02:28:39.771 INFO:tasks.workunit.client.0.smithi103.stdout:8/523: creat d4/d2a/d38/f9c x:0 0 0 2024-03-22T02:28:39.773 INFO:tasks.workunit.client.0.smithi103.stdout:8/524: dread - d4/d19/f9b zero size 2024-03-22T02:28:39.774 INFO:tasks.workunit.client.0.smithi103.stdout:4/444: creat d3/d69/f9e x:0 0 0 2024-03-22T02:28:39.775 INFO:tasks.workunit.client.0.smithi103.stdout:9/381: getdents d2/d10 0 2024-03-22T02:28:39.776 INFO:tasks.workunit.client.0.smithi103.stdout:8/525: mknod d4/d19/d8c/c9d 0 2024-03-22T02:28:39.778 INFO:tasks.workunit.client.0.smithi103.stdout:0/529: dread dd/f24 [0,4194304] 0 2024-03-22T02:28:39.778 INFO:tasks.workunit.client.0.smithi103.stdout:9/382: mknod d2/da/d18/d1e/d34/d3d/d46/c70 0 2024-03-22T02:28:39.780 INFO:tasks.workunit.client.0.smithi103.stdout:0/530: dread - dd/d16/d4b/d96/d31/f8e zero size 2024-03-22T02:28:39.781 INFO:tasks.workunit.client.0.smithi103.stdout:9/383: stat d2/d10/d43/c62 0 2024-03-22T02:28:39.782 INFO:tasks.workunit.client.0.smithi103.stdout:4/445: dwrite d3/d1e/d25/d65/f82 [0,4194304] 0 2024-03-22T02:28:39.782 INFO:tasks.workunit.client.0.smithi103.stdout:9/384: fsync d2/d5/f63 0 2024-03-22T02:28:39.782 INFO:tasks.workunit.client.0.smithi103.stdout:0/531: chown dd/d34/d53/d8c/fa0 3494894 1 2024-03-22T02:28:39.784 INFO:tasks.workunit.client.0.smithi103.stdout:0/532: getdents dd/d16/d4b/d96/d31/d89 0 2024-03-22T02:28:39.787 INFO:tasks.workunit.client.0.smithi103.stdout:0/533: mknod dd/d16/d4b/d96/cb2 0 2024-03-22T02:28:39.788 INFO:tasks.workunit.client.0.smithi103.stdout:9/385: dwrite d2/d5/f63 [0,4194304] 0 2024-03-22T02:28:39.799 INFO:tasks.workunit.client.0.smithi103.stdout:0/534: dread dd/d16/f36 [0,4194304] 0 2024-03-22T02:28:39.801 INFO:tasks.workunit.client.0.smithi103.stdout:0/535: symlink dd/d16/d4b/lb3 0 2024-03-22T02:28:39.802 INFO:tasks.workunit.client.0.smithi103.stdout:0/536: write dd/d16/d4b/f50 [1046285,65709] 0 2024-03-22T02:28:39.803 INFO:tasks.workunit.client.0.smithi103.stdout:0/537: read dd/d16/d4b/d96/d29/f85 [3363784,75833] 0 2024-03-22T02:28:39.805 INFO:tasks.workunit.client.0.smithi103.stdout:0/538: unlink dd/d16/d4b/d96/d29/l6c 0 2024-03-22T02:28:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:39 smithi122 ceph-mon[36403]: pgmap v13: 65 pgs: 65 active+clean; 735 MiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 11 MiB/s rd, 84 MiB/s wr, 348 op/s 2024-03-22T02:28:39.884 INFO:tasks.workunit.client.0.smithi103.stdout:3/458: truncate de/d1a/d25/d26/f53 1906218 0 2024-03-22T02:28:39.884 INFO:tasks.workunit.client.0.smithi103.stdout:9/386: getdents d2/d10/d59 0 2024-03-22T02:28:39.888 INFO:tasks.workunit.client.0.smithi103.stdout:9/387: rename d2/d10/d17/f5a to d2/da/d18/d1e/d34/d3d/f71 0 2024-03-22T02:28:39.890 INFO:tasks.workunit.client.0.smithi103.stdout:9/388: dread - d2/da/d18/d1e/d34/d3d/d46/f49 zero size 2024-03-22T02:28:39.895 INFO:tasks.workunit.client.0.smithi103.stdout:8/526: dread d4/d2a/f6c [0,4194304] 0 2024-03-22T02:28:39.900 INFO:tasks.workunit.client.0.smithi103.stdout:9/389: unlink d2/da/d18/d1e/d34/d3d/f71 0 2024-03-22T02:28:39.902 INFO:tasks.workunit.client.0.smithi103.stdout:0/539: dwrite dd/d16/d4b/d96/d31/f8e [0,4194304] 0 2024-03-22T02:28:39.906 INFO:tasks.workunit.client.0.smithi103.stdout:3/459: dwrite de/f23 [0,4194304] 0 2024-03-22T02:28:39.907 INFO:tasks.workunit.client.0.smithi103.stdout:0/540: write dd/d16/d4b/d96/d29/f4d [1980855,85221] 0 2024-03-22T02:28:39.914 INFO:tasks.workunit.client.0.smithi103.stdout:3/460: dread de/d1a/d25/f6a [0,4194304] 0 2024-03-22T02:28:39.918 INFO:tasks.workunit.client.0.smithi103.stdout:0/541: dwrite dd/d16/d4b/d96/d31/d79/f93 [0,4194304] 0 2024-03-22T02:28:39.918 INFO:tasks.workunit.client.0.smithi103.stdout:3/461: readlink de/d6b/d80/d4f/l55 0 2024-03-22T02:28:39.918 INFO:tasks.workunit.client.0.smithi103.stdout:8/527: dread d4/f34 [0,4194304] 0 2024-03-22T02:28:39.919 INFO:tasks.workunit.client.0.smithi103.stdout:0/542: write dd/d16/d4b/d96/f39 [1314466,77139] 0 2024-03-22T02:28:39.919 INFO:tasks.workunit.client.0.smithi103.stdout:9/390: write d2/da/f23 [1284612,80203] 0 2024-03-22T02:28:39.922 INFO:tasks.workunit.client.0.smithi103.stdout:8/528: chown d4/fd 309048 1 2024-03-22T02:28:39.923 INFO:tasks.workunit.client.0.smithi103.stdout:9/391: creat d2/d10/d17/f72 x:0 0 0 2024-03-22T02:28:39.925 INFO:tasks.workunit.client.0.smithi103.stdout:8/529: creat d4/d19/d1a/d4a/f9e x:0 0 0 2024-03-22T02:28:39.926 INFO:tasks.workunit.client.0.smithi103.stdout:8/530: stat d4/d26/d27/f2c 0 2024-03-22T02:28:39.926 INFO:tasks.workunit.client.0.smithi103.stdout:9/392: mkdir d2/d5/d4c/d56/d73 0 2024-03-22T02:28:39.927 INFO:tasks.workunit.client.0.smithi103.stdout:8/531: write d4/d2a/d3e/f4b [2200465,15914] 0 2024-03-22T02:28:39.927 INFO:tasks.workunit.client.0.smithi103.stdout:0/543: creat dd/d16/d4b/d96/d31/fb4 x:0 0 0 2024-03-22T02:28:39.928 INFO:tasks.workunit.client.0.smithi103.stdout:0/544: dread - dd/d34/d53/fa1 zero size 2024-03-22T02:28:39.931 INFO:tasks.workunit.client.0.smithi103.stdout:8/532: truncate d4/d26/f8e 7863667 0 2024-03-22T02:28:39.934 INFO:tasks.workunit.client.0.smithi103.stdout:0/545: rename dd/f13 to dd/d16/d4b/d96/d29/fb5 0 2024-03-22T02:28:39.936 INFO:tasks.workunit.client.0.smithi103.stdout:8/533: creat d4/d2a/d8b/f9f x:0 0 0 2024-03-22T02:28:39.936 INFO:tasks.workunit.client.0.smithi103.stdout:9/393: dread d2/da/f23 [0,4194304] 0 2024-03-22T02:28:39.939 INFO:tasks.workunit.client.0.smithi103.stdout:8/534: creat d4/d48/d35/fa0 x:0 0 0 2024-03-22T02:28:39.940 INFO:tasks.workunit.client.0.smithi103.stdout:8/535: stat c0 0 2024-03-22T02:28:39.940 INFO:tasks.workunit.client.0.smithi103.stdout:9/394: unlink d2/d10/d17/c2f 0 2024-03-22T02:28:39.945 INFO:tasks.workunit.client.0.smithi103.stdout:8/536: dwrite f1 [4194304,4194304] 0 2024-03-22T02:28:39.947 INFO:tasks.workunit.client.0.smithi103.stdout:8/537: rename d4/d48 to d4/d48/d40/da1 22 2024-03-22T02:28:39.949 INFO:tasks.workunit.client.0.smithi103.stdout:9/395: dwrite d2/d5/f38 [0,4194304] 0 2024-03-22T02:28:39.955 INFO:tasks.workunit.client.0.smithi103.stdout:9/396: symlink d2/da/d18/d1e/d34/l74 0 2024-03-22T02:28:39.957 INFO:tasks.workunit.client.0.smithi103.stdout:8/538: rmdir d4/d26/d27/d7e/d39/d49/d51 39 2024-03-22T02:28:39.958 INFO:tasks.workunit.client.0.smithi103.stdout:3/462: dread de/f21 [0,4194304] 0 2024-03-22T02:28:39.961 INFO:tasks.workunit.client.0.smithi103.stdout:8/539: truncate d4/d48/d35/f44 466377 0 2024-03-22T02:28:39.963 INFO:tasks.workunit.client.0.smithi103.stdout:8/540: chown d4/d2a/f7d 3251602 1 2024-03-22T02:28:39.966 INFO:tasks.workunit.client.0.smithi103.stdout:8/541: creat d4/d26/d27/fa2 x:0 0 0 2024-03-22T02:28:39.966 INFO:tasks.workunit.client.0.smithi103.stdout:3/463: dwrite de/d1a/d25/d26/f69 [0,4194304] 0 2024-03-22T02:28:39.969 INFO:tasks.workunit.client.0.smithi103.stdout:3/464: mknod de/d12/d8f/c98 0 2024-03-22T02:28:39.974 INFO:tasks.workunit.client.0.smithi103.stdout:9/397: fsync d2/d5/f38 0 2024-03-22T02:28:39.974 INFO:tasks.workunit.client.0.smithi103.stdout:8/542: dwrite d4/d48/d40/d68/f95 [0,4194304] 0 2024-03-22T02:28:39.976 INFO:tasks.workunit.client.0.smithi103.stdout:3/465: link de/d1a/d25/d26/f54 de/d12/d5d/f99 0 2024-03-22T02:28:39.978 INFO:tasks.workunit.client.0.smithi103.stdout:9/398: mknod d2/d10/c75 0 2024-03-22T02:28:39.980 INFO:tasks.workunit.client.0.smithi103.stdout:9/399: read d2/f2c [5357790,45591] 0 2024-03-22T02:28:39.981 INFO:tasks.workunit.client.0.smithi103.stdout:8/543: rename d4/d26/d27/d7e/d39/d8d to d4/d26/d27/da3 0 2024-03-22T02:28:39.984 INFO:tasks.workunit.client.0.smithi103.stdout:0/546: dread dd/d16/d4b/d96/d64/d6a/d8d/f95 [0,4194304] 0 2024-03-22T02:28:39.987 INFO:tasks.workunit.client.0.smithi103.stdout:3/466: dwrite de/d6b/d80/d4f/d84/f88 [0,4194304] 0 2024-03-22T02:28:39.987 INFO:tasks.workunit.client.0.smithi103.stdout:8/544: mkdir d4/d2a/d3e/da4 0 2024-03-22T02:28:39.989 INFO:tasks.workunit.client.0.smithi103.stdout:9/400: rename d2/l48 to d2/d5/d4c/d56/d6f/l76 0 2024-03-22T02:28:39.993 INFO:tasks.workunit.client.0.smithi103.stdout:8/545: rmdir d4/d26/d27/d7e 39 2024-03-22T02:28:39.994 INFO:tasks.workunit.client.0.smithi103.stdout:9/401: mkdir d2/da/d18/d1e/d77 0 2024-03-22T02:28:39.997 INFO:tasks.workunit.client.0.smithi103.stdout:9/402: fsync d2/da/d18/f41 0 2024-03-22T02:28:39.999 INFO:tasks.workunit.client.0.smithi103.stdout:3/467: link de/f1f de/d12/f9a 0 2024-03-22T02:28:40.000 INFO:tasks.workunit.client.0.smithi103.stdout:8/546: creat d4/d2a/d3e/da4/fa5 x:0 0 0 2024-03-22T02:28:40.002 INFO:tasks.workunit.client.0.smithi103.stdout:3/468: mknod de/d1a/d25/d3f/c9b 0 2024-03-22T02:28:40.003 INFO:tasks.workunit.client.0.smithi103.stdout:9/403: getdents d2/d5/d4c 0 2024-03-22T02:28:40.005 INFO:tasks.workunit.client.0.smithi103.stdout:3/469: mkdir de/d1a/d25/d3f/d9c 0 2024-03-22T02:28:40.006 INFO:tasks.workunit.client.0.smithi103.stdout:0/547: dwrite f4 [0,4194304] 0 2024-03-22T02:28:40.008 INFO:tasks.workunit.client.0.smithi103.stdout:9/404: mknod d2/da/d18/d1e/d34/d3d/c78 0 2024-03-22T02:28:40.008 INFO:tasks.workunit.client.0.smithi103.stdout:3/470: creat de/d12/d19/f9d x:0 0 0 2024-03-22T02:28:40.009 INFO:tasks.workunit.client.0.smithi103.stdout:9/405: dread d2/f3b [0,4194304] 0 2024-03-22T02:28:40.011 INFO:tasks.workunit.client.0.smithi103.stdout:3/471: creat de/d12/d19/f9e x:0 0 0 2024-03-22T02:28:40.013 INFO:tasks.workunit.client.0.smithi103.stdout:3/472: read - de/d6b/d80/f7c zero size 2024-03-22T02:28:40.014 INFO:tasks.workunit.client.0.smithi103.stdout:0/548: dwrite dd/d16/d4b/d96/d31/fb4 [0,4194304] 0 2024-03-22T02:28:40.015 INFO:tasks.workunit.client.0.smithi103.stdout:9/406: read d2/f2c [3259917,102000] 0 2024-03-22T02:28:40.017 INFO:tasks.workunit.client.0.smithi103.stdout:9/407: write d2/da/f25 [470223,43494] 0 2024-03-22T02:28:40.020 INFO:tasks.workunit.client.0.smithi103.stdout:3/473: symlink de/d12/d97/l9f 0 2024-03-22T02:28:40.025 INFO:tasks.workunit.client.0.smithi103.stdout:3/474: creat de/d12/d30/fa0 x:0 0 0 2024-03-22T02:28:40.026 INFO:tasks.workunit.client.0.smithi103.stdout:9/408: mknod d2/da/d18/d1e/d34/d3d/d61/c79 0 2024-03-22T02:28:40.026 INFO:tasks.workunit.client.0.smithi103.stdout:3/475: read - de/d6b/d80/f7c zero size 2024-03-22T02:28:40.027 INFO:tasks.workunit.client.0.smithi103.stdout:0/549: link dd/d16/l54 dd/d16/d4b/d96/d64/dab/lb6 0 2024-03-22T02:28:40.028 INFO:tasks.workunit.client.0.smithi103.stdout:9/409: dread d2/d10/d43/f3e [0,4194304] 0 2024-03-22T02:28:40.029 INFO:tasks.workunit.client.0.smithi103.stdout:3/476: symlink de/d1a/d25/d3f/la1 0 2024-03-22T02:28:40.030 INFO:tasks.workunit.client.0.smithi103.stdout:3/477: dread - de/d12/d5d/f75 zero size 2024-03-22T02:28:40.030 INFO:tasks.workunit.client.0.smithi103.stdout:0/550: symlink dd/d16/d4b/d96/d64/d91/d9a/d9d/lb7 0 2024-03-22T02:28:40.034 INFO:tasks.workunit.client.0.smithi103.stdout:9/410: dread d2/d10/d27/f51 [0,4194304] 0 2024-03-22T02:28:40.036 INFO:tasks.workunit.client.0.smithi103.stdout:9/411: creat d2/d10/d43/f7a x:0 0 0 2024-03-22T02:28:40.038 INFO:tasks.workunit.client.0.smithi103.stdout:8/547: dread d4/f6e [0,4194304] 0 2024-03-22T02:28:40.047 INFO:tasks.workunit.client.0.smithi103.stdout:8/548: dwrite d4/d26/f81 [0,4194304] 0 2024-03-22T02:28:40.049 INFO:tasks.workunit.client.0.smithi103.stdout:3/478: dread de/f1f [0,4194304] 0 2024-03-22T02:28:40.050 INFO:tasks.workunit.client.0.smithi103.stdout:8/549: mkdir d4/d19/d1a/da6 0 2024-03-22T02:28:40.050 INFO:tasks.workunit.client.0.smithi103.stdout:0/551: dread dd/d16/d4b/d96/d29/f3e [0,4194304] 0 2024-03-22T02:28:40.051 INFO:tasks.workunit.client.0.smithi103.stdout:9/412: dread d2/d10/d43/f1b [0,4194304] 0 2024-03-22T02:28:40.052 INFO:tasks.workunit.client.0.smithi103.stdout:9/413: readlink d2/da/d18/d1e/d34/l5f 0 2024-03-22T02:28:40.052 INFO:tasks.workunit.client.0.smithi103.stdout:0/552: readlink dd/d16/d4b/d96/d31/l41 0 2024-03-22T02:28:40.052 INFO:tasks.workunit.client.0.smithi103.stdout:3/479: dread de/d1a/d25/f6a [0,4194304] 0 2024-03-22T02:28:40.052 INFO:tasks.workunit.client.0.smithi103.stdout:8/550: symlink d4/d19/d1a/d4a/la7 0 2024-03-22T02:28:40.054 INFO:tasks.workunit.client.0.smithi103.stdout:8/551: write d4/d26/f7c [3639607,21899] 0 2024-03-22T02:28:40.054 INFO:tasks.workunit.client.0.smithi103.stdout:0/553: chown dd/d16/d4b/d96/d31/l3d 7063611 1 2024-03-22T02:28:40.055 INFO:tasks.workunit.client.0.smithi103.stdout:9/414: unlink d2/d10/d17/f1f 0 2024-03-22T02:28:40.057 INFO:tasks.workunit.client.0.smithi103.stdout:3/480: mknod de/d12/d19/ca2 0 2024-03-22T02:28:40.059 INFO:tasks.workunit.client.0.smithi103.stdout:0/554: fsync dd/d16/d4b/d96/d29/f3e 0 2024-03-22T02:28:40.059 INFO:tasks.workunit.client.0.smithi103.stdout:8/552: write d4/d26/d27/d7e/f87 [198460,43270] 0 2024-03-22T02:28:40.060 INFO:tasks.workunit.client.0.smithi103.stdout:9/415: mkdir d2/d5/d7b 0 2024-03-22T02:28:40.062 INFO:tasks.workunit.client.0.smithi103.stdout:8/553: write d4/d26/d27/d7e/d39/d49/f74 [1304502,73057] 0 2024-03-22T02:28:40.063 INFO:tasks.workunit.client.0.smithi103.stdout:9/416: unlink d2/d10/d17/c4a 0 2024-03-22T02:28:40.065 INFO:tasks.workunit.client.0.smithi103.stdout:0/555: rename dd/d16/d4b/d96/d35/l3b to dd/d34/d53/d8c/lb8 0 2024-03-22T02:28:40.066 INFO:tasks.workunit.client.0.smithi103.stdout:0/556: dread - dd/d34/d53/d8c/fae zero size 2024-03-22T02:28:40.069 INFO:tasks.workunit.client.0.smithi103.stdout:8/554: creat d4/d26/fa8 x:0 0 0 2024-03-22T02:28:40.070 INFO:tasks.workunit.client.0.smithi103.stdout:8/555: read - d4/d19/d1a/f91 zero size 2024-03-22T02:28:40.071 INFO:tasks.workunit.client.0.smithi103.stdout:0/557: symlink dd/d16/d4b/d96/d29/d48/lb9 0 2024-03-22T02:28:40.072 INFO:tasks.workunit.client.0.smithi103.stdout:9/417: rmdir d2/d5/d4c/d56/d73 0 2024-03-22T02:28:40.073 INFO:tasks.workunit.client.0.smithi103.stdout:9/418: fsync d2/d5/d4c/f5c 0 2024-03-22T02:28:40.073 INFO:tasks.workunit.client.0.smithi103.stdout:8/556: creat d4/d26/d27/d7e/fa9 x:0 0 0 2024-03-22T02:28:40.075 INFO:tasks.workunit.client.0.smithi103.stdout:8/557: read d4/fe [1866927,109112] 0 2024-03-22T02:28:40.076 INFO:tasks.workunit.client.0.smithi103.stdout:9/419: creat d2/d5/d4c/d56/d6f/f7c x:0 0 0 2024-03-22T02:28:40.078 INFO:tasks.workunit.client.0.smithi103.stdout:0/558: mknod dd/d16/cba 0 2024-03-22T02:28:40.080 INFO:tasks.workunit.client.0.smithi103.stdout:8/558: symlink d4/d26/d27/d7e/d39/d49/laa 0 2024-03-22T02:28:40.080 INFO:tasks.workunit.client.0.smithi103.stdout:3/481: dread de/d12/d5d/f99 [0,4194304] 0 2024-03-22T02:28:40.082 INFO:tasks.workunit.client.0.smithi103.stdout:0/559: symlink dd/d16/d4b/d96/d29/lbb 0 2024-03-22T02:28:40.083 INFO:tasks.workunit.client.0.smithi103.stdout:9/420: getdents d2/d5 0 2024-03-22T02:28:40.083 INFO:tasks.workunit.client.0.smithi103.stdout:0/560: fsync dd/d16/d4b/d96/d35/d82/f7b 0 2024-03-22T02:28:40.086 INFO:tasks.workunit.client.0.smithi103.stdout:8/559: rename d4/d48/d40/d68/l80 to d4/d19/d1a/lab 0 2024-03-22T02:28:40.088 INFO:tasks.workunit.client.0.smithi103.stdout:9/421: mkdir d2/d5/d7b/d7d 0 2024-03-22T02:28:40.091 INFO:tasks.workunit.client.0.smithi103.stdout:8/560: creat d4/d2a/d4f/fac x:0 0 0 2024-03-22T02:28:40.093 INFO:tasks.workunit.client.0.smithi103.stdout:8/561: creat d4/d26/d27/d7e/fad x:0 0 0 2024-03-22T02:28:40.095 INFO:tasks.workunit.client.0.smithi103.stdout:3/482: truncate de/f1e 2685384 0 2024-03-22T02:28:40.095 INFO:tasks.workunit.client.0.smithi103.stdout:8/562: symlink d4/d19/d1a/lae 0 2024-03-22T02:28:40.096 INFO:tasks.workunit.client.0.smithi103.stdout:9/422: getdents d2 0 2024-03-22T02:28:40.098 INFO:tasks.workunit.client.0.smithi103.stdout:8/563: unlink d4/ff 0 2024-03-22T02:28:40.099 INFO:tasks.workunit.client.0.smithi103.stdout:9/423: truncate d2/f2c 7917765 0 2024-03-22T02:28:40.100 INFO:tasks.workunit.client.0.smithi103.stdout:9/424: chown d2/d10/l16 5281747 1 2024-03-22T02:28:40.101 INFO:tasks.workunit.client.0.smithi103.stdout:3/483: link de/d6b/d80/d4f/d84/f88 de/d6b/d80/d4f/fa3 0 2024-03-22T02:28:40.102 INFO:tasks.workunit.client.0.smithi103.stdout:3/484: chown de/d1a/d25/d26/f54 1089242 1 2024-03-22T02:28:40.103 INFO:tasks.workunit.client.0.smithi103.stdout:9/425: mkdir d2/d10/d43/d7e 0 2024-03-22T02:28:40.103 INFO:tasks.workunit.client.0.smithi103.stdout:8/564: mkdir d4/d26/d27/d7e/d39/d49/d51/daf 0 2024-03-22T02:28:40.103 INFO:tasks.workunit.client.0.smithi103.stdout:9/426: chown d2/d5/d4c/f5c 48030059 1 2024-03-22T02:28:40.109 INFO:tasks.workunit.client.0.smithi103.stdout:8/565: rename d4/d2a/d3e/f4b to d4/d26/d27/fb0 0 2024-03-22T02:28:40.109 INFO:tasks.workunit.client.0.smithi103.stdout:3/485: dwrite de/d12/d30/f63 [0,4194304] 0 2024-03-22T02:28:40.110 INFO:tasks.workunit.client.0.smithi103.stdout:3/486: chown de 1 1 2024-03-22T02:28:40.111 INFO:tasks.workunit.client.0.smithi103.stdout:8/566: truncate d4/d19/d8c/f8f 281765 0 2024-03-22T02:28:40.114 INFO:tasks.workunit.client.0.smithi103.stdout:8/567: truncate d4/d2a/d4f/fac 630897 0 2024-03-22T02:28:40.119 INFO:tasks.workunit.client.0.smithi103.stdout:3/487: dwrite de/d12/d19/f9e [0,4194304] 0 2024-03-22T02:28:40.121 INFO:tasks.workunit.client.0.smithi103.stdout:3/488: write de/f23 [2806552,84864] 0 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:8/568: link d4/d48/d40/c82 d4/d48/d40/d68/cb1 0 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:9/427: write d2/f3b [517713,66630] 0 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:9/428: dread - d2/da/d18/f6c zero size 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:8/569: write d4/d19/f2e [2252897,9037] 0 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:3/489: symlink de/d6b/d80/d73/la4 0 2024-03-22T02:28:40.127 INFO:tasks.workunit.client.0.smithi103.stdout:8/570: readlink d4/d48/d35/l7a 0 2024-03-22T02:28:40.129 INFO:tasks.workunit.client.0.smithi103.stdout:8/571: write d4/d26/d27/d7e/d39/f9a [404335,401] 0 2024-03-22T02:28:40.132 INFO:tasks.workunit.client.0.smithi103.stdout:8/572: dread d4/d19/d8c/f8f [0,4194304] 0 2024-03-22T02:28:40.135 INFO:tasks.workunit.client.0.smithi103.stdout:3/490: dwrite de/d12/d30/f35 [0,4194304] 0 2024-03-22T02:28:40.139 INFO:tasks.workunit.client.0.smithi103.stdout:8/573: rename d4/d48/l36 to d4/d26/d27/d7e/d39/lb2 0 2024-03-22T02:28:40.140 INFO:tasks.workunit.client.0.smithi103.stdout:3/491: symlink de/d12/d5d/d92/la5 0 2024-03-22T02:28:40.142 INFO:tasks.workunit.client.0.smithi103.stdout:8/574: dread d4/d2a/f6c [0,4194304] 0 2024-03-22T02:28:40.144 INFO:tasks.workunit.client.0.smithi103.stdout:3/492: rename de/d12/d19/f43 to de/d12/d30/fa6 0 2024-03-22T02:28:40.145 INFO:tasks.workunit.client.0.smithi103.stdout:9/429: truncate d2/d5/fb 78236 0 2024-03-22T02:28:40.147 INFO:tasks.workunit.client.0.smithi103.stdout:8/575: creat d4/d19/d1a/fb3 x:0 0 0 2024-03-22T02:28:40.149 INFO:tasks.workunit.client.0.smithi103.stdout:9/430: rename d2/l33 to d2/d5/d7b/l7f 0 2024-03-22T02:28:40.150 INFO:tasks.workunit.client.0.smithi103.stdout:3/493: truncate de/d12/d30/f3d 1071003 0 2024-03-22T02:28:40.151 INFO:tasks.workunit.client.0.smithi103.stdout:8/576: fdatasync d4/d26/d27/fb0 0 2024-03-22T02:28:40.153 INFO:tasks.workunit.client.0.smithi103.stdout:8/577: dread d4/d19/d8c/f8f [0,4194304] 0 2024-03-22T02:28:40.156 INFO:tasks.workunit.client.0.smithi103.stdout:8/578: truncate d4/d26/f8e 8886198 0 2024-03-22T02:28:40.162 INFO:tasks.workunit.client.0.smithi103.stdout:9/431: fdatasync d2/d10/d43/f7a 0 2024-03-22T02:28:40.165 INFO:tasks.workunit.client.0.smithi103.stdout:9/432: chown d2/da/d18/d1e/d34/d3d/d46/c70 77739 1 2024-03-22T02:28:40.169 INFO:tasks.workunit.client.0.smithi103.stdout:9/433: mkdir d2/d5/d4c/d56/d6f/d80 0 2024-03-22T02:28:40.171 INFO:tasks.workunit.client.0.smithi103.stdout:9/434: read d2/da/d18/d1e/d34/d3d/d4d/f58 [6775304,51547] 0 2024-03-22T02:28:40.173 INFO:tasks.workunit.client.0.smithi103.stdout:9/435: write d2/f3 [3554536,74280] 0 2024-03-22T02:28:40.174 INFO:tasks.workunit.client.0.smithi103.stdout:3/494: dread de/f22 [0,4194304] 0 2024-03-22T02:28:40.177 INFO:tasks.workunit.client.0.smithi103.stdout:9/436: mknod d2/d10/d43/c81 0 2024-03-22T02:28:40.177 INFO:tasks.workunit.client.0.smithi103.stdout:8/579: dread d4/d26/d27/d7e/d39/d49/f74 [0,4194304] 0 2024-03-22T02:28:40.178 INFO:tasks.workunit.client.0.smithi103.stdout:3/495: mkdir de/d1a/d25/d3f/d9c/da7 0 2024-03-22T02:28:40.179 INFO:tasks.workunit.client.0.smithi103.stdout:9/437: creat d2/d10/d43/d4b/f82 x:0 0 0 2024-03-22T02:28:40.180 INFO:tasks.workunit.client.0.smithi103.stdout:8/580: write f2 [6482434,63068] 0 2024-03-22T02:28:40.180 INFO:tasks.workunit.client.0.smithi103.stdout:9/438: fsync d2/da/fd 0 2024-03-22T02:28:40.183 INFO:tasks.workunit.client.0.smithi103.stdout:8/581: creat d4/d48/fb4 x:0 0 0 2024-03-22T02:28:40.184 INFO:tasks.workunit.client.0.smithi103.stdout:9/439: symlink d2/d5/d7b/d7d/l83 0 2024-03-22T02:28:40.187 INFO:tasks.workunit.client.0.smithi103.stdout:3/496: truncate de/d1a/d25/d26/f6f 488122 0 2024-03-22T02:28:40.188 INFO:tasks.workunit.client.0.smithi103.stdout:3/497: rmdir de/d12/d97 39 2024-03-22T02:28:40.189 INFO:tasks.workunit.client.0.smithi103.stdout:3/498: readlink de/d6b/d80/d4f/l55 0 2024-03-22T02:28:40.190 INFO:tasks.workunit.client.0.smithi103.stdout:3/499: truncate fa 4629727 0 2024-03-22T02:28:40.190 INFO:tasks.workunit.client.0.smithi103.stdout:3/500: fdatasync de/d12/d19/f71 0 2024-03-22T02:28:40.194 INFO:tasks.workunit.client.0.smithi103.stdout:9/440: truncate d2/f3b 512225 0 2024-03-22T02:28:40.194 INFO:tasks.workunit.client.0.smithi103.stdout:3/501: dread de/f1f [0,4194304] 0 2024-03-22T02:28:40.195 INFO:tasks.workunit.client.0.smithi103.stdout:3/502: stat de/d12/d8f/c98 0 2024-03-22T02:28:40.196 INFO:tasks.workunit.client.0.smithi103.stdout:9/441: creat d2/d5/d7b/f84 x:0 0 0 2024-03-22T02:28:40.197 INFO:tasks.workunit.client.0.smithi103.stdout:3/503: symlink de/d12/d19/la8 0 2024-03-22T02:28:40.198 INFO:tasks.workunit.client.0.smithi103.stdout:3/504: write de/d6b/d80/d4f/f77 [772578,52923] 0 2024-03-22T02:28:40.199 INFO:tasks.workunit.client.0.smithi103.stdout:9/442: creat d2/d5/d4c/f85 x:0 0 0 2024-03-22T02:28:40.199 INFO:tasks.workunit.client.0.smithi103.stdout:9/443: dread - d2/d10/d43/f7a zero size 2024-03-22T02:28:40.201 INFO:tasks.workunit.client.0.smithi103.stdout:9/444: mknod d2/d5/d4c/c86 0 2024-03-22T02:28:40.202 INFO:tasks.workunit.client.0.smithi103.stdout:3/505: dread de/d1a/d25/d26/f2f [0,4194304] 0 2024-03-22T02:28:40.204 INFO:tasks.workunit.client.0.smithi103.stdout:3/506: chown c4 201 1 2024-03-22T02:28:40.206 INFO:tasks.workunit.client.0.smithi103.stdout:3/507: truncate de/d1a/d25/f79 977707 0 2024-03-22T02:28:40.209 INFO:tasks.workunit.client.0.smithi103.stdout:3/508: read de/d6b/d80/d4f/f65 [937975,20503] 0 2024-03-22T02:28:40.211 INFO:tasks.workunit.client.0.smithi103.stdout:3/509: mkdir de/d12/d5d/da9 0 2024-03-22T02:28:40.215 INFO:tasks.workunit.client.0.smithi103.stdout:9/445: dread d2/f3 [0,4194304] 0 2024-03-22T02:28:40.217 INFO:tasks.workunit.client.0.smithi103.stdout:3/510: dwrite de/d12/d19/f9e [4194304,4194304] 0 2024-03-22T02:28:40.219 INFO:tasks.workunit.client.0.smithi103.stdout:9/446: creat d2/da/d18/d1e/d34/d3d/d61/f87 x:0 0 0 2024-03-22T02:28:40.220 INFO:tasks.workunit.client.0.smithi103.stdout:3/511: mkdir de/d1a/daa 0 2024-03-22T02:28:40.222 INFO:tasks.workunit.client.0.smithi103.stdout:9/447: creat d2/da/d18/d1e/f88 x:0 0 0 2024-03-22T02:28:40.222 INFO:tasks.workunit.client.0.smithi103.stdout:1/482: sync 2024-03-22T02:28:40.222 INFO:tasks.workunit.client.0.smithi103.stdout:3/512: rmdir de/d12/d4d 39 2024-03-22T02:28:40.223 INFO:tasks.workunit.client.0.smithi103.stdout:1/483: truncate db/f9e 359939 0 2024-03-22T02:28:40.224 INFO:tasks.workunit.client.0.smithi103.stdout:1/484: chown db/d10/d29 1 1 2024-03-22T02:28:40.225 INFO:tasks.workunit.client.0.smithi103.stdout:9/448: creat d2/d10/d59/f89 x:0 0 0 2024-03-22T02:28:40.225 INFO:tasks.workunit.client.0.smithi103.stdout:9/449: readlink d2/d10/l16 0 2024-03-22T02:28:40.226 INFO:tasks.workunit.client.0.smithi103.stdout:3/513: symlink de/d12/d97/lab 0 2024-03-22T02:28:40.227 INFO:tasks.workunit.client.0.smithi103.stdout:1/485: creat db/d10/d28/fa7 x:0 0 0 2024-03-22T02:28:40.229 INFO:tasks.workunit.client.0.smithi103.stdout:1/486: truncate db/d10/d29/d55/f7c 1050375 0 2024-03-22T02:28:40.230 INFO:tasks.workunit.client.0.smithi103.stdout:1/487: dread - db/d10/d29/d41/f99 zero size 2024-03-22T02:28:40.232 INFO:tasks.workunit.client.0.smithi103.stdout:9/450: dwrite d2/d10/d17/f26 [0,4194304] 0 2024-03-22T02:28:40.232 INFO:tasks.workunit.client.0.smithi103.stdout:1/488: mkdir db/d4a/d78/d92/da8 0 2024-03-22T02:28:40.234 INFO:tasks.workunit.client.0.smithi103.stdout:1/489: read - db/d10/d2e/f8a zero size 2024-03-22T02:28:40.234 INFO:tasks.workunit.client.0.smithi103.stdout:9/451: read d2/f2c [7905741,107590] 0 2024-03-22T02:28:40.235 INFO:tasks.workunit.client.0.smithi103.stdout:9/452: dread - d2/da/f52 zero size 2024-03-22T02:28:40.235 INFO:tasks.workunit.client.0.smithi103.stdout:3/514: dread de/d1a/d25/d26/f6f [0,4194304] 0 2024-03-22T02:28:40.235 INFO:tasks.workunit.client.0.smithi103.stdout:1/490: fsync db/d10/d29/d41/f90 0 2024-03-22T02:28:40.239 INFO:tasks.workunit.client.0.smithi103.stdout:3/515: fdatasync de/d1a/d25/d26/f58 0 2024-03-22T02:28:40.239 INFO:tasks.workunit.client.0.smithi103.stdout:9/453: dread d2/d5/fb [0,4194304] 0 2024-03-22T02:28:40.241 INFO:tasks.workunit.client.0.smithi103.stdout:1/491: creat db/fa9 x:0 0 0 2024-03-22T02:28:40.242 INFO:tasks.workunit.client.0.smithi103.stdout:3/516: write de/d12/d5d/f99 [654867,56651] 0 2024-03-22T02:28:40.242 INFO:tasks.workunit.client.0.smithi103.stdout:1/492: fdatasync db/d10/d28/f43 0 2024-03-22T02:28:40.243 INFO:tasks.workunit.client.0.smithi103.stdout:9/454: truncate d2/da/d18/d1e/d34/d3d/d46/f49 700189 0 2024-03-22T02:28:40.243 INFO:tasks.workunit.client.0.smithi103.stdout:1/493: readlink db/d10/l85 0 2024-03-22T02:28:40.245 INFO:tasks.workunit.client.0.smithi103.stdout:9/455: creat d2/d10/f8a x:0 0 0 2024-03-22T02:28:40.246 INFO:tasks.workunit.client.0.smithi103.stdout:3/517: creat de/d12/d8a/d93/fac x:0 0 0 2024-03-22T02:28:40.250 INFO:tasks.workunit.client.0.smithi103.stdout:9/456: truncate d2/d10/d27/f4f 2540098 0 2024-03-22T02:28:40.251 INFO:tasks.workunit.client.0.smithi103.stdout:1/494: dwrite db/d10/d18/f20 [4194304,4194304] 0 2024-03-22T02:28:40.254 INFO:tasks.workunit.client.0.smithi103.stdout:1/495: write db/d10/d28/f43 [326525,14372] 0 2024-03-22T02:28:40.256 INFO:tasks.workunit.client.0.smithi103.stdout:3/518: dwrite de/d12/d30/f63 [4194304,4194304] 0 2024-03-22T02:28:40.256 INFO:tasks.workunit.client.0.smithi103.stdout:9/457: rename d2/d10/d17 to d2/da/d18/d1e/d77/d8b 0 2024-03-22T02:28:40.258 INFO:tasks.workunit.client.0.smithi103.stdout:3/519: fsync de/d12/d30/f6e 0 2024-03-22T02:28:40.259 INFO:tasks.workunit.client.0.smithi103.stdout:1/496: creat db/d10/d6b/faa x:0 0 0 2024-03-22T02:28:40.260 INFO:tasks.workunit.client.0.smithi103.stdout:9/458: readlink d2/d5/d4c/d56/d6f/l76 0 2024-03-22T02:28:40.261 INFO:tasks.workunit.client.0.smithi103.stdout:1/497: fdatasync db/d10/d29/d41/f89 0 2024-03-22T02:28:40.261 INFO:tasks.workunit.client.0.smithi103.stdout:3/520: unlink de/c34 0 2024-03-22T02:28:40.262 INFO:tasks.workunit.client.0.smithi103.stdout:1/498: stat db/d10/d18/c42 0 2024-03-22T02:28:40.262 INFO:tasks.workunit.client.0.smithi103.stdout:3/521: fdatasync de/d1a/d25/d26/f58 0 2024-03-22T02:28:40.263 INFO:tasks.workunit.client.0.smithi103.stdout:3/522: chown de/d1a/d25/d26 36931 1 2024-03-22T02:28:40.264 INFO:tasks.workunit.client.0.smithi103.stdout:1/499: chown db/d10/c60 16359 1 2024-03-22T02:28:40.265 INFO:tasks.workunit.client.0.smithi103.stdout:1/500: chown db/d10/d28/fa7 26 1 2024-03-22T02:28:40.265 INFO:tasks.workunit.client.0.smithi103.stdout:1/501: chown db/d4a/l52 241468884 1 2024-03-22T02:28:40.266 INFO:tasks.workunit.client.0.smithi103.stdout:9/459: rename d2/d5/d4c/d56/d6f/d80 to d2/da/d18/d1e/d77/d8c 0 2024-03-22T02:28:40.267 INFO:tasks.workunit.client.0.smithi103.stdout:9/460: chown d2/da/d18/d1e/c4e 115 1 2024-03-22T02:28:40.268 INFO:tasks.workunit.client.0.smithi103.stdout:3/523: creat de/d1a/d25/d3f/d9c/da7/fad x:0 0 0 2024-03-22T02:28:40.269 INFO:tasks.workunit.client.0.smithi103.stdout:1/502: truncate db/d10/d18/f3b 5658518 0 2024-03-22T02:28:40.273 INFO:tasks.workunit.client.0.smithi103.stdout:3/524: rmdir de/d1a/daa 0 2024-03-22T02:28:40.277 INFO:tasks.workunit.client.0.smithi103.stdout:3/525: dwrite de/f1f [0,4194304] 0 2024-03-22T02:28:40.354 INFO:tasks.workunit.client.0.smithi103.stdout:9/461: write d2/d10/d43/f1b [1167830,93666] 0 2024-03-22T02:28:40.357 INFO:tasks.workunit.client.0.smithi103.stdout:7/459: sync 2024-03-22T02:28:40.357 INFO:tasks.workunit.client.0.smithi103.stdout:5/468: sync 2024-03-22T02:28:40.359 INFO:tasks.workunit.client.0.smithi103.stdout:1/503: truncate db/d10/f1c 3068577 0 2024-03-22T02:28:40.360 INFO:tasks.workunit.client.0.smithi103.stdout:7/460: stat d1/da/d1c/c66 0 2024-03-22T02:28:40.364 INFO:tasks.workunit.client.0.smithi103.stdout:5/469: symlink d0/d14/d23/d36/d95/l9f 0 2024-03-22T02:28:40.364 INFO:tasks.workunit.client.0.smithi103.stdout:9/462: dwrite d2/d10/f8a [0,4194304] 0 2024-03-22T02:28:40.365 INFO:tasks.workunit.client.0.smithi103.stdout:3/526: write de/d12/f16 [2937213,128685] 0 2024-03-22T02:28:40.365 INFO:tasks.workunit.client.0.smithi103.stdout:7/461: write d1/d10/d11/d19/d29/d33/f5f [1394256,119328] 0 2024-03-22T02:28:40.369 INFO:tasks.workunit.client.0.smithi103.stdout:3/527: dread de/d1a/d25/d3f/f85 [0,4194304] 0 2024-03-22T02:28:40.369 INFO:tasks.workunit.client.0.smithi103.stdout:3/528: write de/d12/d5d/f75 [901152,48162] 0 2024-03-22T02:28:40.370 INFO:tasks.workunit.client.0.smithi103.stdout:3/529: read de/d12/f9a [273711,81277] 0 2024-03-22T02:28:40.372 INFO:tasks.workunit.client.0.smithi103.stdout:9/463: chown d2/d5/f63 10099316 1 2024-03-22T02:28:40.372 INFO:tasks.workunit.client.0.smithi103.stdout:1/504: mkdir db/d10/d18/d84/dab 0 2024-03-22T02:28:40.374 INFO:tasks.workunit.client.0.smithi103.stdout:3/530: rmdir de/d12 39 2024-03-22T02:28:40.380 INFO:tasks.workunit.client.0.smithi103.stdout:3/531: dread de/d1a/d25/d3f/f68 [0,4194304] 0 2024-03-22T02:28:40.382 INFO:tasks.workunit.client.0.smithi103.stdout:9/464: unlink d2/da/d18/c53 0 2024-03-22T02:28:40.382 INFO:tasks.workunit.client.0.smithi103.stdout:1/505: symlink db/d4a/d78/lac 0 2024-03-22T02:28:40.383 INFO:tasks.workunit.client.0.smithi103.stdout:7/462: mknod d1/d10/d11/d19/d29/d33/d4b/c94 0 2024-03-22T02:28:40.385 INFO:tasks.workunit.client.0.smithi103.stdout:3/532: mknod de/d1a/d25/d3f/d9c/cae 0 2024-03-22T02:28:40.387 INFO:tasks.workunit.client.0.smithi103.stdout:5/470: getdents d0/d14/d23/d36/d95 0 2024-03-22T02:28:40.388 INFO:tasks.workunit.client.0.smithi103.stdout:1/506: truncate db/d10/d18/d2d/f88 1248767 0 2024-03-22T02:28:40.390 INFO:tasks.workunit.client.0.smithi103.stdout:7/463: symlink d1/d10/d11/d19/d29/d57/l95 0 2024-03-22T02:28:40.391 INFO:tasks.workunit.client.0.smithi103.stdout:3/533: fdatasync de/d12/f16 0 2024-03-22T02:28:40.394 INFO:tasks.workunit.client.0.smithi103.stdout:7/464: truncate d1/d10/d11/d19/d29/d33/d4b/f83 636142 0 2024-03-22T02:28:40.394 INFO:tasks.workunit.client.0.smithi103.stdout:1/507: creat db/d10/d18/fad x:0 0 0 2024-03-22T02:28:40.395 INFO:tasks.workunit.client.0.smithi103.stdout:3/534: dread - de/d1a/d25/d3f/d9c/da7/fad zero size 2024-03-22T02:28:40.396 INFO:tasks.workunit.client.0.smithi103.stdout:1/508: dread - db/d10/d2e/f8a zero size 2024-03-22T02:28:40.396 INFO:tasks.workunit.client.0.smithi103.stdout:9/465: getdents d2/d5 0 2024-03-22T02:28:40.397 INFO:tasks.workunit.client.0.smithi103.stdout:3/535: chown de/d12/c2c 677766 1 2024-03-22T02:28:40.401 INFO:tasks.workunit.client.0.smithi103.stdout:1/509: mkdir db/d10/d2e/dae 0 2024-03-22T02:28:40.402 INFO:tasks.workunit.client.0.smithi103.stdout:3/536: symlink de/d12/d30/d70/laf 0 2024-03-22T02:28:40.404 INFO:tasks.workunit.client.0.smithi103.stdout:5/471: dread d0/d14/d23/d2a/f63 [0,4194304] 0 2024-03-22T02:28:40.406 INFO:tasks.workunit.client.0.smithi103.stdout:7/465: dwrite d1/d10/d11/d19/d29/d61/d1d/f41 [0,4194304] 0 2024-03-22T02:28:40.408 INFO:tasks.workunit.client.0.smithi103.stdout:1/510: unlink db/d10/d21/l80 0 2024-03-22T02:28:40.409 INFO:tasks.workunit.client.0.smithi103.stdout:7/466: chown d1/d10/d11/d19/d29/d61/d69/c27 71669130 1 2024-03-22T02:28:40.409 INFO:tasks.workunit.client.0.smithi103.stdout:1/511: dread - db/d10/d2e/f8a zero size 2024-03-22T02:28:40.411 INFO:tasks.workunit.client.0.smithi103.stdout:3/537: rename de/d12/d97 to de/d12/d8a/db0 0 2024-03-22T02:28:40.412 INFO:tasks.workunit.client.0.smithi103.stdout:5/472: fsync d0/d14/d23/f44 0 2024-03-22T02:28:40.413 INFO:tasks.workunit.client.0.smithi103.stdout:5/473: write d0/f97 [367562,58551] 0 2024-03-22T02:28:40.415 INFO:tasks.workunit.client.0.smithi103.stdout:1/512: write db/d10/f83 [1008719,50080] 0 2024-03-22T02:28:40.417 INFO:tasks.workunit.client.0.smithi103.stdout:0/561: sync 2024-03-22T02:28:40.417 INFO:tasks.workunit.client.0.smithi103.stdout:2/426: sync 2024-03-22T02:28:40.417 INFO:tasks.workunit.client.0.smithi103.stdout:6/350: sync 2024-03-22T02:28:40.417 INFO:tasks.workunit.client.0.smithi103.stdout:8/582: sync 2024-03-22T02:28:40.417 INFO:tasks.workunit.client.0.smithi103.stdout:4/446: sync 2024-03-22T02:28:40.421 INFO:tasks.workunit.client.0.smithi103.stdout:1/513: mknod db/d4a/d78/d92/caf 0 2024-03-22T02:28:40.421 INFO:tasks.workunit.client.0.smithi103.stdout:2/427: stat d11/d19/d5d/f68 0 2024-03-22T02:28:40.421 INFO:tasks.workunit.client.0.smithi103.stdout:3/538: chown de/d12/f1c 161 1 2024-03-22T02:28:40.425 INFO:tasks.workunit.client.0.smithi103.stdout:0/562: write dd/d16/d4b/d96/d35/d82/f7b [181127,65046] 0 2024-03-22T02:28:40.426 INFO:tasks.workunit.client.0.smithi103.stdout:3/539: symlink de/d12/d30/lb1 0 2024-03-22T02:28:40.426 INFO:tasks.workunit.client.0.smithi103.stdout:6/351: read d0/d3/d22/f3c [2411772,4844] 0 2024-03-22T02:28:40.431 INFO:tasks.workunit.client.0.smithi103.stdout:6/352: write d0/d3/f49 [876369,79856] 0 2024-03-22T02:28:40.431 INFO:tasks.workunit.client.0.smithi103.stdout:9/466: dwrite d2/f9 [4194304,4194304] 0 2024-03-22T02:28:40.433 INFO:tasks.workunit.client.0.smithi103.stdout:2/428: write d11/d25/d63/d45/f79 [1618307,39676] 0 2024-03-22T02:28:40.436 INFO:tasks.workunit.client.0.smithi103.stdout:0/563: creat dd/d34/d53/d8c/fbc x:0 0 0 2024-03-22T02:28:40.439 INFO:tasks.workunit.client.0.smithi103.stdout:3/540: creat de/d12/d8a/db0/fb2 x:0 0 0 2024-03-22T02:28:40.440 INFO:tasks.workunit.client.0.smithi103.stdout:3/541: write de/d12/d19/f71 [1818965,50599] 0 2024-03-22T02:28:40.442 INFO:tasks.workunit.client.0.smithi103.stdout:3/542: readlink de/d6b/d80/d73/la4 0 2024-03-22T02:28:40.442 INFO:tasks.workunit.client.0.smithi103.stdout:7/467: link d1/d10/d11/f28 d1/d10/d11/d19/d29/d61/f96 0 2024-03-22T02:28:40.443 INFO:tasks.workunit.client.0.smithi103.stdout:4/447: link d3/d1e/d25/d65/c9d d3/d1e/d78/c9f 0 2024-03-22T02:28:40.446 INFO:tasks.workunit.client.0.smithi103.stdout:5/474: dread d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:40.448 INFO:tasks.workunit.client.0.smithi103.stdout:8/583: rename d4/d26/d27/fa2 to d4/d26/d27/d7e/d39/d49/fb5 0 2024-03-22T02:28:40.450 INFO:tasks.workunit.client.0.smithi103.stdout:8/584: write d4/f89 [1986078,100748] 0 2024-03-22T02:28:40.453 INFO:tasks.workunit.client.0.smithi103.stdout:0/564: chown dd/d16/c4a 7548618 1 2024-03-22T02:28:40.453 INFO:tasks.workunit.client.0.smithi103.stdout:1/514: dwrite db/d10/d18/f72 [0,4194304] 0 2024-03-22T02:28:40.455 INFO:tasks.workunit.client.0.smithi103.stdout:5/475: creat d0/d14/d23/d36/d95/fa0 x:0 0 0 2024-03-22T02:28:40.457 INFO:tasks.workunit.client.0.smithi103.stdout:1/515: truncate db/d4a/f65 38410 0 2024-03-22T02:28:40.460 INFO:tasks.workunit.client.0.smithi103.stdout:3/543: symlink de/d1a/lb3 0 2024-03-22T02:28:40.461 INFO:tasks.workunit.client.0.smithi103.stdout:6/353: dread d0/d3/d22/f15 [0,4194304] 0 2024-03-22T02:28:40.461 INFO:tasks.workunit.client.0.smithi103.stdout:9/467: rename d2/da/d18/d1e/d34/d3d/d46/f49 to d2/da/d18/d1e/d34/f8d 0 2024-03-22T02:28:40.461 INFO:tasks.workunit.client.0.smithi103.stdout:5/476: write d0/d14/d23/d36/f85 [4867989,81047] 0 2024-03-22T02:28:40.463 INFO:tasks.workunit.client.0.smithi103.stdout:3/544: stat de/d12/f86 0 2024-03-22T02:28:40.465 INFO:tasks.workunit.client.0.smithi103.stdout:2/429: link d11/d25/d63/d45/l5e d11/d19/d7e/l8c 0 2024-03-22T02:28:40.466 INFO:tasks.workunit.client.0.smithi103.stdout:9/468: write d2/d5/f63 [2624729,60977] 0 2024-03-22T02:28:40.467 INFO:tasks.workunit.client.0.smithi103.stdout:4/448: truncate d3/d30/f49 1845045 0 2024-03-22T02:28:40.472 INFO:tasks.workunit.client.0.smithi103.stdout:4/449: fdatasync d3/d1e/d25/f52 0 2024-03-22T02:28:40.473 INFO:tasks.workunit.client.0.smithi103.stdout:0/565: mkdir dd/d34/dbd 0 2024-03-22T02:28:40.481 INFO:tasks.workunit.client.0.smithi103.stdout:6/354: dwrite d0/f1a [0,4194304] 0 2024-03-22T02:28:40.482 INFO:tasks.workunit.client.0.smithi103.stdout:1/516: creat db/d10/d18/d2d/fb0 x:0 0 0 2024-03-22T02:28:40.485 INFO:tasks.workunit.client.0.smithi103.stdout:5/477: rename d0/d14/d23/d2a/f63 to d0/d14/d23/d6f/d83/fa1 0 2024-03-22T02:28:40.486 INFO:tasks.workunit.client.0.smithi103.stdout:2/430: dwrite d11/d25/d63/d45/f8b [0,4194304] 0 2024-03-22T02:28:40.490 INFO:tasks.workunit.client.0.smithi103.stdout:8/585: dwrite d4/d26/d27/d7e/d39/d49/d5f/f66 [0,4194304] 0 2024-03-22T02:28:40.495 INFO:tasks.workunit.client.0.smithi103.stdout:4/450: stat d3/d1e/f2f 0 2024-03-22T02:28:40.499 INFO:tasks.workunit.client.0.smithi103.stdout:0/566: mknod dd/d16/d4b/d96/d29/d48/d5c/cbe 0 2024-03-22T02:28:40.501 INFO:tasks.workunit.client.0.smithi103.stdout:6/355: write d0/d3/d22/f3c [3699568,37777] 0 2024-03-22T02:28:40.503 INFO:tasks.workunit.client.0.smithi103.stdout:2/431: creat d11/d19/d5d/f8d x:0 0 0 2024-03-22T02:28:40.506 INFO:tasks.workunit.client.0.smithi103.stdout:1/517: rename db/d4a/l6a to db/d10/d28/d97/lb1 0 2024-03-22T02:28:40.508 INFO:tasks.workunit.client.0.smithi103.stdout:7/468: dwrite d1/d10/d11/d19/d29/d61/f96 [0,4194304] 0 2024-03-22T02:28:40.511 INFO:tasks.workunit.client.0.smithi103.stdout:5/478: dwrite d0/d14/d23/d46/f7a [0,4194304] 0 2024-03-22T02:28:40.513 INFO:tasks.workunit.client.0.smithi103.stdout:3/545: creat de/fb4 x:0 0 0 2024-03-22T02:28:40.514 INFO:tasks.workunit.client.0.smithi103.stdout:0/567: write dd/d16/d4b/d96/d29/f3e [3854754,109568] 0 2024-03-22T02:28:40.514 INFO:tasks.workunit.client.0.smithi103.stdout:2/432: unlink d11/d19/d4e/f88 0 2024-03-22T02:28:40.516 INFO:tasks.workunit.client.0.smithi103.stdout:9/469: dwrite d2/da/f23 [0,4194304] 0 2024-03-22T02:28:40.517 INFO:tasks.workunit.client.0.smithi103.stdout:4/451: creat d3/fa0 x:0 0 0 2024-03-22T02:28:40.517 INFO:tasks.workunit.client.0.smithi103.stdout:3/546: fsync de/d6b/d80/f7a 0 2024-03-22T02:28:40.519 INFO:tasks.workunit.client.0.smithi103.stdout:7/469: chown d1/d10/d11/d19/d29 2612365 1 2024-03-22T02:28:40.520 INFO:tasks.workunit.client.0.smithi103.stdout:5/479: mkdir d0/da2 0 2024-03-22T02:28:40.521 INFO:tasks.workunit.client.0.smithi103.stdout:4/452: dread - d3/fa0 zero size 2024-03-22T02:28:40.521 INFO:tasks.workunit.client.0.smithi103.stdout:0/568: write dd/d16/d4b/d96/d64/d91/f99 [1408900,2720] 0 2024-03-22T02:28:40.525 INFO:tasks.workunit.client.0.smithi103.stdout:5/480: readlink d0/d14/d23/d2a/d2c/l90 0 2024-03-22T02:28:40.526 INFO:tasks.workunit.client.0.smithi103.stdout:5/481: readlink d0/d14/d23/l5e 0 2024-03-22T02:28:40.527 INFO:tasks.workunit.client.0.smithi103.stdout:9/470: mknod d2/d10/c8e 0 2024-03-22T02:28:40.533 INFO:tasks.workunit.client.0.smithi103.stdout:6/356: dread d0/d3/d22/f1c [0,4194304] 0 2024-03-22T02:28:40.535 INFO:tasks.workunit.client.0.smithi103.stdout:5/482: truncate d0/d14/f8d 273604 0 2024-03-22T02:28:40.538 INFO:tasks.workunit.client.0.smithi103.stdout:4/453: dwrite d3/d69/f46 [0,4194304] 0 2024-03-22T02:28:40.540 INFO:tasks.workunit.client.0.smithi103.stdout:7/470: link d1/da/d2f/d36/f91 d1/d10/d11/d19/d29/d33/d4b/f97 0 2024-03-22T02:28:40.540 INFO:tasks.workunit.client.0.smithi103.stdout:2/433: dwrite d11/d19/d5d/f8d [0,4194304] 0 2024-03-22T02:28:40.542 INFO:tasks.workunit.client.0.smithi103.stdout:2/434: readlink d11/d19/l22 0 2024-03-22T02:28:40.542 INFO:tasks.workunit.client.0.smithi103.stdout:3/547: dwrite de/d12/f2b [0,4194304] 0 2024-03-22T02:28:40.550 INFO:tasks.workunit.client.0.smithi103.stdout:6/357: rename d0/d14/c4b to d0/d3/c70 0 2024-03-22T02:28:40.553 INFO:tasks.workunit.client.0.smithi103.stdout:2/435: rmdir d11/d25/d46/d48 39 2024-03-22T02:28:40.556 INFO:tasks.workunit.client.0.smithi103.stdout:5/483: creat d0/d14/d23/fa3 x:0 0 0 2024-03-22T02:28:40.557 INFO:tasks.workunit.client.0.smithi103.stdout:8/586: truncate d4/d26/d27/d7e/d39/d49/d5f/f66 3821204 0 2024-03-22T02:28:40.558 INFO:tasks.workunit.client.0.smithi103.stdout:8/587: write d4/d2a/f94 [694744,35596] 0 2024-03-22T02:28:40.559 INFO:tasks.workunit.client.0.smithi103.stdout:6/358: creat d0/d14/d57/d68/f71 x:0 0 0 2024-03-22T02:28:40.561 INFO:tasks.workunit.client.0.smithi103.stdout:6/359: truncate d0/d3/d22/f50 145511 0 2024-03-22T02:28:40.561 INFO:tasks.workunit.client.0.smithi103.stdout:6/360: write d0/d14/f56 [514246,88128] 0 2024-03-22T02:28:40.562 INFO:tasks.workunit.client.0.smithi103.stdout:5/484: creat d0/d14/d23/d5c/d77/fa4 x:0 0 0 2024-03-22T02:28:40.562 INFO:tasks.workunit.client.0.smithi103.stdout:3/548: link de/d12/d19/la8 de/d1a/d25/d3f/d9c/da7/lb5 0 2024-03-22T02:28:40.564 INFO:tasks.workunit.client.0.smithi103.stdout:6/361: chown d0/d14/d57/d68/f28 604440630 1 2024-03-22T02:28:40.565 INFO:tasks.workunit.client.0.smithi103.stdout:6/362: fsync d0/d14/f56 0 2024-03-22T02:28:40.565 INFO:tasks.workunit.client.0.smithi103.stdout:8/588: mknod d4/d26/d27/d7e/d39/d49/cb6 0 2024-03-22T02:28:40.569 INFO:tasks.workunit.client.0.smithi103.stdout:7/471: write d1/d10/d11/d19/f55 [427409,113272] 0 2024-03-22T02:28:40.570 INFO:tasks.workunit.client.0.smithi103.stdout:5/485: fsync d0/d14/f41 0 2024-03-22T02:28:40.572 INFO:tasks.workunit.client.0.smithi103.stdout:5/486: dread - d0/f73 zero size 2024-03-22T02:28:40.572 INFO:tasks.workunit.client.0.smithi103.stdout:5/487: readlink d0/d3e/l6b 0 2024-03-22T02:28:40.573 INFO:tasks.workunit.client.0.smithi103.stdout:6/363: dwrite d0/d14/d57/d68/f2d [0,4194304] 0 2024-03-22T02:28:40.573 INFO:tasks.workunit.client.0.smithi103.stdout:8/589: creat d4/d2a/d4f/fb7 x:0 0 0 2024-03-22T02:28:40.575 INFO:tasks.workunit.client.0.smithi103.stdout:8/590: stat d4/d19/f5c 0 2024-03-22T02:28:40.577 INFO:tasks.workunit.client.0.smithi103.stdout:2/436: getdents d11/d25/d46 0 2024-03-22T02:28:40.578 INFO:tasks.workunit.client.0.smithi103.stdout:2/437: write d11/f30 [4114359,112431] 0 2024-03-22T02:28:40.579 INFO:tasks.workunit.client.0.smithi103.stdout:6/364: dread d0/f5b [0,4194304] 0 2024-03-22T02:28:40.581 INFO:tasks.workunit.client.0.smithi103.stdout:6/365: chown d0/d14/c17 24 1 2024-03-22T02:28:40.582 INFO:tasks.workunit.client.0.smithi103.stdout:6/366: stat d0/d3/d22/f5c 0 2024-03-22T02:28:40.583 INFO:tasks.workunit.client.0.smithi103.stdout:2/438: dread d11/d25/d46/d40/f6f [0,4194304] 0 2024-03-22T02:28:40.583 INFO:tasks.workunit.client.0.smithi103.stdout:7/472: unlink d1/d10/d11/d19/f56 0 2024-03-22T02:28:40.585 INFO:tasks.workunit.client.0.smithi103.stdout:5/488: write d0/d14/d23/f40 [3101727,61099] 0 2024-03-22T02:28:40.588 INFO:tasks.workunit.client.0.smithi103.stdout:8/591: creat d4/d26/d27/d7e/d39/d7b/fb8 x:0 0 0 2024-03-22T02:28:40.594 INFO:tasks.workunit.client.0.smithi103.stdout:3/549: dwrite de/d12/d19/f28 [0,4194304] 0 2024-03-22T02:28:40.595 INFO:tasks.workunit.client.0.smithi103.stdout:3/550: chown de/d1a/lb3 1716 1 2024-03-22T02:28:40.598 INFO:tasks.workunit.client.0.smithi103.stdout:5/489: dread d0/d14/d23/d2a/d33/f4c [0,4194304] 0 2024-03-22T02:28:40.602 INFO:tasks.workunit.client.0.smithi103.stdout:8/592: creat d4/d26/d27/d7e/d39/d49/d79/fb9 x:0 0 0 2024-03-22T02:28:40.603 INFO:tasks.workunit.client.0.smithi103.stdout:8/593: dread - d4/d2a/d8b/f9f zero size 2024-03-22T02:28:40.605 INFO:tasks.workunit.client.0.smithi103.stdout:6/367: read d0/d3/d33/f60 [2731997,2254] 0 2024-03-22T02:28:40.607 INFO:tasks.workunit.client.0.smithi103.stdout:7/473: truncate d1/d10/d11/d19/d29/d61/d69/f32 462809 0 2024-03-22T02:28:40.608 INFO:tasks.workunit.client.0.smithi103.stdout:3/551: unlink de/d12/d8f/c98 0 2024-03-22T02:28:40.611 INFO:tasks.workunit.client.0.smithi103.stdout:6/368: mknod d0/d14/d57/d68/c72 0 2024-03-22T02:28:40.612 INFO:tasks.workunit.client.0.smithi103.stdout:2/439: dwrite d11/d25/d46/f37 [0,4194304] 0 2024-03-22T02:28:40.614 INFO:tasks.workunit.client.0.smithi103.stdout:6/369: write d0/d3/f39 [4160567,31744] 0 2024-03-22T02:28:40.615 INFO:tasks.workunit.client.0.smithi103.stdout:6/370: dread - d0/d14/d57/d68/f6b zero size 2024-03-22T02:28:40.622 INFO:tasks.workunit.client.0.smithi103.stdout:2/440: dread d11/d25/d46/d40/f6f [0,4194304] 0 2024-03-22T02:28:40.624 INFO:tasks.workunit.client.0.smithi103.stdout:7/474: write d1/da/f30 [3974149,114555] 0 2024-03-22T02:28:40.624 INFO:tasks.workunit.client.0.smithi103.stdout:8/594: link d4/fd d4/d48/d40/fba 0 2024-03-22T02:28:40.627 INFO:tasks.workunit.client.0.smithi103.stdout:5/490: getdents d0/d14 0 2024-03-22T02:28:40.629 INFO:tasks.workunit.client.0.smithi103.stdout:6/371: rename d0/d3/c43 to d0/d14/d57/c73 0 2024-03-22T02:28:40.629 INFO:tasks.workunit.client.0.smithi103.stdout:5/491: truncate d0/d14/f8d 1145262 0 2024-03-22T02:28:40.630 INFO:tasks.workunit.client.0.smithi103.stdout:8/595: mknod d4/d2a/d38/cbb 0 2024-03-22T02:28:40.631 INFO:tasks.workunit.client.0.smithi103.stdout:6/372: chown d0/d14/d57 5264 1 2024-03-22T02:28:40.632 INFO:tasks.workunit.client.0.smithi103.stdout:3/552: dwrite de/d1a/d25/d26/f2f [0,4194304] 0 2024-03-22T02:28:40.633 INFO:tasks.workunit.client.0.smithi103.stdout:5/492: chown d0/d14/d23/d36/d95/fa0 5764 1 2024-03-22T02:28:40.643 INFO:tasks.workunit.client.0.smithi103.stdout:2/441: getdents d11/d25/d46/d40 0 2024-03-22T02:28:40.646 INFO:tasks.workunit.client.0.smithi103.stdout:7/475: rename d1/da/d1c/l3a to d1/d10/d11/d19/d29/d33/d4b/l98 0 2024-03-22T02:28:40.646 INFO:tasks.workunit.client.0.smithi103.stdout:5/493: write d0/d14/d23/d36/d95/fa0 [619446,28543] 0 2024-03-22T02:28:40.653 INFO:tasks.workunit.client.0.smithi103.stdout:2/442: mknod d11/d25/d63/d45/d47/d83/d78/c8e 0 2024-03-22T02:28:40.654 INFO:tasks.workunit.client.0.smithi103.stdout:7/476: write d1/da/d2f/f38 [230596,74565] 0 2024-03-22T02:28:40.656 INFO:tasks.workunit.client.0.smithi103.stdout:8/596: dwrite d4/d26/d27/f63 [0,4194304] 0 2024-03-22T02:28:40.657 INFO:tasks.workunit.client.0.smithi103.stdout:3/553: dwrite de/d12/d5d/f75 [0,4194304] 0 2024-03-22T02:28:40.657 INFO:tasks.workunit.client.0.smithi103.stdout:6/373: dwrite d0/d14/f63 [0,4194304] 0 2024-03-22T02:28:40.660 INFO:tasks.workunit.client.0.smithi103.stdout:6/374: read d0/d14/f56 [590376,107373] 0 2024-03-22T02:28:40.660 INFO:tasks.workunit.client.0.smithi103.stdout:7/477: chown d1/da/d2f/d36/l48 17 1 2024-03-22T02:28:40.660 INFO:tasks.workunit.client.0.smithi103.stdout:2/443: chown d11/d25/d46/d34/f36 8785532 1 2024-03-22T02:28:40.664 INFO:tasks.workunit.client.0.smithi103.stdout:3/554: write de/d1a/d25/f6c [374770,88866] 0 2024-03-22T02:28:40.665 INFO:tasks.workunit.client.0.smithi103.stdout:6/375: readlink d0/d14/l5f 0 2024-03-22T02:28:40.665 INFO:tasks.workunit.client.0.smithi103.stdout:8/597: creat d4/d2a/d8b/fbc x:0 0 0 2024-03-22T02:28:40.667 INFO:tasks.workunit.client.0.smithi103.stdout:6/376: read - d0/f69 zero size 2024-03-22T02:28:40.668 INFO:tasks.workunit.client.0.smithi103.stdout:6/377: stat d0/d14/d57/d68/f71 0 2024-03-22T02:28:40.669 INFO:tasks.workunit.client.0.smithi103.stdout:6/378: readlink d0/d14/l5f 0 2024-03-22T02:28:40.671 INFO:tasks.workunit.client.0.smithi103.stdout:2/444: rename d11/d25/d63/d45/f62 to d11/d19/f8f 0 2024-03-22T02:28:40.676 INFO:tasks.workunit.client.0.smithi103.stdout:6/379: creat d0/d3/f74 x:0 0 0 2024-03-22T02:28:40.677 INFO:tasks.workunit.client.0.smithi103.stdout:3/555: dwrite de/d1a/d25/f29 [0,4194304] 0 2024-03-22T02:28:40.678 INFO:tasks.workunit.client.0.smithi103.stdout:7/478: dwrite d1/d10/d11/d19/d29/d61/d69/f84 [0,4194304] 0 2024-03-22T02:28:40.682 INFO:tasks.workunit.client.0.smithi103.stdout:3/556: write de/d12/d30/f35 [3607713,102735] 0 2024-03-22T02:28:40.682 INFO:tasks.workunit.client.0.smithi103.stdout:7/479: chown d1/da/c2c 134495 1 2024-03-22T02:28:40.684 INFO:tasks.workunit.client.0.smithi103.stdout:7/480: readlink d1/da/l17 0 2024-03-22T02:28:40.685 INFO:tasks.workunit.client.0.smithi103.stdout:8/598: rmdir d4/d2a 39 2024-03-22T02:28:40.686 INFO:tasks.workunit.client.0.smithi103.stdout:5/494: truncate d0/d14/d23/d2a/d2c/f4d 3311978 0 2024-03-22T02:28:40.688 INFO:tasks.workunit.client.0.smithi103.stdout:2/445: dwrite d11/d25/d46/d48/f7a [0,4194304] 0 2024-03-22T02:28:40.688 INFO:tasks.workunit.client.0.smithi103.stdout:5/495: chown d0/f97 287 1 2024-03-22T02:28:40.689 INFO:tasks.workunit.client.0.smithi103.stdout:6/380: dread d0/f1a [0,4194304] 0 2024-03-22T02:28:40.694 INFO:tasks.workunit.client.0.smithi103.stdout:7/481: mkdir d1/d6f/d99 0 2024-03-22T02:28:40.695 INFO:tasks.workunit.client.0.smithi103.stdout:8/599: rmdir d4/d19/d1a/d4a 39 2024-03-22T02:28:40.696 INFO:tasks.workunit.client.0.smithi103.stdout:2/446: dwrite f10 [0,4194304] 0 2024-03-22T02:28:40.697 INFO:tasks.workunit.client.0.smithi103.stdout:7/482: chown d1/d10/d11/d19/d29/d61/d1d 105537 1 2024-03-22T02:28:40.701 INFO:tasks.workunit.client.0.smithi103.stdout:6/381: unlink d0/d3/f53 0 2024-03-22T02:28:40.702 INFO:tasks.workunit.client.0.smithi103.stdout:6/382: read d0/d3/d22/f15 [3238526,95191] 0 2024-03-22T02:28:40.703 INFO:tasks.workunit.client.0.smithi103.stdout:5/496: rename d0/d14/d23/d36/f6a to d0/d14/d23/d36/d95/fa5 0 2024-03-22T02:28:40.705 INFO:tasks.workunit.client.0.smithi103.stdout:2/447: mkdir d11/d25/d46/d40/d90 0 2024-03-22T02:28:40.706 INFO:tasks.workunit.client.0.smithi103.stdout:2/448: dread - d11/d19/d4e/f53 zero size 2024-03-22T02:28:40.706 INFO:tasks.workunit.client.0.smithi103.stdout:3/557: link de/d1a/f39 de/d12/d19/fb6 0 2024-03-22T02:28:40.707 INFO:tasks.workunit.client.0.smithi103.stdout:2/449: write d11/d25/f7c [987106,69642] 0 2024-03-22T02:28:40.710 INFO:tasks.workunit.client.0.smithi103.stdout:2/450: truncate d11/d19/d5d/f68 210376 0 2024-03-22T02:28:40.710 INFO:tasks.workunit.client.0.smithi103.stdout:2/451: stat d11/d25/d46/d40/d90 0 2024-03-22T02:28:40.712 INFO:tasks.workunit.client.0.smithi103.stdout:5/497: rename d0/d14/d23/d5c/d70/l9b to d0/d14/d23/d2a/d7b/la6 0 2024-03-22T02:28:40.713 INFO:tasks.workunit.client.0.smithi103.stdout:6/383: mkdir d0/d3/d22/d58/d6f/d75 0 2024-03-22T02:28:40.714 INFO:tasks.workunit.client.0.smithi103.stdout:8/600: mknod d4/d2a/cbd 0 2024-03-22T02:28:40.716 INFO:tasks.workunit.client.0.smithi103.stdout:3/558: mknod de/d12/d4d/cb7 0 2024-03-22T02:28:40.717 INFO:tasks.workunit.client.0.smithi103.stdout:8/601: dread - d4/d26/d27/d7e/d39/d49/fb5 zero size 2024-03-22T02:28:40.717 INFO:tasks.workunit.client.0.smithi103.stdout:2/452: dwrite d11/d19/d5d/f8d [4194304,4194304] 0 2024-03-22T02:28:40.719 INFO:tasks.workunit.client.0.smithi103.stdout:2/453: dread - d11/d25/d46/d48/f6d zero size 2024-03-22T02:28:40.719 INFO:tasks.workunit.client.0.smithi103.stdout:5/498: rename d0/d14/f51 to d0/d14/d23/d6f/fa7 0 2024-03-22T02:28:40.719 INFO:tasks.workunit.client.0.smithi103.stdout:6/384: symlink d0/d14/d57/d68/l76 0 2024-03-22T02:28:40.721 INFO:tasks.workunit.client.0.smithi103.stdout:3/559: creat de/d12/d8f/fb8 x:0 0 0 2024-03-22T02:28:40.723 INFO:tasks.workunit.client.0.smithi103.stdout:2/454: creat d11/d19/d5d/f91 x:0 0 0 2024-03-22T02:28:40.724 INFO:tasks.workunit.client.0.smithi103.stdout:6/385: rmdir d0/d3/d22/d58 39 2024-03-22T02:28:40.730 INFO:tasks.workunit.client.0.smithi103.stdout:3/560: dwrite de/d12/d30/f63 [4194304,4194304] 0 2024-03-22T02:28:40.734 INFO:tasks.workunit.client.0.smithi103.stdout:2/455: rename d11/d25/d63/d45/d47/f49 to d11/d19/d5d/f92 0 2024-03-22T02:28:40.734 INFO:tasks.workunit.client.0.smithi103.stdout:6/386: truncate d0/d3/f42 705398 0 2024-03-22T02:28:40.736 INFO:tasks.workunit.client.0.smithi103.stdout:3/561: write de/d12/d19/f48 [1459666,7669] 0 2024-03-22T02:28:40.736 INFO:tasks.workunit.client.0.smithi103.stdout:3/562: dread - de/f96 zero size 2024-03-22T02:28:40.738 INFO:tasks.workunit.client.0.smithi103.stdout:2/456: creat d11/d19/d7e/f93 x:0 0 0 2024-03-22T02:28:40.738 INFO:tasks.workunit.client.0.smithi103.stdout:2/457: chown f10 10570380 1 2024-03-22T02:28:40.740 INFO:tasks.workunit.client.0.smithi103.stdout:6/387: chown d0/c10 46 1 2024-03-22T02:28:40.741 INFO:tasks.workunit.client.0.smithi103.stdout:3/563: mkdir de/d12/d5d/da9/db9 0 2024-03-22T02:28:40.743 INFO:tasks.workunit.client.0.smithi103.stdout:2/458: write d11/d19/d5d/f8d [9128105,95165] 0 2024-03-22T02:28:40.743 INFO:tasks.workunit.client.0.smithi103.stdout:6/388: write d0/d14/d57/d68/f6b [989787,49521] 0 2024-03-22T02:28:40.745 INFO:tasks.workunit.client.0.smithi103.stdout:7/483: dwrite d1/d10/d11/d19/d29/d61/d69/f32 [0,4194304] 0 2024-03-22T02:28:40.748 INFO:tasks.workunit.client.0.smithi103.stdout:2/459: dread d11/d25/d46/d34/f52 [0,4194304] 0 2024-03-22T02:28:40.748 INFO:tasks.workunit.client.0.smithi103.stdout:7/484: chown d1/d10/d11/d19/d29/d61/c31 464 1 2024-03-22T02:28:40.751 INFO:tasks.workunit.client.0.smithi103.stdout:3/564: dwrite de/d6b/d80/d4f/f77 [4194304,4194304] 0 2024-03-22T02:28:40.751 INFO:tasks.workunit.client.0.smithi103.stdout:7/485: fsync d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f 0 2024-03-22T02:28:40.765 INFO:tasks.workunit.client.0.smithi103.stdout:3/565: creat de/d6b/d80/d73/fba x:0 0 0 2024-03-22T02:28:40.767 INFO:tasks.workunit.client.0.smithi103.stdout:7/486: rename d1/d10/d11/d19/d29/d33/d4b/f89 to d1/d10/d11/d19/d29/d61/d69/f9a 0 2024-03-22T02:28:40.769 INFO:tasks.workunit.client.0.smithi103.stdout:8/602: truncate d4/d26/d27/f2c 1607091 0 2024-03-22T02:28:40.773 INFO:tasks.workunit.client.0.smithi103.stdout:7/487: symlink d1/d10/d11/d19/d29/l9b 0 2024-03-22T02:28:40.774 INFO:tasks.workunit.client.0.smithi103.stdout:3/566: dwrite de/d6b/d80/d73/fba [0,4194304] 0 2024-03-22T02:28:40.775 INFO:tasks.workunit.client.0.smithi103.stdout:5/499: creat d0/d14/d23/d5c/d77/fa8 x:0 0 0 2024-03-22T02:28:40.776 INFO:tasks.workunit.client.0.smithi103.stdout:3/567: fdatasync de/d6b/d80/f7c 0 2024-03-22T02:28:40.777 INFO:tasks.workunit.client.0.smithi103.stdout:6/389: dread d0/d3/f39 [0,4194304] 0 2024-03-22T02:28:40.778 INFO:tasks.workunit.client.0.smithi103.stdout:6/390: chown d0/c26 130 1 2024-03-22T02:28:40.779 INFO:tasks.workunit.client.0.smithi103.stdout:8/603: rmdir d4/d19/d1a/d4a 39 2024-03-22T02:28:40.780 INFO:tasks.workunit.client.0.smithi103.stdout:6/391: write d0/d14/f56 [1174597,73320] 0 2024-03-22T02:28:40.781 INFO:tasks.workunit.client.0.smithi103.stdout:7/488: symlink d1/d10/d11/d70/l9c 0 2024-03-22T02:28:40.783 INFO:tasks.workunit.client.0.smithi103.stdout:2/460: dread f4 [0,4194304] 0 2024-03-22T02:28:40.784 INFO:tasks.workunit.client.0.smithi103.stdout:7/489: write d1/d10/d11/d19/d29/d33/d4b/f83 [1425809,57875] 0 2024-03-22T02:28:40.784 INFO:tasks.workunit.client.0.smithi103.stdout:3/568: creat de/d12/d8f/fbb x:0 0 0 2024-03-22T02:28:40.786 INFO:tasks.workunit.client.0.smithi103.stdout:3/569: readlink de/d12/d30/d70/laf 0 2024-03-22T02:28:40.788 INFO:tasks.workunit.client.0.smithi103.stdout:3/570: dread de/d1a/d25/d3f/f85 [0,4194304] 0 2024-03-22T02:28:40.788 INFO:tasks.workunit.client.0.smithi103.stdout:5/500: unlink d0/d14/d23/d2a/d1f/l64 0 2024-03-22T02:28:40.793 INFO:tasks.workunit.client.0.smithi103.stdout:8/604: mknod d4/d19/d1a/d4a/cbe 0 2024-03-22T02:28:40.795 INFO:tasks.workunit.client.0.smithi103.stdout:7/490: mkdir d1/d10/d11/d70/d8a/d9d 0 2024-03-22T02:28:40.796 INFO:tasks.workunit.client.0.smithi103.stdout:6/392: symlink d0/d3/d22/d58/d6f/d75/l77 0 2024-03-22T02:28:40.797 INFO:tasks.workunit.client.0.smithi103.stdout:8/605: symlink d4/d26/d27/d7e/d39/d49/d5f/lbf 0 2024-03-22T02:28:40.799 INFO:tasks.workunit.client.0.smithi103.stdout:2/461: creat d11/d25/d63/d45/f94 x:0 0 0 2024-03-22T02:28:40.800 INFO:tasks.workunit.client.0.smithi103.stdout:6/393: read d0/f5b [362852,129095] 0 2024-03-22T02:28:40.803 INFO:tasks.workunit.client.0.smithi103.stdout:7/491: dwrite d1/f7e [0,4194304] 0 2024-03-22T02:28:40.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/394: dread d0/d14/f4c [0,4194304] 0 2024-03-22T02:28:40.806 INFO:tasks.workunit.client.0.smithi103.stdout:2/462: unlink d11/d25/d46/d40/l42 0 2024-03-22T02:28:40.807 INFO:tasks.workunit.client.0.smithi103.stdout:2/463: fdatasync d11/f30 0 2024-03-22T02:28:40.808 INFO:tasks.workunit.client.0.smithi103.stdout:7/492: creat d1/d10/d11/d19/f9e x:0 0 0 2024-03-22T02:28:40.811 INFO:tasks.workunit.client.0.smithi103.stdout:2/464: mknod d11/d25/d63/d45/d47/c95 0 2024-03-22T02:28:40.813 INFO:tasks.workunit.client.0.smithi103.stdout:6/395: creat d0/d3/d33/f78 x:0 0 0 2024-03-22T02:28:40.814 INFO:tasks.workunit.client.0.smithi103.stdout:7/493: creat d1/d10/d11/f9f x:0 0 0 2024-03-22T02:28:40.816 INFO:tasks.workunit.client.0.smithi103.stdout:6/396: creat d0/d3/d22/d58/f79 x:0 0 0 2024-03-22T02:28:40.817 INFO:tasks.workunit.client.0.smithi103.stdout:6/397: stat d0/d14/c2f 0 2024-03-22T02:28:40.819 INFO:tasks.workunit.client.0.smithi103.stdout:6/398: rmdir d0/d14/d57/d68 39 2024-03-22T02:28:40.822 INFO:tasks.workunit.client.0.smithi103.stdout:6/399: write d0/d3/f39 [2377564,94671] 0 2024-03-22T02:28:40.825 INFO:tasks.workunit.client.0.smithi103.stdout:7/494: dwrite d1/d10/d11/d19/d29/d61/d69/f84 [0,4194304] 0 2024-03-22T02:28:40.826 INFO:tasks.workunit.client.0.smithi103.stdout:8/606: dread f1 [4194304,4194304] 0 2024-03-22T02:28:40.828 INFO:tasks.workunit.client.0.smithi103.stdout:7/495: creat d1/da/d2f/fa0 x:0 0 0 2024-03-22T02:28:40.829 INFO:tasks.workunit.client.0.smithi103.stdout:6/400: symlink d0/d14/d57/l7a 0 2024-03-22T02:28:40.831 INFO:tasks.workunit.client.0.smithi103.stdout:7/496: creat d1/d10/d11/d19/d29/d61/d69/fa1 x:0 0 0 2024-03-22T02:28:40.832 INFO:tasks.workunit.client.0.smithi103.stdout:6/401: readlink d0/d3/d33/l4d 0 2024-03-22T02:28:40.836 INFO:tasks.workunit.client.0.smithi103.stdout:8/607: dwrite d4/d26/d27/d7e/d39/d49/fb5 [0,4194304] 0 2024-03-22T02:28:40.837 INFO:tasks.workunit.client.0.smithi103.stdout:7/497: mknod d1/d10/d11/d19/d29/d61/d69/ca2 0 2024-03-22T02:28:40.839 INFO:tasks.workunit.client.0.smithi103.stdout:6/402: creat d0/d3/d22/d58/d6f/d75/f7b x:0 0 0 2024-03-22T02:28:40.839 INFO:tasks.workunit.client.0.smithi103.stdout:8/608: symlink d4/d48/d35/lc0 0 2024-03-22T02:28:40.841 INFO:tasks.workunit.client.0.smithi103.stdout:6/403: read - d0/d14/d57/d68/f71 zero size 2024-03-22T02:28:40.847 INFO:tasks.workunit.client.0.smithi103.stdout:6/404: creat d0/d3/d33/f7c x:0 0 0 2024-03-22T02:28:40.847 INFO:tasks.workunit.client.0.smithi103.stdout:8/609: link d4/d2a/d8b/l99 d4/d2a/d3e/da4/lc1 0 2024-03-22T02:28:40.848 INFO:tasks.workunit.client.0.smithi103.stdout:8/610: stat d4/d26/d27/d7e/f52 0 2024-03-22T02:28:40.867 INFO:tasks.workunit.client.0.smithi103.stdout:6/405: write d0/d3/d22/f1c [797517,91034] 0 2024-03-22T02:28:40.868 INFO:tasks.workunit.client.0.smithi103.stdout:8/611: truncate d4/d26/f81 501634 0 2024-03-22T02:28:40.873 INFO:tasks.workunit.client.0.smithi103.stdout:7/498: dread d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:40.875 INFO:tasks.workunit.client.0.smithi103.stdout:8/612: dwrite d4/d19/d1a/fb3 [0,4194304] 0 2024-03-22T02:28:40.878 INFO:tasks.workunit.client.0.smithi103.stdout:8/613: write d4/d26/d27/d7e/d39/d49/d5f/f60 [444999,15417] 0 2024-03-22T02:28:40.882 INFO:tasks.workunit.client.0.smithi103.stdout:8/614: mkdir d4/d26/d27/d7e/d39/d49/d51/daf/dc2 0 2024-03-22T02:28:40.883 INFO:tasks.workunit.client.0.smithi103.stdout:7/499: write d1/d10/d11/d19/d29/d33/d4b/f97 [981985,20561] 0 2024-03-22T02:28:40.886 INFO:tasks.workunit.client.0.smithi103.stdout:8/615: mkdir d4/d19/d1a/dc3 0 2024-03-22T02:28:40.887 INFO:tasks.workunit.client.0.smithi103.stdout:8/616: truncate d4/d2a/d38/f6a 711507 0 2024-03-22T02:28:40.889 INFO:tasks.workunit.client.0.smithi103.stdout:8/617: mknod d4/d19/d1a/dc3/cc4 0 2024-03-22T02:28:40.891 INFO:tasks.workunit.client.0.smithi103.stdout:8/618: truncate d4/f9 5381765 0 2024-03-22T02:28:40.892 INFO:tasks.workunit.client.0.smithi103.stdout:7/500: write d1/da/fe [280065,87698] 0 2024-03-22T02:28:40.893 INFO:tasks.workunit.client.0.smithi103.stdout:8/619: mknod d4/d26/d27/d7e/d39/cc5 0 2024-03-22T02:28:40.895 INFO:tasks.workunit.client.0.smithi103.stdout:8/620: dread - d4/d2a/d3e/f6b zero size 2024-03-22T02:28:40.895 INFO:tasks.workunit.client.0.smithi103.stdout:7/501: mkdir d1/d10/d11/d19/d29/d61/d1d/da3 0 2024-03-22T02:28:40.899 INFO:tasks.workunit.client.0.smithi103.stdout:7/502: rename d1/d10/d11/d19/d29/d61/d69/l3d to d1/d10/d11/d19/d29/d61/d69/la4 0 2024-03-22T02:28:40.900 INFO:tasks.workunit.client.0.smithi103.stdout:7/503: chown d1/d10/d11/d19/d29/d33/d4b/c65 1717368029 1 2024-03-22T02:28:40.901 INFO:tasks.workunit.client.0.smithi103.stdout:8/621: rename d4/d2a/d4f/fac to d4/d26/d27/d7e/d39/d49/fc6 0 2024-03-22T02:28:40.901 INFO:tasks.workunit.client.0.smithi103.stdout:8/622: stat d4/d26/d27/d7e/d39/cc5 0 2024-03-22T02:28:40.904 INFO:tasks.workunit.client.0.smithi103.stdout:8/623: mkdir d4/d26/d27/d7e/d39/d7b/dc7 0 2024-03-22T02:28:40.905 INFO:tasks.workunit.client.0.smithi103.stdout:8/624: dread - d4/d26/d27/d7e/d39/d49/d79/fb9 zero size 2024-03-22T02:28:40.906 INFO:tasks.workunit.client.0.smithi103.stdout:7/504: truncate d1/da/d2f/d36/f91 1604649 0 2024-03-22T02:28:40.907 INFO:tasks.workunit.client.0.smithi103.stdout:7/505: fsync d1/f7e 0 2024-03-22T02:28:40.908 INFO:tasks.workunit.client.0.smithi103.stdout:8/625: mkdir d4/d26/d27/d7e/dc8 0 2024-03-22T02:28:40.910 INFO:tasks.workunit.client.0.smithi103.stdout:8/626: truncate d4/d26/d27/d7e/d39/d49/d79/fb9 345913 0 2024-03-22T02:28:40.910 INFO:tasks.workunit.client.0.smithi103.stdout:8/627: stat d4/d26/d27/f93 0 2024-03-22T02:28:40.911 INFO:tasks.workunit.client.0.smithi103.stdout:8/628: dread - d4/d48/d35/fa0 zero size 2024-03-22T02:28:40.912 INFO:tasks.workunit.client.0.smithi103.stdout:8/629: truncate d4/d19/f5c 10481 0 2024-03-22T02:28:40.916 INFO:tasks.workunit.client.0.smithi103.stdout:7/506: truncate d1/da/fe 536109 0 2024-03-22T02:28:40.917 INFO:tasks.workunit.client.0.smithi103.stdout:8/630: dwrite d4/f89 [0,4194304] 0 2024-03-22T02:28:40.919 INFO:tasks.workunit.client.0.smithi103.stdout:7/507: dread d1/f4e [4194304,4194304] 0 2024-03-22T02:28:40.922 INFO:tasks.workunit.client.0.smithi103.stdout:8/631: dread f1 [4194304,4194304] 0 2024-03-22T02:28:40.922 INFO:tasks.workunit.client.0.smithi103.stdout:7/508: chown d1/d10/d11/d70/l9c 6879 1 2024-03-22T02:28:40.923 INFO:tasks.workunit.client.0.smithi103.stdout:7/509: stat d1/d10/d11/d19/d29/d61/d1d 0 2024-03-22T02:28:40.924 INFO:tasks.workunit.client.0.smithi103.stdout:7/510: fsync d1/d10/d11/f5a 0 2024-03-22T02:28:40.926 INFO:tasks.workunit.client.0.smithi103.stdout:7/511: creat d1/d10/d11/d19/d29/d61/d69/fa5 x:0 0 0 2024-03-22T02:28:40.932 INFO:tasks.workunit.client.0.smithi103.stdout:7/512: dwrite d1/da/d1c/f8b [0,4194304] 0 2024-03-22T02:28:40.936 INFO:tasks.workunit.client.0.smithi103.stdout:7/513: getdents d1/d10/d11 0 2024-03-22T02:28:40.937 INFO:tasks.workunit.client.0.smithi103.stdout:7/514: readlink d1/d10/d11/l34 0 2024-03-22T02:28:40.942 INFO:tasks.workunit.client.0.smithi103.stdout:7/515: dread d1/d10/d11/d19/d29/d61/d69/f32 [0,4194304] 0 2024-03-22T02:28:40.948 INFO:tasks.workunit.client.0.smithi103.stdout:7/516: dread d1/da/d2f/f38 [0,4194304] 0 2024-03-22T02:28:40.949 INFO:tasks.workunit.client.0.smithi103.stdout:7/517: write d1/da/d2f/f38 [1834437,44465] 0 2024-03-22T02:28:40.951 INFO:tasks.workunit.client.0.smithi103.stdout:7/518: truncate d1/d10/d11/d19/d29/d61/d69/fa5 65706 0 2024-03-22T02:28:40.952 INFO:tasks.workunit.client.0.smithi103.stdout:7/519: mknod d1/d10/d11/d70/d8a/ca6 0 2024-03-22T02:28:40.953 INFO:tasks.workunit.client.0.smithi103.stdout:7/520: truncate d1/d10/d11/d19/d29/d61/d69/fa1 476898 0 2024-03-22T02:28:40.955 INFO:tasks.workunit.client.0.smithi103.stdout:7/521: truncate d1/d10/d11/d19/d29/d33/d4b/f83 2407173 0 2024-03-22T02:28:40.960 INFO:tasks.workunit.client.0.smithi103.stdout:7/522: dwrite d1/f76 [0,4194304] 0 2024-03-22T02:28:40.961 INFO:tasks.workunit.client.0.smithi103.stdout:7/523: stat d1/d6f/f8c 0 2024-03-22T02:28:40.961 INFO:tasks.workunit.client.0.smithi103.stdout:7/524: fdatasync d1/da/f75 0 2024-03-22T02:28:40.968 INFO:tasks.workunit.client.0.smithi103.stdout:7/525: dwrite d1/d10/d11/f5a [0,4194304] 0 2024-03-22T02:28:40.972 INFO:tasks.workunit.client.0.smithi103.stdout:7/526: rename d1/d10/d11/d19/d29/d33/d4b/f83 to d1/d10/d11/d19/d29/d61/d1d/fa7 0 2024-03-22T02:28:40.973 INFO:tasks.workunit.client.0.smithi103.stdout:7/527: dread - d1/da/d2f/fa0 zero size 2024-03-22T02:28:40.977 INFO:tasks.workunit.client.0.smithi103.stdout:7/528: truncate d1/da/f75 797520 0 2024-03-22T02:28:40.978 INFO:tasks.workunit.client.0.smithi103.stdout:7/529: chown d1/d10/d11/d19/d86/d92 495797990 1 2024-03-22T02:28:40.979 INFO:tasks.workunit.client.0.smithi103.stdout:7/530: fdatasync d1/da/d1c/f60 0 2024-03-22T02:28:40.981 INFO:tasks.workunit.client.0.smithi103.stdout:7/531: dread d1/d10/d11/d19/d29/d61/d69/fa1 [0,4194304] 0 2024-03-22T02:28:40.986 INFO:tasks.workunit.client.0.smithi103.stdout:7/532: dread d1/d10/d11/d19/d29/d61/d69/f32 [0,4194304] 0 2024-03-22T02:28:40.987 INFO:tasks.workunit.client.0.smithi103.stdout:7/533: write d1/d10/d11/d19/d29/d61/f7b [887629,15087] 0 2024-03-22T02:28:41.001 INFO:tasks.workunit.client.0.smithi103.stdout:7/534: dread d1/d10/d11/d19/d29/f52 [0,4194304] 0 2024-03-22T02:28:41.002 INFO:tasks.workunit.client.0.smithi103.stdout:7/535: mkdir d1/d10/d11/d19/d29/d33/da8 0 2024-03-22T02:28:41.004 INFO:tasks.workunit.client.0.smithi103.stdout:7/536: write d1/d10/d11/d19/d29/d61/f96 [2768013,18795] 0 2024-03-22T02:28:41.004 INFO:tasks.workunit.client.0.smithi103.stdout:7/537: fdatasync d1/d6f/f8c 0 2024-03-22T02:28:41.007 INFO:tasks.workunit.client.0.smithi103.stdout:7/538: creat d1/d10/d11/d19/d29/d61/d1d/fa9 x:0 0 0 2024-03-22T02:28:41.007 INFO:tasks.workunit.client.0.smithi103.stdout:7/539: chown d1/da/d2f/d36/f91 7 1 2024-03-22T02:28:41.009 INFO:tasks.workunit.client.0.smithi103.stdout:7/540: getdents d1/d10/d11/d19/d86/d92 0 2024-03-22T02:28:41.010 INFO:tasks.workunit.client.0.smithi103.stdout:7/541: stat d1/da/l58 0 2024-03-22T02:28:41.186 INFO:tasks.workunit.client.0.smithi103.stdout:1/518: sync 2024-03-22T02:28:41.187 INFO:tasks.workunit.client.0.smithi103.stdout:1/519: chown db/d10/d28/f43 6537187 1 2024-03-22T02:28:41.188 INFO:tasks.workunit.client.0.smithi103.stdout:1/520: mkdir db/d10/d28/db2 0 2024-03-22T02:28:41.189 INFO:tasks.workunit.client.0.smithi103.stdout:1/521: getdents db/d4a/d78/d92/da8 0 2024-03-22T02:28:41.190 INFO:tasks.workunit.client.0.smithi103.stdout:1/522: write db/d10/d29/d55/f6d [3383246,48833] 0 2024-03-22T02:28:41.194 INFO:tasks.workunit.client.0.smithi103.stdout:1/523: getdents db/d10/d29/d41/d68 0 2024-03-22T02:28:41.196 INFO:tasks.workunit.client.0.smithi103.stdout:1/524: creat db/d10/d28/d97/fb3 x:0 0 0 2024-03-22T02:28:41.197 INFO:tasks.workunit.client.0.smithi103.stdout:1/525: dread - db/d4a/fa0 zero size 2024-03-22T02:28:41.197 INFO:tasks.workunit.client.0.smithi103.stdout:1/526: write db/d10/d18/d2d/f16 [1909303,39764] 0 2024-03-22T02:28:41.200 INFO:tasks.workunit.client.0.smithi103.stdout:1/527: truncate db/f35 4397064 0 2024-03-22T02:28:41.214 INFO:tasks.workunit.client.0.smithi103.stdout:1/528: dread db/d10/d29/d55/f6d [0,4194304] 0 2024-03-22T02:28:41.216 INFO:tasks.workunit.client.0.smithi103.stdout:1/529: rmdir db/d10/d18/d84/dab 0 2024-03-22T02:28:41.217 INFO:tasks.workunit.client.0.smithi103.stdout:1/530: chown db/d10/d29/d55/f6d 543 1 2024-03-22T02:28:41.219 INFO:tasks.workunit.client.0.smithi103.stdout:1/531: creat db/d10/d18/d98/fb4 x:0 0 0 2024-03-22T02:28:41.222 INFO:tasks.workunit.client.0.smithi103.stdout:1/532: getdents db/d10/d18 0 2024-03-22T02:28:41.223 INFO:tasks.workunit.client.0.smithi103.stdout:1/533: write db/d10/d2e/f51 [1299623,91420] 0 2024-03-22T02:28:41.224 INFO:tasks.workunit.client.0.smithi103.stdout:1/534: fdatasync db/d10/d28/fa7 0 2024-03-22T02:28:41.226 INFO:tasks.workunit.client.0.smithi103.stdout:1/535: chown db/d10/d18/d2d/f48 33681 1 2024-03-22T02:28:41.227 INFO:tasks.workunit.client.0.smithi103.stdout:1/536: dread - db/d10/d18/d2d/fb0 zero size 2024-03-22T02:28:41.228 INFO:tasks.workunit.client.0.smithi103.stdout:1/537: truncate db/f2b 4343051 0 2024-03-22T02:28:41.229 INFO:tasks.workunit.client.0.smithi103.stdout:1/538: mkdir db/d10/d6b/db5 0 2024-03-22T02:28:41.230 INFO:tasks.workunit.client.0.smithi103.stdout:1/539: creat db/d10/d18/d84/fb6 x:0 0 0 2024-03-22T02:28:41.231 INFO:tasks.workunit.client.0.smithi103.stdout:1/540: truncate db/d10/d6b/f8d 714148 0 2024-03-22T02:28:41.233 INFO:tasks.workunit.client.0.smithi103.stdout:1/541: getdents db/d10/d2e 0 2024-03-22T02:28:41.234 INFO:tasks.workunit.client.0.smithi103.stdout:1/542: chown db/d10/d18/f37 15 1 2024-03-22T02:28:41.236 INFO:tasks.workunit.client.0.smithi103.stdout:1/543: mkdir db/d4a/d78/d92/db7 0 2024-03-22T02:28:41.241 INFO:tasks.workunit.client.0.smithi103.stdout:1/544: dwrite db/d4a/fa0 [0,4194304] 0 2024-03-22T02:28:41.242 INFO:tasks.workunit.client.0.smithi103.stdout:1/545: chown db/d10/d29/d41/d68 1948006 1 2024-03-22T02:28:41.243 INFO:tasks.workunit.client.0.smithi103.stdout:1/546: read - db/d10/d29/f39 zero size 2024-03-22T02:28:41.250 INFO:tasks.workunit.client.0.smithi103.stdout:1/547: dwrite db/d10/d6b/f7f [0,4194304] 0 2024-03-22T02:28:41.252 INFO:tasks.workunit.client.0.smithi103.stdout:1/548: write db/d10/d28/f43 [68330,53548] 0 2024-03-22T02:28:41.257 INFO:tasks.workunit.client.0.smithi103.stdout:1/549: dread db/d10/d29/d55/f6d [0,4194304] 0 2024-03-22T02:28:41.269 INFO:tasks.workunit.client.0.smithi103.stdout:1/550: dwrite db/d10/d18/f3b [4194304,4194304] 0 2024-03-22T02:28:41.270 INFO:tasks.workunit.client.0.smithi103.stdout:1/551: read db/d10/d29/d55/f6d [1283483,103152] 0 2024-03-22T02:28:41.274 INFO:tasks.workunit.client.0.smithi103.stdout:1/552: rename db/d10/d6b/f7f to db/d10/d29/fb8 0 2024-03-22T02:28:41.275 INFO:tasks.workunit.client.0.smithi103.stdout:6/406: sync 2024-03-22T02:28:41.275 INFO:tasks.workunit.client.0.smithi103.stdout:5/501: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:3/571: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:8/632: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:9/471: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:0/569: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:2/465: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:7/542: sync 2024-03-22T02:28:41.276 INFO:tasks.workunit.client.0.smithi103.stdout:4/454: sync 2024-03-22T02:28:41.277 INFO:tasks.workunit.client.0.smithi103.stdout:1/553: creat db/d10/d28/db2/fb9 x:0 0 0 2024-03-22T02:28:41.278 INFO:tasks.workunit.client.0.smithi103.stdout:8/633: stat d4/d19/d1a/lae 0 2024-03-22T02:28:41.279 INFO:tasks.workunit.client.0.smithi103.stdout:9/472: readlink d2/d10/l16 0 2024-03-22T02:28:41.280 INFO:tasks.workunit.client.0.smithi103.stdout:7/543: fdatasync d1/da/d1c/f8b 0 2024-03-22T02:28:41.280 INFO:tasks.workunit.client.0.smithi103.stdout:2/466: creat d11/d25/d63/f96 x:0 0 0 2024-03-22T02:28:41.281 INFO:tasks.workunit.client.0.smithi103.stdout:6/407: mkdir d0/d3/d33/d7d 0 2024-03-22T02:28:41.282 INFO:tasks.workunit.client.0.smithi103.stdout:0/570: stat dd/d16/d4b/d96/d29/d48/d5c 0 2024-03-22T02:28:41.283 INFO:tasks.workunit.client.0.smithi103.stdout:5/502: stat d0/d14/d23/d2a/d1f/c29 0 2024-03-22T02:28:41.286 INFO:tasks.workunit.client.0.smithi103.stdout:3/572: dread de/d1a/d25/d26/f6f [0,4194304] 0 2024-03-22T02:28:41.288 INFO:tasks.workunit.client.0.smithi103.stdout:2/467: creat d11/d25/d46/d48/f97 x:0 0 0 2024-03-22T02:28:41.289 INFO:tasks.workunit.client.0.smithi103.stdout:4/455: mkdir d3/d1e/d25/d65/d83/da1 0 2024-03-22T02:28:41.291 INFO:tasks.workunit.client.0.smithi103.stdout:2/468: write d11/d19/d7e/f93 [379278,69212] 0 2024-03-22T02:28:41.293 INFO:tasks.workunit.client.0.smithi103.stdout:0/571: fsync fc 0 2024-03-22T02:28:41.293 INFO:tasks.workunit.client.0.smithi103.stdout:9/473: dwrite d2/f9 [0,4194304] 0 2024-03-22T02:28:41.294 INFO:tasks.workunit.client.0.smithi103.stdout:5/503: mkdir d0/d14/d23/d2a/d2c/da9 0 2024-03-22T02:28:41.299 INFO:tasks.workunit.client.0.smithi103.stdout:6/408: rmdir d0/d3/d33/d7d 0 2024-03-22T02:28:41.300 INFO:tasks.workunit.client.0.smithi103.stdout:9/474: write d2/d10/d43/f7a [977457,99034] 0 2024-03-22T02:28:41.302 INFO:tasks.workunit.client.0.smithi103.stdout:3/573: dwrite de/d1a/d25/f8b [0,4194304] 0 2024-03-22T02:28:41.302 INFO:tasks.workunit.client.0.smithi103.stdout:2/469: fsync d11/d19/d5d/f92 0 2024-03-22T02:28:41.303 INFO:tasks.workunit.client.0.smithi103.stdout:1/554: getdents db/d4a/d78/d92 0 2024-03-22T02:28:41.306 INFO:tasks.workunit.client.0.smithi103.stdout:0/572: mknod dd/d16/d4b/d96/d31/d89/cbf 0 2024-03-22T02:28:41.306 INFO:tasks.workunit.client.0.smithi103.stdout:1/555: dread db/d4a/f65 [0,4194304] 0 2024-03-22T02:28:41.307 INFO:tasks.workunit.client.0.smithi103.stdout:8/634: truncate d4/d26/d27/f71 621236 0 2024-03-22T02:28:41.310 INFO:tasks.workunit.client.0.smithi103.stdout:7/544: truncate d1/d10/d11/d19/d29/f52 1573430 0 2024-03-22T02:28:41.311 INFO:tasks.workunit.client.0.smithi103.stdout:2/470: creat d11/d19/d7e/f98 x:0 0 0 2024-03-22T02:28:41.313 INFO:tasks.workunit.client.0.smithi103.stdout:3/574: unlink de/d12/d19/c67 0 2024-03-22T02:28:41.314 INFO:tasks.workunit.client.0.smithi103.stdout:5/504: getdents d0/d14/d23/d36/d95/d96 0 2024-03-22T02:28:41.314 INFO:tasks.workunit.client.0.smithi103.stdout:4/456: dwrite d3/f7 [0,4194304] 0 2024-03-22T02:28:41.316 INFO:tasks.workunit.client.0.smithi103.stdout:5/505: chown d0/d14/d23/d8c 134258208 1 2024-03-22T02:28:41.316 INFO:tasks.workunit.client.0.smithi103.stdout:4/457: dread - d3/fa0 zero size 2024-03-22T02:28:41.318 INFO:tasks.workunit.client.0.smithi103.stdout:8/635: unlink d4/d48/fb4 0 2024-03-22T02:28:41.319 INFO:tasks.workunit.client.0.smithi103.stdout:4/458: write d3/fb [1145576,91133] 0 2024-03-22T02:28:41.320 INFO:tasks.workunit.client.0.smithi103.stdout:1/556: mkdir db/d10/d29/d41/d68/d74/dba 0 2024-03-22T02:28:41.323 INFO:tasks.workunit.client.0.smithi103.stdout:3/575: mknod de/d1a/d25/d3f/d9c/da7/cbc 0 2024-03-22T02:28:41.325 INFO:tasks.workunit.client.0.smithi103.stdout:4/459: dread d3/d69/f46 [0,4194304] 0 2024-03-22T02:28:41.326 INFO:tasks.workunit.client.0.smithi103.stdout:7/545: rename d1/d10/d11/d19/d29/c3e to d1/d10/d11/d19/d29/d61/d1d/da3/caa 0 2024-03-22T02:28:41.327 INFO:tasks.workunit.client.0.smithi103.stdout:5/506: fdatasync d0/d14/d23/d2a/d1f/f38 0 2024-03-22T02:28:41.329 INFO:tasks.workunit.client.0.smithi103.stdout:8/636: mknod d4/d2a/d3e/da4/cc9 0 2024-03-22T02:28:41.329 INFO:tasks.workunit.client.0.smithi103.stdout:6/409: link d0/d14/c17 d0/d3/d22/c7e 0 2024-03-22T02:28:41.331 INFO:tasks.workunit.client.0.smithi103.stdout:6/410: write d0/d3/d22/f3c [5490113,60938] 0 2024-03-22T02:28:41.332 INFO:tasks.workunit.client.0.smithi103.stdout:6/411: fdatasync d0/d14/f56 0 2024-03-22T02:28:41.333 INFO:tasks.workunit.client.0.smithi103.stdout:9/475: getdents d2/d10 0 2024-03-22T02:28:41.333 INFO:tasks.workunit.client.0.smithi103.stdout:1/557: mknod db/d10/d18/d84/cbb 0 2024-03-22T02:28:41.334 INFO:tasks.workunit.client.0.smithi103.stdout:6/412: write d0/d14/d57/d68/f62 [520154,75402] 0 2024-03-22T02:28:41.335 INFO:tasks.workunit.client.0.smithi103.stdout:1/558: chown db/d10/d18/f37 5147 1 2024-03-22T02:28:41.336 INFO:tasks.workunit.client.0.smithi103.stdout:0/573: creat dd/d34/d53/d8c/fc0 x:0 0 0 2024-03-22T02:28:41.337 INFO:tasks.workunit.client.0.smithi103.stdout:1/559: chown db/d10/d29/d55/f6d 12 1 2024-03-22T02:28:41.338 INFO:tasks.workunit.client.0.smithi103.stdout:1/560: write db/d10/d29/d41/d68/fa3 [315151,8074] 0 2024-03-22T02:28:41.345 INFO:tasks.workunit.client.0.smithi103.stdout:9/476: creat d2/da/d18/d1e/d34/d3d/d4d/f8f x:0 0 0 2024-03-22T02:28:41.346 INFO:tasks.workunit.client.0.smithi103.stdout:9/477: fsync d2/da/d18/d1e/d34/d3d/d61/f87 0 2024-03-22T02:28:41.350 INFO:tasks.workunit.client.0.smithi103.stdout:0/574: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/fc1 x:0 0 0 2024-03-22T02:28:41.350 INFO:tasks.workunit.client.0.smithi103.stdout:5/507: mkdir d0/d14/d23/d36/d95/d96/daa 0 2024-03-22T02:28:41.350 INFO:tasks.workunit.client.0.smithi103.stdout:3/576: rename de/d12/f86 to de/d1a/fbd 0 2024-03-22T02:28:41.350 INFO:tasks.workunit.client.0.smithi103.stdout:9/478: fsync d2/d10/d27/f51 0 2024-03-22T02:28:41.352 INFO:tasks.workunit.client.0.smithi103.stdout:8/637: rename d4/d19/d1a/f2d to d4/d19/d1a/fca 0 2024-03-22T02:28:41.353 INFO:tasks.workunit.client.0.smithi103.stdout:3/577: write de/d12/d19/f28 [2523576,45751] 0 2024-03-22T02:28:41.356 INFO:tasks.workunit.client.0.smithi103.stdout:3/578: chown de/d12/d4d/cb7 40329 1 2024-03-22T02:28:41.356 INFO:tasks.workunit.client.0.smithi103.stdout:1/561: rename db/d10/d29/c44 to db/d10/d29/d41/d68/cbc 0 2024-03-22T02:28:41.357 INFO:tasks.workunit.client.0.smithi103.stdout:8/638: truncate d4/d26/d27/d7e/fad 895820 0 2024-03-22T02:28:41.358 INFO:tasks.workunit.client.0.smithi103.stdout:0/575: dwrite dd/d16/d4b/d96/d64/d91/f99 [0,4194304] 0 2024-03-22T02:28:41.358 INFO:tasks.workunit.client.0.smithi103.stdout:3/579: readlink de/d1a/lb3 0 2024-03-22T02:28:41.358 INFO:tasks.workunit.client.0.smithi103.stdout:8/639: stat d4/d2a/c32 0 2024-03-22T02:28:41.361 INFO:tasks.workunit.client.0.smithi103.stdout:8/640: fdatasync d4/d26/d27/d7e/d39/d49/d79/fb9 0 2024-03-22T02:28:41.363 INFO:tasks.workunit.client.0.smithi103.stdout:1/562: creat db/d10/d29/d55/fbd x:0 0 0 2024-03-22T02:28:41.363 INFO:tasks.workunit.client.0.smithi103.stdout:3/580: creat de/d12/d8a/db0/fbe x:0 0 0 2024-03-22T02:28:41.365 INFO:tasks.workunit.client.0.smithi103.stdout:5/508: dread d0/fe [4194304,4194304] 0 2024-03-22T02:28:41.367 INFO:tasks.workunit.client.0.smithi103.stdout:8/641: mknod d4/d26/d27/d7e/d39/d49/d51/ccb 0 2024-03-22T02:28:41.367 INFO:tasks.workunit.client.0.smithi103.stdout:1/563: creat db/d10/d18/d2d/fbe x:0 0 0 2024-03-22T02:28:41.369 INFO:tasks.workunit.client.0.smithi103.stdout:3/581: rename de/d12/d19/f48 to de/d12/d19/fbf 0 2024-03-22T02:28:41.370 INFO:tasks.workunit.client.0.smithi103.stdout:6/413: dread d0/d14/f56 [0,4194304] 0 2024-03-22T02:28:41.371 INFO:tasks.workunit.client.0.smithi103.stdout:3/582: fdatasync de/d12/d30/f6e 0 2024-03-22T02:28:41.372 INFO:tasks.workunit.client.0.smithi103.stdout:5/509: creat d0/d14/d23/d8c/fab x:0 0 0 2024-03-22T02:28:41.374 INFO:tasks.workunit.client.0.smithi103.stdout:8/642: mknod d4/d26/d27/d7e/d67/ccc 0 2024-03-22T02:28:41.379 INFO:tasks.workunit.client.0.smithi103.stdout:5/510: mkdir d0/d14/d23/d36/d95/d96/dac 0 2024-03-22T02:28:41.381 INFO:tasks.workunit.client.0.smithi103.stdout:4/460: truncate d3/d69/f46 229812 0 2024-03-22T02:28:41.383 INFO:tasks.workunit.client.0.smithi103.stdout:4/461: chown d3/d1e/f7f 14 1 2024-03-22T02:28:41.383 INFO:tasks.workunit.client.0.smithi103.stdout:7/546: dwrite d1/f4e [0,4194304] 0 2024-03-22T02:28:41.386 INFO:tasks.workunit.client.0.smithi103.stdout:7/547: chown d1/da/d1c/f8b 35 1 2024-03-22T02:28:41.387 INFO:tasks.workunit.client.0.smithi103.stdout:7/548: fsync d1/da/f3 0 2024-03-22T02:28:41.389 INFO:tasks.workunit.client.0.smithi103.stdout:5/511: dwrite d0/d3e/f9e [0,4194304] 0 2024-03-22T02:28:41.390 INFO:tasks.workunit.client.0.smithi103.stdout:7/549: read - d1/da/d2f/fa0 zero size 2024-03-22T02:28:41.397 INFO:tasks.workunit.client.0.smithi103.stdout:8/643: rmdir d4/d19 39 2024-03-22T02:28:41.400 INFO:tasks.workunit.client.0.smithi103.stdout:4/462: mkdir d3/da2 0 2024-03-22T02:28:41.401 INFO:tasks.workunit.client.0.smithi103.stdout:6/414: getdents d0/d3/d22/d58/d6f/d75 0 2024-03-22T02:28:41.405 INFO:tasks.workunit.client.0.smithi103.stdout:7/550: mknod d1/da/cab 0 2024-03-22T02:28:41.408 INFO:tasks.workunit.client.0.smithi103.stdout:5/512: creat d0/fad x:0 0 0 2024-03-22T02:28:41.409 INFO:tasks.workunit.client.0.smithi103.stdout:0/576: dwrite dd/f1d [0,4194304] 0 2024-03-22T02:28:41.411 INFO:tasks.workunit.client.0.smithi103.stdout:0/577: chown dd/d16/d4b/d96/d29/d48/c61 0 1 2024-03-22T02:28:41.412 INFO:tasks.workunit.client.0.smithi103.stdout:6/415: mknod d0/d3/c7f 0 2024-03-22T02:28:41.417 INFO:tasks.workunit.client.0.smithi103.stdout:3/583: write de/d6b/d80/d4f/d84/f88 [1022663,111366] 0 2024-03-22T02:28:41.418 INFO:tasks.workunit.client.0.smithi103.stdout:6/416: chown d0/f3e 9486 1 2024-03-22T02:28:41.418 INFO:tasks.workunit.client.0.smithi103.stdout:8/644: link d4/c56 d4/ccd 0 2024-03-22T02:28:41.421 INFO:tasks.workunit.client.0.smithi103.stdout:5/513: mknod d0/cae 0 2024-03-22T02:28:41.423 INFO:tasks.workunit.client.0.smithi103.stdout:7/551: creat d1/d10/d11/d19/d29/d61/d69/d6b/fac x:0 0 0 2024-03-22T02:28:41.426 INFO:tasks.workunit.client.0.smithi103.stdout:1/564: dwrite db/f17 [0,4194304] 0 2024-03-22T02:28:41.429 INFO:tasks.workunit.client.0.smithi103.stdout:6/417: creat d0/d3/d22/d58/f80 x:0 0 0 2024-03-22T02:28:41.430 INFO:tasks.workunit.client.0.smithi103.stdout:4/463: write d3/f11 [1950332,38949] 0 2024-03-22T02:28:41.431 INFO:tasks.workunit.client.0.smithi103.stdout:0/578: dwrite dd/d16/d4b/d96/d29/d48/f73 [0,4194304] 0 2024-03-22T02:28:41.434 INFO:tasks.workunit.client.0.smithi103.stdout:8/645: mknod d4/d19/d1a/d4a/cce 0 2024-03-22T02:28:41.434 INFO:tasks.workunit.client.0.smithi103.stdout:1/565: mknod db/d4a/d78/cbf 0 2024-03-22T02:28:41.436 INFO:tasks.workunit.client.0.smithi103.stdout:7/552: mkdir d1/d10/d11/d70/d8a/d9d/dad 0 2024-03-22T02:28:41.437 INFO:tasks.workunit.client.0.smithi103.stdout:3/584: dwrite de/d12/d30/f63 [0,4194304] 0 2024-03-22T02:28:41.437 INFO:tasks.workunit.client.0.smithi103.stdout:5/514: dwrite d0/f1e [4194304,4194304] 0 2024-03-22T02:28:41.441 INFO:tasks.workunit.client.0.smithi103.stdout:6/418: rename d0/d14/d57/d68/l76 to d0/d3/d22/d58/l81 0 2024-03-22T02:28:41.441 INFO:tasks.workunit.client.0.smithi103.stdout:3/585: chown de/d6b/d80/d4f/f5b 2486 1 2024-03-22T02:28:41.444 INFO:tasks.workunit.client.0.smithi103.stdout:8/646: creat d4/d2a/d3e/fcf x:0 0 0 2024-03-22T02:28:41.448 INFO:tasks.workunit.client.0.smithi103.stdout:7/553: creat d1/d6f/fae x:0 0 0 2024-03-22T02:28:41.448 INFO:tasks.workunit.client.0.smithi103.stdout:6/419: dread d0/d14/f63 [0,4194304] 0 2024-03-22T02:28:41.449 INFO:tasks.workunit.client.0.smithi103.stdout:1/566: creat db/d10/d18/d84/fc0 x:0 0 0 2024-03-22T02:28:41.449 INFO:tasks.workunit.client.0.smithi103.stdout:6/420: readlink d0/d3/d33/l4d 0 2024-03-22T02:28:41.451 INFO:tasks.workunit.client.0.smithi103.stdout:7/554: stat d1/d10/d11/d19/d29/d61/d69/d6b 0 2024-03-22T02:28:41.451 INFO:tasks.workunit.client.0.smithi103.stdout:1/567: dread - db/d10/d18/d2d/fb0 zero size 2024-03-22T02:28:41.453 INFO:tasks.workunit.client.0.smithi103.stdout:0/579: symlink dd/d16/d4b/d96/d31/lc2 0 2024-03-22T02:28:41.453 INFO:tasks.workunit.client.0.smithi103.stdout:6/421: write d0/d3/f1b [1512588,107223] 0 2024-03-22T02:28:41.454 INFO:tasks.workunit.client.0.smithi103.stdout:5/515: rename d0/d14/d23/d8c to d0/d14/d23/d36/d95/d96/daa/daf 0 2024-03-22T02:28:41.456 INFO:tasks.workunit.client.0.smithi103.stdout:7/555: dread - d1/d10/d11/d19/f9e zero size 2024-03-22T02:28:41.456 INFO:tasks.workunit.client.0.smithi103.stdout:3/586: symlink de/d1a/d25/d26/lc0 0 2024-03-22T02:28:41.456 INFO:tasks.workunit.client.0.smithi103.stdout:0/580: fdatasync dd/d16/d4b/d96/d64/fa3 0 2024-03-22T02:28:41.456 INFO:tasks.workunit.client.0.smithi103.stdout:8/647: creat d4/d26/d27/d7e/d39/d7b/fd0 x:0 0 0 2024-03-22T02:28:41.458 INFO:tasks.workunit.client.0.smithi103.stdout:3/587: write fa [4820711,42837] 0 2024-03-22T02:28:41.459 INFO:tasks.workunit.client.0.smithi103.stdout:1/568: creat db/d10/d29/d41/d68/d74/fc1 x:0 0 0 2024-03-22T02:28:41.459 INFO:tasks.workunit.client.0.smithi103.stdout:0/581: readlink dd/d16/d4b/d96/d29/d48/lb9 0 2024-03-22T02:28:41.460 INFO:tasks.workunit.client.0.smithi103.stdout:6/422: symlink d0/d14/l82 0 2024-03-22T02:28:41.462 INFO:tasks.workunit.client.0.smithi103.stdout:4/464: dread d3/f1f [0,4194304] 0 2024-03-22T02:28:41.462 INFO:tasks.workunit.client.0.smithi103.stdout:5/516: creat d0/d14/d23/d36/d95/d96/fb0 x:0 0 0 2024-03-22T02:28:41.463 INFO:tasks.workunit.client.0.smithi103.stdout:5/517: chown d0/d14/d23/d2a/d33 20013 1 2024-03-22T02:28:41.465 INFO:tasks.workunit.client.0.smithi103.stdout:0/582: dread - dd/d34/f43 zero size 2024-03-22T02:28:41.465 INFO:tasks.workunit.client.0.smithi103.stdout:5/518: write d0/d14/d23/d6f/f86 [2240625,47882] 0 2024-03-22T02:28:41.465 INFO:tasks.workunit.client.0.smithi103.stdout:1/569: symlink db/d10/d18/d2d/lc2 0 2024-03-22T02:28:41.468 INFO:tasks.workunit.client.0.smithi103.stdout:7/556: symlink d1/da/d2f/laf 0 2024-03-22T02:28:41.469 INFO:tasks.workunit.client.0.smithi103.stdout:1/570: write f5 [2119572,116205] 0 2024-03-22T02:28:41.471 INFO:tasks.workunit.client.0.smithi103.stdout:8/648: symlink d4/d26/d27/d7e/d39/ld1 0 2024-03-22T02:28:41.471 INFO:tasks.workunit.client.0.smithi103.stdout:4/465: mkdir d3/d1e/d53/d7a/da3 0 2024-03-22T02:28:41.474 INFO:tasks.workunit.client.0.smithi103.stdout:1/571: readlink db/d10/d18/l1d 0 2024-03-22T02:28:41.475 INFO:tasks.workunit.client.0.smithi103.stdout:8/649: chown d4/c10 24671 1 2024-03-22T02:28:41.481 INFO:tasks.workunit.client.0.smithi103.stdout:0/583: dwrite dd/d16/d4b/d96/d29/f83 [4194304,4194304] 0 2024-03-22T02:28:41.481 INFO:tasks.workunit.client.0.smithi103.stdout:1/572: readlink db/l66 0 2024-03-22T02:28:41.482 INFO:tasks.workunit.client.0.smithi103.stdout:5/519: dwrite d0/f2d [4194304,4194304] 0 2024-03-22T02:28:41.485 INFO:tasks.workunit.client.0.smithi103.stdout:8/650: mknod d4/d2a/d3e/da4/cd2 0 2024-03-22T02:28:41.492 INFO:tasks.workunit.client.0.smithi103.stdout:7/557: getdents d1/d10/d11/d19/d29/d61/d69 0 2024-03-22T02:28:41.495 INFO:tasks.workunit.client.0.smithi103.stdout:7/558: readlink d1/d10/d11/d19/d29/d61/d69/la4 0 2024-03-22T02:28:41.496 INFO:tasks.workunit.client.0.smithi103.stdout:8/651: mkdir d4/d26/d27/d7e/d39/d49/d5f/dd3 0 2024-03-22T02:28:41.497 INFO:tasks.workunit.client.0.smithi103.stdout:4/466: dread d3/fd [0,4194304] 0 2024-03-22T02:28:41.498 INFO:tasks.workunit.client.0.smithi103.stdout:8/652: chown d4/d19/d8c/c96 15406 1 2024-03-22T02:28:41.500 INFO:tasks.workunit.client.0.smithi103.stdout:1/573: creat db/d10/d6b/db5/fc3 x:0 0 0 2024-03-22T02:28:41.501 INFO:tasks.workunit.client.0.smithi103.stdout:7/559: write d1/d10/d11/d19/d29/d61/f24 [4796284,46844] 0 2024-03-22T02:28:41.502 INFO:tasks.workunit.client.0.smithi103.stdout:6/423: dread d0/d3/d33/f5d [0,4194304] 0 2024-03-22T02:28:41.502 INFO:tasks.workunit.client.0.smithi103.stdout:2/471: sync 2024-03-22T02:28:41.506 INFO:tasks.workunit.client.0.smithi103.stdout:2/472: write d11/d25/d46/d48/f97 [866514,37968] 0 2024-03-22T02:28:41.506 INFO:tasks.workunit.client.0.smithi103.stdout:1/574: creat db/d10/d18/d84/fc4 x:0 0 0 2024-03-22T02:28:41.507 INFO:tasks.workunit.client.0.smithi103.stdout:5/520: getdents d0/d14/d23/d6f 0 2024-03-22T02:28:41.510 INFO:tasks.workunit.client.0.smithi103.stdout:1/575: fsync db/d10/d2e/f8a 0 2024-03-22T02:28:41.510 INFO:tasks.workunit.client.0.smithi103.stdout:7/560: creat d1/d10/d11/d19/d29/d57/fb0 x:0 0 0 2024-03-22T02:28:41.510 INFO:tasks.workunit.client.0.smithi103.stdout:6/424: dread d0/d14/d57/d68/f2d [0,4194304] 0 2024-03-22T02:28:41.510 INFO:tasks.workunit.client.0.smithi103.stdout:2/473: write d11/d19/d5d/f75 [114238,28958] 0 2024-03-22T02:28:41.510 INFO:tasks.workunit.client.0.smithi103.stdout:3/588: write de/f1f [1737924,82754] 0 2024-03-22T02:28:41.512 INFO:tasks.workunit.client.0.smithi103.stdout:5/521: write d0/d14/d23/d46/f7a [4763396,121828] 0 2024-03-22T02:28:41.515 INFO:tasks.workunit.client.0.smithi103.stdout:3/589: write de/d12/d8a/d93/fac [99057,92587] 0 2024-03-22T02:28:41.517 INFO:tasks.workunit.client.0.smithi103.stdout:2/474: truncate d11/d25/d46/d40/f67 4552946 0 2024-03-22T02:28:41.517 INFO:tasks.workunit.client.0.smithi103.stdout:3/590: readlink de/d12/d8a/db0/l9f 0 2024-03-22T02:28:41.518 INFO:tasks.workunit.client.0.smithi103.stdout:8/653: rename d4/d26/d27/d7e/d67/l85 to d4/d26/d27/ld4 0 2024-03-22T02:28:41.519 INFO:tasks.workunit.client.0.smithi103.stdout:1/576: creat db/d4a/d78/d92/fc5 x:0 0 0 2024-03-22T02:28:41.520 INFO:tasks.workunit.client.0.smithi103.stdout:4/467: getdents d3/d1e/d53/d7a 0 2024-03-22T02:28:41.521 INFO:tasks.workunit.client.0.smithi103.stdout:0/584: dread dd/d16/d4b/d96/d31/f58 [0,4194304] 0 2024-03-22T02:28:41.521 INFO:tasks.workunit.client.0.smithi103.stdout:1/577: write db/d10/d29/d41/f99 [428161,123332] 0 2024-03-22T02:28:41.521 INFO:tasks.workunit.client.0.smithi103.stdout:5/522: creat d0/d14/d23/d36/d95/fb1 x:0 0 0 2024-03-22T02:28:41.523 INFO:tasks.workunit.client.0.smithi103.stdout:2/475: chown d11/d25/d46/d34/c3c 5461666 1 2024-03-22T02:28:41.524 INFO:tasks.workunit.client.0.smithi103.stdout:3/591: rmdir de/d1a/d25/d3f/d9c 39 2024-03-22T02:28:41.527 INFO:tasks.workunit.client.0.smithi103.stdout:4/468: symlink d3/d38/d5c/la4 0 2024-03-22T02:28:41.527 INFO:tasks.workunit.client.0.smithi103.stdout:3/592: write de/d12/d19/f28 [3682176,62386] 0 2024-03-22T02:28:41.528 INFO:tasks.workunit.client.0.smithi103.stdout:7/561: creat d1/da/d2f/fb1 x:0 0 0 2024-03-22T02:28:41.529 INFO:tasks.workunit.client.0.smithi103.stdout:0/585: dread dd/d16/d4b/d96/d31/f58 [0,4194304] 0 2024-03-22T02:28:41.531 INFO:tasks.workunit.client.0.smithi103.stdout:0/586: write dd/d34/d53/fa1 [778464,84587] 0 2024-03-22T02:28:41.534 INFO:tasks.workunit.client.0.smithi103.stdout:5/523: creat d0/da2/fb2 x:0 0 0 2024-03-22T02:28:41.536 INFO:tasks.workunit.client.0.smithi103.stdout:3/593: symlink de/d12/d30/d70/lc1 0 2024-03-22T02:28:41.537 INFO:tasks.workunit.client.0.smithi103.stdout:1/578: dread db/f2b [0,4194304] 0 2024-03-22T02:28:41.537 INFO:tasks.workunit.client.0.smithi103.stdout:1/579: chown db/d10/d21/f33 3 1 2024-03-22T02:28:41.538 INFO:tasks.workunit.client.0.smithi103.stdout:5/524: creat d0/da2/fb3 x:0 0 0 2024-03-22T02:28:41.539 INFO:tasks.workunit.client.0.smithi103.stdout:5/525: fsync d0/d14/d23/fa3 0 2024-03-22T02:28:41.540 INFO:tasks.workunit.client.0.smithi103.stdout:7/562: symlink d1/d10/d11/d19/d29/d33/d4b/lb2 0 2024-03-22T02:28:41.541 INFO:tasks.workunit.client.0.smithi103.stdout:7/563: chown d1/l87 175027 1 2024-03-22T02:28:41.541 INFO:tasks.workunit.client.0.smithi103.stdout:2/476: link d11/d25/d46/d40/f6f d11/d25/d46/d40/d90/f99 0 2024-03-22T02:28:41.542 INFO:tasks.workunit.client.0.smithi103.stdout:8/654: write d4/d19/d8c/f8f [1046332,105746] 0 2024-03-22T02:28:41.544 INFO:tasks.workunit.client.0.smithi103.stdout:5/526: mknod d0/da2/cb4 0 2024-03-22T02:28:41.544 INFO:tasks.workunit.client.0.smithi103.stdout:8/655: dread d4/d19/f5c [0,4194304] 0 2024-03-22T02:28:41.547 INFO:tasks.workunit.client.0.smithi103.stdout:6/425: truncate d0/d3/f39 2687057 0 2024-03-22T02:28:41.548 INFO:tasks.workunit.client.0.smithi103.stdout:1/580: link db/f9e db/d10/d29/d41/d68/d74/dba/fc6 0 2024-03-22T02:28:41.549 INFO:tasks.workunit.client.0.smithi103.stdout:2/477: symlink d11/d25/d46/d34/l9a 0 2024-03-22T02:28:41.549 INFO:tasks.workunit.client.0.smithi103.stdout:1/581: chown db/d4a/f9c 6 1 2024-03-22T02:28:41.550 INFO:tasks.workunit.client.0.smithi103.stdout:2/478: readlink d11/d19/l64 0 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:1/582: write db/d10/d18/f69 [926134,72484] 0 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:5/527: mkdir d0/d14/d23/d2a/d33/db5 0 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:2/479: chown d11/d25/d46/d48/l86 347 1 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:7/564: mkdir d1/d10/d11/d19/d29/d33/d4b/db3 0 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:1/583: chown db/d10/d2e/l3a 429 1 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:2/480: chown d11/d19/l64 85006 1 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:8/656: fsync d4/d26/d27/d7e/d39/d49/f74 0 2024-03-22T02:28:41.555 INFO:tasks.workunit.client.0.smithi103.stdout:6/426: dwrite d0/d3/d33/f78 [0,4194304] 0 2024-03-22T02:28:41.556 INFO:tasks.workunit.client.0.smithi103.stdout:3/594: link de/d12/f1c de/d1a/d25/d3f/d9c/fc2 0 2024-03-22T02:28:41.557 INFO:tasks.workunit.client.0.smithi103.stdout:2/481: read d11/d19/d5d/f75 [117650,92223] 0 2024-03-22T02:28:41.557 INFO:tasks.workunit.client.0.smithi103.stdout:7/565: readlink d1/d10/d11/d19/d29/d61/d69/la4 0 2024-03-22T02:28:41.558 INFO:tasks.workunit.client.0.smithi103.stdout:3/595: write de/d12/d19/f71 [2185109,86920] 0 2024-03-22T02:28:41.561 INFO:tasks.workunit.client.0.smithi103.stdout:7/566: fsync d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f 0 2024-03-22T02:28:41.564 INFO:tasks.workunit.client.0.smithi103.stdout:7/567: chown d1/d10/d11/d19/f26 1 1 2024-03-22T02:28:41.566 INFO:tasks.workunit.client.0.smithi103.stdout:3/596: mknod de/d12/d5d/cc3 0 2024-03-22T02:28:41.566 INFO:tasks.workunit.client.0.smithi103.stdout:7/568: chown d1/d10/d11/d19/d29/d61/d69/d5c 95 1 2024-03-22T02:28:41.566 INFO:tasks.workunit.client.0.smithi103.stdout:5/528: dwrite d0/d14/d23/d2a/d2c/f3f [0,4194304] 0 2024-03-22T02:28:41.567 INFO:tasks.workunit.client.0.smithi103.stdout:1/584: creat db/fc7 x:0 0 0 2024-03-22T02:28:41.568 INFO:tasks.workunit.client.0.smithi103.stdout:7/569: write d1/d6f/fae [125861,46240] 0 2024-03-22T02:28:41.575 INFO:tasks.workunit.client.0.smithi103.stdout:5/529: dwrite d0/da2/fb2 [0,4194304] 0 2024-03-22T02:28:41.576 INFO:tasks.workunit.client.0.smithi103.stdout:1/585: mkdir db/d10/d28/db2/dc8 0 2024-03-22T02:28:41.580 INFO:tasks.workunit.client.0.smithi103.stdout:3/597: unlink de/d12/c13 0 2024-03-22T02:28:41.583 INFO:tasks.workunit.client.0.smithi103.stdout:7/570: mknod d1/cb4 0 2024-03-22T02:28:41.586 INFO:tasks.workunit.client.0.smithi103.stdout:3/598: rename de/d1a/f31 to de/d12/d4d/fc4 0 2024-03-22T02:28:41.588 INFO:tasks.workunit.client.0.smithi103.stdout:1/586: mkdir db/d4a/d78/d92/da8/dc9 0 2024-03-22T02:28:41.591 INFO:tasks.workunit.client.0.smithi103.stdout:3/599: rename de/d1a/d25/d3f/c5a to de/d6b/d80/d4f/d84/cc5 0 2024-03-22T02:28:41.592 INFO:tasks.workunit.client.0.smithi103.stdout:3/600: fdatasync de/d12/d5d/f99 0 2024-03-22T02:28:41.592 INFO:tasks.workunit.client.0.smithi103.stdout:5/530: getdents d0/d14/d23/d2a/d1f 0 2024-03-22T02:28:41.592 INFO:tasks.workunit.client.0.smithi103.stdout:3/601: chown de/d1a/d25 0 1 2024-03-22T02:28:41.593 INFO:tasks.workunit.client.0.smithi103.stdout:7/571: rmdir d1/d6f/d99 0 2024-03-22T02:28:41.596 INFO:tasks.workunit.client.0.smithi103.stdout:7/572: mkdir d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5 0 2024-03-22T02:28:41.598 INFO:tasks.workunit.client.0.smithi103.stdout:7/573: read - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f zero size 2024-03-22T02:28:41.599 INFO:tasks.workunit.client.0.smithi103.stdout:3/602: dwrite de/d12/d5d/f99 [4194304,4194304] 0 2024-03-22T02:28:41.603 INFO:tasks.workunit.client.0.smithi103.stdout:7/574: link d1/d10/d11/d19/d29/d61/d69/c4c d1/d10/cb6 0 2024-03-22T02:28:41.606 INFO:tasks.workunit.client.0.smithi103.stdout:5/531: dread d0/d14/d23/d46/f61 [4194304,4194304] 0 2024-03-22T02:28:41.607 INFO:tasks.workunit.client.0.smithi103.stdout:7/575: rename d1/d6f/fae to d1/da/d2f/fb7 0 2024-03-22T02:28:41.608 INFO:tasks.workunit.client.0.smithi103.stdout:5/532: dread - d0/d14/d23/d36/d95/d96/daa/daf/fab zero size 2024-03-22T02:28:41.609 INFO:tasks.workunit.client.0.smithi103.stdout:5/533: creat d0/d14/d23/d36/fb6 x:0 0 0 2024-03-22T02:28:41.610 INFO:tasks.workunit.client.0.smithi103.stdout:5/534: readlink d0/d14/d23/d2a/l5f 0 2024-03-22T02:28:41.612 INFO:tasks.workunit.client.0.smithi103.stdout:5/535: creat d0/d14/d23/d2a/d1f/fb7 x:0 0 0 2024-03-22T02:28:41.616 INFO:tasks.workunit.client.0.smithi103.stdout:7/576: dread d1/d10/d11/d19/d29/d61/f46 [0,4194304] 0 2024-03-22T02:28:41.622 INFO:tasks.workunit.client.0.smithi103.stdout:7/577: dwrite d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:41.623 INFO:tasks.workunit.client.0.smithi103.stdout:5/536: dread d0/f1e [0,4194304] 0 2024-03-22T02:28:41.623 INFO:tasks.workunit.client.0.smithi103.stdout:7/578: readlink d1/d10/d11/d19/d29/d61/d69/la4 0 2024-03-22T02:28:41.623 INFO:tasks.workunit.client.0.smithi103.stdout:5/537: dread - d0/da2/fb3 zero size 2024-03-22T02:28:41.628 INFO:tasks.workunit.client.0.smithi103.stdout:5/538: dwrite d0/d14/d23/d2a/f6c [0,4194304] 0 2024-03-22T02:28:41.629 INFO:tasks.workunit.client.0.smithi103.stdout:5/539: chown d0/da2 7985 1 2024-03-22T02:28:41.629 INFO:tasks.workunit.client.0.smithi103.stdout:5/540: stat d0/f71 0 2024-03-22T02:28:41.631 INFO:tasks.workunit.client.0.smithi103.stdout:5/541: creat d0/d14/d23/d5c/fb8 x:0 0 0 2024-03-22T02:28:41.632 INFO:tasks.workunit.client.0.smithi103.stdout:5/542: mkdir d0/d14/d23/d2a/d33/db9 0 2024-03-22T02:28:41.734 INFO:tasks.workunit.client.0.smithi103.stdout:9/479: sync 2024-03-22T02:28:41.735 INFO:tasks.workunit.client.0.smithi103.stdout:9/480: truncate d2/d10/d59/f89 190277 0 2024-03-22T02:28:41.736 INFO:tasks.workunit.client.0.smithi103.stdout:9/481: write d2/f9 [3088584,72748] 0 2024-03-22T02:28:41.737 INFO:tasks.workunit.client.0.smithi103.stdout:9/482: write d2/d10/d43/f66 [484451,111750] 0 2024-03-22T02:28:41.738 INFO:tasks.workunit.client.0.smithi103.stdout:9/483: stat d2/da/f52 0 2024-03-22T02:28:41.741 INFO:tasks.workunit.client.0.smithi103.stdout:9/484: rename d2/d10/d43/l40 to d2/d5/l90 0 2024-03-22T02:28:41.743 INFO:tasks.workunit.client.0.smithi103.stdout:9/485: creat d2/d10/d43/d7e/f91 x:0 0 0 2024-03-22T02:28:41.748 INFO:tasks.workunit.client.0.smithi103.stdout:9/486: dwrite d2/da/d18/d1e/d34/d3d/d46/f6a [0,4194304] 0 2024-03-22T02:28:41.752 INFO:tasks.workunit.client.0.smithi103.stdout:9/487: getdents d2/d10/d59 0 2024-03-22T02:28:41.758 INFO:tasks.workunit.client.0.smithi103.stdout:9/488: unlink d2/d5/d7b/l7f 0 2024-03-22T02:28:41.758 INFO:tasks.workunit.client.0.smithi103.stdout:9/489: symlink d2/d5/d4c/l92 0 2024-03-22T02:28:41.758 INFO:tasks.workunit.client.0.smithi103.stdout:9/490: readlink d2/d5/d4c/d56/l57 0 2024-03-22T02:28:41.759 INFO:tasks.workunit.client.0.smithi103.stdout:9/491: write d2/d10/d43/f7a [1646444,41428] 0 2024-03-22T02:28:41.761 INFO:tasks.workunit.client.0.smithi103.stdout:9/492: symlink d2/d5/d4c/l93 0 2024-03-22T02:28:41.763 INFO:tasks.workunit.client.0.smithi103.stdout:9/493: mknod d2/d10/d43/d7e/c94 0 2024-03-22T02:28:41.766 INFO:tasks.workunit.client.0.smithi103.stdout:9/494: getdents d2/da/d18/d1e/d77/d8c 0 2024-03-22T02:28:41.769 INFO:tasks.workunit.client.0.smithi103.stdout:9/495: creat d2/d5/d4c/d56/f95 x:0 0 0 2024-03-22T02:28:41.770 INFO:tasks.workunit.client.0.smithi103.stdout:9/496: chown d2/d10/d27/l67 14878 1 2024-03-22T02:28:41.775 INFO:tasks.workunit.client.0.smithi103.stdout:9/497: mkdir d2/d10/d43/d7e/d96 0 2024-03-22T02:28:41.776 INFO:tasks.workunit.client.0.smithi103.stdout:8/657: dwrite f2 [0,4194304] 0 2024-03-22T02:28:41.781 INFO:tasks.workunit.client.0.smithi103.stdout:2/482: truncate d11/d25/d46/d48/f7a 2451833 0 2024-03-22T02:28:41.782 INFO:tasks.workunit.client.0.smithi103.stdout:9/498: creat d2/da/d18/d1e/d34/f97 x:0 0 0 2024-03-22T02:28:41.784 INFO:tasks.workunit.client.0.smithi103.stdout:8/658: dread f2 [0,4194304] 0 2024-03-22T02:28:41.785 INFO:tasks.workunit.client.0.smithi103.stdout:2/483: symlink d11/d25/d46/l9b 0 2024-03-22T02:28:41.786 INFO:tasks.workunit.client.0.smithi103.stdout:2/484: fsync d11/d19/d5d/f66 0 2024-03-22T02:28:41.789 INFO:tasks.workunit.client.0.smithi103.stdout:8/659: dread d4/f89 [0,4194304] 0 2024-03-22T02:28:41.790 INFO:tasks.workunit.client.0.smithi103.stdout:9/499: rename d2/da/d18/d1e/d77/d8b/f72 to d2/d5/d4c/d56/d6f/f98 0 2024-03-22T02:28:41.795 INFO:tasks.workunit.client.0.smithi103.stdout:8/660: dwrite d4/d2a/d38/f6a [0,4194304] 0 2024-03-22T02:28:41.798 INFO:tasks.workunit.client.0.smithi103.stdout:9/500: unlink d2/d5/ff 0 2024-03-22T02:28:41.803 INFO:tasks.workunit.client.0.smithi103.stdout:2/485: read d11/f20 [1978346,68796] 0 2024-03-22T02:28:41.806 INFO:tasks.workunit.client.0.smithi103.stdout:8/661: mknod d4/d19/d1a/cd5 0 2024-03-22T02:28:41.807 INFO:tasks.workunit.client.0.smithi103.stdout:2/486: creat d11/d25/d46/f9c x:0 0 0 2024-03-22T02:28:41.808 INFO:tasks.workunit.client.0.smithi103.stdout:2/487: write f10 [4523461,67584] 0 2024-03-22T02:28:41.811 INFO:tasks.workunit.client.0.smithi103.stdout:5/543: rmdir d0 39 2024-03-22T02:28:41.812 INFO:tasks.workunit.client.0.smithi103.stdout:8/662: rename d4/d48/d35/f44 to d4/d26/d27/d7e/d39/d49/d51/daf/fd6 0 2024-03-22T02:28:41.815 INFO:tasks.workunit.client.0.smithi103.stdout:8/663: rmdir d4/d26/d27/d7e/d39/d49/d5f 39 2024-03-22T02:28:41.819 INFO:tasks.workunit.client.0.smithi103.stdout:3/603: dread de/d12/d4d/fc4 [8388608,4194304] 0 2024-03-22T02:28:41.820 INFO:tasks.workunit.client.0.smithi103.stdout:8/664: link d4/d26/d27/d7e/d39/f9a d4/d26/d27/d7e/dc8/fd7 0 2024-03-22T02:28:41.821 INFO:tasks.workunit.client.0.smithi103.stdout:3/604: fsync de/d1a/d25/d3f/f85 0 2024-03-22T02:28:41.822 INFO:tasks.workunit.client.0.smithi103.stdout:8/665: symlink d4/d48/ld8 0 2024-03-22T02:28:41.822 INFO:tasks.workunit.client.0.smithi103.stdout:3/605: write de/d12/d30/f35 [3579815,73111] 0 2024-03-22T02:28:41.823 INFO:tasks.workunit.client.0.smithi103.stdout:8/666: dread - d4/d2a/d8b/fbc zero size 2024-03-22T02:28:41.824 INFO:tasks.workunit.client.0.smithi103.stdout:3/606: creat de/d6b/d80/fc6 x:0 0 0 2024-03-22T02:28:41.831 INFO:tasks.workunit.client.0.smithi103.stdout:3/607: dwrite de/d1a/d25/f29 [4194304,4194304] 0 2024-03-22T02:28:41.832 INFO:tasks.workunit.client.0.smithi103.stdout:3/608: chown de/d1a/d25 14032 1 2024-03-22T02:28:41.840 INFO:tasks.workunit.client.0.smithi103.stdout:9/501: write d2/f3b [692350,8196] 0 2024-03-22T02:28:41.842 INFO:tasks.workunit.client.0.smithi103.stdout:5/544: truncate d0/d14/d23/d2a/d2c/f3f 3088454 0 2024-03-22T02:28:41.843 INFO:tasks.workunit.client.0.smithi103.stdout:5/545: chown d0/d14/d23/d36/d95/d96/dac 57602 1 2024-03-22T02:28:41.844 INFO:tasks.workunit.client.0.smithi103.stdout:9/502: creat d2/d10/d59/f99 x:0 0 0 2024-03-22T02:28:41.847 INFO:tasks.workunit.client.0.smithi103.stdout:9/503: rmdir d2/d10/d43 39 2024-03-22T02:28:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:41 smithi122 ceph-mon[36403]: pgmap v14: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 165 MiB/s wr, 509 op/s 2024-03-22T02:28:41.849 INFO:tasks.workunit.client.0.smithi103.stdout:9/504: dread d2/d10/d59/f89 [0,4194304] 0 2024-03-22T02:28:41.850 INFO:tasks.workunit.client.0.smithi103.stdout:5/546: rename d0/d14/d23/f40 to d0/d14/d23/d36/d95/d96/dac/fba 0 2024-03-22T02:28:41.852 INFO:tasks.workunit.client.0.smithi103.stdout:3/609: dread de/d12/d30/fa6 [0,4194304] 0 2024-03-22T02:28:41.852 INFO:tasks.workunit.client.0.smithi103.stdout:8/667: write d4/fd [1891713,50265] 0 2024-03-22T02:28:41.854 INFO:tasks.workunit.client.0.smithi103.stdout:9/505: fsync d2/d10/d43/d4b/f82 0 2024-03-22T02:28:41.858 INFO:tasks.workunit.client.0.smithi103.stdout:3/610: creat de/d6b/d80/d4f/fc7 x:0 0 0 2024-03-22T02:28:41.858 INFO:tasks.workunit.client.0.smithi103.stdout:3/611: chown fa 2947 1 2024-03-22T02:28:41.859 INFO:tasks.workunit.client.0.smithi103.stdout:8/668: creat d4/d2a/d4f/fd9 x:0 0 0 2024-03-22T02:28:41.860 INFO:tasks.workunit.client.0.smithi103.stdout:8/669: chown d4/d19/d1a/d4a/la7 1858233 1 2024-03-22T02:28:41.861 INFO:tasks.workunit.client.0.smithi103.stdout:8/670: write d4/d26/fa8 [197032,48968] 0 2024-03-22T02:28:41.861 INFO:tasks.workunit.client.0.smithi103.stdout:5/547: fsync d0/d14/d23/d5c/d70/f8f 0 2024-03-22T02:28:41.862 INFO:tasks.workunit.client.0.smithi103.stdout:5/548: fdatasync d0/d14/d23/d6f/f86 0 2024-03-22T02:28:41.863 INFO:tasks.workunit.client.0.smithi103.stdout:3/612: symlink de/d12/d30/d70/lc8 0 2024-03-22T02:28:41.864 INFO:tasks.workunit.client.0.smithi103.stdout:5/549: chown d0/d14/d23/d5c/d70/f87 72254810 1 2024-03-22T02:28:41.865 INFO:tasks.workunit.client.0.smithi103.stdout:9/506: link d2/da/d18/d1e/d34/d3d/d46/f6a d2/da/d18/d1e/d34/d3d/d61/f9a 0 2024-03-22T02:28:41.866 INFO:tasks.workunit.client.0.smithi103.stdout:9/507: stat d2/da/d18/d1e/d34/f97 0 2024-03-22T02:28:41.869 INFO:tasks.workunit.client.0.smithi103.stdout:5/550: creat d0/d14/d23/d6f/fbb x:0 0 0 2024-03-22T02:28:41.870 INFO:tasks.workunit.client.0.smithi103.stdout:9/508: mkdir d2/d10/d43/d9b 0 2024-03-22T02:28:41.871 INFO:tasks.workunit.client.0.smithi103.stdout:5/551: fsync d0/d14/d23/d5c/d77/fa4 0 2024-03-22T02:28:41.871 INFO:tasks.workunit.client.0.smithi103.stdout:9/509: write d2/f3b [225433,122883] 0 2024-03-22T02:28:41.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:41 smithi103 ceph-mon[29258]: pgmap v14: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 165 MiB/s wr, 509 op/s 2024-03-22T02:28:41.875 INFO:tasks.workunit.client.0.smithi103.stdout:9/510: creat d2/d10/d43/d7e/f9c x:0 0 0 2024-03-22T02:28:41.876 INFO:tasks.workunit.client.0.smithi103.stdout:5/552: fdatasync d0/d14/d23/d2a/d2c/f4d 0 2024-03-22T02:28:41.880 INFO:tasks.workunit.client.0.smithi103.stdout:9/511: rename d2/d5/d4c/d56/d6f to d2/d10/d43/d7e/d96/d9d 0 2024-03-22T02:28:41.882 INFO:tasks.workunit.client.0.smithi103.stdout:8/671: truncate f1 5754664 0 2024-03-22T02:28:41.883 INFO:tasks.workunit.client.0.smithi103.stdout:3/613: dread de/d6b/d80/d4f/f77 [4194304,4194304] 0 2024-03-22T02:28:41.885 INFO:tasks.workunit.client.0.smithi103.stdout:9/512: symlink d2/da/d18/l9e 0 2024-03-22T02:28:41.887 INFO:tasks.workunit.client.0.smithi103.stdout:5/553: creat d0/d14/d23/fbc x:0 0 0 2024-03-22T02:28:41.887 INFO:tasks.workunit.client.0.smithi103.stdout:5/554: fsync d0/d14/d23/d46/f7a 0 2024-03-22T02:28:41.889 INFO:tasks.workunit.client.0.smithi103.stdout:9/513: unlink d2/da/d18/f41 0 2024-03-22T02:28:41.890 INFO:tasks.workunit.client.0.smithi103.stdout:9/514: write d2/d10/d27/f51 [1064448,108712] 0 2024-03-22T02:28:41.891 INFO:tasks.workunit.client.0.smithi103.stdout:3/614: symlink de/d1a/d25/d3f/lc9 0 2024-03-22T02:28:41.893 INFO:tasks.workunit.client.0.smithi103.stdout:8/672: link d4/d19/d8c/c96 d4/d26/d27/da3/cda 0 2024-03-22T02:28:41.894 INFO:tasks.workunit.client.0.smithi103.stdout:9/515: dread d2/f9 [0,4194304] 0 2024-03-22T02:28:41.895 INFO:tasks.workunit.client.0.smithi103.stdout:3/615: creat de/d1a/d25/d3f/d9c/fca x:0 0 0 2024-03-22T02:28:41.896 INFO:tasks.workunit.client.0.smithi103.stdout:9/516: chown d2/da/d18/d1e/d34/l74 15 1 2024-03-22T02:28:41.896 INFO:tasks.workunit.client.0.smithi103.stdout:5/555: mkdir d0/d14/d23/d2a/d33/db5/dbd 0 2024-03-22T02:28:41.896 INFO:tasks.workunit.client.0.smithi103.stdout:5/556: chown d0/d3e 186826619 1 2024-03-22T02:28:41.899 INFO:tasks.workunit.client.0.smithi103.stdout:5/557: creat d0/d14/d23/d2a/d2c/fbe x:0 0 0 2024-03-22T02:28:41.903 INFO:tasks.workunit.client.0.smithi103.stdout:5/558: symlink d0/d14/d23/d6f/lbf 0 2024-03-22T02:28:41.903 INFO:tasks.workunit.client.0.smithi103.stdout:3/616: dwrite de/d12/d19/fbf [0,4194304] 0 2024-03-22T02:28:41.909 INFO:tasks.workunit.client.0.smithi103.stdout:3/617: rename de/d1a/d25/d3f/d9c/da7/fad to de/d12/d19/fcb 0 2024-03-22T02:28:41.912 INFO:tasks.workunit.client.0.smithi103.stdout:3/618: creat de/d12/d30/d70/fcc x:0 0 0 2024-03-22T02:28:41.913 INFO:tasks.workunit.client.0.smithi103.stdout:5/559: getdents d0/d14/d23/d2a/d33 0 2024-03-22T02:28:41.914 INFO:tasks.workunit.client.0.smithi103.stdout:5/560: read - d0/f73 zero size 2024-03-22T02:28:41.916 INFO:tasks.workunit.client.0.smithi103.stdout:3/619: creat de/d6b/d80/d4f/fcd x:0 0 0 2024-03-22T02:28:41.917 INFO:tasks.workunit.client.0.smithi103.stdout:8/673: write f1 [1307375,103644] 0 2024-03-22T02:28:41.919 INFO:tasks.workunit.client.0.smithi103.stdout:8/674: write d4/f89 [3663706,105772] 0 2024-03-22T02:28:41.921 INFO:tasks.workunit.client.0.smithi103.stdout:4/469: sync 2024-03-22T02:28:41.925 INFO:tasks.workunit.client.0.smithi103.stdout:8/675: creat d4/d26/fdb x:0 0 0 2024-03-22T02:28:41.927 INFO:tasks.workunit.client.0.smithi103.stdout:3/620: dread de/d12/f16 [0,4194304] 0 2024-03-22T02:28:41.928 INFO:tasks.workunit.client.0.smithi103.stdout:3/621: fsync de/d1a/d25/f29 0 2024-03-22T02:28:41.931 INFO:tasks.workunit.client.0.smithi103.stdout:4/470: truncate d3/f8 3137274 0 2024-03-22T02:28:41.931 INFO:tasks.workunit.client.0.smithi103.stdout:3/622: dread de/d1a/fbd [0,4194304] 0 2024-03-22T02:28:41.932 INFO:tasks.workunit.client.0.smithi103.stdout:8/676: dread d4/d26/d27/d7e/d39/d49/d51/daf/fd6 [0,4194304] 0 2024-03-22T02:28:41.934 INFO:tasks.workunit.client.0.smithi103.stdout:8/677: truncate d4/d19/d1a/d4a/f9e 36155 0 2024-03-22T02:28:41.935 INFO:tasks.workunit.client.0.smithi103.stdout:3/623: creat de/d12/d5d/d92/fce x:0 0 0 2024-03-22T02:28:41.935 INFO:tasks.workunit.client.0.smithi103.stdout:3/624: fdatasync de/d1a/d25/d26/f58 0 2024-03-22T02:28:41.937 INFO:tasks.workunit.client.0.smithi103.stdout:4/471: rename d3/d1e/c2a to d3/d30/d45/d6e/ca5 0 2024-03-22T02:28:41.939 INFO:tasks.workunit.client.0.smithi103.stdout:5/561: dread d0/d14/d23/d46/f7a [0,4194304] 0 2024-03-22T02:28:41.940 INFO:tasks.workunit.client.0.smithi103.stdout:8/678: creat d4/fdc x:0 0 0 2024-03-22T02:28:41.940 INFO:tasks.workunit.client.0.smithi103.stdout:4/472: readlink d3/d1e/d25/d65/l8d 0 2024-03-22T02:28:41.945 INFO:tasks.workunit.client.0.smithi103.stdout:8/679: creat d4/d2a/d4f/fdd x:0 0 0 2024-03-22T02:28:41.945 INFO:tasks.workunit.client.0.smithi103.stdout:3/625: dwrite de/d1a/d25/f42 [0,4194304] 0 2024-03-22T02:28:41.949 INFO:tasks.workunit.client.0.smithi103.stdout:8/680: rmdir d4/d48/d35 39 2024-03-22T02:28:41.953 INFO:tasks.workunit.client.0.smithi103.stdout:3/626: rename de/d12/d5d/c95 to de/d12/d4d/ccf 0 2024-03-22T02:28:41.954 INFO:tasks.workunit.client.0.smithi103.stdout:4/473: dwrite d3/f9b [0,4194304] 0 2024-03-22T02:28:41.955 INFO:tasks.workunit.client.0.smithi103.stdout:5/562: dwrite d0/d14/d23/d36/fb6 [0,4194304] 0 2024-03-22T02:28:41.958 INFO:tasks.workunit.client.0.smithi103.stdout:3/627: chown de/d12/d19/fb6 529263987 1 2024-03-22T02:28:41.958 INFO:tasks.workunit.client.0.smithi103.stdout:4/474: truncate d3/d1e/d53/d7a/f7d 476827 0 2024-03-22T02:28:41.961 INFO:tasks.workunit.client.0.smithi103.stdout:8/681: mknod d4/d26/d27/d7e/d39/d49/d5f/cde 0 2024-03-22T02:28:41.967 INFO:tasks.workunit.client.0.smithi103.stdout:5/563: mkdir d0/d14/dc0 0 2024-03-22T02:28:41.967 INFO:tasks.workunit.client.0.smithi103.stdout:8/682: rename d4/d26/l55 to d4/d2a/d8b/ldf 0 2024-03-22T02:28:41.968 INFO:tasks.workunit.client.0.smithi103.stdout:3/628: fdatasync de/f1e 0 2024-03-22T02:28:41.969 INFO:tasks.workunit.client.0.smithi103.stdout:5/564: stat d0/d14/d23/d36/d95/c8e 0 2024-03-22T02:28:41.974 INFO:tasks.workunit.client.0.smithi103.stdout:8/683: fdatasync d4/d26/f81 0 2024-03-22T02:28:41.974 INFO:tasks.workunit.client.0.smithi103.stdout:5/565: truncate d0/d14/d23/d2a/d2c/fbe 468444 0 2024-03-22T02:28:41.975 INFO:tasks.workunit.client.0.smithi103.stdout:8/684: write d4/d26/d27/d7e/d39/d7b/fd0 [448831,84201] 0 2024-03-22T02:28:41.976 INFO:tasks.workunit.client.0.smithi103.stdout:5/566: readlink d0/d14/l43 0 2024-03-22T02:28:41.976 INFO:tasks.workunit.client.0.smithi103.stdout:8/685: chown d4/d2a/d4f/fb7 0 1 2024-03-22T02:28:41.977 INFO:tasks.workunit.client.0.smithi103.stdout:4/475: dread d3/d31/f4e [0,4194304] 0 2024-03-22T02:28:41.978 INFO:tasks.workunit.client.0.smithi103.stdout:4/476: dread - d3/d38/f85 zero size 2024-03-22T02:28:41.979 INFO:tasks.workunit.client.0.smithi103.stdout:4/477: chown d3/d69/l74 367 1 2024-03-22T02:28:41.979 INFO:tasks.workunit.client.0.smithi103.stdout:8/686: creat d4/d48/d40/d68/fe0 x:0 0 0 2024-03-22T02:28:41.980 INFO:tasks.workunit.client.0.smithi103.stdout:8/687: dread - d4/d2a/f7d zero size 2024-03-22T02:28:41.980 INFO:tasks.workunit.client.0.smithi103.stdout:5/567: symlink d0/d14/d23/d2a/d33/db5/lc1 0 2024-03-22T02:28:41.981 INFO:tasks.workunit.client.0.smithi103.stdout:8/688: dread - d4/d2a/d38/f9c zero size 2024-03-22T02:28:41.982 INFO:tasks.workunit.client.0.smithi103.stdout:4/478: mkdir d3/da2/da6 0 2024-03-22T02:28:41.982 INFO:tasks.workunit.client.0.smithi103.stdout:8/689: write d4/d2a/d38/f6a [1807613,75675] 0 2024-03-22T02:28:41.987 INFO:tasks.workunit.client.0.smithi103.stdout:8/690: mknod d4/d26/d27/d7e/d39/ce1 0 2024-03-22T02:28:41.991 INFO:tasks.workunit.client.0.smithi103.stdout:5/568: dwrite d0/d14/f8d [0,4194304] 0 2024-03-22T02:28:41.993 INFO:tasks.workunit.client.0.smithi103.stdout:4/479: dwrite d3/f1f [0,4194304] 0 2024-03-22T02:28:41.993 INFO:tasks.workunit.client.0.smithi103.stdout:8/691: unlink d4/d26/d27/d7e/d39/d7b/fd0 0 2024-03-22T02:28:41.996 INFO:tasks.workunit.client.0.smithi103.stdout:5/569: mknod d0/d14/d23/d36/d95/d96/cc2 0 2024-03-22T02:28:41.997 INFO:tasks.workunit.client.0.smithi103.stdout:3/629: write de/d1a/d25/d26/f6f [1406498,75118] 0 2024-03-22T02:28:41.999 INFO:tasks.workunit.client.0.smithi103.stdout:8/692: creat d4/d26/d27/d7e/d39/d7b/fe2 x:0 0 0 2024-03-22T02:28:42.000 INFO:tasks.workunit.client.0.smithi103.stdout:4/480: symlink d3/d1e/d25/d65/d83/da1/la7 0 2024-03-22T02:28:42.001 INFO:tasks.workunit.client.0.smithi103.stdout:8/693: read f1 [52066,122968] 0 2024-03-22T02:28:42.002 INFO:tasks.workunit.client.0.smithi103.stdout:3/630: symlink de/d1a/d25/d3f/d9c/ld0 0 2024-03-22T02:28:42.003 INFO:tasks.workunit.client.0.smithi103.stdout:3/631: write de/d6b/d80/f7a [413967,28900] 0 2024-03-22T02:28:42.011 INFO:tasks.workunit.client.0.smithi103.stdout:3/632: dwrite de/d1a/d25/d3f/f68 [0,4194304] 0 2024-03-22T02:28:42.011 INFO:tasks.workunit.client.0.smithi103.stdout:4/481: rename d3/d38/f75 to d3/d1e/d78/fa8 0 2024-03-22T02:28:42.015 INFO:tasks.workunit.client.0.smithi103.stdout:4/482: chown d3/d1e/d53/l5f 263691389 1 2024-03-22T02:28:42.016 INFO:tasks.workunit.client.0.smithi103.stdout:4/483: chown d3/d1e/d25/c72 4638612 1 2024-03-22T02:28:42.019 INFO:tasks.workunit.client.0.smithi103.stdout:3/633: rename de/d1a/f39 to de/d1a/d25/fd1 0 2024-03-22T02:28:42.023 INFO:tasks.workunit.client.0.smithi103.stdout:5/570: truncate d0/d14/d23/d36/fb6 3472363 0 2024-03-22T02:28:42.024 INFO:tasks.workunit.client.0.smithi103.stdout:4/484: link d3/d1e/d53/l5f d3/d1e/d53/d8a/la9 0 2024-03-22T02:28:42.026 INFO:tasks.workunit.client.0.smithi103.stdout:8/694: dread d4/fe [0,4194304] 0 2024-03-22T02:28:42.026 INFO:tasks.workunit.client.0.smithi103.stdout:5/571: write d0/d14/d23/d36/f82 [366317,104073] 0 2024-03-22T02:28:42.029 INFO:tasks.workunit.client.0.smithi103.stdout:8/695: fdatasync d4/d2a/d8b/fbc 0 2024-03-22T02:28:42.029 INFO:tasks.workunit.client.0.smithi103.stdout:3/634: dwrite de/d1a/d25/d26/f6f [0,4194304] 0 2024-03-22T02:28:42.030 INFO:tasks.workunit.client.0.smithi103.stdout:5/572: truncate d0/f97 1235233 0 2024-03-22T02:28:42.033 INFO:tasks.workunit.client.0.smithi103.stdout:8/696: creat d4/d2a/d4f/fe3 x:0 0 0 2024-03-22T02:28:42.034 INFO:tasks.workunit.client.0.smithi103.stdout:3/635: mknod de/d1a/cd2 0 2024-03-22T02:28:42.034 INFO:tasks.workunit.client.0.smithi103.stdout:8/697: chown d4/d19/d1a/d4a/cbe 462 1 2024-03-22T02:28:42.035 INFO:tasks.workunit.client.0.smithi103.stdout:5/573: mknod d0/d14/d23/d6f/cc3 0 2024-03-22T02:28:42.042 INFO:tasks.workunit.client.0.smithi103.stdout:8/698: link d4/d19/d1a/d4a/cce d4/d2a/d38/ce4 0 2024-03-22T02:28:42.044 INFO:tasks.workunit.client.0.smithi103.stdout:5/574: dwrite d0/da2/fb2 [0,4194304] 0 2024-03-22T02:28:42.044 INFO:tasks.workunit.client.0.smithi103.stdout:4/485: dwrite d3/d38/f4c [0,4194304] 0 2024-03-22T02:28:42.049 INFO:tasks.workunit.client.0.smithi103.stdout:8/699: creat d4/d26/d27/d7e/d39/d49/d51/daf/fe5 x:0 0 0 2024-03-22T02:28:42.051 INFO:tasks.workunit.client.0.smithi103.stdout:4/486: creat d3/d38/d5c/faa x:0 0 0 2024-03-22T02:28:42.052 INFO:tasks.workunit.client.0.smithi103.stdout:4/487: readlink d3/d69/l73 0 2024-03-22T02:28:42.054 INFO:tasks.workunit.client.0.smithi103.stdout:4/488: write d3/d38/f85 [568287,129936] 0 2024-03-22T02:28:42.054 INFO:tasks.workunit.client.0.smithi103.stdout:8/700: rmdir d4/d26/d27/da3 39 2024-03-22T02:28:42.055 INFO:tasks.workunit.client.0.smithi103.stdout:5/575: creat d0/d14/d23/d5c/d70/fc4 x:0 0 0 2024-03-22T02:28:42.056 INFO:tasks.workunit.client.0.smithi103.stdout:4/489: dread - d3/d69/f9e zero size 2024-03-22T02:28:42.061 INFO:tasks.workunit.client.0.smithi103.stdout:4/490: rename d3/d31/f57 to d3/d38/d5c/d90/fab 0 2024-03-22T02:28:42.064 INFO:tasks.workunit.client.0.smithi103.stdout:3/636: dread de/d1a/d25/d26/f69 [0,4194304] 0 2024-03-22T02:28:42.065 INFO:tasks.workunit.client.0.smithi103.stdout:5/576: dwrite d0/d14/d23/d36/d95/fa0 [0,4194304] 0 2024-03-22T02:28:42.066 INFO:tasks.workunit.client.0.smithi103.stdout:4/491: creat d3/d1e/fac x:0 0 0 2024-03-22T02:28:42.066 INFO:tasks.workunit.client.0.smithi103.stdout:5/577: readlink d0/d14/d23/d6f/l89 0 2024-03-22T02:28:42.068 INFO:tasks.workunit.client.0.smithi103.stdout:3/637: mkdir de/d12/d5d/d92/dd3 0 2024-03-22T02:28:42.072 INFO:tasks.workunit.client.0.smithi103.stdout:4/492: dread d3/f9b [0,4194304] 0 2024-03-22T02:28:42.073 INFO:tasks.workunit.client.0.smithi103.stdout:5/578: creat d0/d14/d23/d2a/d33/db5/dbd/fc5 x:0 0 0 2024-03-22T02:28:42.073 INFO:tasks.workunit.client.0.smithi103.stdout:4/493: dread - d3/fa0 zero size 2024-03-22T02:28:42.074 INFO:tasks.workunit.client.0.smithi103.stdout:3/638: symlink de/d6b/d80/d4f/d84/ld4 0 2024-03-22T02:28:42.076 INFO:tasks.workunit.client.0.smithi103.stdout:4/494: creat d3/d1e/d25/d65/d83/da1/fad x:0 0 0 2024-03-22T02:28:42.077 INFO:tasks.workunit.client.0.smithi103.stdout:8/701: write d4/fe [2750213,10547] 0 2024-03-22T02:28:42.078 INFO:tasks.workunit.client.0.smithi103.stdout:3/639: write de/d1a/fbd [949804,87822] 0 2024-03-22T02:28:42.079 INFO:tasks.workunit.client.0.smithi103.stdout:4/495: dread - d3/d30/d45/f84 zero size 2024-03-22T02:28:42.080 INFO:tasks.workunit.client.0.smithi103.stdout:5/579: rename d0/d14/d23/d2a/f6c to d0/d14/d23/d36/d95/d96/daa/daf/fc6 0 2024-03-22T02:28:42.081 INFO:tasks.workunit.client.0.smithi103.stdout:5/580: readlink d0/d14/d23/d2a/d2c/l90 0 2024-03-22T02:28:42.082 INFO:tasks.workunit.client.0.smithi103.stdout:4/496: chown d3/d1e/d53/d7a/c8b 195520 1 2024-03-22T02:28:42.083 INFO:tasks.workunit.client.0.smithi103.stdout:3/640: unlink de/d12/f16 0 2024-03-22T02:28:42.142 INFO:tasks.workunit.client.0.smithi103.stdout:6/427: sync 2024-03-22T02:28:42.142 INFO:tasks.workunit.client.0.smithi103.stdout:1/587: sync 2024-03-22T02:28:42.142 INFO:tasks.workunit.client.0.smithi103.stdout:0/587: sync 2024-03-22T02:28:42.142 INFO:tasks.workunit.client.0.smithi103.stdout:4/497: mknod d3/d38/d5c/cae 0 2024-03-22T02:28:42.144 INFO:tasks.workunit.client.0.smithi103.stdout:4/498: stat d3/d30/d45/d6e/c88 0 2024-03-22T02:28:42.145 INFO:tasks.workunit.client.0.smithi103.stdout:8/702: dwrite d4/d48/d40/d68/f95 [0,4194304] 0 2024-03-22T02:28:42.145 INFO:tasks.workunit.client.0.smithi103.stdout:3/641: creat de/d12/d19/fd5 x:0 0 0 2024-03-22T02:28:42.146 INFO:tasks.workunit.client.0.smithi103.stdout:4/499: stat d3/d38/f4c 0 2024-03-22T02:28:42.147 INFO:tasks.workunit.client.0.smithi103.stdout:5/581: unlink d0/d14/d23/d2a/d7b/c7c 0 2024-03-22T02:28:42.147 INFO:tasks.workunit.client.0.smithi103.stdout:1/588: mkdir db/d10/d18/dca 0 2024-03-22T02:28:42.149 INFO:tasks.workunit.client.0.smithi103.stdout:6/428: symlink d0/d14/d57/l83 0 2024-03-22T02:28:42.150 INFO:tasks.workunit.client.0.smithi103.stdout:6/429: chown d0/d3/f49 56 1 2024-03-22T02:28:42.153 INFO:tasks.workunit.client.0.smithi103.stdout:0/588: creat dd/fc3 x:0 0 0 2024-03-22T02:28:42.155 INFO:tasks.workunit.client.0.smithi103.stdout:5/582: symlink d0/d14/d23/d36/d95/d96/lc7 0 2024-03-22T02:28:42.157 INFO:tasks.workunit.client.0.smithi103.stdout:1/589: dwrite db/d10/d29/d41/d68/f76 [0,4194304] 0 2024-03-22T02:28:42.157 INFO:tasks.workunit.client.0.smithi103.stdout:5/583: read d0/f71 [5321826,13937] 0 2024-03-22T02:28:42.159 INFO:tasks.workunit.client.0.smithi103.stdout:1/590: chown db/d10/d28/d97 130283590 1 2024-03-22T02:28:42.159 INFO:tasks.workunit.client.0.smithi103.stdout:5/584: chown d0/d3e/c34 4657 1 2024-03-22T02:28:42.163 INFO:tasks.workunit.client.0.smithi103.stdout:8/703: write f1 [2182696,50821] 0 2024-03-22T02:28:42.165 INFO:tasks.workunit.client.0.smithi103.stdout:4/500: link d3/d1e/d25/f3d d3/d30/d45/d6e/faf 0 2024-03-22T02:28:42.170 INFO:tasks.workunit.client.0.smithi103.stdout:0/589: mkdir dd/d16/d4b/d96/d35/dc4 0 2024-03-22T02:28:42.170 INFO:tasks.workunit.client.0.smithi103.stdout:0/590: chown dd/d16/d4b/d96/d64/d91/d9a/d9d/lb7 32095682 1 2024-03-22T02:28:42.170 INFO:tasks.workunit.client.0.smithi103.stdout:3/642: rename de/d12/d4d/l72 to de/d12/ld6 0 2024-03-22T02:28:42.171 INFO:tasks.workunit.client.0.smithi103.stdout:5/585: mknod d0/d14/d23/d2a/d2c/cc8 0 2024-03-22T02:28:42.174 INFO:tasks.workunit.client.0.smithi103.stdout:6/430: dread d0/d3/d22/f3d [0,4194304] 0 2024-03-22T02:28:42.177 INFO:tasks.workunit.client.0.smithi103.stdout:3/643: dwrite de/d1a/d25/d3f/f85 [0,4194304] 0 2024-03-22T02:28:42.183 INFO:tasks.workunit.client.0.smithi103.stdout:8/704: mknod d4/d19/d1a/ce6 0 2024-03-22T02:28:42.183 INFO:tasks.workunit.client.0.smithi103.stdout:8/705: readlink d4/d19/l3f 0 2024-03-22T02:28:42.184 INFO:tasks.workunit.client.0.smithi103.stdout:4/501: fsync d3/f77 0 2024-03-22T02:28:42.185 INFO:tasks.workunit.client.0.smithi103.stdout:1/591: dread db/d10/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:42.188 INFO:tasks.workunit.client.0.smithi103.stdout:6/431: mknod d0/d3/d22/d58/c84 0 2024-03-22T02:28:42.190 INFO:tasks.workunit.client.0.smithi103.stdout:3/644: symlink de/d12/d8a/db0/ld7 0 2024-03-22T02:28:42.191 INFO:tasks.workunit.client.0.smithi103.stdout:0/591: dwrite dd/d16/d4b/d96/d35/f46 [0,4194304] 0 2024-03-22T02:28:42.193 INFO:tasks.workunit.client.0.smithi103.stdout:1/592: creat db/d10/d29/d41/fcb x:0 0 0 2024-03-22T02:28:42.197 INFO:tasks.workunit.client.0.smithi103.stdout:8/706: symlink d4/d26/le7 0 2024-03-22T02:28:42.199 INFO:tasks.workunit.client.0.smithi103.stdout:1/593: mknod db/d4a/d78/d92/da8/ccc 0 2024-03-22T02:28:42.200 INFO:tasks.workunit.client.0.smithi103.stdout:1/594: write db/d10/d29/d41/d68/f76 [2159201,15940] 0 2024-03-22T02:28:42.201 INFO:tasks.workunit.client.0.smithi103.stdout:5/586: unlink d0/d14/d23/d6f/c94 0 2024-03-22T02:28:42.203 INFO:tasks.workunit.client.0.smithi103.stdout:1/595: read db/d10/d18/d2d/f5d [1607349,52240] 0 2024-03-22T02:28:42.209 INFO:tasks.workunit.client.0.smithi103.stdout:6/432: rename d0/d3/c13 to d0/d3/d22/c85 0 2024-03-22T02:28:42.209 INFO:tasks.workunit.client.0.smithi103.stdout:4/502: dwrite d3/d30/d45/d6e/faf [0,4194304] 0 2024-03-22T02:28:42.211 INFO:tasks.workunit.client.0.smithi103.stdout:6/433: write d0/d14/f6c [568785,84328] 0 2024-03-22T02:28:42.214 INFO:tasks.workunit.client.0.smithi103.stdout:4/503: chown d3/d1e/d25/c8e 17 1 2024-03-22T02:28:42.215 INFO:tasks.workunit.client.0.smithi103.stdout:5/587: creat d0/d14/d23/d5c/fc9 x:0 0 0 2024-03-22T02:28:42.216 INFO:tasks.workunit.client.0.smithi103.stdout:3/645: dread de/f23 [0,4194304] 0 2024-03-22T02:28:42.216 INFO:tasks.workunit.client.0.smithi103.stdout:6/434: creat d0/f86 x:0 0 0 2024-03-22T02:28:42.219 INFO:tasks.workunit.client.0.smithi103.stdout:4/504: creat d3/d69/fb0 x:0 0 0 2024-03-22T02:28:42.219 INFO:tasks.workunit.client.0.smithi103.stdout:3/646: dread - de/d12/d30/d70/fcc zero size 2024-03-22T02:28:42.221 INFO:tasks.workunit.client.0.smithi103.stdout:1/596: getdents db/d10/d6b 0 2024-03-22T02:28:42.223 INFO:tasks.workunit.client.0.smithi103.stdout:4/505: symlink d3/d1e/d68/lb1 0 2024-03-22T02:28:42.223 INFO:tasks.workunit.client.0.smithi103.stdout:8/707: dwrite d4/d26/d27/d7e/d39/d49/d51/daf/fd6 [0,4194304] 0 2024-03-22T02:28:42.223 INFO:tasks.workunit.client.0.smithi103.stdout:3/647: write de/d1a/d25/d3f/f68 [1106439,48900] 0 2024-03-22T02:28:42.224 INFO:tasks.workunit.client.0.smithi103.stdout:4/506: chown d3/l13 4733 1 2024-03-22T02:28:42.224 INFO:tasks.workunit.client.0.smithi103.stdout:8/708: dread - d4/d26/fdb zero size 2024-03-22T02:28:42.225 INFO:tasks.workunit.client.0.smithi103.stdout:5/588: rename d0/d14/d23/d36/d95/d96/dac to d0/da2/dca 0 2024-03-22T02:28:42.226 INFO:tasks.workunit.client.0.smithi103.stdout:3/648: chown de/d1a/d25/d26/f6f 36 1 2024-03-22T02:28:42.227 INFO:tasks.workunit.client.0.smithi103.stdout:3/649: write de/d12/d19/f71 [1667046,113289] 0 2024-03-22T02:28:42.229 INFO:tasks.workunit.client.0.smithi103.stdout:4/507: creat d3/da2/da6/fb2 x:0 0 0 2024-03-22T02:28:42.230 INFO:tasks.workunit.client.0.smithi103.stdout:6/435: rename d0/d3/d22/d58/c84 to d0/d3/d22/d58/d6f/c87 0 2024-03-22T02:28:42.232 INFO:tasks.workunit.client.0.smithi103.stdout:8/709: symlink d4/d19/le8 0 2024-03-22T02:28:42.234 INFO:tasks.workunit.client.0.smithi103.stdout:3/650: creat de/d1a/d25/d26/fd8 x:0 0 0 2024-03-22T02:28:42.235 INFO:tasks.workunit.client.0.smithi103.stdout:4/508: symlink d3/d1e/d25/d65/lb3 0 2024-03-22T02:28:42.236 INFO:tasks.workunit.client.0.smithi103.stdout:6/436: rmdir d0/d14/d57 39 2024-03-22T02:28:42.237 INFO:tasks.workunit.client.0.smithi103.stdout:6/437: write d0/f64 [614978,126008] 0 2024-03-22T02:28:42.237 INFO:tasks.workunit.client.0.smithi103.stdout:8/710: symlink d4/d2a/d3e/le9 0 2024-03-22T02:28:42.238 INFO:tasks.workunit.client.0.smithi103.stdout:8/711: read - d4/d2a/d3e/da4/fa5 zero size 2024-03-22T02:28:42.239 INFO:tasks.workunit.client.0.smithi103.stdout:8/712: readlink d4/d26/d27/d7e/d67/l97 0 2024-03-22T02:28:42.240 INFO:tasks.workunit.client.0.smithi103.stdout:5/589: link d0/d14/d23/d5c/d70/fc4 d0/d14/d23/d36/d95/d96/fcb 0 2024-03-22T02:28:42.241 INFO:tasks.workunit.client.0.smithi103.stdout:8/713: write d4/d2a/d4f/fd9 [218485,17747] 0 2024-03-22T02:28:42.242 INFO:tasks.workunit.client.0.smithi103.stdout:3/651: mknod de/d6b/d80/d4f/cd9 0 2024-03-22T02:28:42.245 INFO:tasks.workunit.client.0.smithi103.stdout:4/509: symlink d3/d38/lb4 0 2024-03-22T02:28:42.246 INFO:tasks.workunit.client.0.smithi103.stdout:5/590: truncate d0/d14/d23/d5c/d70/f87 624922 0 2024-03-22T02:28:42.247 INFO:tasks.workunit.client.0.smithi103.stdout:8/714: creat d4/d26/d27/d7e/d39/d49/d51/fea x:0 0 0 2024-03-22T02:28:42.248 INFO:tasks.workunit.client.0.smithi103.stdout:3/652: dread - de/d6b/d80/d4f/f64 zero size 2024-03-22T02:28:42.249 INFO:tasks.workunit.client.0.smithi103.stdout:4/510: write d3/d1e/d25/f2b [8708312,28170] 0 2024-03-22T02:28:42.250 INFO:tasks.workunit.client.0.smithi103.stdout:3/653: read de/d1a/d25/f79 [648337,45081] 0 2024-03-22T02:28:42.251 INFO:tasks.workunit.client.0.smithi103.stdout:5/591: dread d0/f12 [4194304,4194304] 0 2024-03-22T02:28:42.252 INFO:tasks.workunit.client.0.smithi103.stdout:4/511: mkdir d3/d69/db5 0 2024-03-22T02:28:42.253 INFO:tasks.workunit.client.0.smithi103.stdout:5/592: chown d0/d14/d23/d36/d95/c8e 42240899 1 2024-03-22T02:28:42.253 INFO:tasks.workunit.client.0.smithi103.stdout:8/715: dread d4/fe [4194304,4194304] 0 2024-03-22T02:28:42.255 INFO:tasks.workunit.client.0.smithi103.stdout:5/593: write d0/d14/d23/fa3 [312573,62502] 0 2024-03-22T02:28:42.257 INFO:tasks.workunit.client.0.smithi103.stdout:5/594: write d0/d14/d23/d36/f85 [5813588,42985] 0 2024-03-22T02:28:42.259 INFO:tasks.workunit.client.0.smithi103.stdout:8/716: rmdir d4/d19/d1a/da6 0 2024-03-22T02:28:42.260 INFO:tasks.workunit.client.0.smithi103.stdout:8/717: chown d4/d26/d27/d7e/d39/d49/d51 1480988 1 2024-03-22T02:28:42.261 INFO:tasks.workunit.client.0.smithi103.stdout:8/718: dread - d4/d2a/d3e/fcf zero size 2024-03-22T02:28:42.267 INFO:tasks.workunit.client.0.smithi103.stdout:8/719: getdents d4/d2a 0 2024-03-22T02:28:42.267 INFO:tasks.workunit.client.0.smithi103.stdout:5/595: dread d0/d14/d23/d2a/d1f/f78 [0,4194304] 0 2024-03-22T02:28:42.268 INFO:tasks.workunit.client.0.smithi103.stdout:5/596: fdatasync d0/d14/d23/d5c/d77/fa4 0 2024-03-22T02:28:42.268 INFO:tasks.workunit.client.0.smithi103.stdout:3/654: write de/d6b/d80/d4f/f65 [2390935,61357] 0 2024-03-22T02:28:42.270 INFO:tasks.workunit.client.0.smithi103.stdout:8/720: mkdir d4/d26/deb 0 2024-03-22T02:28:42.271 INFO:tasks.workunit.client.0.smithi103.stdout:3/655: unlink f1 0 2024-03-22T02:28:42.273 INFO:tasks.workunit.client.0.smithi103.stdout:8/721: fsync d4/d26/f81 0 2024-03-22T02:28:42.276 INFO:tasks.workunit.client.0.smithi103.stdout:8/722: dread - d4/d19/f9b zero size 2024-03-22T02:28:42.276 INFO:tasks.workunit.client.0.smithi103.stdout:5/597: dwrite d0/d14/d23/d2a/d1f/fb7 [0,4194304] 0 2024-03-22T02:28:42.278 INFO:tasks.workunit.client.0.smithi103.stdout:5/598: write d0/f9a [534799,95851] 0 2024-03-22T02:28:42.280 INFO:tasks.workunit.client.0.smithi103.stdout:8/723: readlink d4/d2a/d8b/l99 0 2024-03-22T02:28:42.282 INFO:tasks.workunit.client.0.smithi103.stdout:8/724: fdatasync d4/d26/d27/d7e/d39/d49/d79/fb9 0 2024-03-22T02:28:42.284 INFO:tasks.workunit.client.0.smithi103.stdout:8/725: read - d4/d26/d27/d7e/d39/d7b/fb8 zero size 2024-03-22T02:28:42.285 INFO:tasks.workunit.client.0.smithi103.stdout:3/656: truncate de/d6b/d80/d4f/fa3 3468940 0 2024-03-22T02:28:42.286 INFO:tasks.workunit.client.0.smithi103.stdout:5/599: dwrite d0/d14/d23/d2a/d1f/fb7 [0,4194304] 0 2024-03-22T02:28:42.291 INFO:tasks.workunit.client.0.smithi103.stdout:5/600: symlink d0/d14/d23/d2a/d33/db5/dbd/lcc 0 2024-03-22T02:28:42.294 INFO:tasks.workunit.client.0.smithi103.stdout:5/601: write d0/d14/d23/fbc [871268,127002] 0 2024-03-22T02:28:42.294 INFO:tasks.workunit.client.0.smithi103.stdout:3/657: getdents de/d1a/d25/d26 0 2024-03-22T02:28:42.296 INFO:tasks.workunit.client.0.smithi103.stdout:3/658: write de/d12/d5d/f99 [7073276,109702] 0 2024-03-22T02:28:42.296 INFO:tasks.workunit.client.0.smithi103.stdout:5/602: chown d0/d14/d23/f24 1666931 1 2024-03-22T02:28:42.299 INFO:tasks.workunit.client.0.smithi103.stdout:8/726: truncate d4/f89 1752534 0 2024-03-22T02:28:42.300 INFO:tasks.workunit.client.0.smithi103.stdout:8/727: stat d4/d26/d27/d7e/d39/d49/cb6 0 2024-03-22T02:28:42.300 INFO:tasks.workunit.client.0.smithi103.stdout:3/659: mkdir de/d12/d8a/dda 0 2024-03-22T02:28:42.302 INFO:tasks.workunit.client.0.smithi103.stdout:5/603: mknod d0/d14/ccd 0 2024-03-22T02:28:42.304 INFO:tasks.workunit.client.0.smithi103.stdout:5/604: symlink d0/d14/d23/d2a/d33/db9/lce 0 2024-03-22T02:28:42.305 INFO:tasks.workunit.client.0.smithi103.stdout:5/605: stat d0/d14/d23 0 2024-03-22T02:28:42.323 INFO:tasks.workunit.client.0.smithi103.stdout:5/606: dread d0/f71 [0,4194304] 0 2024-03-22T02:28:42.323 INFO:tasks.workunit.client.0.smithi103.stdout:3/660: dread de/d12/d19/f51 [0,4194304] 0 2024-03-22T02:28:42.325 INFO:tasks.workunit.client.0.smithi103.stdout:3/661: symlink de/d12/d8a/ldb 0 2024-03-22T02:28:42.327 INFO:tasks.workunit.client.0.smithi103.stdout:5/607: link d0/d3e/f9e d0/d14/d23/fcf 0 2024-03-22T02:28:42.331 INFO:tasks.workunit.client.0.smithi103.stdout:5/608: unlink d0/d14/f8d 0 2024-03-22T02:28:42.332 INFO:tasks.workunit.client.0.smithi103.stdout:3/662: link c4 de/d12/d5d/da9/db9/cdc 0 2024-03-22T02:28:42.332 INFO:tasks.workunit.client.0.smithi103.stdout:3/663: fsync de/d12/d5d/f75 0 2024-03-22T02:28:42.335 INFO:tasks.workunit.client.0.smithi103.stdout:3/664: mknod de/d1a/d25/d26/cdd 0 2024-03-22T02:28:42.340 INFO:tasks.workunit.client.0.smithi103.stdout:3/665: link de/d6b/d80/d73/fba de/d1a/fde 0 2024-03-22T02:28:42.341 INFO:tasks.workunit.client.0.smithi103.stdout:8/728: write d4/d26/d27/d7e/d39/f9a [1367719,84527] 0 2024-03-22T02:28:42.448 INFO:tasks.workunit.client.0.smithi103.stdout:7/579: sync 2024-03-22T02:28:42.470 INFO:tasks.workunit.client.0.smithi103.stdout:7/580: fdatasync d1/d10/d11/d19/d29/d57/fb0 0 2024-03-22T02:28:42.470 INFO:tasks.workunit.client.0.smithi103.stdout:8/729: rename d4/d26/d27/d7e/c16 to d4/d2a/cec 0 2024-03-22T02:28:42.475 INFO:tasks.workunit.client.0.smithi103.stdout:8/730: fdatasync d4/d26/f8e 0 2024-03-22T02:28:42.477 INFO:tasks.workunit.client.0.smithi103.stdout:3/666: dwrite de/d1a/fde [4194304,4194304] 0 2024-03-22T02:28:42.480 INFO:tasks.workunit.client.0.smithi103.stdout:8/731: symlink d4/d26/d27/d7e/d39/d7b/led 0 2024-03-22T02:28:42.481 INFO:tasks.workunit.client.0.smithi103.stdout:7/581: dwrite d1/d10/d11/d19/d29/d61/d69/f32 [0,4194304] 0 2024-03-22T02:28:42.481 INFO:tasks.workunit.client.0.smithi103.stdout:3/667: creat de/d1a/d25/d3f/d9c/da7/fdf x:0 0 0 2024-03-22T02:28:42.482 INFO:tasks.workunit.client.0.smithi103.stdout:7/582: stat d1/d10/d11/d70/l93 0 2024-03-22T02:28:42.482 INFO:tasks.workunit.client.0.smithi103.stdout:7/583: fdatasync d1/da/d2f/fb1 0 2024-03-22T02:28:42.486 INFO:tasks.workunit.client.0.smithi103.stdout:7/584: creat d1/d10/d11/d19/d29/d61/d69/d5c/fb8 x:0 0 0 2024-03-22T02:28:42.487 INFO:tasks.workunit.client.0.smithi103.stdout:8/732: dwrite d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:42.488 INFO:tasks.workunit.client.0.smithi103.stdout:8/733: write d4/f15 [4225928,128818] 0 2024-03-22T02:28:42.490 INFO:tasks.workunit.client.0.smithi103.stdout:8/734: chown d4/d19/c23 269 1 2024-03-22T02:28:42.491 INFO:tasks.workunit.client.0.smithi103.stdout:7/585: mkdir d1/d10/d11/d19/d29/d61/d1d/da3/db9 0 2024-03-22T02:28:42.491 INFO:tasks.workunit.client.0.smithi103.stdout:8/735: write d4/d19/d1a/fb3 [116859,96273] 0 2024-03-22T02:28:42.492 INFO:tasks.workunit.client.0.smithi103.stdout:7/586: chown d1/d10/d11/d19/d29/d61/d69/c25 927 1 2024-03-22T02:28:42.494 INFO:tasks.workunit.client.0.smithi103.stdout:8/736: symlink d4/d26/d27/d7e/d39/d49/lee 0 2024-03-22T02:28:42.501 INFO:tasks.workunit.client.0.smithi103.stdout:8/737: dwrite d4/d19/d1a/f91 [0,4194304] 0 2024-03-22T02:28:42.506 INFO:tasks.workunit.client.0.smithi103.stdout:8/738: dread d4/d26/d27/d7e/d39/d49/d51/daf/fd6 [0,4194304] 0 2024-03-22T02:28:42.507 INFO:tasks.workunit.client.0.smithi103.stdout:8/739: symlink d4/d19/d8c/lef 0 2024-03-22T02:28:42.510 INFO:tasks.workunit.client.0.smithi103.stdout:8/740: rename d4/f9 to d4/d19/ff0 0 2024-03-22T02:28:42.512 INFO:tasks.workunit.client.0.smithi103.stdout:8/741: truncate d4/d2a/d8b/fbc 616662 0 2024-03-22T02:28:42.514 INFO:tasks.workunit.client.0.smithi103.stdout:8/742: truncate d4/d26/d27/d7e/d39/d49/d51/fea 104497 0 2024-03-22T02:28:42.514 INFO:tasks.workunit.client.0.smithi103.stdout:8/743: fdatasync d4/d2a/d8b/f9f 0 2024-03-22T02:28:42.516 INFO:tasks.workunit.client.0.smithi103.stdout:8/744: chown d4/d26/d27/d7e/d39/d49/d5f/f66 1 1 2024-03-22T02:28:42.518 INFO:tasks.workunit.client.0.smithi103.stdout:8/745: creat d4/d26/ff1 x:0 0 0 2024-03-22T02:28:42.519 INFO:tasks.workunit.client.0.smithi103.stdout:8/746: creat d4/d26/d27/d7e/d39/d49/d51/daf/ff2 x:0 0 0 2024-03-22T02:28:42.522 INFO:tasks.workunit.client.0.smithi103.stdout:8/747: creat d4/d26/d27/d7e/d39/d7b/dc7/ff3 x:0 0 0 2024-03-22T02:28:42.747 INFO:tasks.workunit.client.0.smithi103.stdout:2/488: sync 2024-03-22T02:28:42.748 INFO:tasks.workunit.client.0.smithi103.stdout:2/489: write d11/d25/d63/d45/d47/f85 [763688,89897] 0 2024-03-22T02:28:42.751 INFO:tasks.workunit.client.0.smithi103.stdout:2/490: creat d11/d25/d46/d48/f9d x:0 0 0 2024-03-22T02:28:42.757 INFO:tasks.workunit.client.0.smithi103.stdout:2/491: dwrite d11/d25/d63/d45/d47/d83/f60 [0,4194304] 0 2024-03-22T02:28:42.758 INFO:tasks.workunit.client.0.smithi103.stdout:2/492: fsync d11/f13 0 2024-03-22T02:28:42.760 INFO:tasks.workunit.client.0.smithi103.stdout:2/493: symlink d11/d25/d46/l9e 0 2024-03-22T02:28:42.768 INFO:tasks.workunit.client.0.smithi103.stdout:2/494: dwrite d11/d25/d63/d45/d47/f85 [0,4194304] 0 2024-03-22T02:28:42.770 INFO:tasks.workunit.client.0.smithi103.stdout:2/495: symlink d11/d25/d63/d45/d47/d83/l9f 0 2024-03-22T02:28:42.776 INFO:tasks.workunit.client.0.smithi103.stdout:2/496: dread d11/d25/d63/d45/f77 [0,4194304] 0 2024-03-22T02:28:42.777 INFO:tasks.workunit.client.0.smithi103.stdout:2/497: read d11/d25/f4c [59038,52218] 0 2024-03-22T02:28:42.790 INFO:tasks.workunit.client.0.smithi103.stdout:2/498: dwrite d11/d25/d46/d40/d90/f99 [0,4194304] 0 2024-03-22T02:28:42.843 INFO:tasks.workunit.client.0.smithi103.stdout:9/517: sync 2024-03-22T02:28:42.843 INFO:tasks.workunit.client.0.smithi103.stdout:0/592: sync 2024-03-22T02:28:42.843 INFO:tasks.workunit.client.0.smithi103.stdout:9/518: dread - d2/d10/d43/d4b/f82 zero size 2024-03-22T02:28:42.845 INFO:tasks.workunit.client.0.smithi103.stdout:9/519: write d2/f3 [2351639,21030] 0 2024-03-22T02:28:42.845 INFO:tasks.workunit.client.0.smithi103.stdout:9/520: chown d2/d10 3054685 1 2024-03-22T02:28:42.846 INFO:tasks.workunit.client.0.smithi103.stdout:0/593: getdents dd/d34/d53/d8c 0 2024-03-22T02:28:42.847 INFO:tasks.workunit.client.0.smithi103.stdout:9/521: creat d2/d5/d4c/f9f x:0 0 0 2024-03-22T02:28:42.848 INFO:tasks.workunit.client.0.smithi103.stdout:0/594: write dd/d34/fad [833269,74674] 0 2024-03-22T02:28:42.849 INFO:tasks.workunit.client.0.smithi103.stdout:9/522: fsync d2/da/d18/d1e/d77/d8b/f26 0 2024-03-22T02:28:42.852 INFO:tasks.workunit.client.0.smithi103.stdout:9/523: mkdir d2/da/d18/d1e/d34/d3d/d61/da0 0 2024-03-22T02:28:42.853 INFO:tasks.workunit.client.0.smithi103.stdout:9/524: fsync d2/da/fd 0 2024-03-22T02:28:42.853 INFO:tasks.workunit.client.0.smithi103.stdout:0/595: unlink dd/d16/l54 0 2024-03-22T02:28:42.854 INFO:tasks.workunit.client.0.smithi103.stdout:9/525: read - d2/da/d18/d1e/d34/d3d/d4d/f8f zero size 2024-03-22T02:28:42.855 INFO:tasks.workunit.client.0.smithi103.stdout:0/596: mknod dd/d16/cc5 0 2024-03-22T02:28:42.858 INFO:tasks.workunit.client.0.smithi103.stdout:0/597: chown dd/d16/d4b/d96/d64/c67 7117 1 2024-03-22T02:28:42.860 INFO:tasks.workunit.client.0.smithi103.stdout:9/526: dwrite d2/d5/d4c/f5c [0,4194304] 0 2024-03-22T02:28:42.862 INFO:tasks.workunit.client.0.smithi103.stdout:0/598: rename dd/d16/d4b/d96/d31 to dd/d16/d4b/d96/d64/dab/dc6 0 2024-03-22T02:28:42.864 INFO:tasks.workunit.client.0.smithi103.stdout:9/527: mkdir d2/d10/d27/da1 0 2024-03-22T02:28:42.866 INFO:tasks.workunit.client.0.smithi103.stdout:0/599: creat dd/d16/d4b/d96/d29/d48/fc7 x:0 0 0 2024-03-22T02:28:42.867 INFO:tasks.workunit.client.0.smithi103.stdout:9/528: mknod d2/da/d18/d1e/d77/d8b/ca2 0 2024-03-22T02:28:42.869 INFO:tasks.workunit.client.0.smithi103.stdout:0/600: write dd/d16/d4b/d96/d29/f2a [4649120,56583] 0 2024-03-22T02:28:42.872 INFO:tasks.workunit.client.0.smithi103.stdout:0/601: creat dd/d16/d4b/d96/d29/d48/d5c/fc8 x:0 0 0 2024-03-22T02:28:42.875 INFO:tasks.workunit.client.0.smithi103.stdout:0/602: symlink dd/d16/d4b/d96/d64/d6a/d8d/lc9 0 2024-03-22T02:28:42.881 INFO:tasks.workunit.client.0.smithi103.stdout:0/603: link dd/d16/f36 dd/d16/d4b/d96/fca 0 2024-03-22T02:28:42.891 INFO:tasks.workunit.client.0.smithi103.stdout:9/529: dread d2/d10/f13 [4194304,4194304] 0 2024-03-22T02:28:42.892 INFO:tasks.workunit.client.0.smithi103.stdout:0/604: dread dd/f15 [0,4194304] 0 2024-03-22T02:28:42.893 INFO:tasks.workunit.client.0.smithi103.stdout:0/605: chown dd/d16/d4b/d96/d29/d48/d5c/fc8 1 1 2024-03-22T02:28:42.894 INFO:tasks.workunit.client.0.smithi103.stdout:9/530: creat d2/d10/d27/da1/fa3 x:0 0 0 2024-03-22T02:28:42.898 INFO:tasks.workunit.client.0.smithi103.stdout:0/606: dwrite f4 [0,4194304] 0 2024-03-22T02:28:42.900 INFO:tasks.workunit.client.0.smithi103.stdout:9/531: truncate d2/da/d18/d1e/d34/d3d/d61/f9a 344418 0 2024-03-22T02:28:42.902 INFO:tasks.workunit.client.0.smithi103.stdout:0/607: symlink dd/d16/d4b/d96/d29/d48/d5c/d9b/lcb 0 2024-03-22T02:28:42.903 INFO:tasks.workunit.client.0.smithi103.stdout:0/608: chown dd/d34/d53/c88 47612295 1 2024-03-22T02:28:42.904 INFO:tasks.workunit.client.0.smithi103.stdout:9/532: stat d2/da/d18/d1e/d34/d3d/d46/f6a 0 2024-03-22T02:28:42.906 INFO:tasks.workunit.client.0.smithi103.stdout:9/533: mkdir d2/d10/d43/d7e/da4 0 2024-03-22T02:28:42.907 INFO:tasks.workunit.client.0.smithi103.stdout:9/534: chown d2/da/d18/d1e/d34/d3d/d61 3919 1 2024-03-22T02:28:42.909 INFO:tasks.workunit.client.0.smithi103.stdout:0/609: creat dd/d16/d4b/d96/d64/dab/dc6/d89/fcc x:0 0 0 2024-03-22T02:28:42.911 INFO:tasks.workunit.client.0.smithi103.stdout:0/610: write dd/d16/d4b/d96/d29/fb5 [7855688,27303] 0 2024-03-22T02:28:42.916 INFO:tasks.workunit.client.0.smithi103.stdout:9/535: dwrite d2/d10/f8a [4194304,4194304] 0 2024-03-22T02:28:42.919 INFO:tasks.workunit.client.0.smithi103.stdout:9/536: write d2/da/f25 [1863123,61098] 0 2024-03-22T02:28:42.923 INFO:tasks.workunit.client.0.smithi103.stdout:0/611: dwrite dd/d16/d4b/d96/d35/d82/f7b [0,4194304] 0 2024-03-22T02:28:42.928 INFO:tasks.workunit.client.0.smithi103.stdout:0/612: rename dd/d16/f38 to dd/d34/dbd/fcd 0 2024-03-22T02:28:42.928 INFO:tasks.workunit.client.0.smithi103.stdout:9/537: dwrite d2/d5/f63 [0,4194304] 0 2024-03-22T02:28:42.931 INFO:tasks.workunit.client.0.smithi103.stdout:9/538: mkdir d2/d10/d59/da5 0 2024-03-22T02:28:42.932 INFO:tasks.workunit.client.0.smithi103.stdout:9/539: chown d2/da/d18/d1e/c4e 5 1 2024-03-22T02:28:42.932 INFO:tasks.workunit.client.0.smithi103.stdout:0/613: rmdir dd/d16/d4b/d96/d29/d48/d5c/d9b 39 2024-03-22T02:28:42.938 INFO:tasks.workunit.client.0.smithi103.stdout:1/597: sync 2024-03-22T02:28:42.938 INFO:tasks.workunit.client.0.smithi103.stdout:4/512: sync 2024-03-22T02:28:42.938 INFO:tasks.workunit.client.0.smithi103.stdout:6/438: sync 2024-03-22T02:28:42.938 INFO:tasks.workunit.client.0.smithi103.stdout:1/598: readlink db/d10/l85 0 2024-03-22T02:28:42.939 INFO:tasks.workunit.client.0.smithi103.stdout:6/439: chown d0/d3/d22/d58/d6f 41 1 2024-03-22T02:28:42.940 INFO:tasks.workunit.client.0.smithi103.stdout:4/513: write d3/da2/da6/fb2 [674512,53134] 0 2024-03-22T02:28:42.940 INFO:tasks.workunit.client.0.smithi103.stdout:6/440: chown d0 11631 1 2024-03-22T02:28:42.941 INFO:tasks.workunit.client.0.smithi103.stdout:4/514: write d3/fb [3843250,130276] 0 2024-03-22T02:28:42.944 INFO:tasks.workunit.client.0.smithi103.stdout:6/441: mkdir d0/d88 0 2024-03-22T02:28:42.947 INFO:tasks.workunit.client.0.smithi103.stdout:1/599: creat db/d10/d28/db2/dc8/fcd x:0 0 0 2024-03-22T02:28:42.948 INFO:tasks.workunit.client.0.smithi103.stdout:4/515: creat d3/d38/d5c/d86/fb6 x:0 0 0 2024-03-22T02:28:42.950 INFO:tasks.workunit.client.0.smithi103.stdout:1/600: rmdir db/d4a 39 2024-03-22T02:28:42.951 INFO:tasks.workunit.client.0.smithi103.stdout:9/540: dread d2/da/f25 [0,4194304] 0 2024-03-22T02:28:42.952 INFO:tasks.workunit.client.0.smithi103.stdout:4/516: creat d3/d1e/d53/fb7 x:0 0 0 2024-03-22T02:28:42.954 INFO:tasks.workunit.client.0.smithi103.stdout:1/601: chown db/d10/d18/c24 1268 1 2024-03-22T02:28:42.957 INFO:tasks.workunit.client.0.smithi103.stdout:4/517: rename d3/d1e/f7e to d3/d1e/d53/d7a/da3/fb8 0 2024-03-22T02:28:42.958 INFO:tasks.workunit.client.0.smithi103.stdout:4/518: dread - d3/d30/d45/f84 zero size 2024-03-22T02:28:42.960 INFO:tasks.workunit.client.0.smithi103.stdout:9/541: link d2/da/f24 d2/d5/d7b/d7d/fa6 0 2024-03-22T02:28:42.963 INFO:tasks.workunit.client.0.smithi103.stdout:1/602: rename c0 to db/d10/d29/cce 0 2024-03-22T02:28:42.964 INFO:tasks.workunit.client.0.smithi103.stdout:4/519: mkdir d3/d1e/d78/db9 0 2024-03-22T02:28:42.965 INFO:tasks.workunit.client.0.smithi103.stdout:4/520: truncate d3/da2/da6/fb2 992512 0 2024-03-22T02:28:42.966 INFO:tasks.workunit.client.0.smithi103.stdout:4/521: dread - d3/d1e/d53/fb7 zero size 2024-03-22T02:28:42.966 INFO:tasks.workunit.client.0.smithi103.stdout:1/603: mknod db/ccf 0 2024-03-22T02:28:42.973 INFO:tasks.workunit.client.0.smithi103.stdout:4/522: dread d3/d31/f32 [0,4194304] 0 2024-03-22T02:28:42.974 INFO:tasks.workunit.client.0.smithi103.stdout:9/542: dwrite d2/da/f24 [0,4194304] 0 2024-03-22T02:28:42.975 INFO:tasks.workunit.client.0.smithi103.stdout:4/523: write d3/d1e/d25/f3d [987489,23857] 0 2024-03-22T02:28:42.976 INFO:tasks.workunit.client.0.smithi103.stdout:4/524: stat d3/d1e/d53/d8a/f9c 0 2024-03-22T02:28:42.978 INFO:tasks.workunit.client.0.smithi103.stdout:9/543: rmdir d2/d5/d4c/d56 39 2024-03-22T02:28:42.980 INFO:tasks.workunit.client.0.smithi103.stdout:9/544: write d2/f9 [5771916,115268] 0 2024-03-22T02:28:42.983 INFO:tasks.workunit.client.0.smithi103.stdout:1/604: dread db/f17 [0,4194304] 0 2024-03-22T02:28:42.987 INFO:tasks.workunit.client.0.smithi103.stdout:4/525: write d3/d31/f32 [1409358,44747] 0 2024-03-22T02:28:42.988 INFO:tasks.workunit.client.0.smithi103.stdout:1/605: mknod db/d4a/d78/cd0 0 2024-03-22T02:28:42.989 INFO:tasks.workunit.client.0.smithi103.stdout:9/545: link d2/d10/d43/l11 d2/la7 0 2024-03-22T02:28:42.989 INFO:tasks.workunit.client.0.smithi103.stdout:1/606: fsync db/d10/d29/d41/d68/fa1 0 2024-03-22T02:28:42.990 INFO:tasks.workunit.client.0.smithi103.stdout:4/526: rmdir d3/d1e/d68 39 2024-03-22T02:28:42.991 INFO:tasks.workunit.client.0.smithi103.stdout:4/527: stat d3/d1e/d53/l62 0 2024-03-22T02:28:43.001 INFO:tasks.workunit.client.0.smithi103.stdout:1/607: dwrite db/d10/d29/d41/d68/d74/dba/fc6 [0,4194304] 0 2024-03-22T02:28:43.002 INFO:tasks.workunit.client.0.smithi103.stdout:9/546: dwrite d2/f2c [4194304,4194304] 0 2024-03-22T02:28:43.011 INFO:tasks.workunit.client.0.smithi103.stdout:4/528: dread d3/d1e/f2f [0,4194304] 0 2024-03-22T02:28:43.011 INFO:tasks.workunit.client.0.smithi103.stdout:1/608: rmdir db/d4a/d78/d92/db7 0 2024-03-22T02:28:43.013 INFO:tasks.workunit.client.0.smithi103.stdout:4/529: creat d3/d1e/fba x:0 0 0 2024-03-22T02:28:43.014 INFO:tasks.workunit.client.0.smithi103.stdout:4/530: chown d3/f24 39 1 2024-03-22T02:28:43.014 INFO:tasks.workunit.client.0.smithi103.stdout:1/609: rename db/d10/d21/l4c to db/d10/d29/ld1 0 2024-03-22T02:28:43.018 INFO:tasks.workunit.client.0.smithi103.stdout:4/531: truncate d3/d31/f47 1845806 0 2024-03-22T02:28:43.019 INFO:tasks.workunit.client.0.smithi103.stdout:4/532: symlink d3/d30/d45/lbb 0 2024-03-22T02:28:43.020 INFO:tasks.workunit.client.0.smithi103.stdout:4/533: chown d3/d1e/d78/db9 177 1 2024-03-22T02:28:43.023 INFO:tasks.workunit.client.0.smithi103.stdout:4/534: rmdir d3/d30/d45/d95 0 2024-03-22T02:28:43.023 INFO:tasks.workunit.client.0.smithi103.stdout:4/535: stat d3/d1e/d25 0 2024-03-22T02:28:43.024 INFO:tasks.workunit.client.0.smithi103.stdout:4/536: truncate d3/d38/f70 2085886 0 2024-03-22T02:28:43.026 INFO:tasks.workunit.client.0.smithi103.stdout:4/537: read d3/d1e/d53/d8a/f9a [179544,46526] 0 2024-03-22T02:28:43.030 INFO:tasks.workunit.client.0.smithi103.stdout:4/538: getdents d3/d38 0 2024-03-22T02:28:43.032 INFO:tasks.workunit.client.0.smithi103.stdout:4/539: read d3/d1e/d25/d65/f82 [377258,129754] 0 2024-03-22T02:28:43.032 INFO:tasks.workunit.client.0.smithi103.stdout:4/540: chown d3/d69/f9e 240699 1 2024-03-22T02:28:43.045 INFO:tasks.workunit.client.0.smithi103.stdout:4/541: write d3/d31/f47 [1885275,124190] 0 2024-03-22T02:28:43.046 INFO:tasks.workunit.client.0.smithi103.stdout:4/542: fdatasync d3/d38/f85 0 2024-03-22T02:28:43.048 INFO:tasks.workunit.client.0.smithi103.stdout:4/543: unlink d3/d30/f5d 0 2024-03-22T02:28:43.051 INFO:tasks.workunit.client.0.smithi103.stdout:4/544: rename d3/d69/db5 to d3/d1e/d68/dbc 0 2024-03-22T02:28:43.053 INFO:tasks.workunit.client.0.smithi103.stdout:9/547: dread d2/da/f23 [0,4194304] 0 2024-03-22T02:28:43.054 INFO:tasks.workunit.client.0.smithi103.stdout:4/545: mkdir d3/d1e/d25/d65/d83/da1/dbd 0 2024-03-22T02:28:43.055 INFO:tasks.workunit.client.0.smithi103.stdout:4/546: chown d3/d69/l80 534967447 1 2024-03-22T02:28:43.056 INFO:tasks.workunit.client.0.smithi103.stdout:4/547: fsync d3/da2/da6/fb2 0 2024-03-22T02:28:43.057 INFO:tasks.workunit.client.0.smithi103.stdout:9/548: symlink d2/d10/d43/d7e/d96/d9d/la8 0 2024-03-22T02:28:43.059 INFO:tasks.workunit.client.0.smithi103.stdout:9/549: write d2/da/f23 [3688324,86771] 0 2024-03-22T02:28:43.062 INFO:tasks.workunit.client.0.smithi103.stdout:4/548: dwrite d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:43.064 INFO:tasks.workunit.client.0.smithi103.stdout:9/550: rename d2/d10/d43/c81 to d2/ca9 0 2024-03-22T02:28:43.067 INFO:tasks.workunit.client.0.smithi103.stdout:9/551: fdatasync d2/d10/d43/d7e/d96/d9d/f98 0 2024-03-22T02:28:43.068 INFO:tasks.workunit.client.0.smithi103.stdout:9/552: chown d2/da/d18/d1e/d34/d3d/d4d/l64 21975682 1 2024-03-22T02:28:43.070 INFO:tasks.workunit.client.0.smithi103.stdout:9/553: write d2/da/d18/d1e/d34/f97 [417773,41345] 0 2024-03-22T02:28:43.071 INFO:tasks.workunit.client.0.smithi103.stdout:4/549: dread d3/d31/f47 [0,4194304] 0 2024-03-22T02:28:43.073 INFO:tasks.workunit.client.0.smithi103.stdout:4/550: fdatasync d3/f24 0 2024-03-22T02:28:43.074 INFO:tasks.workunit.client.0.smithi103.stdout:9/554: mknod d2/d5/caa 0 2024-03-22T02:28:43.076 INFO:tasks.workunit.client.0.smithi103.stdout:4/551: rename d3/d30/d45/f84 to d3/d1e/d53/d7a/da3/fbe 0 2024-03-22T02:28:43.077 INFO:tasks.workunit.client.0.smithi103.stdout:9/555: write d2/da/d18/d1e/d34/f8d [661610,5389] 0 2024-03-22T02:28:43.078 INFO:tasks.workunit.client.0.smithi103.stdout:4/552: mknod d3/d38/d5c/d86/cbf 0 2024-03-22T02:28:43.080 INFO:tasks.workunit.client.0.smithi103.stdout:4/553: truncate d3/d1e/fac 12940 0 2024-03-22T02:28:43.083 INFO:tasks.workunit.client.0.smithi103.stdout:4/554: dread d3/d38/f85 [0,4194304] 0 2024-03-22T02:28:43.084 INFO:tasks.workunit.client.0.smithi103.stdout:9/556: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:43.086 INFO:tasks.workunit.client.0.smithi103.stdout:4/555: read d3/da2/da6/fb2 [512072,123157] 0 2024-03-22T02:28:43.088 INFO:tasks.workunit.client.0.smithi103.stdout:9/557: fdatasync d2/da/d18/d1e/d34/d3d/d46/f6a 0 2024-03-22T02:28:43.094 INFO:tasks.workunit.client.0.smithi103.stdout:9/558: getdents d2/d10/d59 0 2024-03-22T02:28:43.095 INFO:tasks.workunit.client.0.smithi103.stdout:9/559: dread - d2/d10/d43/d7e/d96/d9d/f98 zero size 2024-03-22T02:28:43.099 INFO:tasks.workunit.client.0.smithi103.stdout:4/556: dwrite d3/d1e/f43 [0,4194304] 0 2024-03-22T02:28:43.102 INFO:tasks.workunit.client.0.smithi103.stdout:4/557: symlink d3/d69/lc0 0 2024-03-22T02:28:43.106 INFO:tasks.workunit.client.0.smithi103.stdout:9/560: write d2/da/d18/d1e/d34/d3d/d61/f9a [960680,40600] 0 2024-03-22T02:28:43.107 INFO:tasks.workunit.client.0.smithi103.stdout:9/561: write d2/da/d18/d1e/d77/d8b/f26 [4144690,127291] 0 2024-03-22T02:28:43.111 INFO:tasks.workunit.client.0.smithi103.stdout:9/562: creat d2/d10/d43/d9b/fab x:0 0 0 2024-03-22T02:28:43.113 INFO:tasks.workunit.client.0.smithi103.stdout:9/563: symlink d2/d10/d27/da1/lac 0 2024-03-22T02:28:43.114 INFO:tasks.workunit.client.0.smithi103.stdout:9/564: fdatasync d2/d5/d4c/d56/f95 0 2024-03-22T02:28:43.117 INFO:tasks.workunit.client.0.smithi103.stdout:9/565: link d2/d10/d59/f6d d2/da/d18/d1e/d34/d3d/d46/fad 0 2024-03-22T02:28:43.119 INFO:tasks.workunit.client.0.smithi103.stdout:9/566: mkdir d2/da/d18/d1e/d77/d8b/dae 0 2024-03-22T02:28:43.121 INFO:tasks.workunit.client.0.smithi103.stdout:9/567: truncate d2/da/d18/d1e/d34/d3d/d4d/f58 7454160 0 2024-03-22T02:28:43.124 INFO:tasks.workunit.client.0.smithi103.stdout:9/568: link d2/d10/d43/d7e/c94 d2/d10/d43/d4b/d65/caf 0 2024-03-22T02:28:43.129 INFO:tasks.workunit.client.0.smithi103.stdout:9/569: dwrite d2/da/f25 [0,4194304] 0 2024-03-22T02:28:43.132 INFO:tasks.workunit.client.0.smithi103.stdout:9/570: creat d2/da/d18/fb0 x:0 0 0 2024-03-22T02:28:43.132 INFO:tasks.workunit.client.0.smithi103.stdout:9/571: write d2/d10/d43/d9b/fab [763136,121758] 0 2024-03-22T02:28:43.148 INFO:tasks.workunit.client.0.smithi103.stdout:3/668: sync 2024-03-22T02:28:43.149 INFO:tasks.workunit.client.0.smithi103.stdout:2/499: sync 2024-03-22T02:28:43.149 INFO:tasks.workunit.client.0.smithi103.stdout:7/587: sync 2024-03-22T02:28:43.149 INFO:tasks.workunit.client.0.smithi103.stdout:5/609: sync 2024-03-22T02:28:43.149 INFO:tasks.workunit.client.0.smithi103.stdout:8/748: sync 2024-03-22T02:28:43.151 INFO:tasks.workunit.client.0.smithi103.stdout:2/500: dread - d11/f5a zero size 2024-03-22T02:28:43.152 INFO:tasks.workunit.client.0.smithi103.stdout:7/588: dread - d1/da/d2f/fa0 zero size 2024-03-22T02:28:43.152 INFO:tasks.workunit.client.0.smithi103.stdout:9/572: truncate d2/d5/d4c/d56/f95 347573 0 2024-03-22T02:28:43.155 INFO:tasks.workunit.client.0.smithi103.stdout:5/610: rename d0/d14/d23/d36/f85 to d0/da2/dca/fd0 0 2024-03-22T02:28:43.157 INFO:tasks.workunit.client.0.smithi103.stdout:2/501: mkdir d11/da0 0 2024-03-22T02:28:43.159 INFO:tasks.workunit.client.0.smithi103.stdout:2/502: chown d11/f5a 1494322299 1 2024-03-22T02:28:43.160 INFO:tasks.workunit.client.0.smithi103.stdout:3/669: dwrite de/d1a/d25/d26/f58 [4194304,4194304] 0 2024-03-22T02:28:43.162 INFO:tasks.workunit.client.0.smithi103.stdout:9/573: creat d2/d10/d59/da5/fb1 x:0 0 0 2024-03-22T02:28:43.164 INFO:tasks.workunit.client.0.smithi103.stdout:2/503: write d11/f58 [1151683,2976] 0 2024-03-22T02:28:43.167 INFO:tasks.workunit.client.0.smithi103.stdout:2/504: write d11/f18 [10817546,89619] 0 2024-03-22T02:28:43.167 INFO:tasks.workunit.client.0.smithi103.stdout:9/574: fdatasync d2/d5/d4c/f85 0 2024-03-22T02:28:43.169 INFO:tasks.workunit.client.0.smithi103.stdout:3/670: truncate de/d1a/d25/d26/f58 8803825 0 2024-03-22T02:28:43.171 INFO:tasks.workunit.client.0.smithi103.stdout:5/611: dwrite d0/f97 [0,4194304] 0 2024-03-22T02:28:43.175 INFO:tasks.workunit.client.0.smithi103.stdout:7/589: dread d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:43.178 INFO:tasks.workunit.client.0.smithi103.stdout:5/612: truncate d0/d14/d23/d5c/d70/fc4 1007611 0 2024-03-22T02:28:43.178 INFO:tasks.workunit.client.0.smithi103.stdout:5/613: fdatasync d0/d14/d23/d6f/fbb 0 2024-03-22T02:28:43.178 INFO:tasks.workunit.client.0.smithi103.stdout:3/671: unlink de/d12/d19/f9d 0 2024-03-22T02:28:43.183 INFO:tasks.workunit.client.0.smithi103.stdout:3/672: fsync de/d1a/d25/d26/f2f 0 2024-03-22T02:28:43.183 INFO:tasks.workunit.client.0.smithi103.stdout:8/749: dread d4/d19/d1a/fb3 [0,4194304] 0 2024-03-22T02:28:43.183 INFO:tasks.workunit.client.0.smithi103.stdout:2/505: dwrite d11/d25/d46/d48/f6e [0,4194304] 0 2024-03-22T02:28:43.185 INFO:tasks.workunit.client.0.smithi103.stdout:7/590: symlink d1/d10/d11/d19/d29/d61/d69/d5c/lba 0 2024-03-22T02:28:43.186 INFO:tasks.workunit.client.0.smithi103.stdout:3/673: fdatasync de/d12/d19/fd5 0 2024-03-22T02:28:43.186 INFO:tasks.workunit.client.0.smithi103.stdout:8/750: stat d4/d26/d27/d7e/d39/d49/fc6 0 2024-03-22T02:28:43.189 INFO:tasks.workunit.client.1.smithi122.stdout:0/3: sync 2024-03-22T02:28:43.191 INFO:tasks.workunit.client.1.smithi122.stdout:0/4: mknod c1 0 2024-03-22T02:28:43.191 INFO:tasks.workunit.client.1.smithi122.stdout:0/5: readlink l0 0 2024-03-22T02:28:43.192 INFO:tasks.workunit.client.1.smithi122.stdout:0/6: chown c1 15 1 2024-03-22T02:28:43.192 INFO:tasks.workunit.client.1.smithi122.stdout:0/7: read - no filename 2024-03-22T02:28:43.192 INFO:tasks.workunit.client.1.smithi122.stdout:0/8: dread - no filename 2024-03-22T02:28:43.193 INFO:tasks.workunit.client.1.smithi122.stdout:0/9: symlink l2 0 2024-03-22T02:28:43.193 INFO:tasks.workunit.client.1.smithi122.stdout:0/10: rmdir - no directory 2024-03-22T02:28:43.193 INFO:tasks.workunit.client.1.smithi122.stdout:0/11: write - no filename 2024-03-22T02:28:43.193 INFO:tasks.workunit.client.1.smithi122.stdout:0/12: dread - no filename 2024-03-22T02:28:43.194 INFO:tasks.workunit.client.1.smithi122.stdout:0/13: chown c1 57614 1 2024-03-22T02:28:43.194 INFO:tasks.workunit.client.0.smithi103.stdout:2/506: dwrite d11/d19/d4e/f7d [0,4194304] 0 2024-03-22T02:28:43.195 INFO:tasks.workunit.client.1.smithi122.stdout:0/14: rename l2 to l3 0 2024-03-22T02:28:43.196 INFO:tasks.workunit.client.0.smithi103.stdout:9/575: dread d2/d10/d43/f1b [0,4194304] 0 2024-03-22T02:28:43.197 INFO:tasks.workunit.client.1.smithi122.stdout:0/15: mknod c4 0 2024-03-22T02:28:43.198 INFO:tasks.workunit.client.0.smithi103.stdout:9/576: dread - d2/d5/d4c/f9f zero size 2024-03-22T02:28:43.198 INFO:tasks.workunit.client.0.smithi103.stdout:8/751: creat d4/d26/d27/d7e/d39/d49/d51/daf/ff4 x:0 0 0 2024-03-22T02:28:43.199 INFO:tasks.workunit.client.1.smithi122.stdout:0/16: unlink c4 0 2024-03-22T02:28:43.202 INFO:tasks.workunit.client.0.smithi103.stdout:3/674: link fa de/d6b/d80/d73/fe0 0 2024-03-22T02:28:43.203 INFO:tasks.workunit.client.0.smithi103.stdout:2/507: symlink d11/d25/d46/d34/la1 0 2024-03-22T02:28:43.204 INFO:tasks.workunit.client.0.smithi103.stdout:2/508: chown f10 5 1 2024-03-22T02:28:43.207 INFO:tasks.workunit.client.0.smithi103.stdout:2/509: write d11/d19/d5d/f66 [425091,128479] 0 2024-03-22T02:28:43.208 INFO:tasks.workunit.client.0.smithi103.stdout:9/577: dwrite d2/d5/d7b/f84 [0,4194304] 0 2024-03-22T02:28:43.208 INFO:tasks.workunit.client.0.smithi103.stdout:3/675: fdatasync de/d1a/d25/d26/f69 0 2024-03-22T02:28:43.213 INFO:tasks.workunit.client.0.smithi103.stdout:8/752: unlink d4/d2a/l58 0 2024-03-22T02:28:43.213 INFO:tasks.workunit.client.0.smithi103.stdout:3/676: read de/d1a/d25/d26/f6f [1430674,63248] 0 2024-03-22T02:28:43.223 INFO:tasks.workunit.client.0.smithi103.stdout:5/614: dwrite d0/d14/f41 [4194304,4194304] 0 2024-03-22T02:28:43.223 INFO:tasks.workunit.client.0.smithi103.stdout:9/578: creat d2/d10/d43/d4b/d65/fb2 x:0 0 0 2024-03-22T02:28:43.225 INFO:tasks.workunit.client.0.smithi103.stdout:5/615: readlink d0/d14/d23/d2a/l5f 0 2024-03-22T02:28:43.225 INFO:tasks.workunit.client.0.smithi103.stdout:8/753: rmdir d4/d26/d27/d7e/d39/d49/d51/daf 39 2024-03-22T02:28:43.231 INFO:tasks.workunit.client.0.smithi103.stdout:7/591: truncate d1/d10/d11/d19/d29/d61/d1d/f41 3501760 0 2024-03-22T02:28:43.232 INFO:tasks.workunit.client.0.smithi103.stdout:3/677: rmdir de/d12/d5d/d92/dd3 0 2024-03-22T02:28:43.234 INFO:tasks.workunit.client.0.smithi103.stdout:8/754: write d4/d2a/d3e/da4/fa5 [481948,111186] 0 2024-03-22T02:28:43.235 INFO:tasks.workunit.client.0.smithi103.stdout:8/755: chown d4/c28 2 1 2024-03-22T02:28:43.235 INFO:tasks.workunit.client.0.smithi103.stdout:9/579: creat d2/da/d18/d1e/d34/d3d/d46/fb3 x:0 0 0 2024-03-22T02:28:43.237 INFO:tasks.workunit.client.0.smithi103.stdout:5/616: readlink d0/d14/l60 0 2024-03-22T02:28:43.238 INFO:tasks.workunit.client.0.smithi103.stdout:2/510: getdents d11/d25/d46 0 2024-03-22T02:28:43.240 INFO:tasks.workunit.client.0.smithi103.stdout:2/511: fdatasync d11/d19/d5d/f91 0 2024-03-22T02:28:43.243 INFO:tasks.workunit.client.0.smithi103.stdout:9/580: read d2/d10/d27/f4f [35385,38239] 0 2024-03-22T02:28:43.246 INFO:tasks.workunit.client.0.smithi103.stdout:2/512: creat d11/d25/d46/fa2 x:0 0 0 2024-03-22T02:28:43.247 INFO:tasks.workunit.client.0.smithi103.stdout:8/756: dwrite d4/d19/d1a/d4a/f9e [0,4194304] 0 2024-03-22T02:28:43.250 INFO:tasks.workunit.client.0.smithi103.stdout:5/617: truncate d0/f12 4169053 0 2024-03-22T02:28:43.252 INFO:tasks.workunit.client.0.smithi103.stdout:7/592: dwrite d1/d10/d11/d19/f26 [0,4194304] 0 2024-03-22T02:28:43.255 INFO:tasks.workunit.client.1.smithi122.stdout:2/2: sync 2024-03-22T02:28:43.257 INFO:tasks.workunit.client.0.smithi103.stdout:9/581: dwrite d2/da/d18/d1e/d34/d3d/d46/f6a [0,4194304] 0 2024-03-22T02:28:43.257 INFO:tasks.workunit.client.0.smithi103.stdout:3/678: dread de/d1a/d25/d26/f54 [4194304,4194304] 0 2024-03-22T02:28:43.258 INFO:tasks.workunit.client.1.smithi122.stdout:2/3: mknod c0 0 2024-03-22T02:28:43.260 INFO:tasks.workunit.client.0.smithi103.stdout:9/582: write d2/da/f23 [603830,8286] 0 2024-03-22T02:28:43.260 INFO:tasks.workunit.client.0.smithi103.stdout:8/757: mkdir d4/d26/d27/da3/df5 0 2024-03-22T02:28:43.260 INFO:tasks.workunit.client.1.smithi122.stdout:2/4: symlink l1 0 2024-03-22T02:28:43.260 INFO:tasks.workunit.client.1.smithi122.stdout:2/5: dread - no filename 2024-03-22T02:28:43.261 INFO:tasks.workunit.client.1.smithi122.stdout:2/6: chown c0 204 1 2024-03-22T02:28:43.261 INFO:tasks.workunit.client.1.smithi122.stdout:2/7: chown l1 3 1 2024-03-22T02:28:43.262 INFO:tasks.workunit.client.1.smithi122.stdout:2/8: dread - no filename 2024-03-22T02:28:43.262 INFO:tasks.workunit.client.1.smithi122.stdout:2/9: truncate - no filename 2024-03-22T02:28:43.262 INFO:tasks.workunit.client.1.smithi122.stdout:2/10: dwrite - no filename 2024-03-22T02:28:43.262 INFO:tasks.workunit.client.0.smithi103.stdout:2/513: mkdir d11/d25/d63/d45/d47/d83/da3 0 2024-03-22T02:28:43.263 INFO:tasks.workunit.client.1.smithi122.stdout:2/11: symlink l2 0 2024-03-22T02:28:43.266 INFO:tasks.workunit.client.1.smithi122.stdout:2/12: creat f3 x:0 0 0 2024-03-22T02:28:43.267 INFO:tasks.workunit.client.1.smithi122.stdout:2/13: chown c0 6319838 1 2024-03-22T02:28:43.267 INFO:tasks.workunit.client.1.smithi122.stdout:2/14: dread - f3 zero size 2024-03-22T02:28:43.270 INFO:tasks.workunit.client.1.smithi122.stdout:2/15: creat f4 x:0 0 0 2024-03-22T02:28:43.270 INFO:tasks.workunit.client.0.smithi103.stdout:9/583: dwrite d2/da/d18/fb0 [0,4194304] 0 2024-03-22T02:28:43.271 INFO:tasks.workunit.client.1.smithi122.stdout:2/16: unlink f3 0 2024-03-22T02:28:43.272 INFO:tasks.workunit.client.0.smithi103.stdout:9/584: chown d2/da/d18/d1e/d34/d3d/c78 37312743 1 2024-03-22T02:28:43.272 INFO:tasks.workunit.client.1.smithi122.stdout:2/17: write f4 [337946,112830] 0 2024-03-22T02:28:43.273 INFO:tasks.workunit.client.0.smithi103.stdout:5/618: rename d0/d14/d23/d2a/d33/l79 to d0/ld1 0 2024-03-22T02:28:43.274 INFO:tasks.workunit.client.1.smithi122.stdout:2/18: symlink l5 0 2024-03-22T02:28:43.274 INFO:tasks.workunit.client.0.smithi103.stdout:0/614: sync 2024-03-22T02:28:43.274 INFO:tasks.workunit.client.0.smithi103.stdout:3/679: dread de/d1a/d25/f29 [4194304,4194304] 0 2024-03-22T02:28:43.275 INFO:tasks.workunit.client.1.smithi122.stdout:2/19: creat f6 x:0 0 0 2024-03-22T02:28:43.277 INFO:tasks.workunit.client.0.smithi103.stdout:8/758: creat d4/ff6 x:0 0 0 2024-03-22T02:28:43.277 INFO:tasks.workunit.client.1.smithi122.stdout:2/20: creat f7 x:0 0 0 2024-03-22T02:28:43.278 INFO:tasks.workunit.client.0.smithi103.stdout:7/593: getdents d1 0 2024-03-22T02:28:43.278 INFO:tasks.workunit.client.0.smithi103.stdout:2/514: creat d11/d25/d63/d45/d47/d83/d78/fa4 x:0 0 0 2024-03-22T02:28:43.279 INFO:tasks.workunit.client.1.smithi122.stdout:2/21: mknod c8 0 2024-03-22T02:28:43.280 INFO:tasks.workunit.client.0.smithi103.stdout:7/594: readlink d1/da/d1c/l47 0 2024-03-22T02:28:43.280 INFO:tasks.workunit.client.0.smithi103.stdout:2/515: chown d11/d19/d5d/f91 533420652 1 2024-03-22T02:28:43.281 INFO:tasks.workunit.client.1.smithi122.stdout:2/22: write f7 [337111,69380] 0 2024-03-22T02:28:43.282 INFO:tasks.workunit.client.0.smithi103.stdout:0/615: unlink dd/d16/d4b/d96/d29/d48/d5c/cbe 0 2024-03-22T02:28:43.282 INFO:tasks.workunit.client.1.smithi122.stdout:2/23: symlink l9 0 2024-03-22T02:28:43.284 INFO:tasks.workunit.client.1.smithi122.stdout:2/24: symlink la 0 2024-03-22T02:28:43.284 INFO:tasks.workunit.client.0.smithi103.stdout:5/619: read d0/d14/d23/d2a/d1f/f32 [3933738,107425] 0 2024-03-22T02:28:43.284 INFO:tasks.workunit.client.1.smithi122.stdout:2/25: chown f6 165898456 1 2024-03-22T02:28:43.284 INFO:tasks.workunit.client.1.smithi122.stdout:2/26: read - f6 zero size 2024-03-22T02:28:43.286 INFO:tasks.workunit.client.1.smithi122.stdout:2/27: rename c8 to cb 0 2024-03-22T02:28:43.287 INFO:tasks.workunit.client.0.smithi103.stdout:0/616: stat dd/d16/d4b/d96/d64/dab/dc6/d79/f93 0 2024-03-22T02:28:43.288 INFO:tasks.workunit.client.1.smithi122.stdout:2/28: rename l9 to lc 0 2024-03-22T02:28:43.290 INFO:tasks.workunit.client.0.smithi103.stdout:9/585: dwrite d2/da/d18/d1e/d34/d3d/d61/f9a [0,4194304] 0 2024-03-22T02:28:43.291 INFO:tasks.workunit.client.0.smithi103.stdout:8/759: creat d4/d26/d27/d7e/d39/d49/d51/daf/dc2/ff7 x:0 0 0 2024-03-22T02:28:43.292 INFO:tasks.workunit.client.1.smithi122.stdout:0/17: unlink l3 0 2024-03-22T02:28:43.292 INFO:tasks.workunit.client.1.smithi122.stdout:0/18: dwrite - no filename 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:4/41: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:5/16: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:7/38: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:6/6: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:1/4: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:9/19: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:8/65: sync 2024-03-22T02:28:43.294 INFO:tasks.workunit.client.1.smithi122.stdout:3/3: sync 2024-03-22T02:28:43.295 INFO:tasks.workunit.client.0.smithi103.stdout:3/680: truncate de/d6b/d80/d4f/fa3 4021958 0 2024-03-22T02:28:43.296 INFO:tasks.workunit.client.1.smithi122.stdout:0/19: link c1 c5 0 2024-03-22T02:28:43.296 INFO:tasks.workunit.client.1.smithi122.stdout:2/29: dwrite f7 [0,4194304] 0 2024-03-22T02:28:43.296 INFO:tasks.workunit.client.1.smithi122.stdout:2/30: rmdir - no directory 2024-03-22T02:28:43.297 INFO:tasks.workunit.client.0.smithi103.stdout:8/760: chown d4/d19/d1a/f91 198220608 1 2024-03-22T02:28:43.298 INFO:tasks.workunit.client.0.smithi103.stdout:7/595: rename d1/d10/d11/d19/d29/d61/d1d/fa7 to d1/d10/d11/d19/d29/fbb 0 2024-03-22T02:28:43.298 INFO:tasks.workunit.client.0.smithi103.stdout:3/681: write de/d12/d8a/db0/fbe [541069,105659] 0 2024-03-22T02:28:43.301 INFO:tasks.workunit.client.0.smithi103.stdout:9/586: symlink d2/d10/d43/d7e/d96/d9d/lb4 0 2024-03-22T02:28:43.302 INFO:tasks.workunit.client.1.smithi122.stdout:6/7: dread - f4 zero size 2024-03-22T02:28:43.303 INFO:tasks.workunit.client.0.smithi103.stdout:2/516: write d11/d25/d46/d34/f36 [2542008,126248] 0 2024-03-22T02:28:43.304 INFO:tasks.workunit.client.1.smithi122.stdout:9/20: chown f5 399211 1 2024-03-22T02:28:43.304 INFO:tasks.workunit.client.0.smithi103.stdout:5/620: dwrite d0/d14/d23/d2a/d2c/f5d [0,4194304] 0 2024-03-22T02:28:43.305 INFO:tasks.workunit.client.1.smithi122.stdout:7/39: creat d0/f8 x:0 0 0 2024-03-22T02:28:43.306 INFO:tasks.workunit.client.1.smithi122.stdout:3/4: unlink l0 0 2024-03-22T02:28:43.306 INFO:tasks.workunit.client.1.smithi122.stdout:3/5: rename - no filename 2024-03-22T02:28:43.307 INFO:tasks.workunit.client.1.smithi122.stdout:5/17: fdatasync f0 0 2024-03-22T02:28:43.308 INFO:tasks.workunit.client.0.smithi103.stdout:0/617: mknod dd/d16/d4b/cce 0 2024-03-22T02:28:43.308 INFO:tasks.workunit.client.1.smithi122.stdout:1/5: rename f0 to f2 0 2024-03-22T02:28:43.309 INFO:tasks.workunit.client.1.smithi122.stdout:6/8: readlink l1 0 2024-03-22T02:28:43.310 INFO:tasks.workunit.client.1.smithi122.stdout:6/9: chown f4 978854210 1 2024-03-22T02:28:43.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/20: chown c5 6 1 2024-03-22T02:28:43.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/21: dread - no filename 2024-03-22T02:28:43.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/22: dread - no filename 2024-03-22T02:28:43.311 INFO:tasks.workunit.client.0.smithi103.stdout:7/596: creat d1/d10/d11/d19/d29/fbc x:0 0 0 2024-03-22T02:28:43.313 INFO:tasks.workunit.client.0.smithi103.stdout:8/761: unlink d4/d2a/l3c 0 2024-03-22T02:28:43.313 INFO:tasks.workunit.client.1.smithi122.stdout:4/42: rename d0/d2/d7/de/fd to d0/f10 0 2024-03-22T02:28:43.313 INFO:tasks.workunit.client.0.smithi103.stdout:0/618: rmdir dd/d34/dbd 39 2024-03-22T02:28:43.315 INFO:tasks.workunit.client.1.smithi122.stdout:3/6: creat f1 x:0 0 0 2024-03-22T02:28:43.315 INFO:tasks.workunit.client.0.smithi103.stdout:5/621: write d0/d14/d23/d36/d95/d96/daa/daf/fc6 [2063053,117986] 0 2024-03-22T02:28:43.316 INFO:tasks.workunit.client.1.smithi122.stdout:3/7: write f1 [559603,48348] 0 2024-03-22T02:28:43.317 INFO:tasks.workunit.client.1.smithi122.stdout:5/18: symlink l3 0 2024-03-22T02:28:43.318 INFO:tasks.workunit.client.0.smithi103.stdout:3/682: rename de/d1a/d25/d3f/d9c/fc2 to de/d12/d5d/d92/fe1 0 2024-03-22T02:28:43.318 INFO:tasks.workunit.client.0.smithi103.stdout:9/587: dread d2/d10/d43/f7a [0,4194304] 0 2024-03-22T02:28:43.318 INFO:tasks.workunit.client.0.smithi103.stdout:7/597: creat d1/d10/d11/d19/d29/fbd x:0 0 0 2024-03-22T02:28:43.319 INFO:tasks.workunit.client.0.smithi103.stdout:8/762: rmdir d4/d2a/d8b 39 2024-03-22T02:28:43.320 INFO:tasks.workunit.client.1.smithi122.stdout:2/31: link f4 fd 0 2024-03-22T02:28:43.320 INFO:tasks.workunit.client.1.smithi122.stdout:6/10: creat f5 x:0 0 0 2024-03-22T02:28:43.320 INFO:tasks.workunit.client.0.smithi103.stdout:8/763: readlink d4/l21 0 2024-03-22T02:28:43.321 INFO:tasks.workunit.client.0.smithi103.stdout:5/622: mknod d0/d14/d23/d2a/d33/db5/dbd/cd2 0 2024-03-22T02:28:43.322 INFO:tasks.workunit.client.1.smithi122.stdout:0/23: mknod c6 0 2024-03-22T02:28:43.322 INFO:tasks.workunit.client.0.smithi103.stdout:8/764: write d4/d26/d27/d7e/d39/d7b/fe2 [942906,65337] 0 2024-03-22T02:28:43.323 INFO:tasks.workunit.client.1.smithi122.stdout:9/21: truncate f5 3206484 0 2024-03-22T02:28:43.324 INFO:tasks.workunit.client.1.smithi122.stdout:9/22: chown f5 58105477 1 2024-03-22T02:28:43.324 INFO:tasks.workunit.client.0.smithi103.stdout:5/623: readlink d0/d14/l16 0 2024-03-22T02:28:43.325 INFO:tasks.workunit.client.1.smithi122.stdout:4/43: creat d0/d2/d7/de/f11 x:0 0 0 2024-03-22T02:28:43.325 INFO:tasks.workunit.client.0.smithi103.stdout:8/765: write d4/fd [2696200,37963] 0 2024-03-22T02:28:43.326 INFO:tasks.workunit.client.1.smithi122.stdout:7/40: link d0/f8 d0/d4/f9 0 2024-03-22T02:28:43.327 INFO:tasks.workunit.client.0.smithi103.stdout:1/610: sync 2024-03-22T02:28:43.327 INFO:tasks.workunit.client.0.smithi103.stdout:6/442: sync 2024-03-22T02:28:43.327 INFO:tasks.workunit.client.0.smithi103.stdout:4/558: sync 2024-03-22T02:28:43.328 INFO:tasks.workunit.client.0.smithi103.stdout:2/517: rename d11/d19/d4e/f53 to d11/d25/d63/d45/d47/d83/fa5 0 2024-03-22T02:28:43.328 INFO:tasks.workunit.client.1.smithi122.stdout:7/41: write d0/d4/f5 [1065954,45815] 0 2024-03-22T02:28:43.330 INFO:tasks.workunit.client.1.smithi122.stdout:2/32: creat fe x:0 0 0 2024-03-22T02:28:43.331 INFO:tasks.workunit.client.0.smithi103.stdout:3/683: creat de/d12/fe2 x:0 0 0 2024-03-22T02:28:43.332 INFO:tasks.workunit.client.1.smithi122.stdout:6/11: rename f4 to f6 0 2024-03-22T02:28:43.332 INFO:tasks.workunit.client.0.smithi103.stdout:5/624: symlink d0/d14/d23/d36/d95/d96/ld3 0 2024-03-22T02:28:43.333 INFO:tasks.workunit.client.0.smithi103.stdout:0/619: dread dd/d16/d4b/d96/f39 [0,4194304] 0 2024-03-22T02:28:43.334 INFO:tasks.workunit.client.1.smithi122.stdout:9/23: creat f6 x:0 0 0 2024-03-22T02:28:43.334 INFO:tasks.workunit.client.1.smithi122.stdout:9/24: readlink - no filename 2024-03-22T02:28:43.334 INFO:tasks.workunit.client.0.smithi103.stdout:6/443: creat d0/d3/d22/d58/f89 x:0 0 0 2024-03-22T02:28:43.335 INFO:tasks.workunit.client.0.smithi103.stdout:9/588: rename d2/da/f52 to d2/d5/fb5 0 2024-03-22T02:28:43.335 INFO:tasks.workunit.client.0.smithi103.stdout:1/611: truncate db/d10/d29/d55/f8f 92457 0 2024-03-22T02:28:43.337 INFO:tasks.workunit.client.0.smithi103.stdout:4/559: mkdir d3/d1e/d25/d65/d83/dc1 0 2024-03-22T02:28:43.338 INFO:tasks.workunit.client.0.smithi103.stdout:2/518: fsync d11/d25/d46/d34/f52 0 2024-03-22T02:28:43.339 INFO:tasks.workunit.client.0.smithi103.stdout:6/444: readlink d0/d3/d22/d58/l81 0 2024-03-22T02:28:43.341 INFO:tasks.workunit.client.1.smithi122.stdout:7/42: write d0/f8 [72661,4079] 0 2024-03-22T02:28:43.342 INFO:tasks.workunit.client.1.smithi122.stdout:8/66: link d1/d9/l16 d1/l17 0 2024-03-22T02:28:43.342 INFO:tasks.workunit.client.1.smithi122.stdout:7/43: chown d0/f2 17541713 1 2024-03-22T02:28:43.344 INFO:tasks.workunit.client.0.smithi103.stdout:3/684: fsync de/d1a/d25/f29 0 2024-03-22T02:28:43.344 INFO:tasks.workunit.client.0.smithi103.stdout:4/560: write d3/d38/f81 [1477929,28359] 0 2024-03-22T02:28:43.344 INFO:tasks.workunit.client.1.smithi122.stdout:2/33: mkdir df 0 2024-03-22T02:28:43.345 INFO:tasks.workunit.client.0.smithi103.stdout:8/766: dwrite d4/d2a/d4f/fdd [0,4194304] 0 2024-03-22T02:28:43.346 INFO:tasks.workunit.client.0.smithi103.stdout:6/445: read - d0/d3/d22/d58/f80 zero size 2024-03-22T02:28:43.349 INFO:tasks.workunit.client.0.smithi103.stdout:7/598: dread d1/da/d1c/f8b [0,4194304] 0 2024-03-22T02:28:43.350 INFO:tasks.workunit.client.0.smithi103.stdout:6/446: readlink d0/d3/d33/l4d 0 2024-03-22T02:28:43.350 INFO:tasks.workunit.client.1.smithi122.stdout:9/25: creat f7 x:0 0 0 2024-03-22T02:28:43.350 INFO:tasks.workunit.client.1.smithi122.stdout:7/44: dwrite d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:43.352 INFO:tasks.workunit.client.0.smithi103.stdout:9/589: dwrite d2/d5/d4c/d56/f95 [0,4194304] 0 2024-03-22T02:28:43.352 INFO:tasks.workunit.client.0.smithi103.stdout:7/599: stat d1/d10/d11/d19/d29/d33/d4b/db3 0 2024-03-22T02:28:43.354 INFO:tasks.workunit.client.1.smithi122.stdout:1/6: getdents . 0 2024-03-22T02:28:43.355 INFO:tasks.workunit.client.1.smithi122.stdout:5/19: dwrite f0 [0,4194304] 0 2024-03-22T02:28:43.355 INFO:tasks.workunit.client.0.smithi103.stdout:2/519: mkdir d11/d25/d46/d48/da6 0 2024-03-22T02:28:43.356 INFO:tasks.workunit.client.1.smithi122.stdout:1/7: truncate f2 961252 0 2024-03-22T02:28:43.357 INFO:tasks.workunit.client.1.smithi122.stdout:7/45: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:43.358 INFO:tasks.workunit.client.1.smithi122.stdout:4/44: unlink d0/f10 0 2024-03-22T02:28:43.359 INFO:tasks.workunit.client.0.smithi103.stdout:0/620: creat dd/d16/d4b/d96/d64/d6a/d8d/fcf x:0 0 0 2024-03-22T02:28:43.360 INFO:tasks.workunit.client.0.smithi103.stdout:2/520: write d11/d19/d4e/f7d [2929884,89591] 0 2024-03-22T02:28:43.362 INFO:tasks.workunit.client.1.smithi122.stdout:2/34: unlink fe 0 2024-03-22T02:28:43.364 INFO:tasks.workunit.client.0.smithi103.stdout:1/612: dwrite db/d10/d18/d2d/f4d [0,4194304] 0 2024-03-22T02:28:43.365 INFO:tasks.workunit.client.1.smithi122.stdout:0/24: link c5 c7 0 2024-03-22T02:28:43.366 INFO:tasks.workunit.client.1.smithi122.stdout:9/26: creat f8 x:0 0 0 2024-03-22T02:28:43.367 INFO:tasks.workunit.client.0.smithi103.stdout:5/625: creat d0/fd4 x:0 0 0 2024-03-22T02:28:43.368 INFO:tasks.workunit.client.0.smithi103.stdout:8/767: chown d4/f89 0 1 2024-03-22T02:28:43.368 INFO:tasks.workunit.client.1.smithi122.stdout:9/27: dread - f7 zero size 2024-03-22T02:28:43.369 INFO:tasks.workunit.client.0.smithi103.stdout:1/613: dread - db/d10/d29/d41/fcb zero size 2024-03-22T02:28:43.370 INFO:tasks.workunit.client.1.smithi122.stdout:5/20: rename f0 to f4 0 2024-03-22T02:28:43.370 INFO:tasks.workunit.client.0.smithi103.stdout:4/561: mkdir d3/da2/da6/dc2 0 2024-03-22T02:28:43.370 INFO:tasks.workunit.client.1.smithi122.stdout:4/45: dwrite d0/d2/d7/fb [0,4194304] 0 2024-03-22T02:28:43.374 INFO:tasks.workunit.client.1.smithi122.stdout:7/46: symlink d0/d4/la 0 2024-03-22T02:28:43.377 INFO:tasks.workunit.client.0.smithi103.stdout:7/600: chown d1/da/fe 102099756 1 2024-03-22T02:28:43.378 INFO:tasks.workunit.client.1.smithi122.stdout:7/47: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:43.378 INFO:tasks.workunit.client.0.smithi103.stdout:7/601: chown d1/f7e 5 1 2024-03-22T02:28:43.378 INFO:tasks.workunit.client.0.smithi103.stdout:9/590: rename d2/d10/d27/da1/fa3 to d2/da/d18/d1e/d77/d8b/fb6 0 2024-03-22T02:28:43.379 INFO:tasks.workunit.client.1.smithi122.stdout:0/25: mknod c8 0 2024-03-22T02:28:43.381 INFO:tasks.workunit.client.0.smithi103.stdout:7/602: chown d1/da/d2f/fb7 136 1 2024-03-22T02:28:43.381 INFO:tasks.workunit.client.1.smithi122.stdout:8/67: dread - d1/d9/f13 zero size 2024-03-22T02:28:43.381 INFO:tasks.workunit.client.0.smithi103.stdout:2/521: mkdir d11/d19/da7 0 2024-03-22T02:28:43.383 INFO:tasks.workunit.client.1.smithi122.stdout:9/28: mknod c9 0 2024-03-22T02:28:43.384 INFO:tasks.workunit.client.0.smithi103.stdout:0/621: mknod dd/d16/d4b/d96/d29/d48/d94/cd0 0 2024-03-22T02:28:43.386 INFO:tasks.workunit.client.1.smithi122.stdout:3/8: truncate f1 275211 0 2024-03-22T02:28:43.387 INFO:tasks.workunit.client.0.smithi103.stdout:5/626: creat d0/d3e/fd5 x:0 0 0 2024-03-22T02:28:43.389 INFO:tasks.workunit.client.0.smithi103.stdout:0/622: dread - dd/d34/d53/d8c/fbc zero size 2024-03-22T02:28:43.389 INFO:tasks.workunit.client.1.smithi122.stdout:6/12: getdents . 0 2024-03-22T02:28:43.390 INFO:tasks.workunit.client.0.smithi103.stdout:2/522: dwrite d11/d25/d46/d40/f67 [0,4194304] 0 2024-03-22T02:28:43.390 INFO:tasks.workunit.client.1.smithi122.stdout:6/13: dread - f3 zero size 2024-03-22T02:28:43.393 INFO:tasks.workunit.client.0.smithi103.stdout:2/523: chown d11/d19/d7e/f98 3505 1 2024-03-22T02:28:43.393 INFO:tasks.workunit.client.1.smithi122.stdout:5/21: dwrite f4 [0,4194304] 0 2024-03-22T02:28:43.393 INFO:tasks.workunit.client.1.smithi122.stdout:7/48: creat d0/d4/fb x:0 0 0 2024-03-22T02:28:43.394 INFO:tasks.workunit.client.1.smithi122.stdout:0/26: creat f9 x:0 0 0 2024-03-22T02:28:43.395 INFO:tasks.workunit.client.0.smithi103.stdout:3/685: dread fa [0,4194304] 0 2024-03-22T02:28:43.396 INFO:tasks.workunit.client.1.smithi122.stdout:6/14: dread - f5 zero size 2024-03-22T02:28:43.397 INFO:tasks.workunit.client.1.smithi122.stdout:8/68: chown d1/l7 281 1 2024-03-22T02:28:43.397 INFO:tasks.workunit.client.0.smithi103.stdout:9/591: mkdir d2/da/d18/d1e/d34/d3d/d61/db7 0 2024-03-22T02:28:43.398 INFO:tasks.workunit.client.1.smithi122.stdout:8/69: dread - d1/d9/f13 zero size 2024-03-22T02:28:43.399 INFO:tasks.workunit.client.1.smithi122.stdout:4/46: unlink d0/d2/d7/de/f11 0 2024-03-22T02:28:43.404 INFO:tasks.workunit.client.0.smithi103.stdout:1/614: rename db/d10/f26 to db/d10/d6b/fd2 0 2024-03-22T02:28:43.404 INFO:tasks.workunit.client.1.smithi122.stdout:0/27: symlink la 0 2024-03-22T02:28:43.407 INFO:tasks.workunit.client.1.smithi122.stdout:5/22: mknod c5 0 2024-03-22T02:28:43.407 INFO:tasks.workunit.client.0.smithi103.stdout:6/447: dwrite d0/d14/d57/d68/f2d [0,4194304] 0 2024-03-22T02:28:43.408 INFO:tasks.workunit.client.1.smithi122.stdout:7/49: mkdir d0/dc 0 2024-03-22T02:28:43.408 INFO:tasks.workunit.client.0.smithi103.stdout:3/686: mknod de/d1a/d25/d3f/d9c/da7/ce3 0 2024-03-22T02:28:43.409 INFO:tasks.workunit.client.1.smithi122.stdout:9/29: chown f5 19881353 1 2024-03-22T02:28:43.410 INFO:tasks.workunit.client.1.smithi122.stdout:6/15: mkdir d7 0 2024-03-22T02:28:43.410 INFO:tasks.workunit.client.0.smithi103.stdout:0/623: creat dd/d34/dbd/fd1 x:0 0 0 2024-03-22T02:28:43.412 INFO:tasks.workunit.client.1.smithi122.stdout:1/8: dwrite f2 [0,4194304] 0 2024-03-22T02:28:43.412 INFO:tasks.workunit.client.1.smithi122.stdout:8/70: chown d1/d8 17 1 2024-03-22T02:28:43.412 INFO:tasks.workunit.client.1.smithi122.stdout:1/9: rmdir - no directory 2024-03-22T02:28:43.412 INFO:tasks.workunit.client.1.smithi122.stdout:1/10: readlink - no filename 2024-03-22T02:28:43.412 INFO:tasks.workunit.client.1.smithi122.stdout:4/47: mkdir d0/d12 0 2024-03-22T02:28:43.413 INFO:tasks.workunit.client.0.smithi103.stdout:0/624: rename dd/d16/d4b/d96/d35/d82 to dd/d16/d4b/d96/d35/d82/dd2 22 2024-03-22T02:28:43.413 INFO:tasks.workunit.client.0.smithi103.stdout:2/524: symlink d11/d25/d63/d45/la8 0 2024-03-22T02:28:43.413 INFO:tasks.workunit.client.1.smithi122.stdout:3/9: stat f1 0 2024-03-22T02:28:43.414 INFO:tasks.workunit.client.1.smithi122.stdout:8/71: dread - d1/d9/f13 zero size 2024-03-22T02:28:43.416 INFO:tasks.workunit.client.1.smithi122.stdout:0/28: chown c1 0 1 2024-03-22T02:28:43.416 INFO:tasks.workunit.client.0.smithi103.stdout:6/448: unlink d0/f86 0 2024-03-22T02:28:43.417 INFO:tasks.workunit.client.0.smithi103.stdout:6/449: chown d0/d14/d57 30633 1 2024-03-22T02:28:43.417 INFO:tasks.workunit.client.1.smithi122.stdout:9/30: rename f3 to fa 0 2024-03-22T02:28:43.417 INFO:tasks.workunit.client.1.smithi122.stdout:7/50: write d0/d4/f9 [146264,53103] 0 2024-03-22T02:28:43.419 INFO:tasks.workunit.client.1.smithi122.stdout:2/35: dwrite f4 [0,4194304] 0 2024-03-22T02:28:43.420 INFO:tasks.workunit.client.1.smithi122.stdout:1/11: mknod c3 0 2024-03-22T02:28:43.425 INFO:tasks.workunit.client.1.smithi122.stdout:4/48: link d0/d2/fa d0/d2/f13 0 2024-03-22T02:28:43.425 INFO:tasks.workunit.client.1.smithi122.stdout:4/49: readlink - no filename 2024-03-22T02:28:43.426 INFO:tasks.workunit.client.0.smithi103.stdout:3/687: symlink de/d12/d8a/le4 0 2024-03-22T02:28:43.428 INFO:tasks.workunit.client.1.smithi122.stdout:8/72: link d1/d9/f11 d1/d8/f18 0 2024-03-22T02:28:43.428 INFO:tasks.workunit.client.0.smithi103.stdout:9/592: creat d2/da/d18/d1e/d34/d3d/d61/da0/fb8 x:0 0 0 2024-03-22T02:28:43.430 INFO:tasks.workunit.client.1.smithi122.stdout:2/36: rename f7 to df/f10 0 2024-03-22T02:28:43.431 INFO:tasks.workunit.client.1.smithi122.stdout:0/29: dwrite f9 [0,4194304] 0 2024-03-22T02:28:43.431 INFO:tasks.workunit.client.1.smithi122.stdout:2/37: readlink l2 0 2024-03-22T02:28:43.431 INFO:tasks.workunit.client.0.smithi103.stdout:4/562: dwrite d3/d38/f70 [0,4194304] 0 2024-03-22T02:28:43.436 INFO:tasks.workunit.client.0.smithi103.stdout:2/525: fsync d11/d25/d46/d40/f41 0 2024-03-22T02:28:43.438 INFO:tasks.workunit.client.1.smithi122.stdout:0/30: creat fb x:0 0 0 2024-03-22T02:28:43.439 INFO:tasks.workunit.client.0.smithi103.stdout:3/688: rmdir de/d12/d8a 39 2024-03-22T02:28:43.439 INFO:tasks.workunit.client.0.smithi103.stdout:8/768: dwrite d4/d26/d27/d7e/d39/d49/d5f/f66 [0,4194304] 0 2024-03-22T02:28:43.440 INFO:tasks.workunit.client.1.smithi122.stdout:8/73: write d1/f10 [3087433,36784] 0 2024-03-22T02:28:43.440 INFO:tasks.workunit.client.0.smithi103.stdout:9/593: chown d2/d5/d7b/d7d 13806730 1 2024-03-22T02:28:43.441 INFO:tasks.workunit.client.0.smithi103.stdout:7/603: dwrite d1/d10/d11/d19/d29/d61/f46 [0,4194304] 0 2024-03-22T02:28:43.443 INFO:tasks.workunit.client.1.smithi122.stdout:5/23: fsync f4 0 2024-03-22T02:28:43.449 INFO:tasks.workunit.client.0.smithi103.stdout:8/769: chown d4/d26/d27/d7e/d39/d49/d51/ccb 10003245 1 2024-03-22T02:28:43.449 INFO:tasks.workunit.client.0.smithi103.stdout:1/615: link db/d10/d29/d6c/c91 db/d10/d28/cd3 0 2024-03-22T02:28:43.449 INFO:tasks.workunit.client.1.smithi122.stdout:4/50: dwrite d0/d2/f13 [0,4194304] 0 2024-03-22T02:28:43.450 INFO:tasks.workunit.client.1.smithi122.stdout:2/38: dwrite fd [0,4194304] 0 2024-03-22T02:28:43.451 INFO:tasks.workunit.client.1.smithi122.stdout:5/24: write f4 [4243174,123539] 0 2024-03-22T02:28:43.452 INFO:tasks.workunit.client.0.smithi103.stdout:3/689: mkdir de/d12/d30/d70/de5 0 2024-03-22T02:28:43.452 INFO:tasks.workunit.client.1.smithi122.stdout:7/51: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:43.452 INFO:tasks.workunit.client.1.smithi122.stdout:5/25: stat f4 0 2024-03-22T02:28:43.456 INFO:tasks.workunit.client.0.smithi103.stdout:0/625: dwrite dd/f24 [0,4194304] 0 2024-03-22T02:28:43.456 INFO:tasks.workunit.client.0.smithi103.stdout:2/526: write d11/d25/f4c [736054,34712] 0 2024-03-22T02:28:43.456 INFO:tasks.workunit.client.0.smithi103.stdout:4/563: fsync d3/d1e/d53/d7a/da3/fb8 0 2024-03-22T02:28:43.457 INFO:tasks.workunit.client.1.smithi122.stdout:4/51: chown d0/d2/d7/fb 7679285 1 2024-03-22T02:28:43.458 INFO:tasks.workunit.client.1.smithi122.stdout:3/10: fsync f1 0 2024-03-22T02:28:43.458 INFO:tasks.workunit.client.1.smithi122.stdout:0/31: dread f9 [0,4194304] 0 2024-03-22T02:28:43.459 INFO:tasks.workunit.client.1.smithi122.stdout:4/52: read d0/d2/f8 [636609,99364] 0 2024-03-22T02:28:43.459 INFO:tasks.workunit.client.0.smithi103.stdout:5/627: dwrite d0/d14/d23/fcf [0,4194304] 0 2024-03-22T02:28:43.460 INFO:tasks.workunit.client.1.smithi122.stdout:8/74: dwrite d1/f10 [0,4194304] 0 2024-03-22T02:28:43.460 INFO:tasks.workunit.client.1.smithi122.stdout:0/32: readlink l0 0 2024-03-22T02:28:43.460 INFO:tasks.workunit.client.1.smithi122.stdout:5/26: symlink l6 0 2024-03-22T02:28:43.460 INFO:tasks.workunit.client.0.smithi103.stdout:5/628: write d0/d14/d23/d5c/f92 [928926,79848] 0 2024-03-22T02:28:43.465 INFO:tasks.workunit.client.1.smithi122.stdout:5/27: write f4 [2355150,73665] 0 2024-03-22T02:28:43.467 INFO:tasks.workunit.client.1.smithi122.stdout:8/75: chown d1/l17 1217022 1 2024-03-22T02:28:43.468 INFO:tasks.workunit.client.0.smithi103.stdout:7/604: read - d1/d10/d11/d19/d29/d61/d1d/fa9 zero size 2024-03-22T02:28:43.468 INFO:tasks.workunit.client.0.smithi103.stdout:1/616: read - db/d10/d18/d98/fb4 zero size 2024-03-22T02:28:43.468 INFO:tasks.workunit.client.0.smithi103.stdout:6/450: truncate d0/d3/d22/f5c 3645213 0 2024-03-22T02:28:43.468 INFO:tasks.workunit.client.1.smithi122.stdout:5/28: chown c2 387606 1 2024-03-22T02:28:43.471 INFO:tasks.workunit.client.0.smithi103.stdout:3/690: stat de/d12/d5d/da9/db9/cdc 0 2024-03-22T02:28:43.472 INFO:tasks.workunit.client.1.smithi122.stdout:7/52: rmdir d0/dc 0 2024-03-22T02:28:43.474 INFO:tasks.workunit.client.0.smithi103.stdout:3/691: chown de/d1a/fbd 8 1 2024-03-22T02:28:43.474 INFO:tasks.workunit.client.0.smithi103.stdout:3/692: readlink de/d1a/lb3 0 2024-03-22T02:28:43.474 INFO:tasks.workunit.client.1.smithi122.stdout:9/31: getdents . 0 2024-03-22T02:28:43.477 INFO:tasks.workunit.client.1.smithi122.stdout:0/33: dwrite fb [0,4194304] 0 2024-03-22T02:28:43.479 INFO:tasks.workunit.client.0.smithi103.stdout:2/527: creat d11/d19/d7e/fa9 x:0 0 0 2024-03-22T02:28:43.481 INFO:tasks.workunit.client.0.smithi103.stdout:7/605: rmdir d1/d10/d11/d19/d29/d57 39 2024-03-22T02:28:43.482 INFO:tasks.workunit.client.0.smithi103.stdout:7/606: chown d1/d10/d11/d19/f26 1011169784 1 2024-03-22T02:28:43.484 INFO:tasks.workunit.client.0.smithi103.stdout:7/607: write d1/d10/d11/f9f [555630,95468] 0 2024-03-22T02:28:43.487 INFO:tasks.workunit.client.0.smithi103.stdout:2/528: dread d11/d25/d46/d40/f67 [4194304,4194304] 0 2024-03-22T02:28:43.487 INFO:tasks.workunit.client.0.smithi103.stdout:1/617: readlink db/l50 0 2024-03-22T02:28:43.488 INFO:tasks.workunit.client.0.smithi103.stdout:6/451: chown d0/d3/d22/c7e 2 1 2024-03-22T02:28:43.491 INFO:tasks.workunit.client.0.smithi103.stdout:9/594: unlink d2/da/d18/d1e/c5d 0 2024-03-22T02:28:43.492 INFO:tasks.workunit.client.0.smithi103.stdout:0/626: unlink dd/d16/d4b/d96/d64/dab/lb0 0 2024-03-22T02:28:43.493 INFO:tasks.workunit.client.1.smithi122.stdout:7/53: write d0/d4/f5 [3759044,42362] 0 2024-03-22T02:28:43.493 INFO:tasks.workunit.client.1.smithi122.stdout:4/53: link d0/d2/f13 d0/d2/d7/de/f14 0 2024-03-22T02:28:43.493 INFO:tasks.workunit.client.0.smithi103.stdout:1/618: write db/d10/d29/d41/d68/d74/f82 [432199,80732] 0 2024-03-22T02:28:43.494 INFO:tasks.workunit.client.0.smithi103.stdout:2/529: chown d11/d25/d63/d45/d47/f72 954991 1 2024-03-22T02:28:43.494 INFO:tasks.workunit.client.0.smithi103.stdout:3/693: symlink de/d12/d5d/le6 0 2024-03-22T02:28:43.495 INFO:tasks.workunit.client.1.smithi122.stdout:9/32: mkdir db 0 2024-03-22T02:28:43.496 INFO:tasks.workunit.client.0.smithi103.stdout:1/619: dread - db/d10/d29/d41/fcb zero size 2024-03-22T02:28:43.500 INFO:tasks.workunit.client.1.smithi122.stdout:5/29: dwrite f4 [0,4194304] 0 2024-03-22T02:28:43.500 INFO:tasks.workunit.client.1.smithi122.stdout:1/12: truncate f2 37607 0 2024-03-22T02:28:43.501 INFO:tasks.workunit.client.0.smithi103.stdout:5/629: mknod d0/d14/d23/d2a/d2c/da9/cd6 0 2024-03-22T02:28:43.502 INFO:tasks.workunit.client.0.smithi103.stdout:8/770: dread d4/d26/d27/d7e/d39/f9a [0,4194304] 0 2024-03-22T02:28:43.504 INFO:tasks.workunit.client.0.smithi103.stdout:2/530: dread d11/d25/d46/d40/f6f [0,4194304] 0 2024-03-22T02:28:43.505 INFO:tasks.workunit.client.1.smithi122.stdout:6/16: dwrite f3 [0,4194304] 0 2024-03-22T02:28:43.505 INFO:tasks.workunit.client.1.smithi122.stdout:5/30: read f4 [2771109,42116] 0 2024-03-22T02:28:43.510 INFO:tasks.workunit.client.1.smithi122.stdout:4/54: mkdir d0/d12/d15 0 2024-03-22T02:28:43.511 INFO:tasks.workunit.client.1.smithi122.stdout:0/34: dwrite fb [0,4194304] 0 2024-03-22T02:28:43.513 INFO:tasks.workunit.client.0.smithi103.stdout:3/694: mknod de/d1a/d25/d3f/d9c/da7/ce7 0 2024-03-22T02:28:43.513 INFO:tasks.workunit.client.0.smithi103.stdout:5/630: dwrite d0/f9a [0,4194304] 0 2024-03-22T02:28:43.515 INFO:tasks.workunit.client.1.smithi122.stdout:9/33: link f5 db/fc 0 2024-03-22T02:28:43.515 INFO:tasks.workunit.client.1.smithi122.stdout:2/39: truncate df/f10 2372593 0 2024-03-22T02:28:43.517 INFO:tasks.workunit.client.1.smithi122.stdout:2/40: write f4 [1308329,36831] 0 2024-03-22T02:28:43.517 INFO:tasks.workunit.client.1.smithi122.stdout:0/35: symlink lc 0 2024-03-22T02:28:43.518 INFO:tasks.workunit.client.0.smithi103.stdout:8/771: rename d4/d26/f7c to d4/d19/d1a/d4a/ff8 0 2024-03-22T02:28:43.519 INFO:tasks.workunit.client.1.smithi122.stdout:6/17: mknod d7/c8 0 2024-03-22T02:28:43.520 INFO:tasks.workunit.client.0.smithi103.stdout:2/531: symlink d11/d25/d63/d45/d47/laa 0 2024-03-22T02:28:43.522 INFO:tasks.workunit.client.1.smithi122.stdout:3/11: dwrite f1 [0,4194304] 0 2024-03-22T02:28:43.526 INFO:tasks.workunit.client.0.smithi103.stdout:5/631: write d0/d14/d23/d2a/d33/f4c [368502,76615] 0 2024-03-22T02:28:43.528 INFO:tasks.workunit.client.0.smithi103.stdout:8/772: dwrite d4/f15 [0,4194304] 0 2024-03-22T02:28:43.533 INFO:tasks.workunit.client.0.smithi103.stdout:2/532: symlink d11/d25/lab 0 2024-03-22T02:28:43.533 INFO:tasks.workunit.client.0.smithi103.stdout:5/632: write d0/d14/d23/d6f/fbb [175954,87503] 0 2024-03-22T02:28:43.534 INFO:tasks.workunit.client.1.smithi122.stdout:4/55: link d0/d2/d7/fb d0/d2/d7/f16 0 2024-03-22T02:28:43.534 INFO:tasks.workunit.client.1.smithi122.stdout:3/12: dread f1 [0,4194304] 0 2024-03-22T02:28:43.534 INFO:tasks.workunit.client.1.smithi122.stdout:9/34: dwrite fa [4194304,4194304] 0 2024-03-22T02:28:43.535 INFO:tasks.workunit.client.1.smithi122.stdout:9/35: readlink - no filename 2024-03-22T02:28:43.536 INFO:tasks.workunit.client.0.smithi103.stdout:8/773: mkdir d4/d2a/d3e/df9 0 2024-03-22T02:28:43.537 INFO:tasks.workunit.client.1.smithi122.stdout:1/13: dread f2 [0,4194304] 0 2024-03-22T02:28:43.537 INFO:tasks.workunit.client.0.smithi103.stdout:5/633: truncate d0/d14/d23/d5c/d77/fa8 137602 0 2024-03-22T02:28:43.538 INFO:tasks.workunit.client.1.smithi122.stdout:4/56: chown d0/d2/f8 8456650 1 2024-03-22T02:28:43.538 INFO:tasks.workunit.client.1.smithi122.stdout:2/41: creat df/f11 x:0 0 0 2024-03-22T02:28:43.539 INFO:tasks.workunit.client.0.smithi103.stdout:1/620: link db/c13 db/d10/cd4 0 2024-03-22T02:28:43.540 INFO:tasks.workunit.client.0.smithi103.stdout:5/634: write d0/d14/d23/d5c/d77/fa8 [141685,857] 0 2024-03-22T02:28:43.541 INFO:tasks.workunit.client.0.smithi103.stdout:1/621: chown db/d10/d29/fb8 0 1 2024-03-22T02:28:43.541 INFO:tasks.workunit.client.1.smithi122.stdout:4/57: write d0/f9 [948356,120661] 0 2024-03-22T02:28:43.542 INFO:tasks.workunit.client.1.smithi122.stdout:1/14: fsync f2 0 2024-03-22T02:28:43.542 INFO:tasks.workunit.client.1.smithi122.stdout:3/13: mkdir d2 0 2024-03-22T02:28:43.542 INFO:tasks.workunit.client.1.smithi122.stdout:2/42: chown f4 15 1 2024-03-22T02:28:43.543 INFO:tasks.workunit.client.0.smithi103.stdout:3/695: getdents de/d1a/d25/d3f/d9c 0 2024-03-22T02:28:43.546 INFO:tasks.workunit.client.1.smithi122.stdout:1/15: symlink l4 0 2024-03-22T02:28:43.546 INFO:tasks.workunit.client.0.smithi103.stdout:4/564: truncate d3/d1e/d25/f3d 634843 0 2024-03-22T02:28:43.547 INFO:tasks.workunit.client.0.smithi103.stdout:4/565: chown d3/f8 135337 1 2024-03-22T02:28:43.547 INFO:tasks.workunit.client.1.smithi122.stdout:2/43: symlink df/l12 0 2024-03-22T02:28:43.548 INFO:tasks.workunit.client.1.smithi122.stdout:1/16: write f2 [41873,59880] 0 2024-03-22T02:28:43.549 INFO:tasks.workunit.client.1.smithi122.stdout:4/58: mknod d0/d2/d7/de/c17 0 2024-03-22T02:28:43.550 INFO:tasks.workunit.client.0.smithi103.stdout:9/595: write d2/d10/d27/f4f [1384437,54600] 0 2024-03-22T02:28:43.552 INFO:tasks.workunit.client.0.smithi103.stdout:4/566: write d3/f7 [1717124,119988] 0 2024-03-22T02:28:43.552 INFO:tasks.workunit.client.1.smithi122.stdout:2/44: read - df/f11 zero size 2024-03-22T02:28:43.553 INFO:tasks.workunit.client.1.smithi122.stdout:1/17: truncate f2 886672 0 2024-03-22T02:28:43.553 INFO:tasks.workunit.client.1.smithi122.stdout:5/31: fsync f4 0 2024-03-22T02:28:43.554 INFO:tasks.workunit.client.0.smithi103.stdout:6/452: write d0/d3/f9 [1539127,117155] 0 2024-03-22T02:28:43.554 INFO:tasks.workunit.client.0.smithi103.stdout:1/622: write db/d10/d18/d2d/f5d [2010388,53843] 0 2024-03-22T02:28:43.554 INFO:tasks.workunit.client.1.smithi122.stdout:9/36: truncate f5 2504561 0 2024-03-22T02:28:43.555 INFO:tasks.workunit.client.0.smithi103.stdout:3/696: readlink de/d12/d8a/db0/lab 0 2024-03-22T02:28:43.556 INFO:tasks.workunit.client.1.smithi122.stdout:6/18: dwrite f6 [0,4194304] 0 2024-03-22T02:28:43.557 INFO:tasks.workunit.client.1.smithi122.stdout:3/14: dread f1 [0,4194304] 0 2024-03-22T02:28:43.560 INFO:tasks.workunit.client.1.smithi122.stdout:4/59: creat d0/d12/d15/f18 x:0 0 0 2024-03-22T02:28:43.561 INFO:tasks.workunit.client.0.smithi103.stdout:6/453: unlink d0/d3/d22/f1c 0 2024-03-22T02:28:43.563 INFO:tasks.workunit.client.1.smithi122.stdout:2/45: link f4 df/f13 0 2024-03-22T02:28:43.564 INFO:tasks.workunit.client.1.smithi122.stdout:9/37: symlink db/ld 0 2024-03-22T02:28:43.564 INFO:tasks.workunit.client.0.smithi103.stdout:9/596: symlink d2/d10/d43/lb9 0 2024-03-22T02:28:43.564 INFO:tasks.workunit.client.0.smithi103.stdout:8/774: dwrite d4/d26/d27/d7e/d39/f9a [0,4194304] 0 2024-03-22T02:28:43.565 INFO:tasks.workunit.client.0.smithi103.stdout:3/697: unlink de/d12/d30/d70/laf 0 2024-03-22T02:28:43.566 INFO:tasks.workunit.client.1.smithi122.stdout:2/46: chown f4 9 1 2024-03-22T02:28:43.568 INFO:tasks.workunit.client.1.smithi122.stdout:4/60: write d0/d2/d7/fb [3637808,38873] 0 2024-03-22T02:28:43.569 INFO:tasks.workunit.client.1.smithi122.stdout:3/15: link f1 d2/f3 0 2024-03-22T02:28:43.569 INFO:tasks.workunit.client.1.smithi122.stdout:5/32: dwrite f4 [4194304,4194304] 0 2024-03-22T02:28:43.569 INFO:tasks.workunit.client.0.smithi103.stdout:1/623: dwrite db/d10/d18/d2d/fbe [0,4194304] 0 2024-03-22T02:28:43.572 INFO:tasks.workunit.client.1.smithi122.stdout:4/61: chown d0/d2/d7/de 8284 1 2024-03-22T02:28:43.575 INFO:tasks.workunit.client.1.smithi122.stdout:2/47: creat df/f14 x:0 0 0 2024-03-22T02:28:43.575 INFO:tasks.workunit.client.1.smithi122.stdout:6/19: dwrite f5 [0,4194304] 0 2024-03-22T02:28:43.577 INFO:tasks.workunit.client.1.smithi122.stdout:2/48: truncate fd 4606330 0 2024-03-22T02:28:43.581 INFO:tasks.workunit.client.1.smithi122.stdout:2/49: dread - df/f14 zero size 2024-03-22T02:28:43.581 INFO:tasks.workunit.client.0.smithi103.stdout:3/698: read de/f22 [3090400,122787] 0 2024-03-22T02:28:43.582 INFO:tasks.workunit.client.0.smithi103.stdout:2/533: write d11/d25/d46/d34/f52 [164157,86532] 0 2024-03-22T02:28:43.582 INFO:tasks.workunit.client.0.smithi103.stdout:7/608: dread d1/f7e [0,4194304] 0 2024-03-22T02:28:43.583 INFO:tasks.workunit.client.1.smithi122.stdout:1/18: read f2 [542966,7587] 0 2024-03-22T02:28:43.583 INFO:tasks.workunit.client.1.smithi122.stdout:5/33: rename l6 to l7 0 2024-03-22T02:28:43.583 INFO:tasks.workunit.client.1.smithi122.stdout:7/54: truncate d0/d4/f5 251356 0 2024-03-22T02:28:43.583 INFO:tasks.workunit.client.0.smithi103.stdout:4/567: creat d3/d1e/fc3 x:0 0 0 2024-03-22T02:28:43.585 INFO:tasks.workunit.client.1.smithi122.stdout:1/19: stat f2 0 2024-03-22T02:28:43.585 INFO:tasks.workunit.client.1.smithi122.stdout:6/20: mkdir d7/d9 0 2024-03-22T02:28:43.586 INFO:tasks.workunit.client.0.smithi103.stdout:4/568: truncate d3/d1e/fc3 800475 0 2024-03-22T02:28:43.587 INFO:tasks.workunit.client.0.smithi103.stdout:5/635: write d0/d14/d23/d5c/d70/fc4 [266170,103602] 0 2024-03-22T02:28:43.588 INFO:tasks.workunit.client.1.smithi122.stdout:4/62: creat d0/f19 x:0 0 0 2024-03-22T02:28:43.588 INFO:tasks.workunit.client.1.smithi122.stdout:7/55: write d0/d4/fb [959242,126497] 0 2024-03-22T02:28:43.590 INFO:tasks.workunit.client.0.smithi103.stdout:8/775: creat d4/ffa x:0 0 0 2024-03-22T02:28:43.590 INFO:tasks.workunit.client.1.smithi122.stdout:4/63: write d0/d12/d15/f18 [94693,46513] 0 2024-03-22T02:28:43.591 INFO:tasks.workunit.client.0.smithi103.stdout:8/776: dread - d4/ff6 zero size 2024-03-22T02:28:43.591 INFO:tasks.workunit.client.0.smithi103.stdout:6/454: creat d0/d88/f8a x:0 0 0 2024-03-22T02:28:43.591 INFO:tasks.workunit.client.0.smithi103.stdout:3/699: creat de/d6b/d80/d4f/d84/fe8 x:0 0 0 2024-03-22T02:28:43.594 INFO:tasks.workunit.client.0.smithi103.stdout:4/569: mknod d3/d1e/d25/d65/d83/cc4 0 2024-03-22T02:28:43.594 INFO:tasks.workunit.client.1.smithi122.stdout:2/50: creat df/f15 x:0 0 0 2024-03-22T02:28:43.594 INFO:tasks.workunit.client.1.smithi122.stdout:5/34: mkdir d8 0 2024-03-22T02:28:43.594 INFO:tasks.workunit.client.0.smithi103.stdout:5/636: read d0/d14/d23/fcf [1660321,6857] 0 2024-03-22T02:28:43.595 INFO:tasks.workunit.client.1.smithi122.stdout:1/20: symlink l5 0 2024-03-22T02:28:43.595 INFO:tasks.workunit.client.1.smithi122.stdout:5/35: readlink l3 0 2024-03-22T02:28:43.595 INFO:tasks.workunit.client.1.smithi122.stdout:6/21: creat d7/fa x:0 0 0 2024-03-22T02:28:43.596 INFO:tasks.workunit.client.0.smithi103.stdout:4/570: stat d3/da2 0 2024-03-22T02:28:43.596 INFO:tasks.workunit.client.1.smithi122.stdout:4/64: rmdir d0/d12 39 2024-03-22T02:28:43.598 INFO:tasks.workunit.client.0.smithi103.stdout:6/455: stat d0/d3/d22/d58/d6f 0 2024-03-22T02:28:43.601 INFO:tasks.workunit.client.1.smithi122.stdout:5/36: dread f4 [4194304,4194304] 0 2024-03-22T02:28:43.601 INFO:tasks.workunit.client.0.smithi103.stdout:4/571: chown d3/d1e/f43 193712 1 2024-03-22T02:28:43.602 INFO:tasks.workunit.client.1.smithi122.stdout:2/51: creat df/f16 x:0 0 0 2024-03-22T02:28:43.603 INFO:tasks.workunit.client.0.smithi103.stdout:0/627: dread dd/d34/fac [0,4194304] 0 2024-03-22T02:28:43.603 INFO:tasks.workunit.client.0.smithi103.stdout:5/637: write d0/d14/d23/d2a/d2c/f4b [5114974,109548] 0 2024-03-22T02:28:43.604 INFO:tasks.workunit.client.1.smithi122.stdout:1/21: write f2 [97037,91719] 0 2024-03-22T02:28:43.604 INFO:tasks.workunit.client.1.smithi122.stdout:1/22: rmdir - no directory 2024-03-22T02:28:43.605 INFO:tasks.workunit.client.0.smithi103.stdout:4/572: write d3/d69/fb0 [904443,41784] 0 2024-03-22T02:28:43.606 INFO:tasks.workunit.client.1.smithi122.stdout:5/37: write f4 [1803736,13544] 0 2024-03-22T02:28:43.606 INFO:tasks.workunit.client.0.smithi103.stdout:8/777: symlink d4/d2a/d3e/da4/lfb 0 2024-03-22T02:28:43.606 INFO:tasks.workunit.client.0.smithi103.stdout:9/597: write d2/d10/d59/f89 [185304,91041] 0 2024-03-22T02:28:43.607 INFO:tasks.workunit.client.0.smithi103.stdout:6/456: creat d0/d3/d22/d58/d6f/f8b x:0 0 0 2024-03-22T02:28:43.608 INFO:tasks.workunit.client.1.smithi122.stdout:3/16: dwrite f1 [4194304,4194304] 0 2024-03-22T02:28:43.610 INFO:tasks.workunit.client.0.smithi103.stdout:0/628: readlink dd/d34/l40 0 2024-03-22T02:28:43.611 INFO:tasks.workunit.client.1.smithi122.stdout:6/22: symlink d7/lb 0 2024-03-22T02:28:43.611 INFO:tasks.workunit.client.0.smithi103.stdout:2/534: creat d11/d25/d63/d45/d47/d83/fac x:0 0 0 2024-03-22T02:28:43.613 INFO:tasks.workunit.client.0.smithi103.stdout:9/598: read d2/d10/d59/f5e [2769551,79135] 0 2024-03-22T02:28:43.614 INFO:tasks.workunit.client.0.smithi103.stdout:5/638: readlink d0/d14/d23/d2a/d7b/la6 0 2024-03-22T02:28:43.615 INFO:tasks.workunit.client.1.smithi122.stdout:1/23: symlink l6 0 2024-03-22T02:28:43.615 INFO:tasks.workunit.client.1.smithi122.stdout:4/65: fdatasync d0/d2/f13 0 2024-03-22T02:28:43.615 INFO:tasks.workunit.client.0.smithi103.stdout:2/535: dread - d11/d25/d63/d45/f94 zero size 2024-03-22T02:28:43.616 INFO:tasks.workunit.client.0.smithi103.stdout:6/457: creat d0/d3/d22/d58/f8c x:0 0 0 2024-03-22T02:28:43.616 INFO:tasks.workunit.client.1.smithi122.stdout:1/24: chown l6 17208 1 2024-03-22T02:28:43.617 INFO:tasks.workunit.client.1.smithi122.stdout:9/38: dread f5 [0,4194304] 0 2024-03-22T02:28:43.617 INFO:tasks.workunit.client.0.smithi103.stdout:4/573: symlink d3/d1e/d25/d65/d83/lc5 0 2024-03-22T02:28:43.618 INFO:tasks.workunit.client.0.smithi103.stdout:8/778: mkdir d4/d26/d27/dfc 0 2024-03-22T02:28:43.619 INFO:tasks.workunit.client.1.smithi122.stdout:2/52: rename df/f16 to df/f17 0 2024-03-22T02:28:43.619 INFO:tasks.workunit.client.1.smithi122.stdout:9/39: readlink db/ld 0 2024-03-22T02:28:43.621 INFO:tasks.workunit.client.1.smithi122.stdout:1/25: write f2 [1608488,40815] 0 2024-03-22T02:28:43.622 INFO:tasks.workunit.client.0.smithi103.stdout:3/700: dwrite de/d12/d30/fa6 [4194304,4194304] 0 2024-03-22T02:28:43.622 INFO:tasks.workunit.client.0.smithi103.stdout:9/599: dread - d2/da/d18/d1e/d77/d8b/fb6 zero size 2024-03-22T02:28:43.623 INFO:tasks.workunit.client.0.smithi103.stdout:5/639: unlink d0/f71 0 2024-03-22T02:28:43.624 INFO:tasks.workunit.client.1.smithi122.stdout:4/66: write d0/d2/f13 [3246952,73889] 0 2024-03-22T02:28:43.624 INFO:tasks.workunit.client.1.smithi122.stdout:5/38: mknod d8/c9 0 2024-03-22T02:28:43.626 INFO:tasks.workunit.client.1.smithi122.stdout:3/17: dwrite d2/f3 [4194304,4194304] 0 2024-03-22T02:28:43.629 INFO:tasks.workunit.client.1.smithi122.stdout:6/23: dwrite f3 [0,4194304] 0 2024-03-22T02:28:43.629 INFO:tasks.workunit.client.1.smithi122.stdout:3/18: write f1 [5634697,13458] 0 2024-03-22T02:28:43.632 INFO:tasks.workunit.client.0.smithi103.stdout:0/629: creat dd/d16/d4b/d96/d64/d91/d9a/d9d/fd3 x:0 0 0 2024-03-22T02:28:43.634 INFO:tasks.workunit.client.0.smithi103.stdout:6/458: dread d0/d14/d57/d68/f2d [0,4194304] 0 2024-03-22T02:28:43.635 INFO:tasks.workunit.client.0.smithi103.stdout:1/624: dwrite db/f25 [0,4194304] 0 2024-03-22T02:28:43.635 INFO:tasks.workunit.client.0.smithi103.stdout:4/574: symlink d3/d1e/d53/d8a/lc6 0 2024-03-22T02:28:43.636 INFO:tasks.workunit.client.1.smithi122.stdout:4/67: write d0/d2/f8 [889589,83683] 0 2024-03-22T02:28:43.636 INFO:tasks.workunit.client.0.smithi103.stdout:7/609: truncate d1/d10/d11/d19/d29/d61/d69/f62 2458714 0 2024-03-22T02:28:43.638 INFO:tasks.workunit.client.1.smithi122.stdout:4/68: stat d0 0 2024-03-22T02:28:43.639 INFO:tasks.workunit.client.0.smithi103.stdout:5/640: rename d0/d14/d23/d6f/d83/fa1 to d0/d14/d23/d6f/fd7 0 2024-03-22T02:28:43.640 INFO:tasks.workunit.client.0.smithi103.stdout:0/630: chown dd/d16/d4b/d96/d29/d48/l63 48132451 1 2024-03-22T02:28:43.642 INFO:tasks.workunit.client.1.smithi122.stdout:6/24: mkdir d7/dc 0 2024-03-22T02:28:43.642 INFO:tasks.workunit.client.1.smithi122.stdout:3/19: dwrite d2/f3 [0,4194304] 0 2024-03-22T02:28:43.643 INFO:tasks.workunit.client.0.smithi103.stdout:7/610: chown d1/d10/d11/d19/d29/d61/d69/c27 600034 1 2024-03-22T02:28:43.645 INFO:tasks.workunit.client.0.smithi103.stdout:7/611: write d1/d10/d11/d19/d29/d61/f7b [522926,105504] 0 2024-03-22T02:28:43.645 INFO:tasks.workunit.client.1.smithi122.stdout:5/39: dwrite f4 [4194304,4194304] 0 2024-03-22T02:28:43.645 INFO:tasks.workunit.client.1.smithi122.stdout:9/40: link c9 db/ce 0 2024-03-22T02:28:43.645 INFO:tasks.workunit.client.0.smithi103.stdout:9/600: dread d2/da/d18/d1e/d34/f97 [0,4194304] 0 2024-03-22T02:28:43.646 INFO:tasks.workunit.client.1.smithi122.stdout:9/41: write f6 [614143,123062] 0 2024-03-22T02:28:43.648 INFO:tasks.workunit.client.1.smithi122.stdout:9/42: write f7 [443597,14779] 0 2024-03-22T02:28:43.649 INFO:tasks.workunit.client.1.smithi122.stdout:4/69: symlink d0/d2/l1a 0 2024-03-22T02:28:43.650 INFO:tasks.workunit.client.1.smithi122.stdout:4/70: write d0/f9 [2360374,98953] 0 2024-03-22T02:28:43.652 INFO:tasks.workunit.client.1.smithi122.stdout:3/20: rename f1 to d2/f4 0 2024-03-22T02:28:43.652 INFO:tasks.workunit.client.1.smithi122.stdout:3/21: readlink - no filename 2024-03-22T02:28:43.653 INFO:tasks.workunit.client.1.smithi122.stdout:4/71: write d0/f9 [1053779,10690] 0 2024-03-22T02:28:43.654 INFO:tasks.workunit.client.1.smithi122.stdout:5/40: mkdir d8/da 0 2024-03-22T02:28:43.657 INFO:tasks.workunit.client.0.smithi103.stdout:1/625: mkdir db/d10/d29/d41/d68/d74/dd5 0 2024-03-22T02:28:43.657 INFO:tasks.workunit.client.0.smithi103.stdout:7/612: chown d1/d10/d11/d19/d29/d61/d69/c27 239838 1 2024-03-22T02:28:43.658 INFO:tasks.workunit.client.1.smithi122.stdout:7/56: dread d0/d4/fb [0,4194304] 0 2024-03-22T02:28:43.659 INFO:tasks.workunit.client.0.smithi103.stdout:8/779: creat d4/d26/d27/d7e/d39/d49/d51/daf/dc2/ffd x:0 0 0 2024-03-22T02:28:43.661 INFO:tasks.workunit.client.0.smithi103.stdout:7/613: chown d1/d10/d11/d19/d29/d33/d4b/f97 92685323 1 2024-03-22T02:28:43.662 INFO:tasks.workunit.client.1.smithi122.stdout:2/53: rename df/f17 to df/f18 0 2024-03-22T02:28:43.663 INFO:tasks.workunit.client.1.smithi122.stdout:1/26: dread f2 [0,4194304] 0 2024-03-22T02:28:43.663 INFO:tasks.workunit.client.0.smithi103.stdout:5/641: dread d0/d14/f41 [4194304,4194304] 0 2024-03-22T02:28:43.665 INFO:tasks.workunit.client.0.smithi103.stdout:1/626: truncate db/d10/d28/fa7 709876 0 2024-03-22T02:28:43.671 INFO:tasks.workunit.client.1.smithi122.stdout:9/43: fsync f6 0 2024-03-22T02:28:43.672 INFO:tasks.workunit.client.1.smithi122.stdout:7/57: creat d0/fd x:0 0 0 2024-03-22T02:28:43.672 INFO:tasks.workunit.client.1.smithi122.stdout:2/54: dwrite df/f11 [0,4194304] 0 2024-03-22T02:28:43.673 INFO:tasks.workunit.client.1.smithi122.stdout:1/27: creat f7 x:0 0 0 2024-03-22T02:28:43.673 INFO:tasks.workunit.client.1.smithi122.stdout:2/55: readlink l5 0 2024-03-22T02:28:43.674 INFO:tasks.workunit.client.1.smithi122.stdout:4/72: mknod d0/d12/d15/c1b 0 2024-03-22T02:28:43.674 INFO:tasks.workunit.client.1.smithi122.stdout:5/41: chown l7 122736 1 2024-03-22T02:28:43.676 INFO:tasks.workunit.client.1.smithi122.stdout:1/28: write f7 [799868,19874] 0 2024-03-22T02:28:43.677 INFO:tasks.workunit.client.1.smithi122.stdout:7/58: write d0/fd [390058,68131] 0 2024-03-22T02:28:43.677 INFO:tasks.workunit.client.1.smithi122.stdout:2/56: creat df/f19 x:0 0 0 2024-03-22T02:28:43.678 INFO:tasks.workunit.client.1.smithi122.stdout:2/57: chown la 436034 1 2024-03-22T02:28:43.681 INFO:tasks.workunit.client.1.smithi122.stdout:1/29: dread f2 [0,4194304] 0 2024-03-22T02:28:43.683 INFO:tasks.workunit.client.1.smithi122.stdout:9/44: mkdir db/df 0 2024-03-22T02:28:43.684 INFO:tasks.workunit.client.1.smithi122.stdout:7/59: write d0/f2 [3220816,128472] 0 2024-03-22T02:28:43.685 INFO:tasks.workunit.client.1.smithi122.stdout:2/58: mknod df/c1a 0 2024-03-22T02:28:43.685 INFO:tasks.workunit.client.1.smithi122.stdout:4/73: dread d0/d2/fa [0,4194304] 0 2024-03-22T02:28:43.686 INFO:tasks.workunit.client.1.smithi122.stdout:5/42: fsync f4 0 2024-03-22T02:28:43.686 INFO:tasks.workunit.client.1.smithi122.stdout:1/30: rename c3 to c8 0 2024-03-22T02:28:43.690 INFO:tasks.workunit.client.1.smithi122.stdout:2/59: write f4 [4371616,70966] 0 2024-03-22T02:28:43.691 INFO:tasks.workunit.client.1.smithi122.stdout:2/60: chown l5 144073 1 2024-03-22T02:28:43.694 INFO:tasks.workunit.client.1.smithi122.stdout:5/43: rename l3 to d8/da/lb 0 2024-03-22T02:28:43.695 INFO:tasks.workunit.client.0.smithi103.stdout:6/459: rmdir d0/d3/d22/d58/d6f 39 2024-03-22T02:28:43.697 INFO:tasks.workunit.client.0.smithi103.stdout:9/601: fdatasync d2/f3b 0 2024-03-22T02:28:43.699 INFO:tasks.workunit.client.1.smithi122.stdout:3/22: getdents d2 0 2024-03-22T02:28:43.701 INFO:tasks.workunit.client.1.smithi122.stdout:1/31: dwrite f2 [0,4194304] 0 2024-03-22T02:28:43.706 INFO:tasks.workunit.client.1.smithi122.stdout:4/74: rename d0/d12/d15/f18 to d0/d12/f1c 0 2024-03-22T02:28:43.706 INFO:tasks.workunit.client.1.smithi122.stdout:5/44: rename d8/da to d8/da/dc 22 2024-03-22T02:28:43.707 INFO:tasks.workunit.client.0.smithi103.stdout:4/575: dwrite d3/d69/f54 [0,4194304] 0 2024-03-22T02:28:43.708 INFO:tasks.workunit.client.1.smithi122.stdout:6/25: truncate f6 4130192 0 2024-03-22T02:28:43.709 INFO:tasks.workunit.client.1.smithi122.stdout:6/26: write f5 [3115788,83290] 0 2024-03-22T02:28:43.711 INFO:tasks.workunit.client.0.smithi103.stdout:4/576: truncate d3/d1e/d25/d65/f99 346415 0 2024-03-22T02:28:43.711 INFO:tasks.workunit.client.1.smithi122.stdout:5/45: write f4 [9418783,34732] 0 2024-03-22T02:28:43.713 INFO:tasks.workunit.client.1.smithi122.stdout:1/32: dwrite f2 [0,4194304] 0 2024-03-22T02:28:43.714 INFO:tasks.workunit.client.1.smithi122.stdout:5/46: read f4 [3027482,12235] 0 2024-03-22T02:28:43.714 INFO:tasks.workunit.client.1.smithi122.stdout:1/33: readlink l6 0 2024-03-22T02:28:43.717 INFO:tasks.workunit.client.1.smithi122.stdout:4/75: creat d0/d2/f1d x:0 0 0 2024-03-22T02:28:43.720 INFO:tasks.workunit.client.1.smithi122.stdout:7/60: getdents d0 0 2024-03-22T02:28:43.721 INFO:tasks.workunit.client.1.smithi122.stdout:4/76: readlink d0/d2/l1a 0 2024-03-22T02:28:43.722 INFO:tasks.workunit.client.1.smithi122.stdout:7/61: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:43.723 INFO:tasks.workunit.client.1.smithi122.stdout:2/61: write df/f11 [4365937,10965] 0 2024-03-22T02:28:43.724 INFO:tasks.workunit.client.1.smithi122.stdout:3/23: dwrite d2/f4 [4194304,4194304] 0 2024-03-22T02:28:43.725 INFO:tasks.workunit.client.1.smithi122.stdout:5/47: rename l7 to d8/ld 0 2024-03-22T02:28:43.726 INFO:tasks.workunit.client.1.smithi122.stdout:4/77: write d0/d2/f8 [4119370,86479] 0 2024-03-22T02:28:43.730 INFO:tasks.workunit.client.1.smithi122.stdout:6/27: mkdir d7/dc/dd 0 2024-03-22T02:28:43.734 INFO:tasks.workunit.client.1.smithi122.stdout:6/28: read f5 [292630,117183] 0 2024-03-22T02:28:43.735 INFO:tasks.workunit.client.1.smithi122.stdout:5/48: mknod d8/ce 0 2024-03-22T02:28:43.736 INFO:tasks.workunit.client.0.smithi103.stdout:3/701: creat de/d12/d30/d70/de5/fe9 x:0 0 0 2024-03-22T02:28:43.736 INFO:tasks.workunit.client.1.smithi122.stdout:7/62: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:43.737 INFO:tasks.workunit.client.1.smithi122.stdout:3/24: rename d2/f3 to d2/f5 0 2024-03-22T02:28:43.739 INFO:tasks.workunit.client.0.smithi103.stdout:0/631: rename dd/d34/d53/fb1 to dd/d16/d4b/d96/d35/dc4/fd4 0 2024-03-22T02:28:43.739 INFO:tasks.workunit.client.1.smithi122.stdout:2/62: dwrite f4 [4194304,4194304] 0 2024-03-22T02:28:43.742 INFO:tasks.workunit.client.1.smithi122.stdout:7/63: dread d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:43.743 INFO:tasks.workunit.client.0.smithi103.stdout:7/614: symlink d1/d10/d11/d70/lbe 0 2024-03-22T02:28:43.745 INFO:tasks.workunit.client.0.smithi103.stdout:1/627: rmdir db/d4a 39 2024-03-22T02:28:43.746 INFO:tasks.workunit.client.0.smithi103.stdout:3/702: dwrite de/d12/d8a/db0/fb2 [0,4194304] 0 2024-03-22T02:28:43.747 INFO:tasks.workunit.client.1.smithi122.stdout:9/45: dwrite f5 [0,4194304] 0 2024-03-22T02:28:43.750 INFO:tasks.workunit.client.0.smithi103.stdout:1/628: write db/d10/d28/d97/fb3 [758228,31761] 0 2024-03-22T02:28:43.750 INFO:tasks.workunit.client.1.smithi122.stdout:5/49: readlink d8/da/lb 0 2024-03-22T02:28:43.753 INFO:tasks.workunit.client.0.smithi103.stdout:9/602: symlink d2/d5/d4c/d56/lba 0 2024-03-22T02:28:43.754 INFO:tasks.workunit.client.1.smithi122.stdout:4/78: link d0/d2/f8 d0/d2/f1e 0 2024-03-22T02:28:43.755 INFO:tasks.workunit.client.0.smithi103.stdout:4/577: write d3/d1e/d25/d65/f99 [296050,94965] 0 2024-03-22T02:28:43.757 INFO:tasks.workunit.client.1.smithi122.stdout:6/29: rename d7/dc to d7/de 0 2024-03-22T02:28:43.759 INFO:tasks.workunit.client.1.smithi122.stdout:9/46: dwrite fa [0,4194304] 0 2024-03-22T02:28:43.760 INFO:tasks.workunit.client.1.smithi122.stdout:2/63: rename c0 to df/c1b 0 2024-03-22T02:28:43.761 INFO:tasks.workunit.client.1.smithi122.stdout:7/64: write d0/d4/f5 [39602,14796] 0 2024-03-22T02:28:43.762 INFO:tasks.workunit.client.1.smithi122.stdout:1/34: getdents . 0 2024-03-22T02:28:43.762 INFO:tasks.workunit.client.1.smithi122.stdout:1/35: readlink l6 0 2024-03-22T02:28:43.763 INFO:tasks.workunit.client.1.smithi122.stdout:1/36: fdatasync f2 0 2024-03-22T02:28:43.764 INFO:tasks.workunit.client.1.smithi122.stdout:3/25: truncate d2/f5 1688707 0 2024-03-22T02:28:43.766 INFO:tasks.workunit.client.0.smithi103.stdout:3/703: creat de/d6b/d80/d73/fea x:0 0 0 2024-03-22T02:28:43.767 INFO:tasks.workunit.client.0.smithi103.stdout:0/632: dwrite dd/d16/d4b/d96/f21 [0,4194304] 0 2024-03-22T02:28:43.767 INFO:tasks.workunit.client.0.smithi103.stdout:7/615: truncate d1/d10/d11/d19/d29/d61/d69/fa1 192800 0 2024-03-22T02:28:43.772 INFO:tasks.workunit.client.0.smithi103.stdout:4/578: unlink d3/d30/d45/lbb 0 2024-03-22T02:28:43.772 INFO:tasks.workunit.client.0.smithi103.stdout:4/579: write d3/d1e/fc3 [1085531,109000] 0 2024-03-22T02:28:43.773 INFO:tasks.workunit.client.1.smithi122.stdout:4/79: dwrite d0/d2/fa [0,4194304] 0 2024-03-22T02:28:43.775 INFO:tasks.workunit.client.1.smithi122.stdout:1/37: dwrite f2 [0,4194304] 0 2024-03-22T02:28:43.777 INFO:tasks.workunit.client.1.smithi122.stdout:2/64: rename l2 to df/l1c 0 2024-03-22T02:28:43.777 INFO:tasks.workunit.client.0.smithi103.stdout:6/460: link d0/d14/d57/d68/f62 d0/d3/d22/d58/d6f/d75/f8d 0 2024-03-22T02:28:43.779 INFO:tasks.workunit.client.1.smithi122.stdout:7/65: write d0/d4/f9 [941691,122941] 0 2024-03-22T02:28:43.780 INFO:tasks.workunit.client.0.smithi103.stdout:1/629: mkdir db/d10/d29/d41/d68/d74/dba/dd6 0 2024-03-22T02:28:43.785 INFO:tasks.workunit.client.1.smithi122.stdout:6/30: mkdir d7/de/dd/df 0 2024-03-22T02:28:43.786 INFO:tasks.workunit.client.0.smithi103.stdout:3/704: dwrite de/f22 [0,4194304] 0 2024-03-22T02:28:43.787 INFO:tasks.workunit.client.1.smithi122.stdout:6/31: chown d7/de 82298593 1 2024-03-22T02:28:43.787 INFO:tasks.workunit.client.1.smithi122.stdout:4/80: dwrite d0/f9 [0,4194304] 0 2024-03-22T02:28:43.787 INFO:tasks.workunit.client.0.smithi103.stdout:3/705: chown de/d6b/d80/d4f/f65 78186587 1 2024-03-22T02:28:43.788 INFO:tasks.workunit.client.1.smithi122.stdout:6/32: readlink d7/lb 0 2024-03-22T02:28:43.788 INFO:tasks.workunit.client.0.smithi103.stdout:5/642: rename d0/d14/d23/d2a/d33/f4c to d0/d14/d23/d2a/fd8 0 2024-03-22T02:28:43.791 INFO:tasks.workunit.client.0.smithi103.stdout:6/461: mkdir d0/d88/d8e 0 2024-03-22T02:28:43.794 INFO:tasks.workunit.client.0.smithi103.stdout:1/630: creat db/d10/d29/d55/fd7 x:0 0 0 2024-03-22T02:28:43.796 INFO:tasks.workunit.client.0.smithi103.stdout:0/633: rmdir dd/d16/d4b/d96/d29/d48/d5c/d9b 39 2024-03-22T02:28:43.796 INFO:tasks.workunit.client.1.smithi122.stdout:7/66: fsync d0/f8 0 2024-03-22T02:28:43.798 INFO:tasks.workunit.client.1.smithi122.stdout:7/67: truncate d0/d4/f5 632921 0 2024-03-22T02:28:43.798 INFO:tasks.workunit.client.0.smithi103.stdout:7/616: mkdir d1/d10/d11/d19/d86/d92/dbf 0 2024-03-22T02:28:43.799 INFO:tasks.workunit.client.1.smithi122.stdout:1/38: creat f9 x:0 0 0 2024-03-22T02:28:43.800 INFO:tasks.workunit.client.0.smithi103.stdout:1/631: read db/d10/d29/fb8 [1536181,115624] 0 2024-03-22T02:28:43.801 INFO:tasks.workunit.client.1.smithi122.stdout:5/50: rename f4 to d8/da/ff 0 2024-03-22T02:28:43.802 INFO:tasks.workunit.client.0.smithi103.stdout:1/632: truncate db/d10/d28/f43 742923 0 2024-03-22T02:28:43.804 INFO:tasks.workunit.client.1.smithi122.stdout:4/81: rmdir d0/d2/d7 39 2024-03-22T02:28:43.805 INFO:tasks.workunit.client.0.smithi103.stdout:5/643: mkdir d0/d14/d23/d2a/d2c/dd9 0 2024-03-22T02:28:43.806 INFO:tasks.workunit.client.1.smithi122.stdout:1/39: mknod ca 0 2024-03-22T02:28:43.806 INFO:tasks.workunit.client.0.smithi103.stdout:6/462: rmdir d0/d3/d22 39 2024-03-22T02:28:43.808 INFO:tasks.workunit.client.1.smithi122.stdout:4/82: write d0/d12/f1c [59644,97707] 0 2024-03-22T02:28:43.808 INFO:tasks.workunit.client.1.smithi122.stdout:1/40: mknod cb 0 2024-03-22T02:28:43.808 INFO:tasks.workunit.client.1.smithi122.stdout:6/33: dread f5 [0,4194304] 0 2024-03-22T02:28:43.808 INFO:tasks.workunit.client.1.smithi122.stdout:7/68: getdents d0 0 2024-03-22T02:28:43.809 INFO:tasks.workunit.client.1.smithi122.stdout:4/83: stat d0/d2/f13 0 2024-03-22T02:28:43.810 INFO:tasks.workunit.client.1.smithi122.stdout:7/69: fsync d0/fd 0 2024-03-22T02:28:43.813 INFO:tasks.workunit.client.0.smithi103.stdout:1/633: mknod db/d10/d28/db2/dc8/cd8 0 2024-03-22T02:28:43.814 INFO:tasks.workunit.client.0.smithi103.stdout:1/634: readlink db/d10/d18/d2d/l94 0 2024-03-22T02:28:43.815 INFO:tasks.workunit.client.1.smithi122.stdout:7/70: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:43.816 INFO:tasks.workunit.client.1.smithi122.stdout:1/41: dwrite f7 [0,4194304] 0 2024-03-22T02:28:43.816 INFO:tasks.workunit.client.0.smithi103.stdout:5/644: creat d0/d14/d23/d6f/fda x:0 0 0 2024-03-22T02:28:43.817 INFO:tasks.workunit.client.1.smithi122.stdout:4/84: rename d0/d2/d7/de/c17 to d0/d12/d15/c1f 0 2024-03-22T02:28:43.818 INFO:tasks.workunit.client.1.smithi122.stdout:4/85: read d0/d2/d7/fb [2431617,117500] 0 2024-03-22T02:28:43.819 INFO:tasks.workunit.client.0.smithi103.stdout:0/634: mkdir dd/d16/d4b/d96/d64/d6a/dd5 0 2024-03-22T02:28:43.820 INFO:tasks.workunit.client.0.smithi103.stdout:4/580: getdents d3/d38/d5c/d90 0 2024-03-22T02:28:43.822 INFO:tasks.workunit.client.1.smithi122.stdout:1/42: mkdir dc 0 2024-03-22T02:28:43.822 INFO:tasks.workunit.client.0.smithi103.stdout:1/635: mkdir db/d10/d29/d41/d68/d74/dba/dd9 0 2024-03-22T02:28:43.823 INFO:tasks.workunit.client.0.smithi103.stdout:3/706: link de/d1a/d25/d3f/d9c/da7/ce7 de/d12/d4d/ceb 0 2024-03-22T02:28:43.823 INFO:tasks.workunit.client.0.smithi103.stdout:4/581: chown d3/d38/d5c/faa 3406599 1 2024-03-22T02:28:43.824 INFO:tasks.workunit.client.1.smithi122.stdout:3/26: fsync d2/f5 0 2024-03-22T02:28:43.825 INFO:tasks.workunit.client.1.smithi122.stdout:2/65: rmdir df 39 2024-03-22T02:28:43.826 INFO:tasks.workunit.client.0.smithi103.stdout:5/645: unlink d0/d14/d23/d5c/f92 0 2024-03-22T02:28:43.826 INFO:tasks.workunit.client.0.smithi103.stdout:6/463: dwrite d0/d3/d22/d58/f89 [0,4194304] 0 2024-03-22T02:28:43.828 INFO:tasks.workunit.client.1.smithi122.stdout:3/27: dread d2/f5 [0,4194304] 0 2024-03-22T02:28:43.830 INFO:tasks.workunit.client.1.smithi122.stdout:7/71: dwrite d0/f2 [4194304,4194304] 0 2024-03-22T02:28:43.831 INFO:tasks.workunit.client.1.smithi122.stdout:3/28: write d2/f5 [2179221,69662] 0 2024-03-22T02:28:43.833 INFO:tasks.workunit.client.0.smithi103.stdout:3/707: creat de/d6b/fec x:0 0 0 2024-03-22T02:28:43.835 INFO:tasks.workunit.client.0.smithi103.stdout:1/636: creat db/d10/d29/d41/d68/fda x:0 0 0 2024-03-22T02:28:43.836 INFO:tasks.workunit.client.0.smithi103.stdout:4/582: mkdir d3/d1e/d25/d65/d83/dc7 0 2024-03-22T02:28:43.838 INFO:tasks.workunit.client.1.smithi122.stdout:5/51: unlink d8/da/ff 0 2024-03-22T02:28:43.839 INFO:tasks.workunit.client.1.smithi122.stdout:5/52: chown c5 15519 1 2024-03-22T02:28:43.839 INFO:tasks.workunit.client.1.smithi122.stdout:1/43: creat dc/fd x:0 0 0 2024-03-22T02:28:43.840 INFO:tasks.workunit.client.1.smithi122.stdout:7/72: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:43.840 INFO:tasks.workunit.client.0.smithi103.stdout:3/708: symlink de/d6b/d80/d4f/led 0 2024-03-22T02:28:43.842 INFO:tasks.workunit.client.1.smithi122.stdout:3/29: dread d2/f4 [0,4194304] 0 2024-03-22T02:28:43.843 INFO:tasks.workunit.client.0.smithi103.stdout:3/709: truncate de/d6b/d80/d4f/d84/fe8 770541 0 2024-03-22T02:28:43.844 INFO:tasks.workunit.client.0.smithi103.stdout:6/464: symlink d0/d3/d33/l8f 0 2024-03-22T02:28:43.845 INFO:tasks.workunit.client.1.smithi122.stdout:1/44: creat dc/fe x:0 0 0 2024-03-22T02:28:43.846 INFO:tasks.workunit.client.0.smithi103.stdout:4/583: symlink d3/d1e/lc8 0 2024-03-22T02:28:43.847 INFO:tasks.workunit.client.1.smithi122.stdout:7/73: rmdir d0/d4 39 2024-03-22T02:28:43.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:43 smithi122 ceph-mon[36403]: pgmap v15: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 153 MiB/s wr, 355 op/s 2024-03-22T02:28:43.849 INFO:tasks.workunit.client.0.smithi103.stdout:5/646: truncate d0/d14/d23/d46/f7a 816160 0 2024-03-22T02:28:43.849 INFO:tasks.workunit.client.1.smithi122.stdout:1/45: creat dc/ff x:0 0 0 2024-03-22T02:28:43.850 INFO:tasks.workunit.client.0.smithi103.stdout:3/710: dread de/d6b/d80/d4f/f5b [0,4194304] 0 2024-03-22T02:28:43.851 INFO:tasks.workunit.client.1.smithi122.stdout:1/46: readlink l5 0 2024-03-22T02:28:43.852 INFO:tasks.workunit.client.0.smithi103.stdout:3/711: write de/d1a/d25/d26/f69 [115961,83569] 0 2024-03-22T02:28:43.853 INFO:tasks.workunit.client.1.smithi122.stdout:2/66: dwrite df/f18 [0,4194304] 0 2024-03-22T02:28:43.854 INFO:tasks.workunit.client.0.smithi103.stdout:3/712: stat de/d12/d5d/d92 0 2024-03-22T02:28:43.854 INFO:tasks.workunit.client.0.smithi103.stdout:4/584: fdatasync d3/d1e/d25/f3d 0 2024-03-22T02:28:43.855 INFO:tasks.workunit.client.1.smithi122.stdout:7/74: mknod d0/d4/ce 0 2024-03-22T02:28:43.856 INFO:tasks.workunit.client.0.smithi103.stdout:4/585: chown d3/d31/f4e 993 1 2024-03-22T02:28:43.857 INFO:tasks.workunit.client.0.smithi103.stdout:5/647: dwrite d0/d14/d23/d36/d95/d96/fb0 [0,4194304] 0 2024-03-22T02:28:43.858 INFO:tasks.workunit.client.1.smithi122.stdout:6/34: dwrite f6 [0,4194304] 0 2024-03-22T02:28:43.858 INFO:tasks.workunit.client.1.smithi122.stdout:2/67: symlink df/l1d 0 2024-03-22T02:28:43.859 INFO:tasks.workunit.client.1.smithi122.stdout:4/86: truncate d0/d2/d7/de/f14 2971489 0 2024-03-22T02:28:43.861 INFO:tasks.workunit.client.0.smithi103.stdout:4/586: read d3/d1e/d53/d8a/f9a [109775,95516] 0 2024-03-22T02:28:43.862 INFO:tasks.workunit.client.1.smithi122.stdout:6/35: fsync d7/fa 0 2024-03-22T02:28:43.863 INFO:tasks.workunit.client.1.smithi122.stdout:7/75: truncate d0/d4/f5 1677517 0 2024-03-22T02:28:43.864 INFO:tasks.workunit.client.1.smithi122.stdout:4/87: stat d0/d2 0 2024-03-22T02:28:43.865 INFO:tasks.workunit.client.0.smithi103.stdout:4/587: dread d3/d38/f85 [0,4194304] 0 2024-03-22T02:28:43.866 INFO:tasks.workunit.client.1.smithi122.stdout:6/36: chown d7/de/dd/df 55973657 1 2024-03-22T02:28:43.867 INFO:tasks.workunit.client.0.smithi103.stdout:3/713: link de/d6b/l89 de/d1a/d25/d26/lee 0 2024-03-22T02:28:43.868 INFO:tasks.workunit.client.1.smithi122.stdout:7/76: mknod d0/d4/cf 0 2024-03-22T02:28:43.869 INFO:tasks.workunit.client.1.smithi122.stdout:1/47: dwrite dc/fd [0,4194304] 0 2024-03-22T02:28:43.870 INFO:tasks.workunit.client.0.smithi103.stdout:3/714: mknod de/d1a/d25/d3f/d9c/da7/cef 0 2024-03-22T02:28:43.870 INFO:tasks.workunit.client.1.smithi122.stdout:7/77: readlink d0/d4/l7 0 2024-03-22T02:28:43.871 INFO:tasks.workunit.client.0.smithi103.stdout:8/780: truncate d4/d19/d1a/d4a/ff8 690205 0 2024-03-22T02:28:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:43 smithi103 ceph-mon[29258]: pgmap v15: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 153 MiB/s wr, 355 op/s 2024-03-22T02:28:43.875 INFO:tasks.workunit.client.1.smithi122.stdout:4/88: rename d0/d12/d15/c1b to d0/d2/d7/de/c20 0 2024-03-22T02:28:43.875 INFO:tasks.workunit.client.1.smithi122.stdout:2/68: dwrite df/f11 [0,4194304] 0 2024-03-22T02:28:43.878 INFO:tasks.workunit.client.0.smithi103.stdout:8/781: unlink d4/d19/d1a/d4a/cce 0 2024-03-22T02:28:43.878 INFO:tasks.workunit.client.1.smithi122.stdout:6/37: dread f5 [0,4194304] 0 2024-03-22T02:28:43.879 INFO:tasks.workunit.client.1.smithi122.stdout:1/48: creat dc/f10 x:0 0 0 2024-03-22T02:28:43.879 INFO:tasks.workunit.client.1.smithi122.stdout:2/69: rmdir df 39 2024-03-22T02:28:43.879 INFO:tasks.workunit.client.1.smithi122.stdout:6/38: fdatasync f3 0 2024-03-22T02:28:43.880 INFO:tasks.workunit.client.1.smithi122.stdout:1/49: fdatasync f7 0 2024-03-22T02:28:43.881 INFO:tasks.workunit.client.1.smithi122.stdout:7/78: mkdir d0/d10 0 2024-03-22T02:28:43.884 INFO:tasks.workunit.client.1.smithi122.stdout:6/39: symlink d7/l10 0 2024-03-22T02:28:43.885 INFO:tasks.workunit.client.1.smithi122.stdout:6/40: stat f6 0 2024-03-22T02:28:43.885 INFO:tasks.workunit.client.1.smithi122.stdout:6/41: dread - d7/fa zero size 2024-03-22T02:28:43.886 INFO:tasks.workunit.client.1.smithi122.stdout:1/50: rename f2 to dc/f11 0 2024-03-22T02:28:43.887 INFO:tasks.workunit.client.1.smithi122.stdout:7/79: creat d0/d4/f11 x:0 0 0 2024-03-22T02:28:43.888 INFO:tasks.workunit.client.1.smithi122.stdout:1/51: write dc/fe [460374,80217] 0 2024-03-22T02:28:43.888 INFO:tasks.workunit.client.1.smithi122.stdout:2/70: mknod df/c1e 0 2024-03-22T02:28:43.888 INFO:tasks.workunit.client.1.smithi122.stdout:7/80: chown d0/f2 1722 1 2024-03-22T02:28:43.891 INFO:tasks.workunit.client.1.smithi122.stdout:1/52: mknod dc/c12 0 2024-03-22T02:28:43.892 INFO:tasks.workunit.client.0.smithi103.stdout:7/617: link d1/d10/d11/d19/d29/d61/d69/f62 d1/d10/d11/d19/d29/d61/fc0 0 2024-03-22T02:28:43.892 INFO:tasks.workunit.client.1.smithi122.stdout:7/81: dread d0/fd [0,4194304] 0 2024-03-22T02:28:43.892 INFO:tasks.workunit.client.1.smithi122.stdout:2/71: mkdir df/d1f 0 2024-03-22T02:28:43.893 INFO:tasks.workunit.client.1.smithi122.stdout:6/42: symlink d7/d9/l11 0 2024-03-22T02:28:43.893 INFO:tasks.workunit.client.1.smithi122.stdout:1/53: write dc/ff [846228,122912] 0 2024-03-22T02:28:43.894 INFO:tasks.workunit.client.1.smithi122.stdout:6/43: read f3 [1306958,55175] 0 2024-03-22T02:28:43.896 INFO:tasks.workunit.client.1.smithi122.stdout:2/72: creat df/d1f/f20 x:0 0 0 2024-03-22T02:28:43.896 INFO:tasks.workunit.client.0.smithi103.stdout:1/637: getdents db/d10/d29/d41/d68/d74/dba 0 2024-03-22T02:28:43.898 INFO:tasks.workunit.client.1.smithi122.stdout:7/82: rmdir d0/d10 0 2024-03-22T02:28:43.900 INFO:tasks.workunit.client.0.smithi103.stdout:7/618: dwrite d1/da/d2f/fb7 [0,4194304] 0 2024-03-22T02:28:43.901 INFO:tasks.workunit.client.1.smithi122.stdout:6/44: creat d7/de/dd/df/f12 x:0 0 0 2024-03-22T02:28:43.901 INFO:tasks.workunit.client.1.smithi122.stdout:4/89: stat d0/d2/d7/de/c20 0 2024-03-22T02:28:43.903 INFO:tasks.workunit.client.1.smithi122.stdout:1/54: dwrite dc/ff [0,4194304] 0 2024-03-22T02:28:43.904 INFO:tasks.workunit.client.0.smithi103.stdout:1/638: dread db/d10/d18/d2d/f4d [0,4194304] 0 2024-03-22T02:28:43.905 INFO:tasks.workunit.client.0.smithi103.stdout:7/619: mkdir d1/da/dc1 0 2024-03-22T02:28:43.906 INFO:tasks.workunit.client.0.smithi103.stdout:0/635: write dd/d16/f84 [1831340,113716] 0 2024-03-22T02:28:43.906 INFO:tasks.workunit.client.0.smithi103.stdout:1/639: fsync db/d10/d28/fa7 0 2024-03-22T02:28:43.908 INFO:tasks.workunit.client.0.smithi103.stdout:7/620: creat d1/d10/d80/fc2 x:0 0 0 2024-03-22T02:28:43.910 INFO:tasks.workunit.client.0.smithi103.stdout:1/640: rmdir db/d10/d6b/db5 39 2024-03-22T02:28:43.911 INFO:tasks.workunit.client.1.smithi122.stdout:7/83: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:43.911 INFO:tasks.workunit.client.0.smithi103.stdout:1/641: chown db/d10/d28/c46 19064 1 2024-03-22T02:28:43.912 INFO:tasks.workunit.client.1.smithi122.stdout:4/90: dread d0/d2/f1e [0,4194304] 0 2024-03-22T02:28:43.913 INFO:tasks.workunit.client.1.smithi122.stdout:1/55: dread dc/fd [0,4194304] 0 2024-03-22T02:28:43.913 INFO:tasks.workunit.client.0.smithi103.stdout:7/621: mkdir d1/d10/d11/d19/dc3 0 2024-03-22T02:28:43.914 INFO:tasks.workunit.client.1.smithi122.stdout:7/84: creat d0/d4/f12 x:0 0 0 2024-03-22T02:28:43.914 INFO:tasks.workunit.client.1.smithi122.stdout:4/91: truncate d0/d12/f1c 774215 0 2024-03-22T02:28:43.915 INFO:tasks.workunit.client.1.smithi122.stdout:4/92: chown d0/d2/d7/de/c20 864093989 1 2024-03-22T02:28:43.917 INFO:tasks.workunit.client.1.smithi122.stdout:1/56: mknod dc/c13 0 2024-03-22T02:28:43.917 INFO:tasks.workunit.client.0.smithi103.stdout:3/715: dread de/f2d [4194304,4194304] 0 2024-03-22T02:28:43.917 INFO:tasks.workunit.client.0.smithi103.stdout:1/642: symlink db/d10/d29/d41/d68/d74/dba/ldb 0 2024-03-22T02:28:43.918 INFO:tasks.workunit.client.1.smithi122.stdout:4/93: creat d0/d12/f21 x:0 0 0 2024-03-22T02:28:43.919 INFO:tasks.workunit.client.1.smithi122.stdout:4/94: chown d0/d2/l1a 5678 1 2024-03-22T02:28:43.922 INFO:tasks.workunit.client.1.smithi122.stdout:4/95: creat d0/d2/d7/f22 x:0 0 0 2024-03-22T02:28:43.923 INFO:tasks.workunit.client.0.smithi103.stdout:7/622: fsync d1/d10/d11/d19/d29/f52 0 2024-03-22T02:28:43.924 INFO:tasks.workunit.client.0.smithi103.stdout:4/588: write d3/f77 [910099,95671] 0 2024-03-22T02:28:43.924 INFO:tasks.workunit.client.0.smithi103.stdout:5/648: dwrite d0/d14/d23/d2a/d2c/f4d [0,4194304] 0 2024-03-22T02:28:43.925 INFO:tasks.workunit.client.1.smithi122.stdout:1/57: dwrite dc/ff [0,4194304] 0 2024-03-22T02:28:43.925 INFO:tasks.workunit.client.1.smithi122.stdout:2/73: truncate df/f11 4195077 0 2024-03-22T02:28:43.925 INFO:tasks.workunit.client.1.smithi122.stdout:1/58: chown dc 6043158 1 2024-03-22T02:28:43.926 INFO:tasks.workunit.client.0.smithi103.stdout:4/589: chown d3/d1e/d53/d8a/lc6 6 1 2024-03-22T02:28:43.927 INFO:tasks.workunit.client.1.smithi122.stdout:2/74: write df/f13 [5782724,48926] 0 2024-03-22T02:28:43.928 INFO:tasks.workunit.client.0.smithi103.stdout:0/636: rename dd/d16/d4b/f76 to dd/d16/d4b/fd6 0 2024-03-22T02:28:43.929 INFO:tasks.workunit.client.0.smithi103.stdout:1/643: creat db/d10/d18/dca/fdc x:0 0 0 2024-03-22T02:28:43.930 INFO:tasks.workunit.client.0.smithi103.stdout:0/637: chown dd/d16/d4b/d96/d29/c90 6 1 2024-03-22T02:28:43.932 INFO:tasks.workunit.client.1.smithi122.stdout:1/59: symlink dc/l14 0 2024-03-22T02:28:43.933 INFO:tasks.workunit.client.0.smithi103.stdout:3/716: link de/d12/d8a/db0/fb2 de/d12/d5d/d92/ff0 0 2024-03-22T02:28:43.935 INFO:tasks.workunit.client.1.smithi122.stdout:7/85: fsync d0/d4/f12 0 2024-03-22T02:28:43.935 INFO:tasks.workunit.client.0.smithi103.stdout:5/649: creat d0/d14/d23/d5c/fdb x:0 0 0 2024-03-22T02:28:43.936 INFO:tasks.workunit.client.1.smithi122.stdout:6/45: dwrite f5 [0,4194304] 0 2024-03-22T02:28:43.938 INFO:tasks.workunit.client.0.smithi103.stdout:4/590: mkdir d3/d1e/d25/d65/dc9 0 2024-03-22T02:28:43.939 INFO:tasks.workunit.client.0.smithi103.stdout:0/638: dwrite dd/d34/d53/d8c/fc0 [0,4194304] 0 2024-03-22T02:28:43.940 INFO:tasks.workunit.client.1.smithi122.stdout:7/86: stat d0/fd 0 2024-03-22T02:28:43.941 INFO:tasks.workunit.client.1.smithi122.stdout:6/46: dread - d7/fa zero size 2024-03-22T02:28:43.942 INFO:tasks.workunit.client.0.smithi103.stdout:7/623: mkdir d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4 0 2024-03-22T02:28:43.944 INFO:tasks.workunit.client.1.smithi122.stdout:1/60: dwrite f9 [0,4194304] 0 2024-03-22T02:28:43.946 INFO:tasks.workunit.client.1.smithi122.stdout:1/61: read dc/fe [520025,103086] 0 2024-03-22T02:28:43.949 INFO:tasks.workunit.client.1.smithi122.stdout:6/47: creat d7/de/dd/f13 x:0 0 0 2024-03-22T02:28:43.951 INFO:tasks.workunit.client.0.smithi103.stdout:5/650: rename d0/d14/d23/d36/d95/fa0 to d0/d14/d23/d2a/d2c/fdc 0 2024-03-22T02:28:43.956 INFO:tasks.workunit.client.1.smithi122.stdout:7/87: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:43.961 INFO:tasks.workunit.client.0.smithi103.stdout:4/591: symlink d3/d30/lca 0 2024-03-22T02:28:43.962 INFO:tasks.workunit.client.1.smithi122.stdout:6/48: dwrite f3 [0,4194304] 0 2024-03-22T02:28:43.962 INFO:tasks.workunit.client.0.smithi103.stdout:4/592: dread - d3/d1e/d53/fb7 zero size 2024-03-22T02:28:43.963 INFO:tasks.workunit.client.0.smithi103.stdout:4/593: write d3/d1e/d53/fb7 [89630,39587] 0 2024-03-22T02:28:43.965 INFO:tasks.workunit.client.0.smithi103.stdout:4/594: fsync d3/d69/f5a 0 2024-03-22T02:28:43.965 INFO:tasks.workunit.client.1.smithi122.stdout:6/49: mknod d7/de/c14 0 2024-03-22T02:28:43.967 INFO:tasks.workunit.client.1.smithi122.stdout:6/50: chown d7/de/dd/df/f12 99135155 1 2024-03-22T02:28:43.967 INFO:tasks.workunit.client.1.smithi122.stdout:4/96: dwrite d0/d2/d7/de/f14 [0,4194304] 0 2024-03-22T02:28:43.968 INFO:tasks.workunit.client.0.smithi103.stdout:7/624: mkdir d1/d10/d11/d19/d29/d33/da8/dc5 0 2024-03-22T02:28:43.969 INFO:tasks.workunit.client.0.smithi103.stdout:1/644: rename db/d10 to db/d4a/d78/d92/da8/ddd 0 2024-03-22T02:28:43.970 INFO:tasks.workunit.client.1.smithi122.stdout:6/51: creat d7/f15 x:0 0 0 2024-03-22T02:28:43.971 INFO:tasks.workunit.client.1.smithi122.stdout:4/97: creat d0/d2/d7/f23 x:0 0 0 2024-03-22T02:28:43.972 INFO:tasks.workunit.client.0.smithi103.stdout:1/645: dread - db/d4a/d78/d92/da8/ddd/d18/fa4 zero size 2024-03-22T02:28:43.972 INFO:tasks.workunit.client.1.smithi122.stdout:4/98: rename d0/d2/d7 to d0/d2/d7/de/d24 22 2024-03-22T02:28:43.973 INFO:tasks.workunit.client.0.smithi103.stdout:5/651: link d0/da2/fb2 d0/da2/dca/fdd 0 2024-03-22T02:28:43.977 INFO:tasks.workunit.client.0.smithi103.stdout:0/639: creat dd/d16/d4b/d96/d64/dab/dc6/fd7 x:0 0 0 2024-03-22T02:28:43.978 INFO:tasks.workunit.client.1.smithi122.stdout:1/62: truncate f9 2861518 0 2024-03-22T02:28:43.979 INFO:tasks.workunit.client.0.smithi103.stdout:4/595: creat d3/d1e/d68/fcb x:0 0 0 2024-03-22T02:28:43.979 INFO:tasks.workunit.client.1.smithi122.stdout:1/63: fdatasync dc/ff 0 2024-03-22T02:28:43.980 INFO:tasks.workunit.client.1.smithi122.stdout:1/64: fsync dc/f11 0 2024-03-22T02:28:43.981 INFO:tasks.workunit.client.0.smithi103.stdout:7/625: creat d1/d10/d11/d19/d29/d33/da8/fc6 x:0 0 0 2024-03-22T02:28:43.983 INFO:tasks.workunit.client.1.smithi122.stdout:1/65: mknod dc/c15 0 2024-03-22T02:28:43.985 INFO:tasks.workunit.client.0.smithi103.stdout:5/652: creat d0/da2/dca/fde x:0 0 0 2024-03-22T02:28:43.986 INFO:tasks.workunit.client.1.smithi122.stdout:1/66: dread dc/ff [0,4194304] 0 2024-03-22T02:28:43.987 INFO:tasks.workunit.client.0.smithi103.stdout:5/653: write d0/d14/d23/d36/d95/f7d [190806,87829] 0 2024-03-22T02:28:43.987 INFO:tasks.workunit.client.0.smithi103.stdout:4/596: unlink d3/d1e/d53/l62 0 2024-03-22T02:28:43.990 INFO:tasks.workunit.client.1.smithi122.stdout:1/67: link dc/c13 dc/c16 0 2024-03-22T02:28:43.991 INFO:tasks.workunit.client.1.smithi122.stdout:1/68: dread - dc/f10 zero size 2024-03-22T02:28:43.991 INFO:tasks.workunit.client.1.smithi122.stdout:1/69: readlink l6 0 2024-03-22T02:28:43.992 INFO:tasks.workunit.client.1.smithi122.stdout:1/70: creat dc/f17 x:0 0 0 2024-03-22T02:28:43.994 INFO:tasks.workunit.client.0.smithi103.stdout:0/640: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/fd8 x:0 0 0 2024-03-22T02:28:43.996 INFO:tasks.workunit.client.0.smithi103.stdout:0/641: readlink dd/d16/d4b/d96/d29/d48/d94/l9e 0 2024-03-22T02:28:43.996 INFO:tasks.workunit.client.0.smithi103.stdout:1/646: dwrite db/d4a/d78/d92/da8/ddd/f5a [0,4194304] 0 2024-03-22T02:28:43.998 INFO:tasks.workunit.client.0.smithi103.stdout:5/654: unlink d0/l11 0 2024-03-22T02:28:43.999 INFO:tasks.workunit.client.0.smithi103.stdout:5/655: fdatasync d0/fd4 0 2024-03-22T02:28:43.999 INFO:tasks.workunit.client.1.smithi122.stdout:1/71: rename f7 to dc/f18 0 2024-03-22T02:28:43.999 INFO:tasks.workunit.client.0.smithi103.stdout:7/626: creat d1/da/fc7 x:0 0 0 2024-03-22T02:28:44.001 INFO:tasks.workunit.client.0.smithi103.stdout:4/597: unlink d3/d1e/d53/d7a/l89 0 2024-03-22T02:28:44.001 INFO:tasks.workunit.client.1.smithi122.stdout:1/72: write dc/fd [2158281,130051] 0 2024-03-22T02:28:44.003 INFO:tasks.workunit.client.0.smithi103.stdout:1/647: creat db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fde x:0 0 0 2024-03-22T02:28:44.004 INFO:tasks.workunit.client.1.smithi122.stdout:1/73: mknod dc/c19 0 2024-03-22T02:28:44.005 INFO:tasks.workunit.client.0.smithi103.stdout:5/656: rename d0/d14/d23/d46/l53 to d0/d14/d23/d6f/d83/ldf 0 2024-03-22T02:28:44.006 INFO:tasks.workunit.client.1.smithi122.stdout:6/52: fsync f3 0 2024-03-22T02:28:44.007 INFO:tasks.workunit.client.0.smithi103.stdout:7/627: fsync d1/d10/d11/d19/d29/d57/f6d 0 2024-03-22T02:28:44.009 INFO:tasks.workunit.client.0.smithi103.stdout:1/648: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/cbc 3498 1 2024-03-22T02:28:44.010 INFO:tasks.workunit.client.0.smithi103.stdout:4/598: dwrite d3/d30/f49 [0,4194304] 0 2024-03-22T02:28:44.010 INFO:tasks.workunit.client.1.smithi122.stdout:4/99: rename d0/d2 to d0/d25 0 2024-03-22T02:28:44.012 INFO:tasks.workunit.client.0.smithi103.stdout:5/657: mknod d0/d14/d23/d2a/ce0 0 2024-03-22T02:28:44.013 INFO:tasks.workunit.client.0.smithi103.stdout:4/599: truncate d3/d69/f5a 2037377 0 2024-03-22T02:28:44.014 INFO:tasks.workunit.client.1.smithi122.stdout:6/53: dwrite d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:44.017 INFO:tasks.workunit.client.1.smithi122.stdout:6/54: write f6 [2014764,117638] 0 2024-03-22T02:28:44.022 INFO:tasks.workunit.client.0.smithi103.stdout:5/658: mkdir d0/d14/d23/d36/de1 0 2024-03-22T02:28:44.022 INFO:tasks.workunit.client.0.smithi103.stdout:7/628: dwrite d1/da/d1c/f8b [0,4194304] 0 2024-03-22T02:28:44.023 INFO:tasks.workunit.client.1.smithi122.stdout:4/100: dwrite d0/f19 [0,4194304] 0 2024-03-22T02:28:44.025 INFO:tasks.workunit.client.1.smithi122.stdout:1/74: rename f9 to dc/f1a 0 2024-03-22T02:28:44.026 INFO:tasks.workunit.client.1.smithi122.stdout:6/55: creat d7/de/dd/f16 x:0 0 0 2024-03-22T02:28:44.029 INFO:tasks.workunit.client.1.smithi122.stdout:4/101: symlink d0/d12/d15/l26 0 2024-03-22T02:28:44.029 INFO:tasks.workunit.client.1.smithi122.stdout:4/102: readlink d0/d25/l1a 0 2024-03-22T02:28:44.032 INFO:tasks.workunit.client.1.smithi122.stdout:6/56: mknod d7/de/dd/c17 0 2024-03-22T02:28:44.032 INFO:tasks.workunit.client.0.smithi103.stdout:7/629: creat d1/d10/d11/d19/d29/d61/fc8 x:0 0 0 2024-03-22T02:28:44.033 INFO:tasks.workunit.client.0.smithi103.stdout:5/659: dwrite d0/fd4 [0,4194304] 0 2024-03-22T02:28:44.034 INFO:tasks.workunit.client.1.smithi122.stdout:6/57: write d7/de/dd/df/f12 [248488,107492] 0 2024-03-22T02:28:44.035 INFO:tasks.workunit.client.0.smithi103.stdout:5/660: read - d0/d14/d23/d5c/fb8 zero size 2024-03-22T02:28:44.036 INFO:tasks.workunit.client.1.smithi122.stdout:4/103: creat d0/d25/f27 x:0 0 0 2024-03-22T02:28:44.037 INFO:tasks.workunit.client.1.smithi122.stdout:4/104: write d0/d25/d7/f23 [152009,18746] 0 2024-03-22T02:28:44.037 INFO:tasks.workunit.client.1.smithi122.stdout:6/58: creat d7/d9/f18 x:0 0 0 2024-03-22T02:28:44.040 INFO:tasks.workunit.client.0.smithi103.stdout:7/630: read d1/d10/d11/d19/d29/d61/d69/f62 [628208,63738] 0 2024-03-22T02:28:44.042 INFO:tasks.workunit.client.1.smithi122.stdout:4/105: creat d0/d25/f28 x:0 0 0 2024-03-22T02:28:44.045 INFO:tasks.workunit.client.1.smithi122.stdout:4/106: creat d0/d12/d15/f29 x:0 0 0 2024-03-22T02:28:44.045 INFO:tasks.workunit.client.0.smithi103.stdout:7/631: creat d1/d10/d11/d70/d8a/fc9 x:0 0 0 2024-03-22T02:28:44.047 INFO:tasks.workunit.client.1.smithi122.stdout:4/107: mknod d0/d12/c2a 0 2024-03-22T02:28:44.049 INFO:tasks.workunit.client.1.smithi122.stdout:4/108: symlink d0/d12/d15/l2b 0 2024-03-22T02:28:44.050 INFO:tasks.workunit.client.0.smithi103.stdout:7/632: dread d1/da/d2f/fb7 [0,4194304] 0 2024-03-22T02:28:44.051 INFO:tasks.workunit.client.0.smithi103.stdout:5/661: link d0/d14/d23/d36/d95/f5a d0/d14/d23/d2a/d2c/dd9/fe2 0 2024-03-22T02:28:44.055 INFO:tasks.workunit.client.0.smithi103.stdout:5/662: write d0/d14/f41 [5915986,33882] 0 2024-03-22T02:28:44.056 INFO:tasks.workunit.client.1.smithi122.stdout:4/109: dwrite d0/d25/fa [0,4194304] 0 2024-03-22T02:28:44.057 INFO:tasks.workunit.client.1.smithi122.stdout:4/110: write d0/f9 [834965,80656] 0 2024-03-22T02:28:44.057 INFO:tasks.workunit.client.0.smithi103.stdout:1/649: dread db/d4a/d78/d92/da8/ddd/d18/f20 [0,4194304] 0 2024-03-22T02:28:44.058 INFO:tasks.workunit.client.1.smithi122.stdout:4/111: chown d0/d25/f8 39955125 1 2024-03-22T02:28:44.064 INFO:tasks.workunit.client.0.smithi103.stdout:4/600: dwrite d3/f34 [0,4194304] 0 2024-03-22T02:28:44.064 INFO:tasks.workunit.client.0.smithi103.stdout:7/633: dwrite d1/d10/d11/f9f [0,4194304] 0 2024-03-22T02:28:44.066 INFO:tasks.workunit.client.0.smithi103.stdout:4/601: chown d3/d30/d45/d6e/l91 58857288 1 2024-03-22T02:28:44.068 INFO:tasks.workunit.client.1.smithi122.stdout:6/59: getdents d7/d9 0 2024-03-22T02:28:44.070 INFO:tasks.workunit.client.1.smithi122.stdout:6/60: truncate d7/d9/f18 549473 0 2024-03-22T02:28:44.071 INFO:tasks.workunit.client.0.smithi103.stdout:5/663: rename d0/fad to d0/d14/d23/d36/fe3 0 2024-03-22T02:28:44.073 INFO:tasks.workunit.client.1.smithi122.stdout:6/61: dread d7/d9/f18 [0,4194304] 0 2024-03-22T02:28:44.073 INFO:tasks.workunit.client.1.smithi122.stdout:6/62: chown d7/c8 196403 1 2024-03-22T02:28:44.074 INFO:tasks.workunit.client.0.smithi103.stdout:7/634: truncate d1/d10/d11/d19/d29/d61/fc0 1666012 0 2024-03-22T02:28:44.074 INFO:tasks.workunit.client.1.smithi122.stdout:6/63: write f6 [5235118,47254] 0 2024-03-22T02:28:44.075 INFO:tasks.workunit.client.0.smithi103.stdout:5/664: symlink d0/d14/dc0/le4 0 2024-03-22T02:28:44.078 INFO:tasks.workunit.client.1.smithi122.stdout:6/64: mknod d7/d9/c19 0 2024-03-22T02:28:44.079 INFO:tasks.workunit.client.0.smithi103.stdout:1/650: dwrite db/d4a/f65 [0,4194304] 0 2024-03-22T02:28:44.079 INFO:tasks.workunit.client.0.smithi103.stdout:4/602: write d3/d1e/f7f [2583782,82755] 0 2024-03-22T02:28:44.081 INFO:tasks.workunit.client.1.smithi122.stdout:6/65: creat d7/de/dd/df/f1a x:0 0 0 2024-03-22T02:28:44.081 INFO:tasks.workunit.client.0.smithi103.stdout:4/603: chown d3/d1e/d25/d65/dc9 64142483 1 2024-03-22T02:28:44.082 INFO:tasks.workunit.client.0.smithi103.stdout:5/665: mkdir d0/d14/d23/d36/d95/d96/de5 0 2024-03-22T02:28:44.082 INFO:tasks.workunit.client.0.smithi103.stdout:1/651: fsync db/d4a/d78/d92/da8/ddd/d29/d55/fd7 0 2024-03-22T02:28:44.083 INFO:tasks.workunit.client.1.smithi122.stdout:6/66: creat d7/f1b x:0 0 0 2024-03-22T02:28:44.084 INFO:tasks.workunit.client.0.smithi103.stdout:4/604: truncate d3/d1e/d68/fcb 1043311 0 2024-03-22T02:28:44.085 INFO:tasks.workunit.client.1.smithi122.stdout:6/67: write d7/de/dd/df/f12 [825282,86062] 0 2024-03-22T02:28:44.086 INFO:tasks.workunit.client.0.smithi103.stdout:5/666: creat d0/d14/dc0/fe6 x:0 0 0 2024-03-22T02:28:44.087 INFO:tasks.workunit.client.1.smithi122.stdout:6/68: symlink d7/l1c 0 2024-03-22T02:28:44.088 INFO:tasks.workunit.client.0.smithi103.stdout:1/652: symlink db/d4a/d78/d92/da8/ddd/d29/d6c/ldf 0 2024-03-22T02:28:44.088 INFO:tasks.workunit.client.1.smithi122.stdout:6/69: write f3 [3812255,108127] 0 2024-03-22T02:28:44.089 INFO:tasks.workunit.client.0.smithi103.stdout:4/605: rename d3/d38/f81 to d3/d1e/d25/d65/d83/dc1/fcc 0 2024-03-22T02:28:44.089 INFO:tasks.workunit.client.1.smithi122.stdout:6/70: dread - d7/de/dd/f16 zero size 2024-03-22T02:28:44.090 INFO:tasks.workunit.client.1.smithi122.stdout:6/71: chown f3 46298652 1 2024-03-22T02:28:44.090 INFO:tasks.workunit.client.0.smithi103.stdout:7/635: getdents d1/d10/d11/d19/d29/d33/d4b/d5e 0 2024-03-22T02:28:44.091 INFO:tasks.workunit.client.1.smithi122.stdout:6/72: mknod d7/de/c1d 0 2024-03-22T02:28:44.093 INFO:tasks.workunit.client.0.smithi103.stdout:4/606: read d3/d1e/fac [11527,98259] 0 2024-03-22T02:28:44.093 INFO:tasks.workunit.client.1.smithi122.stdout:6/73: symlink d7/de/l1e 0 2024-03-22T02:28:44.096 INFO:tasks.workunit.client.0.smithi103.stdout:5/667: dwrite d0/d14/d23/d36/d95/fb1 [0,4194304] 0 2024-03-22T02:28:44.098 INFO:tasks.workunit.client.1.smithi122.stdout:6/74: dwrite f6 [0,4194304] 0 2024-03-22T02:28:44.100 INFO:tasks.workunit.client.1.smithi122.stdout:6/75: dread - d7/f1b zero size 2024-03-22T02:28:44.102 INFO:tasks.workunit.client.0.smithi103.stdout:4/607: dwrite d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:44.105 INFO:tasks.workunit.client.1.smithi122.stdout:6/76: dread f3 [0,4194304] 0 2024-03-22T02:28:44.114 INFO:tasks.workunit.client.1.smithi122.stdout:6/77: dwrite f3 [0,4194304] 0 2024-03-22T02:28:44.118 INFO:tasks.workunit.client.0.smithi103.stdout:7/636: rename d1/d10/d11/d19/d29/d33/d4b/l7d to d1/da/d1c/lca 0 2024-03-22T02:28:44.119 INFO:tasks.workunit.client.0.smithi103.stdout:5/668: mkdir d0/d14/d23/d46/de7 0 2024-03-22T02:28:44.121 INFO:tasks.workunit.client.1.smithi122.stdout:6/78: dwrite d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:44.122 INFO:tasks.workunit.client.1.smithi122.stdout:6/79: readlink d7/l10 0 2024-03-22T02:28:44.124 INFO:tasks.workunit.client.1.smithi122.stdout:6/80: write d7/de/dd/df/f12 [596508,128141] 0 2024-03-22T02:28:44.125 INFO:tasks.workunit.client.1.smithi122.stdout:6/81: read f3 [2958307,23635] 0 2024-03-22T02:28:44.125 INFO:tasks.workunit.client.1.smithi122.stdout:6/82: chown d7/fa 1270 1 2024-03-22T02:28:44.129 INFO:tasks.workunit.client.0.smithi103.stdout:7/637: dwrite d1/d10/d11/d19/f55 [0,4194304] 0 2024-03-22T02:28:44.129 INFO:tasks.workunit.client.1.smithi122.stdout:6/83: link d7/d9/c19 d7/de/dd/c1f 0 2024-03-22T02:28:44.131 INFO:tasks.workunit.client.1.smithi122.stdout:6/84: mkdir d7/de/dd/d20 0 2024-03-22T02:28:44.133 INFO:tasks.workunit.client.0.smithi103.stdout:4/608: link d3/d31/f37 d3/d69/fcd 0 2024-03-22T02:28:44.137 INFO:tasks.workunit.client.0.smithi103.stdout:1/653: getdents db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74 0 2024-03-22T02:28:44.138 INFO:tasks.workunit.client.0.smithi103.stdout:5/669: link d0/d14/d23/d2a/d2c/f5d d0/d14/d23/d46/de7/fe8 0 2024-03-22T02:28:44.139 INFO:tasks.workunit.client.1.smithi122.stdout:6/85: dwrite f6 [0,4194304] 0 2024-03-22T02:28:44.140 INFO:tasks.workunit.client.0.smithi103.stdout:7/638: dwrite d1/da/f3 [0,4194304] 0 2024-03-22T02:28:44.141 INFO:tasks.workunit.client.0.smithi103.stdout:4/609: symlink d3/da2/lce 0 2024-03-22T02:28:44.143 INFO:tasks.workunit.client.0.smithi103.stdout:5/670: mknod d0/d14/d23/d2a/d2c/da9/ce9 0 2024-03-22T02:28:44.145 INFO:tasks.workunit.client.0.smithi103.stdout:7/639: write d1/da/d2f/f38 [2106429,101986] 0 2024-03-22T02:28:44.147 INFO:tasks.workunit.client.0.smithi103.stdout:5/671: creat d0/d14/dc0/fea x:0 0 0 2024-03-22T02:28:44.149 INFO:tasks.workunit.client.0.smithi103.stdout:1/654: rmdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd9 0 2024-03-22T02:28:44.149 INFO:tasks.workunit.client.1.smithi122.stdout:6/86: dwrite d7/de/dd/df/f12 [0,4194304] 0 2024-03-22T02:28:44.152 INFO:tasks.workunit.client.1.smithi122.stdout:6/87: getdents d7/de/dd/d20 0 2024-03-22T02:28:44.153 INFO:tasks.workunit.client.0.smithi103.stdout:7/640: symlink d1/d10/d11/d19/d29/d61/d69/d6b/lcb 0 2024-03-22T02:28:44.153 INFO:tasks.workunit.client.0.smithi103.stdout:1/655: rmdir db/d4a/d78/d92 39 2024-03-22T02:28:44.157 INFO:tasks.workunit.client.0.smithi103.stdout:7/641: rmdir d1/d10/d11/d70 39 2024-03-22T02:28:44.159 INFO:tasks.workunit.client.1.smithi122.stdout:6/88: dread f6 [4194304,4194304] 0 2024-03-22T02:28:44.162 INFO:tasks.workunit.client.1.smithi122.stdout:6/89: creat d7/de/dd/d20/f21 x:0 0 0 2024-03-22T02:28:44.162 INFO:tasks.workunit.client.0.smithi103.stdout:1/656: mknod db/d4a/d78/d92/da8/ddd/ce0 0 2024-03-22T02:28:44.163 INFO:tasks.workunit.client.1.smithi122.stdout:6/90: write d7/de/dd/f13 [1161553,17009] 0 2024-03-22T02:28:44.165 INFO:tasks.workunit.client.0.smithi103.stdout:7/642: dread d1/d10/d11/f5a [4194304,4194304] 0 2024-03-22T02:28:44.166 INFO:tasks.workunit.client.0.smithi103.stdout:1/657: write db/d4a/d78/d92/da8/ddd/d29/fb8 [1356686,17178] 0 2024-03-22T02:28:44.167 INFO:tasks.workunit.client.0.smithi103.stdout:4/610: rmdir d3/d69 39 2024-03-22T02:28:44.168 INFO:tasks.workunit.client.1.smithi122.stdout:6/91: dwrite d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:44.169 INFO:tasks.workunit.client.1.smithi122.stdout:6/92: write f5 [2525396,29207] 0 2024-03-22T02:28:44.170 INFO:tasks.workunit.client.1.smithi122.stdout:6/93: readlink d7/d9/l11 0 2024-03-22T02:28:44.171 INFO:tasks.workunit.client.0.smithi103.stdout:1/658: creat db/d4a/d78/d92/da8/fe1 x:0 0 0 2024-03-22T02:28:44.173 INFO:tasks.workunit.client.1.smithi122.stdout:6/94: stat d7/de/dd/c1f 0 2024-03-22T02:28:44.173 INFO:tasks.workunit.client.0.smithi103.stdout:4/611: creat d3/d1e/d25/d65/d83/fcf x:0 0 0 2024-03-22T02:28:44.175 INFO:tasks.workunit.client.0.smithi103.stdout:4/612: write d3/d1e/d25/d65/f6c [38127,91608] 0 2024-03-22T02:28:44.175 INFO:tasks.workunit.client.0.smithi103.stdout:7/643: symlink d1/d10/d11/d70/lcc 0 2024-03-22T02:28:44.180 INFO:tasks.workunit.client.1.smithi122.stdout:6/95: dwrite d7/de/dd/df/f12 [0,4194304] 0 2024-03-22T02:28:44.181 INFO:tasks.workunit.client.1.smithi122.stdout:6/96: read - d7/f15 zero size 2024-03-22T02:28:44.182 INFO:tasks.workunit.client.1.smithi122.stdout:6/97: dread - d7/de/dd/df/f1a zero size 2024-03-22T02:28:44.182 INFO:tasks.workunit.client.1.smithi122.stdout:6/98: chown d7/de 20978946 1 2024-03-22T02:28:44.183 INFO:tasks.workunit.client.1.smithi122.stdout:6/99: dread - d7/de/dd/df/f1a zero size 2024-03-22T02:28:44.184 INFO:tasks.workunit.client.0.smithi103.stdout:5/672: write d0/d14/d23/d2a/d2c/dd9/fe2 [3202914,127878] 0 2024-03-22T02:28:44.186 INFO:tasks.workunit.client.1.smithi122.stdout:6/100: rename d7/de/l1e to d7/l22 0 2024-03-22T02:28:44.187 INFO:tasks.workunit.client.1.smithi122.stdout:6/101: stat d7/de/dd/f13 0 2024-03-22T02:28:44.187 INFO:tasks.workunit.client.1.smithi122.stdout:8/76: sync 2024-03-22T02:28:44.188 INFO:tasks.workunit.client.0.smithi103.stdout:4/613: dwrite d3/d69/f54 [0,4194304] 0 2024-03-22T02:28:44.188 INFO:tasks.workunit.client.1.smithi122.stdout:6/102: mkdir d7/de/dd/d23 0 2024-03-22T02:28:44.188 INFO:tasks.workunit.client.1.smithi122.stdout:6/103: symlink d7/de/dd/l24 2 2024-03-22T02:28:44.189 INFO:tasks.workunit.client.1.smithi122.stdout:8/77: rmdir d1/d8 39 2024-03-22T02:28:44.189 INFO:tasks.workunit.client.1.smithi122.stdout:6/104: write d7/fa [914211,18308] 0 2024-03-22T02:28:44.190 INFO:tasks.workunit.client.0.smithi103.stdout:4/614: chown d3/f1f 1 1 2024-03-22T02:28:44.191 INFO:tasks.workunit.client.1.smithi122.stdout:6/105: write d7/de/dd/df/f1a [826155,86251] 0 2024-03-22T02:28:44.193 INFO:tasks.workunit.client.0.smithi103.stdout:7/644: fsync d1/d10/d11/d19/d29/d61/fc0 0 2024-03-22T02:28:44.196 INFO:tasks.workunit.client.0.smithi103.stdout:1/659: truncate db/d4a/d78/d92/da8/ddd/f5a 2447875 0 2024-03-22T02:28:44.197 INFO:tasks.workunit.client.0.smithi103.stdout:7/645: dread d1/da/f3 [0,4194304] 0 2024-03-22T02:28:44.199 INFO:tasks.workunit.client.0.smithi103.stdout:4/615: rename d3/l13 to d3/d1e/d68/ld0 0 2024-03-22T02:28:44.199 INFO:tasks.workunit.client.1.smithi122.stdout:6/106: dwrite d7/fa [0,4194304] 0 2024-03-22T02:28:44.200 INFO:tasks.workunit.client.0.smithi103.stdout:4/616: readlink d3/d1e/d25/l7c 0 2024-03-22T02:28:44.201 INFO:tasks.workunit.client.0.smithi103.stdout:4/617: readlink d3/d1e/d53/l59 0 2024-03-22T02:28:44.202 INFO:tasks.workunit.client.1.smithi122.stdout:6/107: symlink d7/de/l25 0 2024-03-22T02:28:44.203 INFO:tasks.workunit.client.0.smithi103.stdout:7/646: mkdir d1/d10/d11/d19/d29/d33/dcd 0 2024-03-22T02:28:44.204 INFO:tasks.workunit.client.1.smithi122.stdout:6/108: symlink d7/de/dd/l26 0 2024-03-22T02:28:44.205 INFO:tasks.workunit.client.0.smithi103.stdout:5/673: getdents d0/d14/d23/d46 0 2024-03-22T02:28:44.205 INFO:tasks.workunit.client.0.smithi103.stdout:1/660: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/fb9 [0,4194304] 0 2024-03-22T02:28:44.212 INFO:tasks.workunit.client.0.smithi103.stdout:7/647: write d1/d10/d11/d19/d29/d61/d69/f62 [2280044,127875] 0 2024-03-22T02:28:44.213 INFO:tasks.workunit.client.0.smithi103.stdout:5/674: creat d0/d14/d23/d36/d95/d96/daa/feb x:0 0 0 2024-03-22T02:28:44.214 INFO:tasks.workunit.client.0.smithi103.stdout:4/618: dwrite d3/d30/f55 [4194304,4194304] 0 2024-03-22T02:28:44.218 INFO:tasks.workunit.client.0.smithi103.stdout:4/619: dread - d3/d1e/d25/d65/d83/da1/fad zero size 2024-03-22T02:28:44.220 INFO:tasks.workunit.client.1.smithi122.stdout:8/78: write d1/d9/f11 [773193,48449] 0 2024-03-22T02:28:44.220 INFO:tasks.workunit.client.1.smithi122.stdout:6/109: getdents d7/de 0 2024-03-22T02:28:44.220 INFO:tasks.workunit.client.0.smithi103.stdout:1/661: dwrite db/d4a/fa0 [0,4194304] 0 2024-03-22T02:28:44.226 INFO:tasks.workunit.client.1.smithi122.stdout:8/79: mknod d1/d9/c19 0 2024-03-22T02:28:44.226 INFO:tasks.workunit.client.0.smithi103.stdout:1/662: fsync db/d4a/d78/d92/da8/ddd/d18/d2d/fbe 0 2024-03-22T02:28:44.227 INFO:tasks.workunit.client.1.smithi122.stdout:8/80: creat d1/f1a x:0 0 0 2024-03-22T02:28:44.229 INFO:tasks.workunit.client.1.smithi122.stdout:6/110: dwrite d7/de/dd/df/f12 [0,4194304] 0 2024-03-22T02:28:44.230 INFO:tasks.workunit.client.0.smithi103.stdout:4/620: mknod d3/d1e/d25/d65/d83/da1/dbd/cd1 0 2024-03-22T02:28:44.235 INFO:tasks.workunit.client.0.smithi103.stdout:7/648: write d1/d10/d11/d19/d29/d61/f45 [234606,21247] 0 2024-03-22T02:28:44.238 INFO:tasks.workunit.client.0.smithi103.stdout:4/621: link d3/d1e/d53/l76 d3/d30/d45/d6e/ld2 0 2024-03-22T02:28:44.239 INFO:tasks.workunit.client.0.smithi103.stdout:5/675: truncate d0/d14/d23/d2a/d2c/f4d 3046199 0 2024-03-22T02:28:44.239 INFO:tasks.workunit.client.1.smithi122.stdout:6/111: dwrite f6 [0,4194304] 0 2024-03-22T02:28:44.240 INFO:tasks.workunit.client.0.smithi103.stdout:7/649: rmdir d1/d10/d11/d19/d29/d33/d4b 39 2024-03-22T02:28:44.240 INFO:tasks.workunit.client.1.smithi122.stdout:6/112: chown d7/de/dd/df/f1a 30844 1 2024-03-22T02:28:44.242 INFO:tasks.workunit.client.0.smithi103.stdout:7/650: write d1/d10/d11/d19/d29/d57/fb0 [396113,66668] 0 2024-03-22T02:28:44.242 INFO:tasks.workunit.client.0.smithi103.stdout:4/622: creat d3/d1e/d53/d8a/fd3 x:0 0 0 2024-03-22T02:28:44.244 INFO:tasks.workunit.client.1.smithi122.stdout:6/113: rename d7/d9 to d7/de/d27 0 2024-03-22T02:28:44.244 INFO:tasks.workunit.client.1.smithi122.stdout:6/114: readlink d7/lb 0 2024-03-22T02:28:44.245 INFO:tasks.workunit.client.0.smithi103.stdout:1/663: write db/d4a/d78/d92/da8/ddd/d29/d6c/f87 [1790902,2572] 0 2024-03-22T02:28:44.247 INFO:tasks.workunit.client.1.smithi122.stdout:6/115: creat d7/de/dd/d23/f28 x:0 0 0 2024-03-22T02:28:44.247 INFO:tasks.workunit.client.0.smithi103.stdout:4/623: dread d3/d1e/d53/d8a/f9a [0,4194304] 0 2024-03-22T02:28:44.248 INFO:tasks.workunit.client.0.smithi103.stdout:4/624: dread - d3/d1e/d25/d65/d83/da1/fad zero size 2024-03-22T02:28:44.249 INFO:tasks.workunit.client.0.smithi103.stdout:4/625: chown d3/d1e/d53/d8a/lc6 1740611669 1 2024-03-22T02:28:44.250 INFO:tasks.workunit.client.0.smithi103.stdout:7/651: truncate d1/da/f75 1733305 0 2024-03-22T02:28:44.251 INFO:tasks.workunit.client.1.smithi122.stdout:6/116: dread d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:44.252 INFO:tasks.workunit.client.1.smithi122.stdout:8/81: write d1/f1a [702341,20076] 0 2024-03-22T02:28:44.255 INFO:tasks.workunit.client.1.smithi122.stdout:6/117: unlink d7/de/d27/c19 0 2024-03-22T02:28:44.256 INFO:tasks.workunit.client.0.smithi103.stdout:4/626: dwrite d3/d31/f4e [0,4194304] 0 2024-03-22T02:28:44.256 INFO:tasks.workunit.client.0.smithi103.stdout:7/652: getdents d1/d10/d11/d19/d29/d57 0 2024-03-22T02:28:44.257 INFO:tasks.workunit.client.1.smithi122.stdout:6/118: mknod d7/de/c29 0 2024-03-22T02:28:44.257 INFO:tasks.workunit.client.0.smithi103.stdout:7/653: fsync d1/da/d1c/f60 0 2024-03-22T02:28:44.258 INFO:tasks.workunit.client.1.smithi122.stdout:6/119: write d7/de/dd/df/f1a [311638,125368] 0 2024-03-22T02:28:44.261 INFO:tasks.workunit.client.0.smithi103.stdout:5/676: truncate d0/f97 4044222 0 2024-03-22T02:28:44.263 INFO:tasks.workunit.client.0.smithi103.stdout:5/677: write d0/d14/d23/d36/d95/d96/fb0 [3592471,7659] 0 2024-03-22T02:28:44.263 INFO:tasks.workunit.client.0.smithi103.stdout:7/654: rename d1/f4e to d1/d10/d11/d19/d29/d33/da8/dc5/fce 0 2024-03-22T02:28:44.264 INFO:tasks.workunit.client.0.smithi103.stdout:1/664: write db/d4a/d78/d92/da8/ddd/d18/d2d/f48 [781866,7745] 0 2024-03-22T02:28:44.265 INFO:tasks.workunit.client.1.smithi122.stdout:6/120: rmdir d7 39 2024-03-22T02:28:44.267 INFO:tasks.workunit.client.0.smithi103.stdout:4/627: write d3/d69/fcd [4657135,39375] 0 2024-03-22T02:28:44.268 INFO:tasks.workunit.client.0.smithi103.stdout:5/678: fdatasync d0/d14/d23/d36/fe3 0 2024-03-22T02:28:44.269 INFO:tasks.workunit.client.1.smithi122.stdout:6/121: mknod d7/c2a 0 2024-03-22T02:28:44.270 INFO:tasks.workunit.client.0.smithi103.stdout:4/628: readlink d3/d1e/d53/l76 0 2024-03-22T02:28:44.271 INFO:tasks.workunit.client.1.smithi122.stdout:6/122: mknod d7/de/d27/c2b 0 2024-03-22T02:28:44.272 INFO:tasks.workunit.client.0.smithi103.stdout:4/629: chown d3/d38/d5c/faa 186534 1 2024-03-22T02:28:44.272 INFO:tasks.workunit.client.1.smithi122.stdout:6/123: rmdir d7/de 39 2024-03-22T02:28:44.273 INFO:tasks.workunit.client.0.smithi103.stdout:1/665: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/fe2 x:0 0 0 2024-03-22T02:28:44.276 INFO:tasks.workunit.client.1.smithi122.stdout:6/124: stat d7/de/dd 0 2024-03-22T02:28:44.277 INFO:tasks.workunit.client.0.smithi103.stdout:1/666: dread - db/d4a/d78/d92/da8/ddd/d29/d6c/f71 zero size 2024-03-22T02:28:44.277 INFO:tasks.workunit.client.1.smithi122.stdout:6/125: fsync d7/de/dd/f16 0 2024-03-22T02:28:44.278 INFO:tasks.workunit.client.1.smithi122.stdout:6/126: read f5 [2750652,77882] 0 2024-03-22T02:28:44.278 INFO:tasks.workunit.client.0.smithi103.stdout:4/630: dwrite d3/d31/f4e [0,4194304] 0 2024-03-22T02:28:44.279 INFO:tasks.workunit.client.0.smithi103.stdout:7/655: creat d1/d10/d11/d19/d29/d61/d1d/fcf x:0 0 0 2024-03-22T02:28:44.280 INFO:tasks.workunit.client.0.smithi103.stdout:1/667: chown db/d4a/d78/d92/da8/ddd/d18/d2d/l94 40246103 1 2024-03-22T02:28:44.283 INFO:tasks.workunit.client.1.smithi122.stdout:6/127: dwrite f5 [0,4194304] 0 2024-03-22T02:28:44.283 INFO:tasks.workunit.client.0.smithi103.stdout:7/656: readlink d1/da/d2f/d36/l48 0 2024-03-22T02:28:44.283 INFO:tasks.workunit.client.1.smithi122.stdout:6/128: truncate d7/f15 963204 0 2024-03-22T02:28:44.284 INFO:tasks.workunit.client.0.smithi103.stdout:7/657: chown d1/d10/d11/d19/d86/d92/dbf 1 1 2024-03-22T02:28:44.285 INFO:tasks.workunit.client.0.smithi103.stdout:7/658: readlink d1/da/d1c/l47 0 2024-03-22T02:28:44.288 INFO:tasks.workunit.client.1.smithi122.stdout:6/129: creat d7/de/dd/d20/f2c x:0 0 0 2024-03-22T02:28:44.288 INFO:tasks.workunit.client.0.smithi103.stdout:7/659: symlink d1/d10/d11/d19/d29/d61/d69/d6b/d7a/ld0 0 2024-03-22T02:28:44.289 INFO:tasks.workunit.client.1.smithi122.stdout:6/130: dread - d7/de/dd/d20/f2c zero size 2024-03-22T02:28:44.289 INFO:tasks.workunit.client.1.smithi122.stdout:6/131: fsync d7/fa 0 2024-03-22T02:28:44.291 INFO:tasks.workunit.client.1.smithi122.stdout:6/132: mkdir d7/de/dd/d20/d2d 0 2024-03-22T02:28:44.294 INFO:tasks.workunit.client.0.smithi103.stdout:4/631: dwrite d3/d1e/f36 [0,4194304] 0 2024-03-22T02:28:44.295 INFO:tasks.workunit.client.0.smithi103.stdout:7/660: link d1/d10/d11/f9f d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/fd1 0 2024-03-22T02:28:44.298 INFO:tasks.workunit.client.1.smithi122.stdout:6/133: dwrite d7/de/dd/d20/f21 [0,4194304] 0 2024-03-22T02:28:44.300 INFO:tasks.workunit.client.0.smithi103.stdout:4/632: rename d3/c18 to d3/d1e/d25/cd4 0 2024-03-22T02:28:44.302 INFO:tasks.workunit.client.1.smithi122.stdout:6/134: write d7/de/dd/df/f1a [1897101,101938] 0 2024-03-22T02:28:44.302 INFO:tasks.workunit.client.0.smithi103.stdout:7/661: dwrite d1/da/d2f/d36/f91 [0,4194304] 0 2024-03-22T02:28:44.304 INFO:tasks.workunit.client.1.smithi122.stdout:6/135: rmdir d7/de/dd/df 39 2024-03-22T02:28:44.307 INFO:tasks.workunit.client.1.smithi122.stdout:6/136: truncate d7/de/dd/d23/f28 209104 0 2024-03-22T02:28:44.309 INFO:tasks.workunit.client.1.smithi122.stdout:6/137: symlink d7/de/d27/l2e 0 2024-03-22T02:28:44.309 INFO:tasks.workunit.client.1.smithi122.stdout:6/138: read f6 [154216,22476] 0 2024-03-22T02:28:44.310 INFO:tasks.workunit.client.1.smithi122.stdout:6/139: chown d7/de/dd/d20 7884 1 2024-03-22T02:28:44.311 INFO:tasks.workunit.client.0.smithi103.stdout:1/668: dread db/f2b [0,4194304] 0 2024-03-22T02:28:44.314 INFO:tasks.workunit.client.1.smithi122.stdout:6/140: symlink d7/de/l2f 0 2024-03-22T02:28:44.314 INFO:tasks.workunit.client.0.smithi103.stdout:7/662: dwrite d1/d10/d11/d19/d29/d61/d69/f9a [0,4194304] 0 2024-03-22T02:28:44.315 INFO:tasks.workunit.client.1.smithi122.stdout:6/141: chown d7/de/d27/f18 1706 1 2024-03-22T02:28:44.315 INFO:tasks.workunit.client.0.smithi103.stdout:7/663: dread - d1/d10/d11/d19/d29/d61/d69/d6b/fac zero size 2024-03-22T02:28:44.316 INFO:tasks.workunit.client.0.smithi103.stdout:1/669: rmdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5 39 2024-03-22T02:28:44.317 INFO:tasks.workunit.client.0.smithi103.stdout:7/664: chown d1/d10/d11/d70/d8a/d9d 1316834 1 2024-03-22T02:28:44.318 INFO:tasks.workunit.client.0.smithi103.stdout:1/670: fsync db/d4a/f73 0 2024-03-22T02:28:44.318 INFO:tasks.workunit.client.0.smithi103.stdout:1/671: read - db/fc7 zero size 2024-03-22T02:28:44.319 INFO:tasks.workunit.client.1.smithi122.stdout:6/142: link d7/f15 d7/de/dd/d20/d2d/f30 0 2024-03-22T02:28:44.320 INFO:tasks.workunit.client.0.smithi103.stdout:1/672: write db/d4a/d78/d92/da8/ddd/f83 [1649618,53841] 0 2024-03-22T02:28:44.321 INFO:tasks.workunit.client.0.smithi103.stdout:1/673: stat db/d4a/f65 0 2024-03-22T02:28:44.322 INFO:tasks.workunit.client.1.smithi122.stdout:6/143: fsync d7/de/d27/f18 0 2024-03-22T02:28:44.323 INFO:tasks.workunit.client.1.smithi122.stdout:6/144: chown d7/de/c14 5 1 2024-03-22T02:28:44.324 INFO:tasks.workunit.client.0.smithi103.stdout:1/674: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/le3 0 2024-03-22T02:28:44.327 INFO:tasks.workunit.client.0.smithi103.stdout:1/675: creat db/d4a/d78/d92/da8/ddd/d21/fe4 x:0 0 0 2024-03-22T02:28:44.327 INFO:tasks.workunit.client.1.smithi122.stdout:6/145: rename d7/de/dd/d23/f28 to d7/de/dd/d20/f31 0 2024-03-22T02:28:44.329 INFO:tasks.workunit.client.1.smithi122.stdout:6/146: fdatasync f6 0 2024-03-22T02:28:44.329 INFO:tasks.workunit.client.0.smithi103.stdout:7/665: fsync d1/d10/d11/f9f 0 2024-03-22T02:28:44.330 INFO:tasks.workunit.client.0.smithi103.stdout:2/536: sync 2024-03-22T02:28:44.331 INFO:tasks.workunit.client.0.smithi103.stdout:2/537: readlink d11/l3d 0 2024-03-22T02:28:44.333 INFO:tasks.workunit.client.1.smithi122.stdout:6/147: rename d7/de/dd/c17 to d7/de/dd/df/c32 0 2024-03-22T02:28:44.336 INFO:tasks.workunit.client.1.smithi122.stdout:6/148: link d7/de/dd/c1f d7/de/dd/d20/d2d/c33 0 2024-03-22T02:28:44.339 INFO:tasks.workunit.client.1.smithi122.stdout:6/149: link d7/de/dd/f16 d7/de/f34 0 2024-03-22T02:28:44.344 INFO:tasks.workunit.client.1.smithi122.stdout:6/150: dwrite d7/fa [0,4194304] 0 2024-03-22T02:28:44.348 INFO:tasks.workunit.client.1.smithi122.stdout:6/151: write d7/de/dd/d20/d2d/f30 [1532437,57299] 0 2024-03-22T02:28:44.348 INFO:tasks.workunit.client.0.smithi103.stdout:2/538: write d11/d19/d5d/f91 [392680,59219] 0 2024-03-22T02:28:44.349 INFO:tasks.workunit.client.0.smithi103.stdout:4/633: truncate d3/d31/f47 1735064 0 2024-03-22T02:28:44.349 INFO:tasks.workunit.client.0.smithi103.stdout:1/676: dwrite db/f25 [4194304,4194304] 0 2024-03-22T02:28:44.354 INFO:tasks.workunit.client.0.smithi103.stdout:2/539: creat d11/d25/fad x:0 0 0 2024-03-22T02:28:44.355 INFO:tasks.workunit.client.0.smithi103.stdout:7/666: rmdir d1/da/dc1 0 2024-03-22T02:28:44.371 INFO:tasks.workunit.client.0.smithi103.stdout:1/677: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/fcb [0,4194304] 0 2024-03-22T02:28:44.372 INFO:tasks.workunit.client.0.smithi103.stdout:4/634: link d3/d1e/d53/c87 d3/d1e/d53/cd5 0 2024-03-22T02:28:44.373 INFO:tasks.workunit.client.0.smithi103.stdout:7/667: mknod d1/d10/d11/d19/d86/d92/dbf/cd2 0 2024-03-22T02:28:44.374 INFO:tasks.workunit.client.0.smithi103.stdout:7/668: dread - d1/d10/d11/d19/f9e zero size 2024-03-22T02:28:44.375 INFO:tasks.workunit.client.0.smithi103.stdout:1/678: mkdir db/d4a/d78/d92/da8/ddd/d18/dca/de5 0 2024-03-22T02:28:44.380 INFO:tasks.workunit.client.0.smithi103.stdout:1/679: dread - db/d4a/d78/d92/da8/ddd/d29/d55/f81 zero size 2024-03-22T02:28:44.381 INFO:tasks.workunit.client.0.smithi103.stdout:4/635: dwrite d3/d1e/d25/d65/f99 [0,4194304] 0 2024-03-22T02:28:44.382 INFO:tasks.workunit.client.0.smithi103.stdout:1/680: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/fc6 456399 1 2024-03-22T02:28:44.386 INFO:tasks.workunit.client.0.smithi103.stdout:4/636: symlink d3/d1e/d53/d8a/ld6 0 2024-03-22T02:28:44.388 INFO:tasks.workunit.client.0.smithi103.stdout:4/637: write d3/d1e/d68/fcb [651632,91651] 0 2024-03-22T02:28:44.392 INFO:tasks.workunit.client.0.smithi103.stdout:4/638: creat d3/d1e/d68/dbc/fd7 x:0 0 0 2024-03-22T02:28:44.393 INFO:tasks.workunit.client.0.smithi103.stdout:1/681: mknod db/d4a/d78/d92/da8/ddd/d6b/db5/ce6 0 2024-03-22T02:28:44.397 INFO:tasks.workunit.client.0.smithi103.stdout:4/639: link d3/fb d3/d31/fd8 0 2024-03-22T02:28:44.399 INFO:tasks.workunit.client.0.smithi103.stdout:4/640: truncate d3/d38/d5c/d86/fb6 218240 0 2024-03-22T02:28:44.400 INFO:tasks.workunit.client.0.smithi103.stdout:1/682: link db/d4a/d78/d92/da8/ddd/f12 db/d4a/d78/d92/da8/ddd/d18/d84/fe7 0 2024-03-22T02:28:44.403 INFO:tasks.workunit.client.0.smithi103.stdout:4/641: dread d3/d1e/f28 [0,4194304] 0 2024-03-22T02:28:44.404 INFO:tasks.workunit.client.0.smithi103.stdout:1/683: dread db/d4a/d78/d92/da8/ddd/d29/d41/d68/fa3 [0,4194304] 0 2024-03-22T02:28:44.405 INFO:tasks.workunit.client.0.smithi103.stdout:4/642: mknod d3/d1e/d53/d7a/cd9 0 2024-03-22T02:28:44.407 INFO:tasks.workunit.client.0.smithi103.stdout:4/643: creat d3/d38/fda x:0 0 0 2024-03-22T02:28:44.407 INFO:tasks.workunit.client.0.smithi103.stdout:1/684: truncate db/d4a/d78/d92/da8/ddd/d6b/db5/fc3 302835 0 2024-03-22T02:28:44.409 INFO:tasks.workunit.client.0.smithi103.stdout:4/644: creat d3/d1e/d53/d7a/fdb x:0 0 0 2024-03-22T02:28:44.409 INFO:tasks.workunit.client.0.smithi103.stdout:4/645: readlink d3/d38/lb4 0 2024-03-22T02:28:44.410 INFO:tasks.workunit.client.0.smithi103.stdout:4/646: chown d3/d38/lb4 14951720 1 2024-03-22T02:28:44.412 INFO:tasks.workunit.client.0.smithi103.stdout:4/647: chown d3/d31/f47 9292531 1 2024-03-22T02:28:44.412 INFO:tasks.workunit.client.0.smithi103.stdout:1/685: dread db/d4a/d78/d92/da8/ddd/d29/d55/f7c [0,4194304] 0 2024-03-22T02:28:44.413 INFO:tasks.workunit.client.0.smithi103.stdout:4/648: creat d3/d38/fdc x:0 0 0 2024-03-22T02:28:44.414 INFO:tasks.workunit.client.0.smithi103.stdout:1/686: chown db/d4a/d78/d92/da8/ddd/d28/db2/dc8 111754 1 2024-03-22T02:28:44.414 INFO:tasks.workunit.client.0.smithi103.stdout:4/649: chown d3/d38/fdc 386 1 2024-03-22T02:28:44.416 INFO:tasks.workunit.client.0.smithi103.stdout:4/650: creat d3/d38/d5c/d86/fdd x:0 0 0 2024-03-22T02:28:44.417 INFO:tasks.workunit.client.0.smithi103.stdout:1/687: creat db/d4a/d78/d92/da8/ddd/fe8 x:0 0 0 2024-03-22T02:28:44.419 INFO:tasks.workunit.client.0.smithi103.stdout:4/651: rename d3/le to d3/d1e/d53/d8a/lde 0 2024-03-22T02:28:44.421 INFO:tasks.workunit.client.0.smithi103.stdout:4/652: creat d3/d38/fdf x:0 0 0 2024-03-22T02:28:44.422 INFO:tasks.workunit.client.0.smithi103.stdout:1/688: truncate db/d4a/d78/d92/da8/ddd/f63 320610 0 2024-03-22T02:28:44.423 INFO:tasks.workunit.client.0.smithi103.stdout:4/653: unlink d3/d1e/d25/d65/c8c 0 2024-03-22T02:28:44.430 INFO:tasks.workunit.client.0.smithi103.stdout:1/689: dread db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:44.435 INFO:tasks.workunit.client.0.smithi103.stdout:4/654: rename d3/d1e/d53/d8a/lde to d3/d1e/d25/d65/dc9/le0 0 2024-03-22T02:28:44.438 INFO:tasks.workunit.client.0.smithi103.stdout:1/690: rename db/d4a/d78/d92/da8/ddd/d29/l38 to db/d4a/d78/d92/da8/ddd/d28/db2/dc8/le9 0 2024-03-22T02:28:44.439 INFO:tasks.workunit.client.0.smithi103.stdout:1/691: chown db/d4a/d78/d92/c95 2 1 2024-03-22T02:28:44.440 INFO:tasks.workunit.client.0.smithi103.stdout:4/655: truncate d3/d1e/f2f 1561887 0 2024-03-22T02:28:44.440 INFO:tasks.workunit.client.0.smithi103.stdout:4/656: chown d3/d1e/d25/d65/dc9 887 1 2024-03-22T02:28:44.441 INFO:tasks.workunit.client.0.smithi103.stdout:4/657: chown d3/d38/d5c/la4 32383624 1 2024-03-22T02:28:44.443 INFO:tasks.workunit.client.0.smithi103.stdout:1/692: fdatasync db/d4a/d78/d92/da8/ddd/f5a 0 2024-03-22T02:28:44.443 INFO:tasks.workunit.client.0.smithi103.stdout:4/658: write d3/d1e/d25/d65/d83/da1/fad [337503,129047] 0 2024-03-22T02:28:44.445 INFO:tasks.workunit.client.0.smithi103.stdout:1/693: rmdir db/d4a/d78/d92/da8/ddd/d28/db2 39 2024-03-22T02:28:44.446 INFO:tasks.workunit.client.0.smithi103.stdout:4/659: creat d3/d1e/d25/d65/d83/fe1 x:0 0 0 2024-03-22T02:28:44.448 INFO:tasks.workunit.client.0.smithi103.stdout:1/694: read db/d4a/d78/d92/da8/ddd/d18/f3b [7609662,84136] 0 2024-03-22T02:28:44.448 INFO:tasks.workunit.client.0.smithi103.stdout:4/660: write d3/da2/da6/fb2 [1372203,89385] 0 2024-03-22T02:28:44.450 INFO:tasks.workunit.client.0.smithi103.stdout:1/695: mkdir db/d4a/d78/d92/dea 0 2024-03-22T02:28:44.451 INFO:tasks.workunit.client.0.smithi103.stdout:1/696: stat db/d4a/d78/d92/da8/ddd/d29/d55/f8f 0 2024-03-22T02:28:44.454 INFO:tasks.workunit.client.1.smithi122.stdout:6/152: dwrite d7/de/f34 [0,4194304] 0 2024-03-22T02:28:44.455 INFO:tasks.workunit.client.0.smithi103.stdout:1/697: truncate db/d4a/d78/d92/da8/ddd/d18/d84/fe7 1589487 0 2024-03-22T02:28:44.456 INFO:tasks.workunit.client.1.smithi122.stdout:6/153: symlink d7/de/dd/d20/l35 0 2024-03-22T02:28:44.456 INFO:tasks.workunit.client.1.smithi122.stdout:6/154: stat d7/f1b 0 2024-03-22T02:28:44.458 INFO:tasks.workunit.client.1.smithi122.stdout:6/155: creat d7/de/d27/f36 x:0 0 0 2024-03-22T02:28:44.458 INFO:tasks.workunit.client.0.smithi103.stdout:1/698: mkdir db/d4a/d78/d92/da8/deb 0 2024-03-22T02:28:44.459 INFO:tasks.workunit.client.1.smithi122.stdout:6/156: write d7/f15 [151363,94101] 0 2024-03-22T02:28:44.460 INFO:tasks.workunit.client.1.smithi122.stdout:6/157: read f5 [1529740,61333] 0 2024-03-22T02:28:44.463 INFO:tasks.workunit.client.1.smithi122.stdout:6/158: creat d7/de/f37 x:0 0 0 2024-03-22T02:28:44.465 INFO:tasks.workunit.client.0.smithi103.stdout:1/699: link db/d4a/d78/d92/da8/ddd/d29/d41/fcb db/d4a/d78/d92/da8/deb/fec 0 2024-03-22T02:28:44.528 INFO:tasks.workunit.client.1.smithi122.stdout:6/159: truncate d7/de/dd/d20/f21 2086721 0 2024-03-22T02:28:44.528 INFO:tasks.workunit.client.1.smithi122.stdout:6/160: write d7/de/f37 [591183,36395] 0 2024-03-22T02:28:44.530 INFO:tasks.workunit.client.1.smithi122.stdout:6/161: rmdir d7/de/dd/d20/d2d 39 2024-03-22T02:28:44.532 INFO:tasks.workunit.client.1.smithi122.stdout:6/162: symlink d7/de/d27/l38 0 2024-03-22T02:28:44.533 INFO:tasks.workunit.client.1.smithi122.stdout:6/163: mkdir d7/de/dd/d39 0 2024-03-22T02:28:44.539 INFO:tasks.workunit.client.1.smithi122.stdout:6/164: dwrite d7/de/dd/d20/d2d/f30 [0,4194304] 0 2024-03-22T02:28:44.542 INFO:tasks.workunit.client.1.smithi122.stdout:6/165: creat d7/de/dd/f3a x:0 0 0 2024-03-22T02:28:44.542 INFO:tasks.workunit.client.1.smithi122.stdout:6/166: fsync d7/fa 0 2024-03-22T02:28:44.551 INFO:tasks.workunit.client.1.smithi122.stdout:6/167: truncate d7/de/dd/df/f1a 1120493 0 2024-03-22T02:28:44.555 INFO:tasks.workunit.client.1.smithi122.stdout:6/168: dread f5 [0,4194304] 0 2024-03-22T02:28:44.556 INFO:tasks.workunit.client.1.smithi122.stdout:6/169: write d7/de/dd/d20/f2c [553710,31365] 0 2024-03-22T02:28:44.556 INFO:tasks.workunit.client.1.smithi122.stdout:6/170: chown d7 14372108 1 2024-03-22T02:28:44.562 INFO:tasks.workunit.client.1.smithi122.stdout:6/171: dwrite d7/de/dd/f3a [0,4194304] 0 2024-03-22T02:28:44.569 INFO:tasks.workunit.client.1.smithi122.stdout:6/172: rename d7/de/dd/df/f12 to d7/f3b 0 2024-03-22T02:28:44.573 INFO:tasks.workunit.client.1.smithi122.stdout:6/173: rename d7/de/d27/l38 to d7/de/dd/l3c 0 2024-03-22T02:28:44.574 INFO:tasks.workunit.client.1.smithi122.stdout:6/174: symlink d7/de/l3d 0 2024-03-22T02:28:44.575 INFO:tasks.workunit.client.1.smithi122.stdout:6/175: truncate d7/f1b 535653 0 2024-03-22T02:28:44.585 INFO:tasks.workunit.client.0.smithi103.stdout:9/603: sync 2024-03-22T02:28:44.587 INFO:tasks.workunit.client.0.smithi103.stdout:9/604: mknod d2/d5/d7b/cbb 0 2024-03-22T02:28:44.588 INFO:tasks.workunit.client.0.smithi103.stdout:9/605: chown d2/da/d18/d1e/d34/d3d/c78 10610 1 2024-03-22T02:28:44.589 INFO:tasks.workunit.client.0.smithi103.stdout:9/606: fsync d2/d5/fb5 0 2024-03-22T02:28:44.589 INFO:tasks.workunit.client.0.smithi103.stdout:9/607: stat d2/d10/d43/d7e/d96 0 2024-03-22T02:28:44.589 INFO:tasks.workunit.client.0.smithi103.stdout:9/608: readlink d2/d5/d4c/l93 0 2024-03-22T02:28:44.590 INFO:tasks.workunit.client.0.smithi103.stdout:9/609: chown d2/d10/d27/c2a 257200 1 2024-03-22T02:28:44.592 INFO:tasks.workunit.client.0.smithi103.stdout:9/610: creat d2/d5/d4c/d56/fbc x:0 0 0 2024-03-22T02:28:44.593 INFO:tasks.workunit.client.0.smithi103.stdout:9/611: unlink d2/d5/d4c/l92 0 2024-03-22T02:28:44.594 INFO:tasks.workunit.client.1.smithi122.stdout:6/176: dwrite d7/de/dd/df/f1a [0,4194304] 0 2024-03-22T02:28:44.596 INFO:tasks.workunit.client.1.smithi122.stdout:6/177: dread d7/de/dd/d20/f31 [0,4194304] 0 2024-03-22T02:28:44.599 INFO:tasks.workunit.client.0.smithi103.stdout:9/612: dwrite d2/d10/d43/d7e/f9c [0,4194304] 0 2024-03-22T02:28:44.599 INFO:tasks.workunit.client.1.smithi122.stdout:6/178: creat d7/de/f3e x:0 0 0 2024-03-22T02:28:44.600 INFO:tasks.workunit.client.0.smithi103.stdout:9/613: write d2/d10/d43/d4b/f82 [843427,82018] 0 2024-03-22T02:28:44.601 INFO:tasks.workunit.client.0.smithi103.stdout:9/614: write d2/da/d18/d1e/f88 [178681,81427] 0 2024-03-22T02:28:44.602 INFO:tasks.workunit.client.1.smithi122.stdout:9/47: sync 2024-03-22T02:28:44.602 INFO:tasks.workunit.client.1.smithi122.stdout:0/36: sync 2024-03-22T02:28:44.602 INFO:tasks.workunit.client.1.smithi122.stdout:0/37: readlink lc 0 2024-03-22T02:28:44.603 INFO:tasks.workunit.client.1.smithi122.stdout:9/48: write f8 [235580,118917] 0 2024-03-22T02:28:44.603 INFO:tasks.workunit.client.1.smithi122.stdout:6/179: dread d7/f15 [0,4194304] 0 2024-03-22T02:28:44.605 INFO:tasks.workunit.client.0.smithi103.stdout:9/615: mknod d2/d5/d7b/d7d/cbd 0 2024-03-22T02:28:44.607 INFO:tasks.workunit.client.0.smithi103.stdout:9/616: symlink d2/d5/d4c/d56/lbe 0 2024-03-22T02:28:44.608 INFO:tasks.workunit.client.1.smithi122.stdout:0/38: link fb fd 0 2024-03-22T02:28:44.609 INFO:tasks.workunit.client.1.smithi122.stdout:9/49: unlink c4 0 2024-03-22T02:28:44.612 INFO:tasks.workunit.client.1.smithi122.stdout:0/39: fdatasync fb 0 2024-03-22T02:28:44.613 INFO:tasks.workunit.client.1.smithi122.stdout:9/50: mkdir db/df/d10 0 2024-03-22T02:28:44.614 INFO:tasks.workunit.client.1.smithi122.stdout:0/40: creat fe x:0 0 0 2024-03-22T02:28:44.614 INFO:tasks.workunit.client.1.smithi122.stdout:0/41: dread - fe zero size 2024-03-22T02:28:44.617 INFO:tasks.workunit.client.1.smithi122.stdout:6/180: creat d7/de/dd/d20/f3f x:0 0 0 2024-03-22T02:28:44.617 INFO:tasks.workunit.client.1.smithi122.stdout:0/42: readlink la 0 2024-03-22T02:28:44.618 INFO:tasks.workunit.client.1.smithi122.stdout:0/43: truncate fe 67515 0 2024-03-22T02:28:44.622 INFO:tasks.workunit.client.1.smithi122.stdout:9/51: dwrite f8 [0,4194304] 0 2024-03-22T02:28:44.623 INFO:tasks.workunit.client.1.smithi122.stdout:6/181: dread d7/de/dd/f16 [0,4194304] 0 2024-03-22T02:28:44.624 INFO:tasks.workunit.client.1.smithi122.stdout:9/52: creat db/df/d10/f11 x:0 0 0 2024-03-22T02:28:44.625 INFO:tasks.workunit.client.1.smithi122.stdout:9/53: write db/fc [4358647,49547] 0 2024-03-22T02:28:44.626 INFO:tasks.workunit.client.1.smithi122.stdout:9/54: chown fa 414430 1 2024-03-22T02:28:44.627 INFO:tasks.workunit.client.1.smithi122.stdout:9/55: symlink db/df/d10/l12 0 2024-03-22T02:28:44.634 INFO:tasks.workunit.client.1.smithi122.stdout:9/56: dwrite f6 [0,4194304] 0 2024-03-22T02:28:44.640 INFO:tasks.workunit.client.1.smithi122.stdout:9/57: dwrite f8 [0,4194304] 0 2024-03-22T02:28:44.642 INFO:tasks.workunit.client.1.smithi122.stdout:9/58: creat db/f13 x:0 0 0 2024-03-22T02:28:44.681 INFO:tasks.workunit.client.1.smithi122.stdout:0/44: fsync fd 0 2024-03-22T02:28:44.683 INFO:tasks.workunit.client.1.smithi122.stdout:9/59: getdents db/df 0 2024-03-22T02:28:44.684 INFO:tasks.workunit.client.1.smithi122.stdout:9/60: chown f7 4923 1 2024-03-22T02:28:44.686 INFO:tasks.workunit.client.1.smithi122.stdout:9/61: dread - db/f13 zero size 2024-03-22T02:28:44.690 INFO:tasks.workunit.client.1.smithi122.stdout:0/45: dwrite f9 [0,4194304] 0 2024-03-22T02:28:44.692 INFO:tasks.workunit.client.1.smithi122.stdout:9/62: dread f6 [0,4194304] 0 2024-03-22T02:28:44.694 INFO:tasks.workunit.client.1.smithi122.stdout:9/63: fsync db/f13 0 2024-03-22T02:28:44.695 INFO:tasks.workunit.client.1.smithi122.stdout:0/46: dread f9 [0,4194304] 0 2024-03-22T02:28:44.696 INFO:tasks.workunit.client.1.smithi122.stdout:0/47: truncate fe 407203 0 2024-03-22T02:28:44.697 INFO:tasks.workunit.client.1.smithi122.stdout:9/64: mknod db/df/c14 0 2024-03-22T02:28:44.698 INFO:tasks.workunit.client.1.smithi122.stdout:0/48: mknod cf 0 2024-03-22T02:28:44.699 INFO:tasks.workunit.client.1.smithi122.stdout:9/65: creat db/df/f15 x:0 0 0 2024-03-22T02:28:44.701 INFO:tasks.workunit.client.1.smithi122.stdout:0/49: write f9 [5064078,30504] 0 2024-03-22T02:28:44.701 INFO:tasks.workunit.client.1.smithi122.stdout:0/50: rmdir - no directory 2024-03-22T02:28:44.703 INFO:tasks.workunit.client.1.smithi122.stdout:0/51: creat f10 x:0 0 0 2024-03-22T02:28:44.703 INFO:tasks.workunit.client.1.smithi122.stdout:0/52: readlink la 0 2024-03-22T02:28:44.703 INFO:tasks.workunit.client.1.smithi122.stdout:0/53: dread - f10 zero size 2024-03-22T02:28:44.704 INFO:tasks.workunit.client.1.smithi122.stdout:9/66: dread fa [4194304,4194304] 0 2024-03-22T02:28:44.704 INFO:tasks.workunit.client.1.smithi122.stdout:9/67: read fa [7548076,30193] 0 2024-03-22T02:28:44.705 INFO:tasks.workunit.client.1.smithi122.stdout:0/54: mkdir d11 0 2024-03-22T02:28:44.705 INFO:tasks.workunit.client.1.smithi122.stdout:9/68: rename db/df to db/df/d10/d16 22 2024-03-22T02:28:44.706 INFO:tasks.workunit.client.1.smithi122.stdout:9/69: chown db/f13 1731 1 2024-03-22T02:28:44.706 INFO:tasks.workunit.client.1.smithi122.stdout:0/55: mknod d11/c12 0 2024-03-22T02:28:44.802 INFO:tasks.workunit.client.0.smithi103.stdout:9/617: dread d2/da/f24 [0,4194304] 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/618: chown d2/ca9 25 1 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/619: fsync d2/da/f23 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/620: readlink d2/d10/l16 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/621: symlink d2/d10/d43/d7e/da4/lbf 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/622: fdatasync d2/da/d18/d1e/d34/d3d/d4d/f8f 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/623: link d2/da/f6b d2/d10/d43/d4b/d65/fc0 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/624: getdents d2/d5/d4c/d56 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/625: rmdir d2/da/d18/d1e/d34/d3d/d61/da0 39 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/626: chown d2/d5/f38 1655682284 1 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/627: chown d2/d5/d4c/f85 11837734 1 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/628: stat d2/da 0 2024-03-22T02:28:44.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/629: stat d2/d10/d59 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/630: write d2/da/d18/d1e/d34/d3d/d61/f9a [787313,88521] 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/631: fsync d2/da/d18/d1e/f88 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/632: chown d2/d10/d43/f3e 497430547 1 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/465: sync 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/466: stat d0/d3/d22/c85 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/467: mknod d0/d3/d33/c90 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/468: symlink d0/d88/d8e/l91 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/633: getdents d2/d10/d43/d4b/d65 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/634: mkdir d2/d10/d43/d7e/dc1 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/635: fdatasync d2/d10/d43/d4b/f82 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/636: mknod d2/d10/d27/cc2 0 2024-03-22T02:28:44.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/469: write d0/d3/d33/f36 [3565689,45294] 0 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:9/637: dread - d2/d5/d4c/f9f zero size 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:9/638: dread - d2/da/d18/f6c zero size 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:6/470: getdents d0/d3/d22 0 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:6/471: creat d0/d3/f92 x:0 0 0 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:6/472: write d0/d14/d57/d68/f71 [199838,42841] 0 2024-03-22T02:28:44.805 INFO:tasks.workunit.client.0.smithi103.stdout:6/473: creat d0/d14/f93 x:0 0 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:9/70: mknod db/df/c17 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:9/71: chown db/ld 53327 1 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:9/72: chown f5 1117197 1 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/56: write fd [3470830,5403] 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/57: creat d11/f13 x:0 0 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:9/73: dwrite f8 [0,4194304] 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/58: unlink c8 0 2024-03-22T02:28:44.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/59: truncate fe 1229858 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:9/74: mknod db/df/d10/c18 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/60: write fb [1913442,45518] 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/61: truncate d11/f13 49810 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/62: write fb [1292921,26184] 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/63: mkdir d11/d14 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:9/75: dread db/fc [4194304,4194304] 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:9/76: stat db/df/c14 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/64: creat d11/d14/f15 x:0 0 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/65: mkdir d11/d16 0 2024-03-22T02:28:44.811 INFO:tasks.workunit.client.1.smithi122.stdout:0/66: read f9 [2209947,126295] 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:0/67: dwrite f9 [0,4194304] 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/77: truncate f8 1403904 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/78: creat db/df/f19 x:0 0 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/79: creat db/df/d10/f1a x:0 0 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:0/68: dwrite d11/f13 [0,4194304] 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/80: write f7 [1228192,104304] 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:0/69: mkdir d11/d16/d17 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:0/70: write fb [2277728,103329] 0 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:3/30: sync 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:5/53: sync 2024-03-22T02:28:44.812 INFO:tasks.workunit.client.1.smithi122.stdout:2/75: sync 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:2/76: write df/f18 [514163,75550] 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:3/31: chown d2/f4 32317061 1 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:0/71: dread fb [0,4194304] 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/54: creat d8/da/f10 x:0 0 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:2/77: creat df/d1f/f21 x:0 0 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:2/78: fsync f4 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:3/32: creat d2/f6 x:0 0 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:0/72: write fb [4521805,117982] 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:9/81: rmdir db/df/d10 39 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:2/79: creat df/d1f/f22 x:0 0 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:0/73: truncate d11/d14/f15 1033497 0 2024-03-22T02:28:44.813 INFO:tasks.workunit.client.1.smithi122.stdout:5/55: dwrite d8/da/f10 [0,4194304] 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:9/82: mkdir db/d1b 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:0/74: write fd [3312435,115184] 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/33: link d2/f6 d2/f7 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:2/80: chown df/l1c 7506 1 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:5/56: dread d8/da/f10 [0,4194304] 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/34: write d2/f7 [178404,37343] 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:2/81: mkdir df/d1f/d23 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:2/82: dread - df/d1f/f22 zero size 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:9/83: getdents db/d1b 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/35: rename d2/f5 to d2/f8 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:2/83: unlink l5 0 2024-03-22T02:28:44.814 INFO:tasks.workunit.client.1.smithi122.stdout:0/75: rmdir d11/d16/d17 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/84: chown df/f18 2 1 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:0/76: chown c6 24 1 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/85: write df/f15 [650314,113937] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:5/57: dwrite d8/da/f10 [0,4194304] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/84: dwrite db/f13 [0,4194304] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/86: fdatasync df/f14 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:5/58: write d8/da/f10 [2548112,113285] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/87: write df/f19 [359177,184] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/88: chown df/l1c 42204904 1 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/85: dread f6 [0,4194304] 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/86: dread - db/df/f19 zero size 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:0/77: getdents d11 0 2024-03-22T02:28:44.815 INFO:tasks.workunit.client.1.smithi122.stdout:5/59: mknod d8/da/c11 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:9/87: symlink db/d1b/l1c 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:5/60: write d8/da/f10 [1569084,117773] 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:5/61: stat d8/c9 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:0/78: link c1 d11/d16/c18 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:0/79: symlink d11/d16/l19 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:3/36: dwrite d2/f4 [0,4194304] 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:5/62: symlink d8/da/l12 0 2024-03-22T02:28:44.816 INFO:tasks.workunit.client.1.smithi122.stdout:0/80: mknod d11/c1a 0 2024-03-22T02:28:44.817 INFO:tasks.workunit.client.0.smithi103.stdout:6/474: rename d0/d3/d22/d58/d6f/d75/f8d to d0/d3/d22/d58/d6f/f94 0 2024-03-22T02:28:44.817 INFO:tasks.workunit.client.0.smithi103.stdout:6/475: creat d0/d3/d22/d58/d6f/d75/f95 x:0 0 0 2024-03-22T02:28:44.818 INFO:tasks.workunit.client.1.smithi122.stdout:3/37: write d2/f6 [683780,89445] 0 2024-03-22T02:28:44.818 INFO:tasks.workunit.client.1.smithi122.stdout:0/81: mkdir d11/d14/d1b 0 2024-03-22T02:28:44.818 INFO:tasks.workunit.client.1.smithi122.stdout:9/88: dwrite db/f13 [0,4194304] 0 2024-03-22T02:28:44.818 INFO:tasks.workunit.client.1.smithi122.stdout:3/38: mkdir d2/d9 0 2024-03-22T02:28:44.818 INFO:tasks.workunit.client.1.smithi122.stdout:0/82: mkdir d11/d16/d1c 0 2024-03-22T02:28:44.819 INFO:tasks.workunit.client.1.smithi122.stdout:5/63: link d8/ce d8/da/c13 0 2024-03-22T02:28:44.819 INFO:tasks.workunit.client.1.smithi122.stdout:0/83: mkdir d11/d14/d1d 0 2024-03-22T02:28:44.821 INFO:tasks.workunit.client.1.smithi122.stdout:0/84: mknod d11/d14/d1b/c1e 0 2024-03-22T02:28:44.822 INFO:tasks.workunit.client.1.smithi122.stdout:3/39: dwrite d2/f8 [0,4194304] 0 2024-03-22T02:28:44.822 INFO:tasks.workunit.client.1.smithi122.stdout:0/85: write f10 [922741,12153] 0 2024-03-22T02:28:44.824 INFO:tasks.workunit.client.1.smithi122.stdout:0/86: chown d11/d16/d1c 50 1 2024-03-22T02:28:44.825 INFO:tasks.workunit.client.1.smithi122.stdout:3/40: symlink d2/d9/la 0 2024-03-22T02:28:44.827 INFO:tasks.workunit.client.1.smithi122.stdout:0/87: creat d11/d16/d1c/f1f x:0 0 0 2024-03-22T02:28:44.828 INFO:tasks.workunit.client.1.smithi122.stdout:0/88: mknod d11/c20 0 2024-03-22T02:28:44.832 INFO:tasks.workunit.client.1.smithi122.stdout:0/89: dread fe [0,4194304] 0 2024-03-22T02:28:44.832 INFO:tasks.workunit.client.1.smithi122.stdout:0/90: read - d11/d16/d1c/f1f zero size 2024-03-22T02:28:44.838 INFO:tasks.workunit.client.1.smithi122.stdout:0/91: symlink d11/d16/d1c/l21 0 2024-03-22T02:28:44.839 INFO:tasks.workunit.client.1.smithi122.stdout:0/92: dwrite fd [0,4194304] 0 2024-03-22T02:28:44.840 INFO:tasks.workunit.client.1.smithi122.stdout:0/93: creat d11/d14/f22 x:0 0 0 2024-03-22T02:28:44.842 INFO:tasks.workunit.client.1.smithi122.stdout:0/94: mkdir d11/d14/d23 0 2024-03-22T02:28:44.848 INFO:tasks.workunit.client.1.smithi122.stdout:0/95: dread d11/f13 [0,4194304] 0 2024-03-22T02:28:44.853 INFO:tasks.workunit.client.1.smithi122.stdout:0/96: unlink d11/d16/d1c/f1f 0 2024-03-22T02:28:44.853 INFO:tasks.workunit.client.0.smithi103.stdout:6/476: write d0/f41 [138932,4964] 0 2024-03-22T02:28:44.854 INFO:tasks.workunit.client.0.smithi103.stdout:6/477: write d0/d14/d57/d68/f6b [1502073,15839] 0 2024-03-22T02:28:44.855 INFO:tasks.workunit.client.1.smithi122.stdout:0/97: rmdir d11 39 2024-03-22T02:28:44.855 INFO:tasks.workunit.client.0.smithi103.stdout:6/478: chown d0/d14/d57/d68/f71 7 1 2024-03-22T02:28:44.858 INFO:tasks.workunit.client.1.smithi122.stdout:0/98: mkdir d11/d14/d23/d24 0 2024-03-22T02:28:44.859 INFO:tasks.workunit.client.0.smithi103.stdout:6/479: rename d0/d3/d33/f78 to d0/d3/d22/f96 0 2024-03-22T02:28:44.861 INFO:tasks.workunit.client.1.smithi122.stdout:0/99: creat d11/d14/d23/d24/f25 x:0 0 0 2024-03-22T02:28:44.868 INFO:tasks.workunit.client.1.smithi122.stdout:0/100: dwrite d11/f13 [4194304,4194304] 0 2024-03-22T02:28:44.869 INFO:tasks.workunit.client.1.smithi122.stdout:5/64: truncate d8/da/f10 492875 0 2024-03-22T02:28:44.875 INFO:tasks.workunit.client.0.smithi103.stdout:6/480: mkdir d0/d3/d22/d97 0 2024-03-22T02:28:44.876 INFO:tasks.workunit.client.1.smithi122.stdout:0/101: creat d11/d14/f26 x:0 0 0 2024-03-22T02:28:44.877 INFO:tasks.workunit.client.0.smithi103.stdout:6/481: write d0/d14/f56 [1886294,90802] 0 2024-03-22T02:28:44.877 INFO:tasks.workunit.client.1.smithi122.stdout:3/41: dread d2/f6 [0,4194304] 0 2024-03-22T02:28:44.878 INFO:tasks.workunit.client.1.smithi122.stdout:0/102: unlink d11/d14/f22 0 2024-03-22T02:28:44.879 INFO:tasks.workunit.client.1.smithi122.stdout:3/42: dread d2/f6 [0,4194304] 0 2024-03-22T02:28:44.883 INFO:tasks.workunit.client.1.smithi122.stdout:5/65: truncate d8/da/f10 1232939 0 2024-03-22T02:28:44.887 INFO:tasks.workunit.client.0.smithi103.stdout:5/679: sync 2024-03-22T02:28:44.887 INFO:tasks.workunit.client.0.smithi103.stdout:8/782: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:0/642: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:2/540: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:7/669: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:3/717: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:1/700: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.0.smithi103.stdout:4/661: sync 2024-03-22T02:28:44.888 INFO:tasks.workunit.client.1.smithi122.stdout:3/43: dwrite d2/f4 [0,4194304] 0 2024-03-22T02:28:44.889 INFO:tasks.workunit.client.1.smithi122.stdout:0/103: dwrite fb [0,4194304] 0 2024-03-22T02:28:44.890 INFO:tasks.workunit.client.1.smithi122.stdout:0/104: truncate d11/d14/d23/d24/f25 618265 0 2024-03-22T02:28:44.891 INFO:tasks.workunit.client.0.smithi103.stdout:2/541: chown d11/d19/d4e/f7d 109 1 2024-03-22T02:28:44.891 INFO:tasks.workunit.client.0.smithi103.stdout:1/701: dread db/f2b [4194304,4194304] 0 2024-03-22T02:28:44.892 INFO:tasks.workunit.client.1.smithi122.stdout:0/105: write d11/d14/f15 [38154,44842] 0 2024-03-22T02:28:44.892 INFO:tasks.workunit.client.0.smithi103.stdout:4/662: creat d3/d1e/d53/fe2 x:0 0 0 2024-03-22T02:28:44.893 INFO:tasks.workunit.client.0.smithi103.stdout:8/783: dread - d4/d26/d27/d7e/d39/d49/d51/daf/dc2/ff7 zero size 2024-03-22T02:28:44.893 INFO:tasks.workunit.client.0.smithi103.stdout:2/542: dread - d11/d25/d46/fa2 zero size 2024-03-22T02:28:44.895 INFO:tasks.workunit.client.0.smithi103.stdout:0/643: creat dd/d34/d53/d8c/fd9 x:0 0 0 2024-03-22T02:28:44.895 INFO:tasks.workunit.client.1.smithi122.stdout:3/44: getdents d2 0 2024-03-22T02:28:44.895 INFO:tasks.workunit.client.0.smithi103.stdout:7/670: fdatasync d1/f1e 0 2024-03-22T02:28:44.896 INFO:tasks.workunit.client.1.smithi122.stdout:3/45: write d2/f4 [2283539,54366] 0 2024-03-22T02:28:44.896 INFO:tasks.workunit.client.0.smithi103.stdout:5/680: mknod d0/d14/d23/d2a/d7b/cec 0 2024-03-22T02:28:44.896 INFO:tasks.workunit.client.1.smithi122.stdout:3/46: read d2/f7 [552301,28644] 0 2024-03-22T02:28:44.899 INFO:tasks.workunit.client.1.smithi122.stdout:3/47: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:44.899 INFO:tasks.workunit.client.1.smithi122.stdout:0/106: rename d11/d16/d1c to d11/d14/d1d/d27 0 2024-03-22T02:28:44.899 INFO:tasks.workunit.client.0.smithi103.stdout:3/718: symlink de/d1a/lf1 0 2024-03-22T02:28:44.899 INFO:tasks.workunit.client.0.smithi103.stdout:6/482: dread d0/d3/d22/f5c [0,4194304] 0 2024-03-22T02:28:44.900 INFO:tasks.workunit.client.0.smithi103.stdout:4/663: symlink d3/d1e/d53/le3 0 2024-03-22T02:28:44.902 INFO:tasks.workunit.client.1.smithi122.stdout:5/66: write d8/da/f10 [80199,54956] 0 2024-03-22T02:28:44.904 INFO:tasks.workunit.client.0.smithi103.stdout:3/719: write de/d1a/d25/f6c [218858,80648] 0 2024-03-22T02:28:44.904 INFO:tasks.workunit.client.1.smithi122.stdout:3/48: rename d2/f4 to d2/fb 0 2024-03-22T02:28:44.906 INFO:tasks.workunit.client.0.smithi103.stdout:6/483: dread d0/d14/d57/d68/f71 [0,4194304] 0 2024-03-22T02:28:44.909 INFO:tasks.workunit.client.1.smithi122.stdout:3/49: mkdir d2/d9/dc 0 2024-03-22T02:28:44.910 INFO:tasks.workunit.client.0.smithi103.stdout:7/671: mkdir d1/d10/d11/d19/d29/d33/da8/dd3 0 2024-03-22T02:28:44.910 INFO:tasks.workunit.client.0.smithi103.stdout:8/784: dread - d4/d2a/d38/f9c zero size 2024-03-22T02:28:44.910 INFO:tasks.workunit.client.0.smithi103.stdout:3/720: dread de/d1a/d25/f6a [0,4194304] 0 2024-03-22T02:28:44.911 INFO:tasks.workunit.client.1.smithi122.stdout:5/67: link d8/da/l12 d8/da/l14 0 2024-03-22T02:28:44.911 INFO:tasks.workunit.client.1.smithi122.stdout:5/68: stat d8/c9 0 2024-03-22T02:28:44.911 INFO:tasks.workunit.client.1.smithi122.stdout:3/50: creat d2/d9/dc/fd x:0 0 0 2024-03-22T02:28:44.911 INFO:tasks.workunit.client.1.smithi122.stdout:5/69: truncate d8/da/f10 2007168 0 2024-03-22T02:28:44.912 INFO:tasks.workunit.client.1.smithi122.stdout:3/51: dread d2/f6 [0,4194304] 0 2024-03-22T02:28:44.912 INFO:tasks.workunit.client.1.smithi122.stdout:0/107: fsync fd 0 2024-03-22T02:28:44.912 INFO:tasks.workunit.client.0.smithi103.stdout:1/702: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ded 0 2024-03-22T02:28:44.912 INFO:tasks.workunit.client.1.smithi122.stdout:3/52: chown d2/d9/la 42 1 2024-03-22T02:28:44.913 INFO:tasks.workunit.client.1.smithi122.stdout:5/70: rename d8/da/lb to d8/l15 0 2024-03-22T02:28:44.914 INFO:tasks.workunit.client.0.smithi103.stdout:4/664: dwrite d3/d69/fcd [0,4194304] 0 2024-03-22T02:28:44.914 INFO:tasks.workunit.client.0.smithi103.stdout:3/721: fsync de/d12/d5d/d92/fce 0 2024-03-22T02:28:44.915 INFO:tasks.workunit.client.1.smithi122.stdout:3/53: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:44.916 INFO:tasks.workunit.client.0.smithi103.stdout:5/681: dwrite d0/d14/d23/d2a/d33/db5/dbd/fc5 [0,4194304] 0 2024-03-22T02:28:44.916 INFO:tasks.workunit.client.0.smithi103.stdout:4/665: fsync d3/d38/f4c 0 2024-03-22T02:28:44.916 INFO:tasks.workunit.client.1.smithi122.stdout:5/71: creat d8/da/f16 x:0 0 0 2024-03-22T02:28:44.918 INFO:tasks.workunit.client.1.smithi122.stdout:3/54: mknod d2/ce 0 2024-03-22T02:28:44.918 INFO:tasks.workunit.client.1.smithi122.stdout:0/108: dread fe [0,4194304] 0 2024-03-22T02:28:44.919 INFO:tasks.workunit.client.0.smithi103.stdout:7/672: creat d1/d10/d11/d19/d86/fd4 x:0 0 0 2024-03-22T02:28:44.919 INFO:tasks.workunit.client.1.smithi122.stdout:3/55: dread - d2/d9/dc/fd zero size 2024-03-22T02:28:44.920 INFO:tasks.workunit.client.1.smithi122.stdout:0/109: chown d11/d14/d23 4 1 2024-03-22T02:28:44.920 INFO:tasks.workunit.client.1.smithi122.stdout:7/88: sync 2024-03-22T02:28:44.920 INFO:tasks.workunit.client.1.smithi122.stdout:0/110: write f10 [707096,112020] 0 2024-03-22T02:28:44.920 INFO:tasks.workunit.client.1.smithi122.stdout:7/89: readlink d0/d4/l7 0 2024-03-22T02:28:44.922 INFO:tasks.workunit.client.1.smithi122.stdout:3/56: mkdir d2/d9/dc/df 0 2024-03-22T02:28:44.923 INFO:tasks.workunit.client.1.smithi122.stdout:3/57: stat d2/f6 0 2024-03-22T02:28:44.924 INFO:tasks.workunit.client.1.smithi122.stdout:0/111: mknod d11/d14/d1d/c28 0 2024-03-22T02:28:44.926 INFO:tasks.workunit.client.0.smithi103.stdout:1/703: write db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [1044642,112791] 0 2024-03-22T02:28:44.926 INFO:tasks.workunit.client.0.smithi103.stdout:0/644: dwrite dd/d16/d4b/d96/d64/dab/dc6/f6f [0,4194304] 0 2024-03-22T02:28:44.926 INFO:tasks.workunit.client.0.smithi103.stdout:8/785: mkdir d4/d2a/d38/dfe 0 2024-03-22T02:28:44.927 INFO:tasks.workunit.client.0.smithi103.stdout:2/543: write d11/d25/d46/d48/f7a [1982257,58367] 0 2024-03-22T02:28:44.929 INFO:tasks.workunit.client.0.smithi103.stdout:0/645: chown dd/d16/d4b/d96/c7e 2267 1 2024-03-22T02:28:44.930 INFO:tasks.workunit.client.1.smithi122.stdout:0/112: chown f10 43137 1 2024-03-22T02:28:44.930 INFO:tasks.workunit.client.1.smithi122.stdout:5/72: dwrite d8/da/f16 [0,4194304] 0 2024-03-22T02:28:44.930 INFO:tasks.workunit.client.1.smithi122.stdout:0/113: stat fe 0 2024-03-22T02:28:44.930 INFO:tasks.workunit.client.0.smithi103.stdout:3/722: rename de/d1a/d25/d26 to de/d12/d19/df2 0 2024-03-22T02:28:44.932 INFO:tasks.workunit.client.0.smithi103.stdout:5/682: mkdir d0/d14/d23/d2a/d33/db5/dbd/ded 0 2024-03-22T02:28:44.933 INFO:tasks.workunit.client.0.smithi103.stdout:3/723: write de/d12/d8a/d93/fac [614190,65951] 0 2024-03-22T02:28:44.934 INFO:tasks.workunit.client.1.smithi122.stdout:3/58: symlink d2/d9/dc/l10 0 2024-03-22T02:28:44.934 INFO:tasks.workunit.client.0.smithi103.stdout:8/786: truncate d4/d26/d27/d7e/d39/d49/d79/fb9 1074918 0 2024-03-22T02:28:44.935 INFO:tasks.workunit.client.0.smithi103.stdout:5/683: fdatasync d0/d14/d23/d2a/d1f/f78 0 2024-03-22T02:28:44.938 INFO:tasks.workunit.client.0.smithi103.stdout:3/724: chown de/d6b/d80/d4f/f5b 128743 1 2024-03-22T02:28:44.939 INFO:tasks.workunit.client.1.smithi122.stdout:7/90: dwrite d0/d4/f11 [0,4194304] 0 2024-03-22T02:28:44.939 INFO:tasks.workunit.client.0.smithi103.stdout:4/666: rename d3/d30/d45/f58 to d3/d1e/d25/d65/d83/da1/dbd/fe4 0 2024-03-22T02:28:44.942 INFO:tasks.workunit.client.0.smithi103.stdout:0/646: unlink dd/d16/d4b/d96/d64/dab/dc6/c5e 0 2024-03-22T02:28:44.943 INFO:tasks.workunit.client.1.smithi122.stdout:5/73: dread d8/da/f16 [0,4194304] 0 2024-03-22T02:28:44.945 INFO:tasks.workunit.client.0.smithi103.stdout:1/704: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f82 [0,4194304] 0 2024-03-22T02:28:44.946 INFO:tasks.workunit.client.0.smithi103.stdout:8/787: rmdir d4/d26/d27/d7e/d39/d49/d51/daf/dc2 39 2024-03-22T02:28:44.947 INFO:tasks.workunit.client.1.smithi122.stdout:5/74: creat d8/f17 x:0 0 0 2024-03-22T02:28:44.952 INFO:tasks.workunit.client.1.smithi122.stdout:7/91: rename d0/d4/ce to d0/c13 0 2024-03-22T02:28:44.952 INFO:tasks.workunit.client.1.smithi122.stdout:7/92: read d0/f2 [6603527,21494] 0 2024-03-22T02:28:44.953 INFO:tasks.workunit.client.0.smithi103.stdout:0/647: readlink dd/d16/d4b/d96/d64/dab/dc6/l3d 0 2024-03-22T02:28:44.953 INFO:tasks.workunit.client.0.smithi103.stdout:1/705: dread - db/d4a/d78/d92/da8/ddd/d18/dca/fdc zero size 2024-03-22T02:28:44.954 INFO:tasks.workunit.client.0.smithi103.stdout:6/484: dwrite d0/d3/d22/f47 [0,4194304] 0 2024-03-22T02:28:44.954 INFO:tasks.workunit.client.0.smithi103.stdout:3/725: creat de/d1a/d25/d3f/ff3 x:0 0 0 2024-03-22T02:28:44.955 INFO:tasks.workunit.client.0.smithi103.stdout:4/667: rmdir d3/d1e/d78/db9 0 2024-03-22T02:28:44.958 INFO:tasks.workunit.client.0.smithi103.stdout:5/684: dwrite d0/fd4 [0,4194304] 0 2024-03-22T02:28:44.961 INFO:tasks.workunit.client.0.smithi103.stdout:2/544: dread f4 [0,4194304] 0 2024-03-22T02:28:44.965 INFO:tasks.workunit.client.1.smithi122.stdout:5/75: rmdir d8 39 2024-03-22T02:28:44.966 INFO:tasks.workunit.client.0.smithi103.stdout:5/685: write d0/d14/d23/d2a/d1f/f32 [1970331,5550] 0 2024-03-22T02:28:44.968 INFO:tasks.workunit.client.0.smithi103.stdout:6/485: fsync d0/d14/f4c 0 2024-03-22T02:28:44.969 INFO:tasks.workunit.client.1.smithi122.stdout:5/76: creat d8/da/f18 x:0 0 0 2024-03-22T02:28:44.970 INFO:tasks.workunit.client.1.smithi122.stdout:5/77: dread - d8/f17 zero size 2024-03-22T02:28:44.971 INFO:tasks.workunit.client.0.smithi103.stdout:0/648: read dd/d16/d4b/d96/d64/dab/dc6/fb4 [1422615,53963] 0 2024-03-22T02:28:44.972 INFO:tasks.workunit.client.0.smithi103.stdout:7/673: dread d1/d10/d11/d19/d29/d57/f6d [0,4194304] 0 2024-03-22T02:28:44.973 INFO:tasks.workunit.client.0.smithi103.stdout:0/649: fdatasync fc 0 2024-03-22T02:28:44.974 INFO:tasks.workunit.client.0.smithi103.stdout:0/650: fsync dd/d16/d4b/d96/d64/d91/f99 0 2024-03-22T02:28:44.977 INFO:tasks.workunit.client.0.smithi103.stdout:6/486: rmdir d0/d14/d57/d68 39 2024-03-22T02:28:44.979 INFO:tasks.workunit.client.0.smithi103.stdout:1/706: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ded/cee 0 2024-03-22T02:28:44.980 INFO:tasks.workunit.client.0.smithi103.stdout:5/686: write d0/da2/dca/fdd [2854516,100297] 0 2024-03-22T02:28:44.982 INFO:tasks.workunit.client.1.smithi122.stdout:5/78: rmdir d8/da 39 2024-03-22T02:28:44.983 INFO:tasks.workunit.client.0.smithi103.stdout:7/674: creat d1/d10/d11/d19/d29/d57/fd5 x:0 0 0 2024-03-22T02:28:44.984 INFO:tasks.workunit.client.0.smithi103.stdout:0/651: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/fda x:0 0 0 2024-03-22T02:28:44.985 INFO:tasks.workunit.client.1.smithi122.stdout:5/79: creat d8/f19 x:0 0 0 2024-03-22T02:28:44.986 INFO:tasks.workunit.client.0.smithi103.stdout:4/668: link d3/d1e/d53/c87 d3/d31/ce5 0 2024-03-22T02:28:44.987 INFO:tasks.workunit.client.1.smithi122.stdout:5/80: chown d8/l15 38267 1 2024-03-22T02:28:44.987 INFO:tasks.workunit.client.1.smithi122.stdout:5/81: dread - d8/f17 zero size 2024-03-22T02:28:44.987 INFO:tasks.workunit.client.0.smithi103.stdout:2/545: dwrite d11/d19/d5d/f75 [0,4194304] 0 2024-03-22T02:28:44.989 INFO:tasks.workunit.client.1.smithi122.stdout:5/82: fdatasync d8/da/f16 0 2024-03-22T02:28:44.990 INFO:tasks.workunit.client.0.smithi103.stdout:2/546: write d11/f58 [4065446,107448] 0 2024-03-22T02:28:44.992 INFO:tasks.workunit.client.1.smithi122.stdout:5/83: write d8/da/f16 [2798614,123278] 0 2024-03-22T02:28:44.993 INFO:tasks.workunit.client.0.smithi103.stdout:6/487: symlink d0/d14/d57/d68/l98 0 2024-03-22T02:28:44.995 INFO:tasks.workunit.client.0.smithi103.stdout:4/669: unlink d3/d1e/d25/c79 0 2024-03-22T02:28:44.995 INFO:tasks.workunit.client.0.smithi103.stdout:1/707: dread db/d4a/d78/d92/da8/ddd/d28/d97/fb3 [0,4194304] 0 2024-03-22T02:28:44.995 INFO:tasks.workunit.client.0.smithi103.stdout:3/726: dread de/d6b/d80/d4f/fa3 [0,4194304] 0 2024-03-22T02:28:44.997 INFO:tasks.workunit.client.1.smithi122.stdout:5/84: dread d8/da/f16 [0,4194304] 0 2024-03-22T02:28:44.997 INFO:tasks.workunit.client.0.smithi103.stdout:0/652: truncate dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 58418 0 2024-03-22T02:28:44.999 INFO:tasks.workunit.client.1.smithi122.stdout:5/85: chown d8/da/l14 1112 1 2024-03-22T02:28:44.999 INFO:tasks.workunit.client.1.smithi122.stdout:5/86: chown c1 1054639969 1 2024-03-22T02:28:45.000 INFO:tasks.workunit.client.1.smithi122.stdout:5/87: truncate d8/da/f16 4218936 0 2024-03-22T02:28:45.001 INFO:tasks.workunit.client.0.smithi103.stdout:2/547: mkdir d11/d25/d63/d45/d47/dae 0 2024-03-22T02:28:45.001 INFO:tasks.workunit.client.1.smithi122.stdout:5/88: write d8/da/f16 [2423256,31165] 0 2024-03-22T02:28:45.002 INFO:tasks.workunit.client.0.smithi103.stdout:4/670: creat d3/d31/fe6 x:0 0 0 2024-03-22T02:28:45.003 INFO:tasks.workunit.client.0.smithi103.stdout:4/671: chown d3/d69/f9e 6 1 2024-03-22T02:28:45.003 INFO:tasks.workunit.client.1.smithi122.stdout:5/89: unlink d8/da/c11 0 2024-03-22T02:28:45.003 INFO:tasks.workunit.client.0.smithi103.stdout:0/653: fdatasync dd/f12 0 2024-03-22T02:28:45.006 INFO:tasks.workunit.client.1.smithi122.stdout:5/90: mkdir d8/d1a 0 2024-03-22T02:28:45.006 INFO:tasks.workunit.client.0.smithi103.stdout:7/675: link d1/da/d1c/lca d1/d10/d11/d19/d29/d33/da8/dc5/ld6 0 2024-03-22T02:28:45.008 INFO:tasks.workunit.client.1.smithi122.stdout:5/91: symlink d8/d1a/l1b 0 2024-03-22T02:28:45.008 INFO:tasks.workunit.client.0.smithi103.stdout:0/654: write dd/d34/d53/d8c/fc0 [5103529,83755] 0 2024-03-22T02:28:45.009 INFO:tasks.workunit.client.0.smithi103.stdout:1/708: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/lef 0 2024-03-22T02:28:45.010 INFO:tasks.workunit.client.0.smithi103.stdout:5/687: dread d0/d14/d23/d6f/fa7 [0,4194304] 0 2024-03-22T02:28:45.010 INFO:tasks.workunit.client.1.smithi122.stdout:5/92: creat d8/d1a/f1c x:0 0 0 2024-03-22T02:28:45.010 INFO:tasks.workunit.client.1.smithi122.stdout:5/93: fsync d8/da/f10 0 2024-03-22T02:28:45.011 INFO:tasks.workunit.client.0.smithi103.stdout:7/676: fdatasync d1/da/d1c/f60 0 2024-03-22T02:28:45.011 INFO:tasks.workunit.client.0.smithi103.stdout:6/488: dwrite d0/d3/d22/d58/f89 [4194304,4194304] 0 2024-03-22T02:28:45.012 INFO:tasks.workunit.client.1.smithi122.stdout:5/94: creat d8/d1a/f1d x:0 0 0 2024-03-22T02:28:45.012 INFO:tasks.workunit.client.1.smithi122.stdout:5/95: fdatasync d8/f19 0 2024-03-22T02:28:45.014 INFO:tasks.workunit.client.1.smithi122.stdout:5/96: mkdir d8/d1e 0 2024-03-22T02:28:45.016 INFO:tasks.workunit.client.0.smithi103.stdout:4/672: link d3/d1e/l2e d3/d1e/d53/d7a/da3/le7 0 2024-03-22T02:28:45.018 INFO:tasks.workunit.client.1.smithi122.stdout:5/97: rename d8/ld to d8/l1f 0 2024-03-22T02:28:45.019 INFO:tasks.workunit.client.0.smithi103.stdout:6/489: unlink d0/c10 0 2024-03-22T02:28:45.021 INFO:tasks.workunit.client.1.smithi122.stdout:5/98: unlink d8/da/l12 0 2024-03-22T02:28:45.022 INFO:tasks.workunit.client.0.smithi103.stdout:1/709: dwrite db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:45.024 INFO:tasks.workunit.client.0.smithi103.stdout:3/727: dread de/d1a/d25/f8b [0,4194304] 0 2024-03-22T02:28:45.024 INFO:tasks.workunit.client.0.smithi103.stdout:7/677: unlink d1/d10/d11/d19/d29/d61/d69/f32 0 2024-03-22T02:28:45.026 INFO:tasks.workunit.client.0.smithi103.stdout:0/655: symlink dd/d16/d4b/d96/d35/d82/ldb 0 2024-03-22T02:28:45.026 INFO:tasks.workunit.client.0.smithi103.stdout:0/656: chown fc 2 1 2024-03-22T02:28:45.026 INFO:tasks.workunit.client.0.smithi103.stdout:0/657: stat fc 0 2024-03-22T02:28:45.027 INFO:tasks.workunit.client.0.smithi103.stdout:0/658: fsync dd/d16/d4b/d96/d29/d48/f73 0 2024-03-22T02:28:45.028 INFO:tasks.workunit.client.0.smithi103.stdout:4/673: mknod d3/d1e/d53/ce8 0 2024-03-22T02:28:45.029 INFO:tasks.workunit.client.0.smithi103.stdout:0/659: dread - dd/d16/d4b/d96/d35/dc4/fd4 zero size 2024-03-22T02:28:45.029 INFO:tasks.workunit.client.1.smithi122.stdout:5/99: dwrite d8/d1a/f1c [0,4194304] 0 2024-03-22T02:28:45.030 INFO:tasks.workunit.client.0.smithi103.stdout:5/688: symlink d0/d14/d23/d46/de7/lee 0 2024-03-22T02:28:45.030 INFO:tasks.workunit.client.1.smithi122.stdout:5/100: dread - d8/f17 zero size 2024-03-22T02:28:45.032 INFO:tasks.workunit.client.0.smithi103.stdout:1/710: creat db/d4a/d78/d92/ff0 x:0 0 0 2024-03-22T02:28:45.035 INFO:tasks.workunit.client.0.smithi103.stdout:3/728: rename de/d12/d30/d70/fcc to de/d6b/d80/d73/ff4 0 2024-03-22T02:28:45.037 INFO:tasks.workunit.client.0.smithi103.stdout:0/660: truncate dd/d16/d4b/d96/d64/dab/dc6/fb4 5183528 0 2024-03-22T02:28:45.038 INFO:tasks.workunit.client.0.smithi103.stdout:4/674: dwrite d3/d69/f5a [0,4194304] 0 2024-03-22T02:28:45.038 INFO:tasks.workunit.client.0.smithi103.stdout:9/639: sync 2024-03-22T02:28:45.042 INFO:tasks.workunit.client.0.smithi103.stdout:1/711: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/cf1 0 2024-03-22T02:28:45.043 INFO:tasks.workunit.client.0.smithi103.stdout:7/678: dwrite d1/da/d2f/fb7 [0,4194304] 0 2024-03-22T02:28:45.045 INFO:tasks.workunit.client.0.smithi103.stdout:4/675: creat d3/d1e/d53/d8a/fe9 x:0 0 0 2024-03-22T02:28:45.050 INFO:tasks.workunit.client.0.smithi103.stdout:4/676: chown d3/d38/d5c/d86/fb6 146029290 1 2024-03-22T02:28:45.052 INFO:tasks.workunit.client.0.smithi103.stdout:1/712: creat db/d4a/d78/d92/da8/ddd/d18/d2d/ff2 x:0 0 0 2024-03-22T02:28:45.053 INFO:tasks.workunit.client.0.smithi103.stdout:2/548: dread d11/d25/d46/d40/f6f [0,4194304] 0 2024-03-22T02:28:45.054 INFO:tasks.workunit.client.0.smithi103.stdout:7/679: getdents d1/d10/d11 0 2024-03-22T02:28:45.055 INFO:tasks.workunit.client.0.smithi103.stdout:9/640: dwrite d2/da/d18/d1e/d77/d8b/f26 [0,4194304] 0 2024-03-22T02:28:45.059 INFO:tasks.workunit.client.0.smithi103.stdout:4/677: rename d3/d69/f9e to d3/d1e/d25/d65/d83/da1/dbd/fea 0 2024-03-22T02:28:45.061 INFO:tasks.workunit.client.0.smithi103.stdout:7/680: stat d1/d10/d11/d19/d29/d33/d4b/f97 0 2024-03-22T02:28:45.063 INFO:tasks.workunit.client.0.smithi103.stdout:7/681: readlink d1/d10/d11/d19/d29/d61/d69/d6b/d7a/ld0 0 2024-03-22T02:28:45.064 INFO:tasks.workunit.client.0.smithi103.stdout:1/713: creat db/d4a/d78/d92/da8/ddd/d2e/dae/ff3 x:0 0 0 2024-03-22T02:28:45.066 INFO:tasks.workunit.client.0.smithi103.stdout:9/641: creat d2/da/d18/d1e/d34/d3d/d61/db7/fc3 x:0 0 0 2024-03-22T02:28:45.067 INFO:tasks.workunit.client.0.smithi103.stdout:3/729: dread de/d12/d19/df2/f2f [0,4194304] 0 2024-03-22T02:28:45.069 INFO:tasks.workunit.client.0.smithi103.stdout:4/678: mkdir d3/deb 0 2024-03-22T02:28:45.070 INFO:tasks.workunit.client.0.smithi103.stdout:8/788: sync 2024-03-22T02:28:45.071 INFO:tasks.workunit.client.0.smithi103.stdout:2/549: link d11/d25/d63/d45/l5e d11/d25/d46/d34/laf 0 2024-03-22T02:28:45.075 INFO:tasks.workunit.client.0.smithi103.stdout:7/682: symlink d1/d10/d11/d19/d29/d61/ld7 0 2024-03-22T02:28:45.077 INFO:tasks.workunit.client.0.smithi103.stdout:9/642: dread d2/d10/d27/f51 [0,4194304] 0 2024-03-22T02:28:45.078 INFO:tasks.workunit.client.0.smithi103.stdout:6/490: dread d0/d3/d33/f36 [0,4194304] 0 2024-03-22T02:28:45.078 INFO:tasks.workunit.client.0.smithi103.stdout:3/730: mknod de/d12/d30/d70/de5/cf5 0 2024-03-22T02:28:45.079 INFO:tasks.workunit.client.0.smithi103.stdout:3/731: dread - de/d1a/d25/d3f/ff3 zero size 2024-03-22T02:28:45.081 INFO:tasks.workunit.client.0.smithi103.stdout:5/689: dread d0/d14/f41 [0,4194304] 0 2024-03-22T02:28:45.082 INFO:tasks.workunit.client.0.smithi103.stdout:5/690: truncate d0/d14/d23/fbc 1342731 0 2024-03-22T02:28:45.084 INFO:tasks.workunit.client.1.smithi122.stdout:1/75: sync 2024-03-22T02:28:45.089 INFO:tasks.workunit.client.0.smithi103.stdout:0/661: truncate dd/d16/d4b/d96/f21 3404460 0 2024-03-22T02:28:45.090 INFO:tasks.workunit.client.0.smithi103.stdout:4/679: creat d3/d1e/d25/d65/d83/dc7/fec x:0 0 0 2024-03-22T02:28:45.092 INFO:tasks.workunit.client.0.smithi103.stdout:7/683: creat d1/d10/d11/d19/dc3/fd8 x:0 0 0 2024-03-22T02:28:45.093 INFO:tasks.workunit.client.0.smithi103.stdout:9/643: mkdir d2/da/d18/dc4 0 2024-03-22T02:28:45.095 INFO:tasks.workunit.client.0.smithi103.stdout:2/550: rename d11/d25/d46/d34 to d11/d25/d63/d45/d47/d83/d78/db0 0 2024-03-22T02:28:45.097 INFO:tasks.workunit.client.0.smithi103.stdout:8/789: rmdir d4/d26/deb 0 2024-03-22T02:28:45.099 INFO:tasks.workunit.client.0.smithi103.stdout:1/714: write db/d4a/d78/d92/da8/ddd/d29/f39 [530204,99295] 0 2024-03-22T02:28:45.100 INFO:tasks.workunit.client.0.smithi103.stdout:4/680: symlink d3/d1e/d25/d65/d83/da1/led 0 2024-03-22T02:28:45.102 INFO:tasks.workunit.client.0.smithi103.stdout:9/644: symlink d2/d10/d43/d4b/d65/lc5 0 2024-03-22T02:28:45.102 INFO:tasks.workunit.client.0.smithi103.stdout:8/790: fsync d4/d26/d27/d7e/d39/d49/d51/fea 0 2024-03-22T02:28:45.103 INFO:tasks.workunit.client.0.smithi103.stdout:6/491: creat d0/d14/d57/f99 x:0 0 0 2024-03-22T02:28:45.104 INFO:tasks.workunit.client.0.smithi103.stdout:7/684: dread d1/d10/d11/f28 [0,4194304] 0 2024-03-22T02:28:45.105 INFO:tasks.workunit.client.0.smithi103.stdout:2/551: dwrite d11/d25/d63/f96 [0,4194304] 0 2024-03-22T02:28:45.106 INFO:tasks.workunit.client.0.smithi103.stdout:8/791: dread - d4/d2a/d4f/fb7 zero size 2024-03-22T02:28:45.108 INFO:tasks.workunit.client.0.smithi103.stdout:3/732: rename c6 to de/d6b/cf6 0 2024-03-22T02:28:45.110 INFO:tasks.workunit.client.0.smithi103.stdout:7/685: dread - d1/d10/d11/d19/d29/d61/d69/d5c/fb8 zero size 2024-03-22T02:28:45.117 INFO:tasks.workunit.client.0.smithi103.stdout:0/662: creat dd/d16/d4b/d96/d64/dab/dc6/fdc x:0 0 0 2024-03-22T02:28:45.118 INFO:tasks.workunit.client.0.smithi103.stdout:5/691: link d0/d14/d23/d2a/d2c/l56 d0/d14/d23/d2a/lef 0 2024-03-22T02:28:45.120 INFO:tasks.workunit.client.0.smithi103.stdout:2/552: creat d11/d19/da7/fb1 x:0 0 0 2024-03-22T02:28:45.121 INFO:tasks.workunit.client.0.smithi103.stdout:1/715: link db/d4a/d78/d92/da8/ddd/d18/f3b db/d4a/d78/d92/da8/ddd/d29/d55/ff4 0 2024-03-22T02:28:45.122 INFO:tasks.workunit.client.0.smithi103.stdout:1/716: chown db/d4a/d78/d92/da8/ddd/d28/f43 99 1 2024-03-22T02:28:45.125 INFO:tasks.workunit.client.0.smithi103.stdout:8/792: creat d4/d48/fff x:0 0 0 2024-03-22T02:28:45.127 INFO:tasks.workunit.client.0.smithi103.stdout:5/692: creat d0/d14/d23/d36/d95/d96/daa/daf/ff0 x:0 0 0 2024-03-22T02:28:45.127 INFO:tasks.workunit.client.0.smithi103.stdout:2/553: rmdir d11/d25/d46/d40 39 2024-03-22T02:28:45.129 INFO:tasks.workunit.client.0.smithi103.stdout:6/492: rename d0/d3/d22/d97 to d0/d14/d9a 0 2024-03-22T02:28:45.131 INFO:tasks.workunit.client.0.smithi103.stdout:8/793: readlink d4/d2a/d8b/l99 0 2024-03-22T02:28:45.132 INFO:tasks.workunit.client.0.smithi103.stdout:1/717: mknod db/d4a/d78/d92/da8/ddd/d6b/cf5 0 2024-03-22T02:28:45.135 INFO:tasks.workunit.client.0.smithi103.stdout:5/693: link d0/d14/d23/d36/d95/f5a d0/d14/d23/d36/d95/ff1 0 2024-03-22T02:28:45.136 INFO:tasks.workunit.client.0.smithi103.stdout:5/694: fsync d0/d14/d23/d36/f82 0 2024-03-22T02:28:45.137 INFO:tasks.workunit.client.0.smithi103.stdout:3/733: rename de/d12/f1c to de/d12/d19/ff7 0 2024-03-22T02:28:45.138 INFO:tasks.workunit.client.0.smithi103.stdout:6/493: mknod d0/d3/c9b 0 2024-03-22T02:28:45.140 INFO:tasks.workunit.client.0.smithi103.stdout:8/794: unlink d4/d2a/c32 0 2024-03-22T02:28:45.144 INFO:tasks.workunit.client.0.smithi103.stdout:8/795: creat d4/d19/d8c/f100 x:0 0 0 2024-03-22T02:28:45.145 INFO:tasks.workunit.client.0.smithi103.stdout:8/796: stat d4/d26/d27/da3 0 2024-03-22T02:28:45.146 INFO:tasks.workunit.client.0.smithi103.stdout:5/695: dwrite d0/d14/d23/d46/f61 [0,4194304] 0 2024-03-22T02:28:45.148 INFO:tasks.workunit.client.0.smithi103.stdout:2/554: rename d11/d19/l64 to d11/d25/d63/d45/d47/d83/lb2 0 2024-03-22T02:28:45.148 INFO:tasks.workunit.client.0.smithi103.stdout:3/734: creat de/d1a/d25/ff8 x:0 0 0 2024-03-22T02:28:45.149 INFO:tasks.workunit.client.0.smithi103.stdout:6/494: symlink d0/d14/d57/d68/l9c 0 2024-03-22T02:28:45.153 INFO:tasks.workunit.client.0.smithi103.stdout:8/797: mkdir d4/d26/d27/d7e/d39/d49/d51/daf/d101 0 2024-03-22T02:28:45.154 INFO:tasks.workunit.client.0.smithi103.stdout:3/735: dread de/d12/d19/df2/f2f [0,4194304] 0 2024-03-22T02:28:45.155 INFO:tasks.workunit.client.0.smithi103.stdout:2/555: creat d11/d25/d46/d48/da6/fb3 x:0 0 0 2024-03-22T02:28:45.158 INFO:tasks.workunit.client.0.smithi103.stdout:2/556: write d11/d25/d46/fa2 [23527,74326] 0 2024-03-22T02:28:45.159 INFO:tasks.workunit.client.0.smithi103.stdout:8/798: dread d4/d26/d27/d7e/d39/d49/d79/fb9 [0,4194304] 0 2024-03-22T02:28:45.160 INFO:tasks.workunit.client.0.smithi103.stdout:2/557: read d11/d19/d4e/f7d [2970013,740] 0 2024-03-22T02:28:45.163 INFO:tasks.workunit.client.0.smithi103.stdout:2/558: rmdir d11 39 2024-03-22T02:28:45.165 INFO:tasks.workunit.client.0.smithi103.stdout:3/736: dwrite de/d12/d19/f51 [4194304,4194304] 0 2024-03-22T02:28:45.165 INFO:tasks.workunit.client.0.smithi103.stdout:9/645: dread d2/d10/f8a [4194304,4194304] 0 2024-03-22T02:28:45.167 INFO:tasks.workunit.client.0.smithi103.stdout:5/696: rename d0/d14/d23/d2a/d2c to d0/d14/d23/d5c/d77/df2 0 2024-03-22T02:28:45.170 INFO:tasks.workunit.client.0.smithi103.stdout:4/681: write d3/d31/f47 [862439,40275] 0 2024-03-22T02:28:45.171 INFO:tasks.workunit.client.0.smithi103.stdout:8/799: read d4/d26/d27/d7e/d39/d49/fb5 [452797,101392] 0 2024-03-22T02:28:45.171 INFO:tasks.workunit.client.0.smithi103.stdout:4/682: chown d3/d1e/d25/d65/l66 0 1 2024-03-22T02:28:45.173 INFO:tasks.workunit.client.0.smithi103.stdout:4/683: stat d3/d1e/d53 0 2024-03-22T02:28:45.174 INFO:tasks.workunit.client.0.smithi103.stdout:3/737: rename de/d1a/d25/ff8 to de/d12/d8f/ff9 0 2024-03-22T02:28:45.178 INFO:tasks.workunit.client.0.smithi103.stdout:2/559: dwrite d11/d25/d63/d45/f79 [8388608,4194304] 0 2024-03-22T02:28:45.179 INFO:tasks.workunit.client.0.smithi103.stdout:0/663: write dd/f12 [2706967,8413] 0 2024-03-22T02:28:45.182 INFO:tasks.workunit.client.0.smithi103.stdout:2/560: fdatasync d11/d25/d46/d48/f6e 0 2024-03-22T02:28:45.182 INFO:tasks.workunit.client.0.smithi103.stdout:7/686: write d1/da/f75 [1664174,47184] 0 2024-03-22T02:28:45.182 INFO:tasks.workunit.client.0.smithi103.stdout:5/697: rename d0/d14/d23/d5c/d77/df2/da9 to d0/d14/d23/d5c/df3 0 2024-03-22T02:28:45.184 INFO:tasks.workunit.client.0.smithi103.stdout:2/561: readlink d11/d19/l22 0 2024-03-22T02:28:45.185 INFO:tasks.workunit.client.0.smithi103.stdout:5/698: stat d0/d3e/fd5 0 2024-03-22T02:28:45.185 INFO:tasks.workunit.client.0.smithi103.stdout:4/684: dwrite d3/d1e/d25/d65/f96 [0,4194304] 0 2024-03-22T02:28:45.192 INFO:tasks.workunit.client.0.smithi103.stdout:9/646: rmdir d2/da/d18/d1e/d77/d8c 0 2024-03-22T02:28:45.193 INFO:tasks.workunit.client.0.smithi103.stdout:8/800: symlink d4/d26/d27/d7e/d39/d49/d5f/dd3/l102 0 2024-03-22T02:28:45.195 INFO:tasks.workunit.client.0.smithi103.stdout:9/647: dread - d2/d10/d59/da5/fb1 zero size 2024-03-22T02:28:45.196 INFO:tasks.workunit.client.0.smithi103.stdout:0/664: mkdir dd/d16/d4b/d96/d29/d48/d5c/d9b/ddd 0 2024-03-22T02:28:45.197 INFO:tasks.workunit.client.0.smithi103.stdout:4/685: dwrite d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:45.199 INFO:tasks.workunit.client.0.smithi103.stdout:0/665: write dd/d16/d4b/d96/f3c [2466535,58383] 0 2024-03-22T02:28:45.200 INFO:tasks.workunit.client.0.smithi103.stdout:2/562: unlink d11/d25/d63/d45/la8 0 2024-03-22T02:28:45.200 INFO:tasks.workunit.client.0.smithi103.stdout:3/738: dread de/d1a/d25/f42 [0,4194304] 0 2024-03-22T02:28:45.201 INFO:tasks.workunit.client.0.smithi103.stdout:4/686: truncate d3/da2/da6/fb2 1754376 0 2024-03-22T02:28:45.203 INFO:tasks.workunit.client.0.smithi103.stdout:7/687: fsync d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/fd1 0 2024-03-22T02:28:45.204 INFO:tasks.workunit.client.0.smithi103.stdout:4/687: fdatasync d3/d38/d5c/faa 0 2024-03-22T02:28:45.205 INFO:tasks.workunit.client.0.smithi103.stdout:0/666: write dd/d34/d53/d8c/fae [1025135,120393] 0 2024-03-22T02:28:45.206 INFO:tasks.workunit.client.1.smithi122.stdout:4/112: sync 2024-03-22T02:28:45.209 INFO:tasks.workunit.client.0.smithi103.stdout:9/648: creat d2/d10/d43/d7e/fc6 x:0 0 0 2024-03-22T02:28:45.210 INFO:tasks.workunit.client.1.smithi122.stdout:4/113: mknod d0/d25/c2c 0 2024-03-22T02:28:45.210 INFO:tasks.workunit.client.1.smithi122.stdout:4/114: chown d0/d25/f28 2 1 2024-03-22T02:28:45.211 INFO:tasks.workunit.client.0.smithi103.stdout:2/563: write d11/d25/d63/d45/d47/d83/fac [1043346,86688] 0 2024-03-22T02:28:45.211 INFO:tasks.workunit.client.0.smithi103.stdout:5/699: mkdir d0/d14/d23/d5c/d70/df4 0 2024-03-22T02:28:45.212 INFO:tasks.workunit.client.1.smithi122.stdout:4/115: fdatasync d0/d25/d7/f16 0 2024-03-22T02:28:45.212 INFO:tasks.workunit.client.0.smithi103.stdout:1/718: dwrite db/d4a/f9c [0,4194304] 0 2024-03-22T02:28:45.215 INFO:tasks.workunit.client.1.smithi122.stdout:4/116: unlink d0/d25/d7/de/f14 0 2024-03-22T02:28:45.215 INFO:tasks.workunit.client.0.smithi103.stdout:7/688: mkdir d1/d10/d11/dd9 0 2024-03-22T02:28:45.216 INFO:tasks.workunit.client.0.smithi103.stdout:4/688: mknod d3/d1e/d68/dbc/cee 0 2024-03-22T02:28:45.222 INFO:tasks.workunit.client.0.smithi103.stdout:1/719: creat db/d4a/d78/d92/ff6 x:0 0 0 2024-03-22T02:28:45.224 INFO:tasks.workunit.client.0.smithi103.stdout:5/700: creat d0/d14/d23/d2a/d33/db9/ff5 x:0 0 0 2024-03-22T02:28:45.225 INFO:tasks.workunit.client.0.smithi103.stdout:9/649: dwrite d2/da/f24 [0,4194304] 0 2024-03-22T02:28:45.229 INFO:tasks.workunit.client.0.smithi103.stdout:8/801: link d4/d2a/f94 d4/d2a/d38/dfe/f103 0 2024-03-22T02:28:45.230 INFO:tasks.workunit.client.0.smithi103.stdout:8/802: chown d4/d19/c22 1560644 1 2024-03-22T02:28:45.231 INFO:tasks.workunit.client.0.smithi103.stdout:4/689: mkdir d3/d1e/d53/d8a/def 0 2024-03-22T02:28:45.232 INFO:tasks.workunit.client.0.smithi103.stdout:2/564: mknod d11/d25/d63/d45/cb4 0 2024-03-22T02:28:45.233 INFO:tasks.workunit.client.1.smithi122.stdout:4/117: write d0/d25/fa [3409099,69378] 0 2024-03-22T02:28:45.237 INFO:tasks.workunit.client.0.smithi103.stdout:9/650: mknod d2/d10/d59/cc7 0 2024-03-22T02:28:45.239 INFO:tasks.workunit.client.0.smithi103.stdout:0/667: dread dd/d16/d4b/d96/d64/d91/f99 [0,4194304] 0 2024-03-22T02:28:45.240 INFO:tasks.workunit.client.0.smithi103.stdout:4/690: truncate d3/d1e/d53/d8a/f9a 692632 0 2024-03-22T02:28:45.242 INFO:tasks.workunit.client.0.smithi103.stdout:2/565: mknod d11/d19/d7e/cb5 0 2024-03-22T02:28:45.242 INFO:tasks.workunit.client.0.smithi103.stdout:1/720: dwrite db/d4a/d78/d92/da8/ddd/d18/f20 [0,4194304] 0 2024-03-22T02:28:45.243 INFO:tasks.workunit.client.0.smithi103.stdout:4/691: readlink d3/d31/l5b 0 2024-03-22T02:28:45.243 INFO:tasks.workunit.client.0.smithi103.stdout:3/739: getdents de/d12/d5d/da9/db9 0 2024-03-22T02:28:45.245 INFO:tasks.workunit.client.0.smithi103.stdout:4/692: fdatasync d3/f24 0 2024-03-22T02:28:45.245 INFO:tasks.workunit.client.0.smithi103.stdout:5/701: dwrite d0/da2/dca/fba [0,4194304] 0 2024-03-22T02:28:45.250 INFO:tasks.workunit.client.0.smithi103.stdout:5/702: stat d0/da2/fb2 0 2024-03-22T02:28:45.250 INFO:tasks.workunit.client.0.smithi103.stdout:3/740: write de/d1a/d25/d3f/d9c/da7/fdf [709190,118220] 0 2024-03-22T02:28:45.250 INFO:tasks.workunit.client.0.smithi103.stdout:4/693: chown d3/da2/da6 1130555 1 2024-03-22T02:28:45.251 INFO:tasks.workunit.client.0.smithi103.stdout:4/694: write d3/d38/f70 [1746326,71288] 0 2024-03-22T02:28:45.257 INFO:tasks.workunit.client.0.smithi103.stdout:9/651: link d2/d10/d59/f89 d2/d10/d43/d7e/fc8 0 2024-03-22T02:28:45.258 INFO:tasks.workunit.client.0.smithi103.stdout:2/566: mkdir d11/d25/d46/db6 0 2024-03-22T02:28:45.259 INFO:tasks.workunit.client.0.smithi103.stdout:4/695: dwrite d3/d69/fb0 [0,4194304] 0 2024-03-22T02:28:45.260 INFO:tasks.workunit.client.0.smithi103.stdout:4/696: stat d3 0 2024-03-22T02:28:45.261 INFO:tasks.workunit.client.0.smithi103.stdout:9/652: fsync d2/d10/d43/d9b/fab 0 2024-03-22T02:28:45.264 INFO:tasks.workunit.client.0.smithi103.stdout:8/803: getdents d4/d26/d27/d7e/d39/d49/d51 0 2024-03-22T02:28:45.265 INFO:tasks.workunit.client.0.smithi103.stdout:7/689: dwrite d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:45.265 INFO:tasks.workunit.client.0.smithi103.stdout:4/697: read d3/d1e/fc3 [1058851,107204] 0 2024-03-22T02:28:45.267 INFO:tasks.workunit.client.0.smithi103.stdout:4/698: write d3/d1e/d53/d8a/f9c [855058,87117] 0 2024-03-22T02:28:45.268 INFO:tasks.workunit.client.0.smithi103.stdout:7/690: chown d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4 508542 1 2024-03-22T02:28:45.270 INFO:tasks.workunit.client.0.smithi103.stdout:7/691: fsync d1/d10/d11/d19/d29/d61/f7b 0 2024-03-22T02:28:45.271 INFO:tasks.workunit.client.0.smithi103.stdout:2/567: mknod d11/d19/d7e/cb7 0 2024-03-22T02:28:45.271 INFO:tasks.workunit.client.0.smithi103.stdout:4/699: write d3/fa0 [501685,119404] 0 2024-03-22T02:28:45.272 INFO:tasks.workunit.client.0.smithi103.stdout:7/692: write d1/d10/d11/d19/d29/d61/fc8 [811947,71635] 0 2024-03-22T02:28:45.275 INFO:tasks.workunit.client.0.smithi103.stdout:1/721: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/ff7 x:0 0 0 2024-03-22T02:28:45.277 INFO:tasks.workunit.client.0.smithi103.stdout:5/703: write d0/f12 [1172343,75114] 0 2024-03-22T02:28:45.278 INFO:tasks.workunit.client.0.smithi103.stdout:0/668: link dd/d16/c4a dd/d16/cde 0 2024-03-22T02:28:45.280 INFO:tasks.workunit.client.0.smithi103.stdout:0/669: chown dd/d16/d4b/d96/d64/dab/dc6/f6f 7 1 2024-03-22T02:28:45.281 INFO:tasks.workunit.client.0.smithi103.stdout:9/653: rename d2/da/d18/d1e/d34/d3d/d61/c79 to d2/d10/d43/d4b/cc9 0 2024-03-22T02:28:45.282 INFO:tasks.workunit.client.0.smithi103.stdout:4/700: mknod d3/d1e/d53/cf0 0 2024-03-22T02:28:45.283 INFO:tasks.workunit.client.0.smithi103.stdout:9/654: write d2/d5/d4c/f85 [358126,25304] 0 2024-03-22T02:28:45.283 INFO:tasks.workunit.client.0.smithi103.stdout:2/568: mknod d11/d25/d46/d48/da6/cb8 0 2024-03-22T02:28:45.283 INFO:tasks.workunit.client.0.smithi103.stdout:4/701: write d3/d38/f85 [1139189,86671] 0 2024-03-22T02:28:45.285 INFO:tasks.workunit.client.0.smithi103.stdout:2/569: write d11/d19/da7/fb1 [1011725,36870] 0 2024-03-22T02:28:45.287 INFO:tasks.workunit.client.0.smithi103.stdout:8/804: mkdir d4/d19/d1a/d104 0 2024-03-22T02:28:45.288 INFO:tasks.workunit.client.0.smithi103.stdout:2/570: chown d11/d25/d63/d45/d47/d83/d78/db0/c6c 1730837 1 2024-03-22T02:28:45.289 INFO:tasks.workunit.client.0.smithi103.stdout:5/704: rename d0/d14/d23/d5c/d77/df2/fbe to d0/d14/d23/d2a/d33/ff6 0 2024-03-22T02:28:45.290 INFO:tasks.workunit.client.0.smithi103.stdout:5/705: write d0/d14/d23/fbc [1594518,112481] 0 2024-03-22T02:28:45.290 INFO:tasks.workunit.client.0.smithi103.stdout:7/693: mknod d1/d10/d11/d19/d29/d61/d1d/da3/db9/cda 0 2024-03-22T02:28:45.292 INFO:tasks.workunit.client.0.smithi103.stdout:9/655: unlink d2/d5/f38 0 2024-03-22T02:28:45.293 INFO:tasks.workunit.client.0.smithi103.stdout:3/741: getdents de/d1a 0 2024-03-22T02:28:45.295 INFO:tasks.workunit.client.0.smithi103.stdout:3/742: truncate de/d1a/d25/d3f/ff3 495260 0 2024-03-22T02:28:45.296 INFO:tasks.workunit.client.0.smithi103.stdout:0/670: mknod dd/d16/d4b/cdf 0 2024-03-22T02:28:45.297 INFO:tasks.workunit.client.0.smithi103.stdout:8/805: readlink d4/d26/d27/d7e/d39/lb2 0 2024-03-22T02:28:45.298 INFO:tasks.workunit.client.0.smithi103.stdout:1/722: write db/d4a/d78/d92/da8/deb/fec [1286606,78177] 0 2024-03-22T02:28:45.301 INFO:tasks.workunit.client.0.smithi103.stdout:1/723: dread - db/d4a/d78/d92/da8/ddd/d18/d84/fc0 zero size 2024-03-22T02:28:45.303 INFO:tasks.workunit.client.0.smithi103.stdout:0/671: dwrite dd/d16/d4b/f50 [0,4194304] 0 2024-03-22T02:28:45.304 INFO:tasks.workunit.client.1.smithi122.stdout:6/182: sync 2024-03-22T02:28:45.304 INFO:tasks.workunit.client.1.smithi122.stdout:8/82: sync 2024-03-22T02:28:45.304 INFO:tasks.workunit.client.0.smithi103.stdout:1/724: write db/d4a/d78/d92/da8/ddd/d18/fad [309984,27929] 0 2024-03-22T02:28:45.305 INFO:tasks.workunit.client.1.smithi122.stdout:6/183: dread - d7/de/f3e zero size 2024-03-22T02:28:45.305 INFO:tasks.workunit.client.1.smithi122.stdout:8/83: chown d1/d9/f13 2 1 2024-03-22T02:28:45.306 INFO:tasks.workunit.client.0.smithi103.stdout:7/694: rmdir d1/d10/d11/d19/d29/d61 39 2024-03-22T02:28:45.308 INFO:tasks.workunit.client.1.smithi122.stdout:8/84: mkdir d1/d9/d1b 0 2024-03-22T02:28:45.308 INFO:tasks.workunit.client.0.smithi103.stdout:9/656: symlink d2/d10/d43/d7e/da4/lca 0 2024-03-22T02:28:45.309 INFO:tasks.workunit.client.0.smithi103.stdout:3/743: symlink de/d6b/d80/d73/lfa 0 2024-03-22T02:28:45.310 INFO:tasks.workunit.client.0.smithi103.stdout:1/725: dread db/d4a/d78/d92/da8/ddd/d29/d41/f99 [0,4194304] 0 2024-03-22T02:28:45.311 INFO:tasks.workunit.client.0.smithi103.stdout:2/571: creat d11/d25/d63/d45/d47/d83/da3/fb9 x:0 0 0 2024-03-22T02:28:45.312 INFO:tasks.workunit.client.0.smithi103.stdout:9/657: dread d2/d5/fb [0,4194304] 0 2024-03-22T02:28:45.313 INFO:tasks.workunit.client.1.smithi122.stdout:8/85: unlink d1/d8/f18 0 2024-03-22T02:28:45.314 INFO:tasks.workunit.client.0.smithi103.stdout:0/672: creat dd/d16/d4b/d96/d64/d91/d9a/d9d/fe0 x:0 0 0 2024-03-22T02:28:45.316 INFO:tasks.workunit.client.1.smithi122.stdout:6/184: dwrite d7/de/dd/d20/f3f [0,4194304] 0 2024-03-22T02:28:45.317 INFO:tasks.workunit.client.0.smithi103.stdout:9/658: fsync d2/da/d18/d1e/d34/d3d/d46/fb3 0 2024-03-22T02:28:45.317 INFO:tasks.workunit.client.0.smithi103.stdout:3/744: mkdir de/d12/d8f/dfb 0 2024-03-22T02:28:45.319 INFO:tasks.workunit.client.1.smithi122.stdout:8/86: rename d1/d9/ce to d1/c1c 0 2024-03-22T02:28:45.319 INFO:tasks.workunit.client.0.smithi103.stdout:2/572: dread d11/d19/d5d/f91 [0,4194304] 0 2024-03-22T02:28:45.319 INFO:tasks.workunit.client.0.smithi103.stdout:8/806: mkdir d4/d26/d27/da3/df5/d105 0 2024-03-22T02:28:45.319 INFO:tasks.workunit.client.0.smithi103.stdout:0/673: chown dd/d16/d4b/d96/d64/dab/dc6/l41 0 1 2024-03-22T02:28:45.320 INFO:tasks.workunit.client.1.smithi122.stdout:6/185: rmdir d7/de/dd/d20/d2d 39 2024-03-22T02:28:45.320 INFO:tasks.workunit.client.0.smithi103.stdout:1/726: symlink db/d4a/d78/d92/da8/ddd/d6b/db5/lf8 0 2024-03-22T02:28:45.321 INFO:tasks.workunit.client.0.smithi103.stdout:3/745: chown de/d12/d5d/cc3 3858857 1 2024-03-22T02:28:45.321 INFO:tasks.workunit.client.0.smithi103.stdout:5/706: dwrite d0/d14/d23/d36/d95/fa5 [0,4194304] 0 2024-03-22T02:28:45.323 INFO:tasks.workunit.client.0.smithi103.stdout:2/573: fdatasync d11/d25/d46/f37 0 2024-03-22T02:28:45.325 INFO:tasks.workunit.client.0.smithi103.stdout:4/702: getdents d3/d1e 0 2024-03-22T02:28:45.325 INFO:tasks.workunit.client.1.smithi122.stdout:6/186: mknod d7/de/dd/d20/d2d/c40 0 2024-03-22T02:28:45.326 INFO:tasks.workunit.client.0.smithi103.stdout:3/746: fsync de/f22 0 2024-03-22T02:28:45.328 INFO:tasks.workunit.client.0.smithi103.stdout:7/695: symlink d1/d10/d11/d19/d29/d61/d1d/ldb 0 2024-03-22T02:28:45.328 INFO:tasks.workunit.client.1.smithi122.stdout:6/187: rename d7/de/c14 to d7/c41 0 2024-03-22T02:28:45.331 INFO:tasks.workunit.client.1.smithi122.stdout:8/87: getdents d1/d9 0 2024-03-22T02:28:45.331 INFO:tasks.workunit.client.0.smithi103.stdout:8/807: write d4/d2a/d4f/fe3 [800927,66068] 0 2024-03-22T02:28:45.332 INFO:tasks.workunit.client.0.smithi103.stdout:3/747: fsync de/d12/d30/d70/de5/fe9 0 2024-03-22T02:28:45.333 INFO:tasks.workunit.client.1.smithi122.stdout:8/88: rename d1 to d1/d1d 22 2024-03-22T02:28:45.334 INFO:tasks.workunit.client.1.smithi122.stdout:8/89: truncate d1/f1a 1534188 0 2024-03-22T02:28:45.334 INFO:tasks.workunit.client.1.smithi122.stdout:6/188: creat d7/de/dd/f42 x:0 0 0 2024-03-22T02:28:45.334 INFO:tasks.workunit.client.0.smithi103.stdout:5/707: mkdir d0/d14/d23/d2a/df7 0 2024-03-22T02:28:45.335 INFO:tasks.workunit.client.0.smithi103.stdout:9/659: dwrite d2/d10/d43/d7e/d96/d9d/f98 [0,4194304] 0 2024-03-22T02:28:45.335 INFO:tasks.workunit.client.1.smithi122.stdout:8/90: rename d1 to d1/d8/d1e 22 2024-03-22T02:28:45.336 INFO:tasks.workunit.client.1.smithi122.stdout:8/91: truncate d1/d9/f13 403294 0 2024-03-22T02:28:45.337 INFO:tasks.workunit.client.0.smithi103.stdout:9/660: chown d2/d10/d59 40152 1 2024-03-22T02:28:45.337 INFO:tasks.workunit.client.0.smithi103.stdout:5/708: stat d0/d14/d23/d2a/d33/ff6 0 2024-03-22T02:28:45.339 INFO:tasks.workunit.client.1.smithi122.stdout:8/92: rename d1/f10 to d1/d9/f1f 0 2024-03-22T02:28:45.341 INFO:tasks.workunit.client.0.smithi103.stdout:0/674: creat dd/d16/d4b/d96/fe1 x:0 0 0 2024-03-22T02:28:45.342 INFO:tasks.workunit.client.0.smithi103.stdout:2/574: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:45.343 INFO:tasks.workunit.client.0.smithi103.stdout:5/709: fsync d0/d14/d23/d36/d95/d96/daa/daf/fab 0 2024-03-22T02:28:45.344 INFO:tasks.workunit.client.1.smithi122.stdout:8/93: rename d1/l12 to d1/l20 0 2024-03-22T02:28:45.345 INFO:tasks.workunit.client.1.smithi122.stdout:8/94: chown d1/c3 1646 1 2024-03-22T02:28:45.347 INFO:tasks.workunit.client.0.smithi103.stdout:4/703: truncate d3/d38/f4c 314166 0 2024-03-22T02:28:45.347 INFO:tasks.workunit.client.1.smithi122.stdout:8/95: mknod d1/d8/c21 0 2024-03-22T02:28:45.349 INFO:tasks.workunit.client.0.smithi103.stdout:1/727: write db/d4a/d78/d92/da8/ddd/d29/f6e [4769726,40704] 0 2024-03-22T02:28:45.349 INFO:tasks.workunit.client.0.smithi103.stdout:3/748: rename de/d6b/l8e to de/d12/d8a/lfc 0 2024-03-22T02:28:45.352 INFO:tasks.workunit.client.0.smithi103.stdout:0/675: truncate dd/d16/f2f 598290 0 2024-03-22T02:28:45.353 INFO:tasks.workunit.client.0.smithi103.stdout:7/696: write d1/d10/d11/f9f [3629659,17032] 0 2024-03-22T02:28:45.354 INFO:tasks.workunit.client.0.smithi103.stdout:5/710: dwrite d0/f9a [0,4194304] 0 2024-03-22T02:28:45.356 INFO:tasks.workunit.client.0.smithi103.stdout:6/495: sync 2024-03-22T02:28:45.357 INFO:tasks.workunit.client.0.smithi103.stdout:4/704: rmdir d3/d38/d5c/d90 39 2024-03-22T02:28:45.357 INFO:tasks.workunit.client.1.smithi122.stdout:8/96: getdents d1/d9 0 2024-03-22T02:28:45.358 INFO:tasks.workunit.client.0.smithi103.stdout:8/808: link d4/d19/d1a/d4a/ff8 d4/d2a/d3e/da4/f106 0 2024-03-22T02:28:45.361 INFO:tasks.workunit.client.1.smithi122.stdout:8/97: rename d1/d9/d1b to d1/d22 0 2024-03-22T02:28:45.361 INFO:tasks.workunit.client.0.smithi103.stdout:2/575: dwrite d11/d25/d63/d45/d47/d83/d78/db0/f36 [4194304,4194304] 0 2024-03-22T02:28:45.366 INFO:tasks.workunit.client.0.smithi103.stdout:0/676: write dd/d16/d4b/d96/d64/dab/dc6/fd7 [933388,93672] 0 2024-03-22T02:28:45.368 INFO:tasks.workunit.client.0.smithi103.stdout:5/711: dwrite d0/f12 [0,4194304] 0 2024-03-22T02:28:45.370 INFO:tasks.workunit.client.0.smithi103.stdout:7/697: unlink d1/d10/d11/d19/d29/d33/d4b/f97 0 2024-03-22T02:28:45.374 INFO:tasks.workunit.client.0.smithi103.stdout:1/728: creat db/d4a/d78/ff9 x:0 0 0 2024-03-22T02:28:45.378 INFO:tasks.workunit.client.0.smithi103.stdout:2/576: creat d11/d19/d5d/fba x:0 0 0 2024-03-22T02:28:45.380 INFO:tasks.workunit.client.0.smithi103.stdout:8/809: symlink d4/d26/d27/d7e/d39/d7b/dc7/l107 0 2024-03-22T02:28:45.380 INFO:tasks.workunit.client.0.smithi103.stdout:8/810: chown d4/d26/d27/d7e/d39 3 1 2024-03-22T02:28:45.382 INFO:tasks.workunit.client.0.smithi103.stdout:8/811: write d4/d26/d27/d7e/d39/d7b/dc7/ff3 [229579,93289] 0 2024-03-22T02:28:45.384 INFO:tasks.workunit.client.0.smithi103.stdout:9/661: dread d2/d5/f63 [0,4194304] 0 2024-03-22T02:28:45.385 INFO:tasks.workunit.client.0.smithi103.stdout:5/712: rename d0/d14/d23/d46/de7/fe8 to d0/da2/dca/ff8 0 2024-03-22T02:28:45.387 INFO:tasks.workunit.client.0.smithi103.stdout:0/677: fsync dd/d16/d4b/d96/d29/d48/d5c/fa8 0 2024-03-22T02:28:45.388 INFO:tasks.workunit.client.0.smithi103.stdout:0/678: readlink dd/l92 0 2024-03-22T02:28:45.391 INFO:tasks.workunit.client.0.smithi103.stdout:3/749: dread de/f1e [0,4194304] 0 2024-03-22T02:28:45.391 INFO:tasks.workunit.client.0.smithi103.stdout:8/812: symlink d4/d2a/d8b/l108 0 2024-03-22T02:28:45.393 INFO:tasks.workunit.client.1.smithi122.stdout:8/98: dwrite d1/d9/f11 [0,4194304] 0 2024-03-22T02:28:45.393 INFO:tasks.workunit.client.0.smithi103.stdout:7/698: symlink d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/ldc 0 2024-03-22T02:28:45.396 INFO:tasks.workunit.client.0.smithi103.stdout:8/813: dread d4/d26/d27/d7e/d39/d49/d51/fea [0,4194304] 0 2024-03-22T02:28:45.398 INFO:tasks.workunit.client.0.smithi103.stdout:4/705: dwrite d3/d38/f4c [0,4194304] 0 2024-03-22T02:28:45.399 INFO:tasks.workunit.client.1.smithi122.stdout:8/99: getdents d1/d9 0 2024-03-22T02:28:45.399 INFO:tasks.workunit.client.0.smithi103.stdout:2/577: symlink d11/d25/d46/db6/lbb 0 2024-03-22T02:28:45.401 INFO:tasks.workunit.client.1.smithi122.stdout:8/100: getdents d1/d22 0 2024-03-22T02:28:45.401 INFO:tasks.workunit.client.1.smithi122.stdout:8/101: fdatasync d1/f1a 0 2024-03-22T02:28:45.401 INFO:tasks.workunit.client.0.smithi103.stdout:0/679: mknod dd/d16/d4b/d96/d64/d91/d9a/ce2 0 2024-03-22T02:28:45.405 INFO:tasks.workunit.client.0.smithi103.stdout:3/750: readlink de/d12/l4e 0 2024-03-22T02:28:45.406 INFO:tasks.workunit.client.1.smithi122.stdout:8/102: link d1/c1c d1/d22/c23 0 2024-03-22T02:28:45.407 INFO:tasks.workunit.client.1.smithi122.stdout:8/103: creat d1/d9/f24 x:0 0 0 2024-03-22T02:28:45.408 INFO:tasks.workunit.client.0.smithi103.stdout:6/496: dread d0/f1e [0,4194304] 0 2024-03-22T02:28:45.408 INFO:tasks.workunit.client.0.smithi103.stdout:9/662: mkdir d2/da/d18/d1e/d77/d8b/dae/dcb 0 2024-03-22T02:28:45.410 INFO:tasks.workunit.client.1.smithi122.stdout:8/104: unlink d1/d9/c15 0 2024-03-22T02:28:45.412 INFO:tasks.workunit.client.0.smithi103.stdout:8/814: creat d4/d26/d27/d7e/d39/d49/d51/f109 x:0 0 0 2024-03-22T02:28:45.416 INFO:tasks.workunit.client.0.smithi103.stdout:7/699: rename d1/d10/d11/d19/d29/d61/d69/f9a to d1/d10/d11/d19/d29/d61/d69/d6b/d7a/fdd 0 2024-03-22T02:28:45.416 INFO:tasks.workunit.client.1.smithi122.stdout:8/105: dwrite d1/d9/f24 [0,4194304] 0 2024-03-22T02:28:45.417 INFO:tasks.workunit.client.0.smithi103.stdout:1/729: truncate db/d4a/d78/d92/da8/ddd/f83 106109 0 2024-03-22T02:28:45.418 INFO:tasks.workunit.client.0.smithi103.stdout:0/680: symlink dd/d34/d53/d8c/le3 0 2024-03-22T02:28:45.419 INFO:tasks.workunit.client.0.smithi103.stdout:3/751: mknod de/d12/d8a/db0/cfd 0 2024-03-22T02:28:45.420 INFO:tasks.workunit.client.0.smithi103.stdout:5/713: rmdir d0/d14/d23/d5c/d70/df4 0 2024-03-22T02:28:45.421 INFO:tasks.workunit.client.0.smithi103.stdout:3/752: readlink de/d1a/d25/d3f/d9c/ld0 0 2024-03-22T02:28:45.422 INFO:tasks.workunit.client.0.smithi103.stdout:9/663: creat d2/da/d18/d1e/fcc x:0 0 0 2024-03-22T02:28:45.423 INFO:tasks.workunit.client.0.smithi103.stdout:2/578: mknod d11/d25/d46/cbc 0 2024-03-22T02:28:45.424 INFO:tasks.workunit.client.0.smithi103.stdout:2/579: dread - d11/d25/fad zero size 2024-03-22T02:28:45.425 INFO:tasks.workunit.client.1.smithi122.stdout:8/106: fdatasync d1/d9/f11 0 2024-03-22T02:28:45.426 INFO:tasks.workunit.client.0.smithi103.stdout:8/815: creat d4/d26/d27/d7e/d39/d7b/f10a x:0 0 0 2024-03-22T02:28:45.427 INFO:tasks.workunit.client.0.smithi103.stdout:4/706: truncate d3/d1e/d53/d8a/f9a 527098 0 2024-03-22T02:28:45.428 INFO:tasks.workunit.client.1.smithi122.stdout:8/107: unlink d1/d9/fb 0 2024-03-22T02:28:45.430 INFO:tasks.workunit.client.1.smithi122.stdout:8/108: creat d1/d22/f25 x:0 0 0 2024-03-22T02:28:45.431 INFO:tasks.workunit.client.0.smithi103.stdout:6/497: dread - d0/d14/d57/d68/f55 zero size 2024-03-22T02:28:45.432 INFO:tasks.workunit.client.0.smithi103.stdout:3/753: creat de/d6b/d80/ffe x:0 0 0 2024-03-22T02:28:45.433 INFO:tasks.workunit.client.0.smithi103.stdout:5/714: mknod d0/d14/d23/d2a/d7b/cf9 0 2024-03-22T02:28:45.434 INFO:tasks.workunit.client.1.smithi122.stdout:8/109: link d1/d9/f1f d1/d9/f26 0 2024-03-22T02:28:45.435 INFO:tasks.workunit.client.1.smithi122.stdout:8/110: unlink d1/d22/f25 0 2024-03-22T02:28:45.436 INFO:tasks.workunit.client.0.smithi103.stdout:9/664: fdatasync d2/d10/d59/f89 0 2024-03-22T02:28:45.438 INFO:tasks.workunit.client.1.smithi122.stdout:8/111: symlink d1/l27 0 2024-03-22T02:28:45.438 INFO:tasks.workunit.client.0.smithi103.stdout:6/498: dwrite d0/d3/d22/d58/f79 [0,4194304] 0 2024-03-22T02:28:45.439 INFO:tasks.workunit.client.1.smithi122.stdout:8/112: truncate d1/d9/f13 502705 0 2024-03-22T02:28:45.440 INFO:tasks.workunit.client.1.smithi122.stdout:8/113: creat d1/d9/f28 x:0 0 0 2024-03-22T02:28:45.443 INFO:tasks.workunit.client.0.smithi103.stdout:7/700: getdents d1/d10/d11/d19/d29/d33/d4b/db3 0 2024-03-22T02:28:45.443 INFO:tasks.workunit.client.0.smithi103.stdout:9/665: fsync d2/d10/d43/d7e/d96/d9d/f7c 0 2024-03-22T02:28:45.443 INFO:tasks.workunit.client.1.smithi122.stdout:8/114: write d1/d9/f26 [3181068,3719] 0 2024-03-22T02:28:45.444 INFO:tasks.workunit.client.0.smithi103.stdout:0/681: truncate dd/d16/f36 2720620 0 2024-03-22T02:28:45.446 INFO:tasks.workunit.client.1.smithi122.stdout:8/115: rmdir d1 39 2024-03-22T02:28:45.446 INFO:tasks.workunit.client.0.smithi103.stdout:0/682: stat dd/d16/d4b/d96/d64/d6a/d8d/lc9 0 2024-03-22T02:28:45.450 INFO:tasks.workunit.client.1.smithi122.stdout:8/116: symlink d1/d8/l29 0 2024-03-22T02:28:45.451 INFO:tasks.workunit.client.1.smithi122.stdout:8/117: write d1/d9/f13 [305666,93996] 0 2024-03-22T02:28:45.452 INFO:tasks.workunit.client.0.smithi103.stdout:1/730: write db/d4a/d78/d92/da8/ddd/d29/d6c/f71 [385951,4165] 0 2024-03-22T02:28:45.454 INFO:tasks.workunit.client.0.smithi103.stdout:8/816: creat d4/d26/d27/d7e/dc8/f10b x:0 0 0 2024-03-22T02:28:45.454 INFO:tasks.workunit.client.1.smithi122.stdout:8/118: write d1/d9/f1f [4485716,67786] 0 2024-03-22T02:28:45.454 INFO:tasks.workunit.client.0.smithi103.stdout:2/580: dread d11/d25/d46/d48/f6e [0,4194304] 0 2024-03-22T02:28:45.459 INFO:tasks.workunit.client.0.smithi103.stdout:1/731: fsync db/d4a/d78/d92/da8/ddd/d18/d84/fc4 0 2024-03-22T02:28:45.459 INFO:tasks.workunit.client.0.smithi103.stdout:8/817: chown d4/d2a/d3e/f6b 53845 1 2024-03-22T02:28:45.459 INFO:tasks.workunit.client.0.smithi103.stdout:9/666: rename d2/d10/d27 to d2/d10/d43/d7e/da4/dcd 0 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:8/119: chown d1/d9/f24 69 1 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:3/59: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:9/89: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:5/101: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:0/114: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:1/76: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:2/89: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:7/93: sync 2024-03-22T02:28:45.460 INFO:tasks.workunit.client.1.smithi122.stdout:2/90: fdatasync df/f13 0 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.1.smithi122.stdout:5/102: dread - d8/f17 zero size 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.1.smithi122.stdout:0/115: write fd [2632962,122836] 0 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.1.smithi122.stdout:9/90: fsync fa 0 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.1.smithi122.stdout:8/120: symlink d1/l2a 0 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.1.smithi122.stdout:7/94: fdatasync d0/d4/fb 0 2024-03-22T02:28:45.461 INFO:tasks.workunit.client.0.smithi103.stdout:9/667: write d2/d5/d4c/f85 [1297019,72102] 0 2024-03-22T02:28:45.462 INFO:tasks.workunit.client.0.smithi103.stdout:4/707: rename d3/d1e/d25/d65/c9d to d3/d1e/d25/d65/cf1 0 2024-03-22T02:28:45.463 INFO:tasks.workunit.client.1.smithi122.stdout:5/103: write d8/d1a/f1c [1801165,19552] 0 2024-03-22T02:28:45.463 INFO:tasks.workunit.client.1.smithi122.stdout:2/91: getdents df/d1f/d23 0 2024-03-22T02:28:45.463 INFO:tasks.workunit.client.1.smithi122.stdout:9/91: chown db/df/c14 6 1 2024-03-22T02:28:45.464 INFO:tasks.workunit.client.1.smithi122.stdout:3/60: write d2/f8 [207534,73631] 0 2024-03-22T02:28:45.467 INFO:tasks.workunit.client.1.smithi122.stdout:2/92: rename df/d1f to df/d1f/d23/d24 22 2024-03-22T02:28:45.468 INFO:tasks.workunit.client.0.smithi103.stdout:0/683: unlink dd/d16/d4b/d96/d29/d48/c61 0 2024-03-22T02:28:45.469 INFO:tasks.workunit.client.0.smithi103.stdout:5/715: mknod d0/d14/cfa 0 2024-03-22T02:28:45.471 INFO:tasks.workunit.client.1.smithi122.stdout:3/61: write d2/f8 [2539459,78772] 0 2024-03-22T02:28:45.473 INFO:tasks.workunit.client.0.smithi103.stdout:7/701: creat d1/d10/d11/d19/d29/d33/dcd/fde x:0 0 0 2024-03-22T02:28:45.474 INFO:tasks.workunit.client.0.smithi103.stdout:5/716: read - d0/d14/d23/d5c/fb8 zero size 2024-03-22T02:28:45.474 INFO:tasks.workunit.client.0.smithi103.stdout:9/668: dwrite d2/d5/d7b/d7d/fa6 [0,4194304] 0 2024-03-22T02:28:45.476 INFO:tasks.workunit.client.1.smithi122.stdout:1/77: dread dc/f11 [0,4194304] 0 2024-03-22T02:28:45.478 INFO:tasks.workunit.client.1.smithi122.stdout:4/118: sync 2024-03-22T02:28:45.478 INFO:tasks.workunit.client.1.smithi122.stdout:6/189: sync 2024-03-22T02:28:45.479 INFO:tasks.workunit.client.1.smithi122.stdout:8/121: truncate d1/d9/f26 1830157 0 2024-03-22T02:28:45.479 INFO:tasks.workunit.client.0.smithi103.stdout:4/708: dwrite d3/d1e/d53/d8a/fe9 [0,4194304] 0 2024-03-22T02:28:45.480 INFO:tasks.workunit.client.1.smithi122.stdout:0/116: dwrite d11/d14/f26 [0,4194304] 0 2024-03-22T02:28:45.480 INFO:tasks.workunit.client.0.smithi103.stdout:9/669: write d2/d10/d43/d7e/d96/d9d/f7c [72433,1858] 0 2024-03-22T02:28:45.483 INFO:tasks.workunit.client.0.smithi103.stdout:2/581: rename d11/d25/d63/d45/d47/d83/d78/db0/c6c to d11/d19/d4e/cbd 0 2024-03-22T02:28:45.484 INFO:tasks.workunit.client.1.smithi122.stdout:5/104: creat d8/f20 x:0 0 0 2024-03-22T02:28:45.484 INFO:tasks.workunit.client.1.smithi122.stdout:9/92: dread db/f13 [0,4194304] 0 2024-03-22T02:28:45.486 INFO:tasks.workunit.client.1.smithi122.stdout:1/78: truncate dc/fd 5039235 0 2024-03-22T02:28:45.489 INFO:tasks.workunit.client.0.smithi103.stdout:6/499: dwrite d0/d14/d57/d68/f2d [0,4194304] 0 2024-03-22T02:28:45.489 INFO:tasks.workunit.client.1.smithi122.stdout:2/93: mkdir df/d1f/d23/d25 0 2024-03-22T02:28:45.491 INFO:tasks.workunit.client.1.smithi122.stdout:7/95: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:45.491 INFO:tasks.workunit.client.1.smithi122.stdout:8/122: write d1/d9/f24 [3263132,22177] 0 2024-03-22T02:28:45.493 INFO:tasks.workunit.client.0.smithi103.stdout:4/709: dwrite d3/d31/f4e [0,4194304] 0 2024-03-22T02:28:45.495 INFO:tasks.workunit.client.0.smithi103.stdout:0/684: creat dd/d16/d4b/d96/d35/fe4 x:0 0 0 2024-03-22T02:28:45.495 INFO:tasks.workunit.client.0.smithi103.stdout:4/710: stat d3/d1e/d53/d8a 0 2024-03-22T02:28:45.495 INFO:tasks.workunit.client.1.smithi122.stdout:0/117: read d11/d14/f15 [621028,75790] 0 2024-03-22T02:28:45.495 INFO:tasks.workunit.client.1.smithi122.stdout:3/62: rename d2/d9/la to d2/d9/dc/df/l11 0 2024-03-22T02:28:45.496 INFO:tasks.workunit.client.1.smithi122.stdout:6/190: creat d7/de/f43 x:0 0 0 2024-03-22T02:28:45.496 INFO:tasks.workunit.client.1.smithi122.stdout:2/94: write df/f14 [853323,98199] 0 2024-03-22T02:28:45.497 INFO:tasks.workunit.client.0.smithi103.stdout:0/685: write dd/d34/d53/d8c/fbc [994325,31213] 0 2024-03-22T02:28:45.498 INFO:tasks.workunit.client.1.smithi122.stdout:3/63: read d2/f7 [552120,114553] 0 2024-03-22T02:28:45.500 INFO:tasks.workunit.client.1.smithi122.stdout:1/79: fsync dc/fe 0 2024-03-22T02:28:45.500 INFO:tasks.workunit.client.1.smithi122.stdout:0/118: readlink d11/d14/d1d/d27/l21 0 2024-03-22T02:28:45.501 INFO:tasks.workunit.client.1.smithi122.stdout:8/123: rmdir d1 39 2024-03-22T02:28:45.501 INFO:tasks.workunit.client.1.smithi122.stdout:1/80: chown l6 207 1 2024-03-22T02:28:45.503 INFO:tasks.workunit.client.0.smithi103.stdout:7/702: symlink d1/d10/d11/d19/d29/d61/d1d/da3/ldf 0 2024-03-22T02:28:45.504 INFO:tasks.workunit.client.1.smithi122.stdout:1/81: write dc/f11 [2969930,61369] 0 2024-03-22T02:28:45.505 INFO:tasks.workunit.client.1.smithi122.stdout:6/191: rename d7/de/c29 to d7/de/dd/d39/c44 0 2024-03-22T02:28:45.507 INFO:tasks.workunit.client.1.smithi122.stdout:9/93: symlink db/df/l1d 0 2024-03-22T02:28:45.507 INFO:tasks.workunit.client.1.smithi122.stdout:1/82: write dc/ff [3694409,62239] 0 2024-03-22T02:28:45.509 INFO:tasks.workunit.client.1.smithi122.stdout:9/94: write db/df/f19 [962857,19141] 0 2024-03-22T02:28:45.510 INFO:tasks.workunit.client.0.smithi103.stdout:2/582: mknod d11/d25/d63/cbe 0 2024-03-22T02:28:45.510 INFO:tasks.workunit.client.0.smithi103.stdout:4/711: dwrite d3/d1e/d53/d7a/f7d [0,4194304] 0 2024-03-22T02:28:45.511 INFO:tasks.workunit.client.1.smithi122.stdout:0/119: creat d11/f29 x:0 0 0 2024-03-22T02:28:45.511 INFO:tasks.workunit.client.1.smithi122.stdout:7/96: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:45.514 INFO:tasks.workunit.client.1.smithi122.stdout:3/64: unlink d2/fb 0 2024-03-22T02:28:45.516 INFO:tasks.workunit.client.0.smithi103.stdout:5/717: mkdir d0/d14/d23/d2a/d33/db5/dfb 0 2024-03-22T02:28:45.517 INFO:tasks.workunit.client.0.smithi103.stdout:3/754: getdents de/d12/d8a/db0 0 2024-03-22T02:28:45.519 INFO:tasks.workunit.client.0.smithi103.stdout:3/755: chown de/d1a/d25/d3f/c4b 38323 1 2024-03-22T02:28:45.521 INFO:tasks.workunit.client.0.smithi103.stdout:1/732: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ffa x:0 0 0 2024-03-22T02:28:45.522 INFO:tasks.workunit.client.1.smithi122.stdout:8/124: mknod d1/d22/c2b 0 2024-03-22T02:28:45.522 INFO:tasks.workunit.client.1.smithi122.stdout:8/125: fsync d1/f1a 0 2024-03-22T02:28:45.522 INFO:tasks.workunit.client.0.smithi103.stdout:2/583: dwrite d11/d25/d63/d45/d47/d83/f60 [0,4194304] 0 2024-03-22T02:28:45.526 INFO:tasks.workunit.client.0.smithi103.stdout:7/703: unlink d1/d10/d11/d19/d29/f52 0 2024-03-22T02:28:45.526 INFO:tasks.workunit.client.1.smithi122.stdout:2/95: rmdir df/d1f/d23/d25 0 2024-03-22T02:28:45.527 INFO:tasks.workunit.client.1.smithi122.stdout:2/96: chown df/d1f/f22 29 1 2024-03-22T02:28:45.527 INFO:tasks.workunit.client.0.smithi103.stdout:5/718: stat d0/d14/d23/d5c/d77/df2/f4b 0 2024-03-22T02:28:45.528 INFO:tasks.workunit.client.0.smithi103.stdout:0/686: dread dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 [0,4194304] 0 2024-03-22T02:28:45.530 INFO:tasks.workunit.client.0.smithi103.stdout:4/712: creat d3/d1e/d25/d65/d83/da1/ff2 x:0 0 0 2024-03-22T02:28:45.532 INFO:tasks.workunit.client.1.smithi122.stdout:3/65: unlink d2/ce 0 2024-03-22T02:28:45.533 INFO:tasks.workunit.client.1.smithi122.stdout:3/66: chown d2/d9/dc/df 0 1 2024-03-22T02:28:45.533 INFO:tasks.workunit.client.0.smithi103.stdout:6/500: mknod d0/d14/c9d 0 2024-03-22T02:28:45.536 INFO:tasks.workunit.client.1.smithi122.stdout:2/97: symlink df/d1f/l26 0 2024-03-22T02:28:45.537 INFO:tasks.workunit.client.1.smithi122.stdout:5/105: dread d8/da/f10 [0,4194304] 0 2024-03-22T02:28:45.538 INFO:tasks.workunit.client.1.smithi122.stdout:9/95: mkdir db/df/d10/d1e 0 2024-03-22T02:28:45.539 INFO:tasks.workunit.client.0.smithi103.stdout:6/501: write d0/f64 [221423,69556] 0 2024-03-22T02:28:45.539 INFO:tasks.workunit.client.1.smithi122.stdout:4/119: write d0/d25/d7/f16 [168748,46840] 0 2024-03-22T02:28:45.540 INFO:tasks.workunit.client.1.smithi122.stdout:3/67: mknod d2/d9/dc/df/c12 0 2024-03-22T02:28:45.541 INFO:tasks.workunit.client.0.smithi103.stdout:2/584: dwrite d11/d25/d63/d45/d47/d83/fac [0,4194304] 0 2024-03-22T02:28:45.542 INFO:tasks.workunit.client.1.smithi122.stdout:2/98: mknod df/d1f/c27 0 2024-03-22T02:28:45.543 INFO:tasks.workunit.client.1.smithi122.stdout:9/96: mkdir db/df/d10/d1f 0 2024-03-22T02:28:45.544 INFO:tasks.workunit.client.1.smithi122.stdout:4/120: mkdir d0/d12/d15/d2d 0 2024-03-22T02:28:45.545 INFO:tasks.workunit.client.0.smithi103.stdout:8/818: rename d4/d26/d27/d7e/d39/d49/d51/daf/ff2 to d4/d2a/d38/f10c 0 2024-03-22T02:28:45.547 INFO:tasks.workunit.client.0.smithi103.stdout:1/733: dwrite db/d4a/d78/d92/da8/fe1 [0,4194304] 0 2024-03-22T02:28:45.547 INFO:tasks.workunit.client.1.smithi122.stdout:9/97: creat db/df/d10/f20 x:0 0 0 2024-03-22T02:28:45.549 INFO:tasks.workunit.client.0.smithi103.stdout:0/687: creat dd/d16/d4b/d96/d64/d91/d9a/d9d/fe5 x:0 0 0 2024-03-22T02:28:45.549 INFO:tasks.workunit.client.0.smithi103.stdout:6/502: chown d0/d14/f63 12003343 1 2024-03-22T02:28:45.550 INFO:tasks.workunit.client.1.smithi122.stdout:2/99: fsync df/f11 0 2024-03-22T02:28:45.556 INFO:tasks.workunit.client.1.smithi122.stdout:4/121: dread d0/d25/f1e [0,4194304] 0 2024-03-22T02:28:45.556 INFO:tasks.workunit.client.1.smithi122.stdout:3/68: dwrite d2/f7 [0,4194304] 0 2024-03-22T02:28:45.557 INFO:tasks.workunit.client.0.smithi103.stdout:5/719: truncate d0/d14/d23/d36/d95/ff1 2136807 0 2024-03-22T02:28:45.557 INFO:tasks.workunit.client.1.smithi122.stdout:2/100: mknod df/c28 0 2024-03-22T02:28:45.558 INFO:tasks.workunit.client.0.smithi103.stdout:3/756: rename de/d6b/d80/f7a to de/d12/d5d/da9/fff 0 2024-03-22T02:28:45.559 INFO:tasks.workunit.client.1.smithi122.stdout:9/98: dread db/df/f19 [0,4194304] 0 2024-03-22T02:28:45.559 INFO:tasks.workunit.client.0.smithi103.stdout:5/720: fsync d0/d14/d23/d36/fe3 0 2024-03-22T02:28:45.560 INFO:tasks.workunit.client.0.smithi103.stdout:2/585: rename d11/d25/d63/l5c to d11/d25/d63/lbf 0 2024-03-22T02:28:45.562 INFO:tasks.workunit.client.0.smithi103.stdout:0/688: mknod dd/d16/d4b/d96/d64/d91/d9a/ce6 0 2024-03-22T02:28:45.564 INFO:tasks.workunit.client.0.smithi103.stdout:1/734: mkdir db/d4a/d78/d92/dea/dfb 0 2024-03-22T02:28:45.565 INFO:tasks.workunit.client.1.smithi122.stdout:6/192: chown d7/de/dd/d39/c44 67271 1 2024-03-22T02:28:45.570 INFO:tasks.workunit.client.0.smithi103.stdout:5/721: write d0/f1e [1746635,112093] 0 2024-03-22T02:28:45.570 INFO:tasks.workunit.client.1.smithi122.stdout:6/193: write d7/f1b [463809,95193] 0 2024-03-22T02:28:45.571 INFO:tasks.workunit.client.0.smithi103.stdout:3/757: symlink de/d12/d8a/l100 0 2024-03-22T02:28:45.571 INFO:tasks.workunit.client.0.smithi103.stdout:8/819: dwrite d4/d19/f2e [0,4194304] 0 2024-03-22T02:28:45.573 INFO:tasks.workunit.client.0.smithi103.stdout:0/689: mknod dd/d16/ce7 0 2024-03-22T02:28:45.573 INFO:tasks.workunit.client.0.smithi103.stdout:2/586: dread - d11/d25/d63/d45/f94 zero size 2024-03-22T02:28:45.573 INFO:tasks.workunit.client.1.smithi122.stdout:8/126: fdatasync d1/d9/f26 0 2024-03-22T02:28:45.574 INFO:tasks.workunit.client.0.smithi103.stdout:7/704: dwrite d1/d10/d11/d19/d29/d61/f96 [0,4194304] 0 2024-03-22T02:28:45.578 INFO:tasks.workunit.client.1.smithi122.stdout:9/99: dwrite f7 [0,4194304] 0 2024-03-22T02:28:45.578 INFO:tasks.workunit.client.0.smithi103.stdout:5/722: unlink d0/d14/d23/fa3 0 2024-03-22T02:28:45.579 INFO:tasks.workunit.client.1.smithi122.stdout:4/122: dread d0/d25/f13 [0,4194304] 0 2024-03-22T02:28:45.580 INFO:tasks.workunit.client.0.smithi103.stdout:6/503: dwrite d0/d3/d22/f50 [0,4194304] 0 2024-03-22T02:28:45.580 INFO:tasks.workunit.client.1.smithi122.stdout:2/101: dwrite fd [0,4194304] 0 2024-03-22T02:28:45.586 INFO:tasks.workunit.client.1.smithi122.stdout:1/83: truncate dc/f11 2630356 0 2024-03-22T02:28:45.586 INFO:tasks.workunit.client.1.smithi122.stdout:3/69: symlink d2/l13 0 2024-03-22T02:28:45.587 INFO:tasks.workunit.client.0.smithi103.stdout:7/705: rmdir d1/d10/d11/d19/d29/d33 39 2024-03-22T02:28:45.588 INFO:tasks.workunit.client.0.smithi103.stdout:4/713: dread d3/d1e/f36 [0,4194304] 0 2024-03-22T02:28:45.588 INFO:tasks.workunit.client.0.smithi103.stdout:8/820: fsync d4/d26/d27/d7e/f52 0 2024-03-22T02:28:45.590 INFO:tasks.workunit.client.1.smithi122.stdout:9/100: dread fa [0,4194304] 0 2024-03-22T02:28:45.591 INFO:tasks.workunit.client.0.smithi103.stdout:5/723: mknod d0/d14/d23/d5c/d77/df2/cfc 0 2024-03-22T02:28:45.593 INFO:tasks.workunit.client.0.smithi103.stdout:1/735: dwrite db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:45.596 INFO:tasks.workunit.client.1.smithi122.stdout:4/123: dread d0/d25/f8 [0,4194304] 0 2024-03-22T02:28:45.599 INFO:tasks.workunit.client.0.smithi103.stdout:4/714: mknod d3/d1e/d25/cf3 0 2024-03-22T02:28:45.599 INFO:tasks.workunit.client.0.smithi103.stdout:1/736: rename db to db/d4a/d78/d92/da8/ddd/d18/d2d/dfc 22 2024-03-22T02:28:45.599 INFO:tasks.workunit.client.0.smithi103.stdout:7/706: mknod d1/d10/d11/d19/d29/d33/dcd/ce0 0 2024-03-22T02:28:45.600 INFO:tasks.workunit.client.1.smithi122.stdout:2/102: chown df/c1b 1 1 2024-03-22T02:28:45.600 INFO:tasks.workunit.client.1.smithi122.stdout:6/194: dread d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:45.601 INFO:tasks.workunit.client.1.smithi122.stdout:9/101: dread f5 [0,4194304] 0 2024-03-22T02:28:45.601 INFO:tasks.workunit.client.0.smithi103.stdout:4/715: fsync d3/d1e/d68/fcb 0 2024-03-22T02:28:45.605 INFO:tasks.workunit.client.0.smithi103.stdout:7/707: write d1/d10/d11/d19/d29/d57/f6d [2018562,123499] 0 2024-03-22T02:28:45.605 INFO:tasks.workunit.client.0.smithi103.stdout:4/716: chown d3/d1e/d53/d7a/da3/fbe 48682 1 2024-03-22T02:28:45.606 INFO:tasks.workunit.client.1.smithi122.stdout:1/84: dread dc/f1a [0,4194304] 0 2024-03-22T02:28:45.607 INFO:tasks.workunit.client.1.smithi122.stdout:1/85: fsync dc/f10 0 2024-03-22T02:28:45.607 INFO:tasks.workunit.client.0.smithi103.stdout:7/708: stat d1/d10/d11/d19/d29/d33/da8/fc6 0 2024-03-22T02:28:45.607 INFO:tasks.workunit.client.0.smithi103.stdout:4/717: chown d3/d30/lca 8 1 2024-03-22T02:28:45.608 INFO:tasks.workunit.client.0.smithi103.stdout:2/587: dread f4 [0,4194304] 0 2024-03-22T02:28:45.608 INFO:tasks.workunit.client.1.smithi122.stdout:1/86: fdatasync dc/fd 0 2024-03-22T02:28:45.608 INFO:tasks.workunit.client.0.smithi103.stdout:2/588: stat d11/d25/d46/f37 0 2024-03-22T02:28:45.610 INFO:tasks.workunit.client.0.smithi103.stdout:3/758: dread de/d12/d5d/f99 [4194304,4194304] 0 2024-03-22T02:28:45.613 INFO:tasks.workunit.client.0.smithi103.stdout:3/759: dread de/f1e [0,4194304] 0 2024-03-22T02:28:45.614 INFO:tasks.workunit.client.1.smithi122.stdout:6/195: dwrite d7/de/dd/f3a [0,4194304] 0 2024-03-22T02:28:45.614 INFO:tasks.workunit.client.1.smithi122.stdout:2/103: rename df/d1f/f22 to df/d1f/d23/f29 0 2024-03-22T02:28:45.615 INFO:tasks.workunit.client.0.smithi103.stdout:1/737: fdatasync db/d4a/d78/d92/da8/ddd/f63 0 2024-03-22T02:28:45.616 INFO:tasks.workunit.client.1.smithi122.stdout:8/127: link d1/d8/lc d1/d8/l2c 0 2024-03-22T02:28:45.616 INFO:tasks.workunit.client.1.smithi122.stdout:6/196: fsync d7/de/f43 0 2024-03-22T02:28:45.617 INFO:tasks.workunit.client.1.smithi122.stdout:4/124: symlink d0/d12/d15/d2d/l2e 0 2024-03-22T02:28:45.618 INFO:tasks.workunit.client.0.smithi103.stdout:7/709: rmdir d1/d10/d11/d19/d86/d92 39 2024-03-22T02:28:45.618 INFO:tasks.workunit.client.1.smithi122.stdout:8/128: write d1/d9/f13 [1529580,57445] 0 2024-03-22T02:28:45.619 INFO:tasks.workunit.client.1.smithi122.stdout:4/125: chown d0/d25/f27 2781 1 2024-03-22T02:28:45.619 INFO:tasks.workunit.client.1.smithi122.stdout:6/197: truncate d7/de/f3e 1025329 0 2024-03-22T02:28:45.620 INFO:tasks.workunit.client.1.smithi122.stdout:9/102: mkdir db/df/d10/d1f/d21 0 2024-03-22T02:28:45.621 INFO:tasks.workunit.client.0.smithi103.stdout:3/760: fsync de/d6b/d80/d4f/f77 0 2024-03-22T02:28:45.621 INFO:tasks.workunit.client.1.smithi122.stdout:4/126: truncate d0/d25/d7/f22 171818 0 2024-03-22T02:28:45.622 INFO:tasks.workunit.client.1.smithi122.stdout:9/103: dread - db/df/d10/f20 zero size 2024-03-22T02:28:45.622 INFO:tasks.workunit.client.1.smithi122.stdout:2/104: unlink df/f19 0 2024-03-22T02:28:45.625 INFO:tasks.workunit.client.1.smithi122.stdout:6/198: fsync d7/de/f34 0 2024-03-22T02:28:45.626 INFO:tasks.workunit.client.0.smithi103.stdout:0/690: dread dd/f15 [0,4194304] 0 2024-03-22T02:28:45.626 INFO:tasks.workunit.client.1.smithi122.stdout:4/127: symlink d0/d12/l2f 0 2024-03-22T02:28:45.627 INFO:tasks.workunit.client.0.smithi103.stdout:0/691: chown dd/c27 799679328 1 2024-03-22T02:28:45.628 INFO:tasks.workunit.client.1.smithi122.stdout:4/128: write d0/d25/f1d [78685,67821] 0 2024-03-22T02:28:45.628 INFO:tasks.workunit.client.1.smithi122.stdout:9/104: creat db/d1b/f22 x:0 0 0 2024-03-22T02:28:45.630 INFO:tasks.workunit.client.0.smithi103.stdout:1/738: mkdir db/d4a/d78/d92/da8/ddd/d28/db2/dfd 0 2024-03-22T02:28:45.632 INFO:tasks.workunit.client.1.smithi122.stdout:2/105: mkdir df/d1f/d2a 0 2024-03-22T02:28:45.632 INFO:tasks.workunit.client.1.smithi122.stdout:2/106: readlink l1 0 2024-03-22T02:28:45.634 INFO:tasks.workunit.client.0.smithi103.stdout:0/692: unlink dd/d16/d4b/d96/d64/dab/dc6/d79/l8f 0 2024-03-22T02:28:45.635 INFO:tasks.workunit.client.0.smithi103.stdout:3/761: fsync de/d12/d19/ff7 0 2024-03-22T02:28:45.638 INFO:tasks.workunit.client.1.smithi122.stdout:4/129: symlink d0/d25/d7/l30 0 2024-03-22T02:28:45.639 INFO:tasks.workunit.client.0.smithi103.stdout:7/710: dwrite d1/d10/d11/f5a [4194304,4194304] 0 2024-03-22T02:28:45.641 INFO:tasks.workunit.client.1.smithi122.stdout:4/130: dread d0/d25/f1d [0,4194304] 0 2024-03-22T02:28:45.641 INFO:tasks.workunit.client.0.smithi103.stdout:7/711: fsync d1/d10/d11/d19/f26 0 2024-03-22T02:28:45.642 INFO:tasks.workunit.client.1.smithi122.stdout:8/129: creat d1/f2d x:0 0 0 2024-03-22T02:28:45.642 INFO:tasks.workunit.client.1.smithi122.stdout:8/130: chown d1/d9/f13 1380 1 2024-03-22T02:28:45.643 INFO:tasks.workunit.client.1.smithi122.stdout:4/131: truncate d0/d12/d15/f29 885495 0 2024-03-22T02:28:45.643 INFO:tasks.workunit.client.0.smithi103.stdout:7/712: fdatasync d1/d10/d11/d19/dc3/fd8 0 2024-03-22T02:28:45.644 INFO:tasks.workunit.client.1.smithi122.stdout:8/131: fsync d1/d9/f26 0 2024-03-22T02:28:45.646 INFO:tasks.workunit.client.0.smithi103.stdout:5/724: getdents d0/d14/d23 0 2024-03-22T02:28:45.649 INFO:tasks.workunit.client.0.smithi103.stdout:7/713: fdatasync d1/d10/d11/d19/d29/d33/dcd/fde 0 2024-03-22T02:28:45.650 INFO:tasks.workunit.client.0.smithi103.stdout:0/693: creat dd/d16/d4b/d96/d29/d48/d5c/fe8 x:0 0 0 2024-03-22T02:28:45.650 INFO:tasks.workunit.client.0.smithi103.stdout:3/762: dwrite de/d12/d19/df2/fd8 [0,4194304] 0 2024-03-22T02:28:45.651 INFO:tasks.workunit.client.1.smithi122.stdout:4/132: creat d0/d12/f31 x:0 0 0 2024-03-22T02:28:45.652 INFO:tasks.workunit.client.1.smithi122.stdout:8/132: dread d1/d9/f13 [0,4194304] 0 2024-03-22T02:28:45.652 INFO:tasks.workunit.client.1.smithi122.stdout:3/70: symlink d2/l14 0 2024-03-22T02:28:45.653 INFO:tasks.workunit.client.1.smithi122.stdout:4/133: chown d0/d25/d7/de/c20 36 1 2024-03-22T02:28:45.653 INFO:tasks.workunit.client.0.smithi103.stdout:2/589: dread d11/d19/d5d/f92 [0,4194304] 0 2024-03-22T02:28:45.653 INFO:tasks.workunit.client.1.smithi122.stdout:6/199: creat d7/de/dd/d20/f45 x:0 0 0 2024-03-22T02:28:45.657 INFO:tasks.workunit.client.1.smithi122.stdout:8/133: creat d1/d22/f2e x:0 0 0 2024-03-22T02:28:45.658 INFO:tasks.workunit.client.0.smithi103.stdout:2/590: dread - d11/d19/d7e/fa9 zero size 2024-03-22T02:28:45.661 INFO:tasks.workunit.client.1.smithi122.stdout:8/134: write d1/f2d [535363,27187] 0 2024-03-22T02:28:45.662 INFO:tasks.workunit.client.1.smithi122.stdout:9/105: getdents db/d1b 0 2024-03-22T02:28:45.662 INFO:tasks.workunit.client.1.smithi122.stdout:8/135: dread - d1/d22/f2e zero size 2024-03-22T02:28:45.662 INFO:tasks.workunit.client.1.smithi122.stdout:6/200: mkdir d7/de/dd/d39/d46 0 2024-03-22T02:28:45.663 INFO:tasks.workunit.client.1.smithi122.stdout:8/136: write d1/d9/f11 [4706870,53914] 0 2024-03-22T02:28:45.664 INFO:tasks.workunit.client.1.smithi122.stdout:4/134: dwrite d0/d25/d7/f23 [0,4194304] 0 2024-03-22T02:28:45.665 INFO:tasks.workunit.client.1.smithi122.stdout:9/106: write db/df/f15 [677135,36787] 0 2024-03-22T02:28:45.665 INFO:tasks.workunit.client.0.smithi103.stdout:0/694: fsync dd/d16/d4b/d96/fca 0 2024-03-22T02:28:45.665 INFO:tasks.workunit.client.1.smithi122.stdout:2/107: truncate f4 3129050 0 2024-03-22T02:28:45.667 INFO:tasks.workunit.client.1.smithi122.stdout:8/137: write d1/d9/f24 [1056597,29057] 0 2024-03-22T02:28:45.667 INFO:tasks.workunit.client.1.smithi122.stdout:6/201: unlink d7/c8 0 2024-03-22T02:28:45.668 INFO:tasks.workunit.client.1.smithi122.stdout:9/107: write f7 [1536067,121352] 0 2024-03-22T02:28:45.668 INFO:tasks.workunit.client.0.smithi103.stdout:8/821: dwrite d4/d2a/d3e/da4/f106 [0,4194304] 0 2024-03-22T02:28:45.669 INFO:tasks.workunit.client.1.smithi122.stdout:4/135: write d0/d12/f1c [736378,69733] 0 2024-03-22T02:28:45.671 INFO:tasks.workunit.client.1.smithi122.stdout:8/138: write d1/d9/f11 [2813496,75380] 0 2024-03-22T02:28:45.672 INFO:tasks.workunit.client.0.smithi103.stdout:2/591: chown d11/d25/d46/d40/f41 0 1 2024-03-22T02:28:45.672 INFO:tasks.workunit.client.1.smithi122.stdout:2/108: creat df/f2b x:0 0 0 2024-03-22T02:28:45.673 INFO:tasks.workunit.client.0.smithi103.stdout:0/695: chown dd/d16/d4b/d96/d29/d48/d94/cd0 474586 1 2024-03-22T02:28:45.674 INFO:tasks.workunit.client.0.smithi103.stdout:8/822: chown d4/d2a/d8b/f9f 294 1 2024-03-22T02:28:45.674 INFO:tasks.workunit.client.0.smithi103.stdout:2/592: write d11/f30 [9843636,33748] 0 2024-03-22T02:28:45.679 INFO:tasks.workunit.client.1.smithi122.stdout:9/108: rename db/df/d10 to db/d23 0 2024-03-22T02:28:45.680 INFO:tasks.workunit.client.1.smithi122.stdout:6/202: chown d7/de/dd/l26 79956826 1 2024-03-22T02:28:45.680 INFO:tasks.workunit.client.1.smithi122.stdout:8/139: mkdir d1/d8/d2f 0 2024-03-22T02:28:45.681 INFO:tasks.workunit.client.0.smithi103.stdout:0/696: write dd/d16/d4b/d96/d64/dab/dc6/fd7 [1970171,25693] 0 2024-03-22T02:28:45.682 INFO:tasks.workunit.client.0.smithi103.stdout:1/739: dwrite db/d4a/d78/d92/da8/ddd/d18/d84/f96 [0,4194304] 0 2024-03-22T02:28:45.685 INFO:tasks.workunit.client.1.smithi122.stdout:6/203: unlink d7/de/d27/c2b 0 2024-03-22T02:28:45.687 INFO:tasks.workunit.client.1.smithi122.stdout:9/109: mknod db/df/c24 0 2024-03-22T02:28:45.688 INFO:tasks.workunit.client.0.smithi103.stdout:5/725: dread d0/d14/d23/d2a/d1f/f32 [4194304,4194304] 0 2024-03-22T02:28:45.695 INFO:tasks.workunit.client.1.smithi122.stdout:3/71: link d2/f7 d2/f15 0 2024-03-22T02:28:45.695 INFO:tasks.workunit.client.0.smithi103.stdout:7/714: creat d1/da/fe1 x:0 0 0 2024-03-22T02:28:45.695 INFO:tasks.workunit.client.1.smithi122.stdout:8/140: link d1/c3 d1/d22/c30 0 2024-03-22T02:28:45.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/141: fdatasync d1/f1a 0 2024-03-22T02:28:45.697 INFO:tasks.workunit.client.1.smithi122.stdout:8/142: fsync d1/f2d 0 2024-03-22T02:28:45.697 INFO:tasks.workunit.client.1.smithi122.stdout:8/143: fdatasync d1/d9/f11 0 2024-03-22T02:28:45.698 INFO:tasks.workunit.client.1.smithi122.stdout:9/110: mkdir db/d23/d1f/d21/d25 0 2024-03-22T02:28:45.699 INFO:tasks.workunit.client.0.smithi103.stdout:8/823: unlink d4/d19/l1f 0 2024-03-22T02:28:45.701 INFO:tasks.workunit.client.0.smithi103.stdout:0/697: mknod dd/d16/d4b/d96/d64/d91/d9a/d9d/ce9 0 2024-03-22T02:28:45.701 INFO:tasks.workunit.client.1.smithi122.stdout:4/136: truncate d0/d25/d7/f16 2022336 0 2024-03-22T02:28:45.702 INFO:tasks.workunit.client.0.smithi103.stdout:1/740: read - db/d4a/d78/d92/da8/ddd/d29/d41/f90 zero size 2024-03-22T02:28:45.702 INFO:tasks.workunit.client.1.smithi122.stdout:9/111: fsync f5 0 2024-03-22T02:28:45.703 INFO:tasks.workunit.client.0.smithi103.stdout:0/698: readlink dd/d16/d4b/d96/d29/d48/d5c/d9b/lcb 0 2024-03-22T02:28:45.705 INFO:tasks.workunit.client.1.smithi122.stdout:8/144: dwrite d1/d9/f1f [0,4194304] 0 2024-03-22T02:28:45.706 INFO:tasks.workunit.client.1.smithi122.stdout:8/145: read - d1/d9/f28 zero size 2024-03-22T02:28:45.707 INFO:tasks.workunit.client.0.smithi103.stdout:0/699: dread dd/f15 [0,4194304] 0 2024-03-22T02:28:45.708 INFO:tasks.workunit.client.0.smithi103.stdout:8/824: mknod d4/d48/d40/c10d 0 2024-03-22T02:28:45.708 INFO:tasks.workunit.client.0.smithi103.stdout:0/700: stat dd/d34/fac 0 2024-03-22T02:28:45.709 INFO:tasks.workunit.client.1.smithi122.stdout:8/146: readlink d1/d9/l16 0 2024-03-22T02:28:45.710 INFO:tasks.workunit.client.1.smithi122.stdout:4/137: getdents d0/d12 0 2024-03-22T02:28:45.712 INFO:tasks.workunit.client.0.smithi103.stdout:8/825: truncate d4/d26/d27/d7e/d39/d49/d51/daf/fe5 115053 0 2024-03-22T02:28:45.714 INFO:tasks.workunit.client.1.smithi122.stdout:9/112: dwrite db/d23/f20 [0,4194304] 0 2024-03-22T02:28:45.715 INFO:tasks.workunit.client.1.smithi122.stdout:3/72: truncate d2/f6 3730160 0 2024-03-22T02:28:45.715 INFO:tasks.workunit.client.1.smithi122.stdout:3/73: chown d2/f6 41 1 2024-03-22T02:28:45.716 INFO:tasks.workunit.client.1.smithi122.stdout:8/147: mknod d1/d8/c31 0 2024-03-22T02:28:45.718 INFO:tasks.workunit.client.1.smithi122.stdout:8/148: write d1/d9/f11 [302440,17803] 0 2024-03-22T02:28:45.718 INFO:tasks.workunit.client.1.smithi122.stdout:6/204: truncate d7/de/dd/d20/d2d/f30 2856804 0 2024-03-22T02:28:45.719 INFO:tasks.workunit.client.1.smithi122.stdout:8/149: chown d1/d22/c2b 2 1 2024-03-22T02:28:45.721 INFO:tasks.workunit.client.0.smithi103.stdout:3/763: getdents de/d1a/d25/d3f/d9c/da7 0 2024-03-22T02:28:45.722 INFO:tasks.workunit.client.0.smithi103.stdout:3/764: stat de/d12/d8a 0 2024-03-22T02:28:45.723 INFO:tasks.workunit.client.0.smithi103.stdout:8/826: dwrite d4/d26/d27/d7e/d39/d49/d51/daf/fe5 [0,4194304] 0 2024-03-22T02:28:45.724 INFO:tasks.workunit.client.1.smithi122.stdout:9/113: mknod db/d23/d1e/c26 0 2024-03-22T02:28:45.726 INFO:tasks.workunit.client.1.smithi122.stdout:3/74: creat d2/d9/dc/f16 x:0 0 0 2024-03-22T02:28:45.729 INFO:tasks.workunit.client.0.smithi103.stdout:5/726: truncate d0/da2/dca/fba 2195848 0 2024-03-22T02:28:45.730 INFO:tasks.workunit.client.1.smithi122.stdout:8/150: dwrite d1/d22/f2e [0,4194304] 0 2024-03-22T02:28:45.730 INFO:tasks.workunit.client.1.smithi122.stdout:6/205: link d7/de/f43 d7/de/dd/d23/f47 0 2024-03-22T02:28:45.730 INFO:tasks.workunit.client.0.smithi103.stdout:2/593: dread d11/d25/d46/f29 [0,4194304] 0 2024-03-22T02:28:45.731 INFO:tasks.workunit.client.0.smithi103.stdout:2/594: read f4 [924477,119646] 0 2024-03-22T02:28:45.733 INFO:tasks.workunit.client.1.smithi122.stdout:3/75: mkdir d2/d9/dc/d17 0 2024-03-22T02:28:45.734 INFO:tasks.workunit.client.1.smithi122.stdout:8/151: creat d1/d22/f32 x:0 0 0 2024-03-22T02:28:45.735 INFO:tasks.workunit.client.0.smithi103.stdout:7/715: truncate d1/d10/d11/f9f 1951380 0 2024-03-22T02:28:45.735 INFO:tasks.workunit.client.1.smithi122.stdout:9/114: symlink db/d23/d1f/d21/d25/l27 0 2024-03-22T02:28:45.735 INFO:tasks.workunit.client.0.smithi103.stdout:1/741: dread db/d4a/d78/d92/da8/ddd/d29/d41/fcb [0,4194304] 0 2024-03-22T02:28:45.738 INFO:tasks.workunit.client.1.smithi122.stdout:8/152: chown d1/l20 14 1 2024-03-22T02:28:45.739 INFO:tasks.workunit.client.1.smithi122.stdout:8/153: chown d1/d9/f24 2136 1 2024-03-22T02:28:45.739 INFO:tasks.workunit.client.0.smithi103.stdout:8/827: symlink d4/d26/d27/d7e/d39/d7b/l10e 0 2024-03-22T02:28:45.739 INFO:tasks.workunit.client.0.smithi103.stdout:3/765: mknod de/d12/d5d/c101 0 2024-03-22T02:28:45.740 INFO:tasks.workunit.client.0.smithi103.stdout:0/701: mknod dd/d16/d4b/d96/d29/cea 0 2024-03-22T02:28:45.740 INFO:tasks.workunit.client.1.smithi122.stdout:8/154: write d1/f1a [1724995,1892] 0 2024-03-22T02:28:45.741 INFO:tasks.workunit.client.0.smithi103.stdout:8/828: dread - d4/d19/d8c/f100 zero size 2024-03-22T02:28:45.742 INFO:tasks.workunit.client.1.smithi122.stdout:9/115: dread f7 [0,4194304] 0 2024-03-22T02:28:45.743 INFO:tasks.workunit.client.0.smithi103.stdout:1/742: chown db/d4a/d78/d92/da8/ddd/c60 297961109 1 2024-03-22T02:28:45.744 INFO:tasks.workunit.client.1.smithi122.stdout:3/76: dread d2/f15 [0,4194304] 0 2024-03-22T02:28:45.744 INFO:tasks.workunit.client.0.smithi103.stdout:5/727: getdents d0/d14/d23/d2a/d33/db5/dbd/ded 0 2024-03-22T02:28:45.746 INFO:tasks.workunit.client.1.smithi122.stdout:4/138: dread d0/d25/d7/f16 [0,4194304] 0 2024-03-22T02:28:45.747 INFO:tasks.workunit.client.0.smithi103.stdout:5/728: truncate d0/d14/dc0/fea 716976 0 2024-03-22T02:28:45.747 INFO:tasks.workunit.client.0.smithi103.stdout:2/595: symlink d11/d25/d63/d45/lc0 0 2024-03-22T02:28:45.748 INFO:tasks.workunit.client.1.smithi122.stdout:9/116: dread fa [4194304,4194304] 0 2024-03-22T02:28:45.748 INFO:tasks.workunit.client.1.smithi122.stdout:3/77: symlink d2/d9/l18 0 2024-03-22T02:28:45.748 INFO:tasks.workunit.client.1.smithi122.stdout:3/78: stat d2/l13 0 2024-03-22T02:28:45.748 INFO:tasks.workunit.client.0.smithi103.stdout:2/596: dread - d11/d25/d46/d48/f9d zero size 2024-03-22T02:28:45.750 INFO:tasks.workunit.client.1.smithi122.stdout:3/79: rename d2/d9/dc to d2/d9/dc/d17/d19 22 2024-03-22T02:28:45.750 INFO:tasks.workunit.client.0.smithi103.stdout:5/729: unlink d0/d14/d23/d5c/d77/df2/cfc 0 2024-03-22T02:28:45.751 INFO:tasks.workunit.client.0.smithi103.stdout:2/597: write d11/d19/d5d/f68 [792297,43653] 0 2024-03-22T02:28:45.752 INFO:tasks.workunit.client.1.smithi122.stdout:9/117: truncate f5 970448 0 2024-03-22T02:28:45.753 INFO:tasks.workunit.client.0.smithi103.stdout:7/716: creat d1/d10/d11/d19/d29/fe2 x:0 0 0 2024-03-22T02:28:45.754 INFO:tasks.workunit.client.1.smithi122.stdout:3/80: creat d2/d9/f1a x:0 0 0 2024-03-22T02:28:45.754 INFO:tasks.workunit.client.0.smithi103.stdout:5/730: dread - d0/d14/d23/d36/d95/d96/daa/daf/ff0 zero size 2024-03-22T02:28:45.755 INFO:tasks.workunit.client.1.smithi122.stdout:4/139: dwrite d0/d25/d7/f23 [0,4194304] 0 2024-03-22T02:28:45.759 INFO:tasks.workunit.client.1.smithi122.stdout:9/118: unlink db/d23/f20 0 2024-03-22T02:28:45.761 INFO:tasks.workunit.client.0.smithi103.stdout:8/829: creat d4/d26/d27/d7e/d39/f10f x:0 0 0 2024-03-22T02:28:45.765 INFO:tasks.workunit.client.1.smithi122.stdout:9/119: mkdir db/d1b/d28 0 2024-03-22T02:28:45.766 INFO:tasks.workunit.client.0.smithi103.stdout:1/743: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe 0 2024-03-22T02:28:45.766 INFO:tasks.workunit.client.1.smithi122.stdout:3/81: dwrite d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:45.769 INFO:tasks.workunit.client.1.smithi122.stdout:9/120: truncate db/df/f15 1414055 0 2024-03-22T02:28:45.770 INFO:tasks.workunit.client.1.smithi122.stdout:4/140: dwrite d0/d25/d7/f23 [0,4194304] 0 2024-03-22T02:28:45.771 INFO:tasks.workunit.client.1.smithi122.stdout:9/121: truncate db/d23/f1a 985106 0 2024-03-22T02:28:45.771 INFO:tasks.workunit.client.1.smithi122.stdout:3/82: fdatasync d2/f6 0 2024-03-22T02:28:45.771 INFO:tasks.workunit.client.0.smithi103.stdout:2/598: dwrite d11/d25/d63/d45/d47/d83/fa5 [0,4194304] 0 2024-03-22T02:28:45.771 INFO:tasks.workunit.client.0.smithi103.stdout:7/717: rename d1/d10/d11/d19/d29/d33/da8/dc5/fce to d1/d10/d11/d19/d29/d61/d1d/da3/db9/fe3 0 2024-03-22T02:28:45.774 INFO:tasks.workunit.client.0.smithi103.stdout:0/702: write dd/d16/d4b/d96/d64/dab/dc6/d89/f8b [764367,40428] 0 2024-03-22T02:28:45.775 INFO:tasks.workunit.client.0.smithi103.stdout:5/731: symlink d0/d14/d23/d5c/d77/df2/dd9/lfd 0 2024-03-22T02:28:45.776 INFO:tasks.workunit.client.1.smithi122.stdout:3/83: fsync d2/f15 0 2024-03-22T02:28:45.776 INFO:tasks.workunit.client.0.smithi103.stdout:7/718: read d1/f1e [1097434,97259] 0 2024-03-22T02:28:45.781 INFO:tasks.workunit.client.0.smithi103.stdout:1/744: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/dff 0 2024-03-22T02:28:45.785 INFO:tasks.workunit.client.1.smithi122.stdout:4/141: dwrite d0/d25/f27 [0,4194304] 0 2024-03-22T02:28:45.785 INFO:tasks.workunit.client.0.smithi103.stdout:2/599: symlink d11/d25/d63/d45/d47/lc1 0 2024-03-22T02:28:45.786 INFO:tasks.workunit.client.1.smithi122.stdout:9/122: dwrite fa [4194304,4194304] 0 2024-03-22T02:28:45.787 INFO:tasks.workunit.client.0.smithi103.stdout:3/766: dread de/d12/d5d/f75 [0,4194304] 0 2024-03-22T02:28:45.787 INFO:tasks.workunit.client.1.smithi122.stdout:4/142: stat d0/d25/d7/de/c20 0 2024-03-22T02:28:45.788 INFO:tasks.workunit.client.0.smithi103.stdout:2/600: readlink d11/d25/d63/d45/d47/d83/d78/db0/l9a 0 2024-03-22T02:28:45.790 INFO:tasks.workunit.client.0.smithi103.stdout:0/703: rename dd/d16/d4b/d96/d64/d91 to dd/d16/d4b/d96/d64/d6a/dd5/deb 0 2024-03-22T02:28:45.794 INFO:tasks.workunit.client.0.smithi103.stdout:8/830: rmdir d4/d26/d27/dfc 0 2024-03-22T02:28:45.802 INFO:tasks.workunit.client.0.smithi103.stdout:3/767: unlink de/d12/l3e 0 2024-03-22T02:28:45.803 INFO:tasks.workunit.client.0.smithi103.stdout:8/831: mknod d4/d2a/d38/dfe/c110 0 2024-03-22T02:28:45.804 INFO:tasks.workunit.client.0.smithi103.stdout:8/832: fsync d4/d2a/d4f/fb7 0 2024-03-22T02:28:45.805 INFO:tasks.workunit.client.0.smithi103.stdout:2/601: rename d11/d19/d5d/f92 to d11/d25/d46/fc2 0 2024-03-22T02:28:45.805 INFO:tasks.workunit.client.0.smithi103.stdout:1/745: getdents db/d4a/d78/d92/da8/ddd/d6b/db5 0 2024-03-22T02:28:45.810 INFO:tasks.workunit.client.0.smithi103.stdout:8/833: unlink d4/d26/d27/d7e/d39/d7b/f10a 0 2024-03-22T02:28:45.813 INFO:tasks.workunit.client.0.smithi103.stdout:8/834: chown d4/d2a/cec 44563 1 2024-03-22T02:28:45.814 INFO:tasks.workunit.client.0.smithi103.stdout:8/835: readlink d4/d2a/d3e/le9 0 2024-03-22T02:28:45.817 INFO:tasks.workunit.client.0.smithi103.stdout:2/602: link d11/d25/d63/d45/d47/c81 d11/d25/cc3 0 2024-03-22T02:28:45.818 INFO:tasks.workunit.client.0.smithi103.stdout:2/603: chown d11/d25/d63/cbe 8 1 2024-03-22T02:28:45.819 INFO:tasks.workunit.client.0.smithi103.stdout:5/732: dwrite d0/d14/d23/f24 [4194304,4194304] 0 2024-03-22T02:28:45.819 INFO:tasks.workunit.client.0.smithi103.stdout:1/746: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f100 x:0 0 0 2024-03-22T02:28:45.819 INFO:tasks.workunit.client.0.smithi103.stdout:2/604: stat d11/d19/l22 0 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/2178930085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fe040 msgr2=0x7f5e840fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/2178930085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fe040 0x7f5e840fe4b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5e74004940 tx=0x7f5e74009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/2178930085 shutdown_connections 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/2178930085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fe040 0x7f5e840fe4b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/2178930085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fce40 0x7f5e840fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/2178930085 >> 172.21.15.103:0/2178930085 conn(0x7f5e840f8570 msgr2=0x7f5e840fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:45.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/2178930085 shutdown_connections 2024-03-22T02:28:45.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.831+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/2178930085 wait complete. 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: pgmap v16: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 153 MiB/s wr, 355 op/s 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.833 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.834 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 Processor -- start 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 -- start start 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fe040 0x7f5e8410df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8410e500 con 0x7f5e840fce40 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e8c1f6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8410ed50 con 0x7f5e840fe040 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e89f92700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e89f92700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39966/0 (socket says 172.21.15.103:39966) 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.832+0000 7f5e89f92700 1 -- 172.21.15.103:0/3018643661 learned_addr learned my addr 172.21.15.103:0/3018643661 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e89791700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fe040 0x7f5e8410df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e89f92700 1 -- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fe040 msgr2=0x7f5e8410df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e89f92700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fe040 0x7f5e8410df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e89f92700 1 -- 172.21.15.103:0/3018643661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e74005040 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e89f92700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000eb70 tx=0x7f5e8000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e80007c60 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e8000f040 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8410f030 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8000ca90 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.833+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8410f570 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.834+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8405f5f0 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.835+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f5e8000cc10 con 0x7f5e840fce40 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.835+0000 7f5e7a7fc700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 0x7f5e7006dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:45.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.835+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f5e8008a670 con 0x7f5e840fce40 2024-03-22T02:28:45.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.835+0000 7f5e89791700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 0x7f5e7006dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:45.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.836+0000 7f5e89791700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 0x7f5e7006dd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b0d0 tx=0x7f5e74009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:45.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:45.840+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f5e80057b30 con 0x7f5e840fce40 2024-03-22T02:28:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: pgmap v16: 65 pgs: 65 active+clean; 1.2 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 32 MiB/s rd, 153 MiB/s wr, 355 op/s 2024-03-22T02:28:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:45.866 INFO:tasks.workunit.client.1.smithi122.stdout:9/123: write f8 [2370543,8715] 0 2024-03-22T02:28:45.866 INFO:tasks.workunit.client.1.smithi122.stdout:4/143: mkdir d0/d12/d15/d2d/d32 0 2024-03-22T02:28:45.867 INFO:tasks.workunit.client.1.smithi122.stdout:7/97: sync 2024-03-22T02:28:45.867 INFO:tasks.workunit.client.1.smithi122.stdout:5/106: sync 2024-03-22T02:28:45.867 INFO:tasks.workunit.client.1.smithi122.stdout:0/120: sync 2024-03-22T02:28:45.868 INFO:tasks.workunit.client.1.smithi122.stdout:7/98: dread - d0/d4/f12 zero size 2024-03-22T02:28:45.870 INFO:tasks.workunit.client.1.smithi122.stdout:9/124: getdents db/d23/d1f/d21 0 2024-03-22T02:28:45.871 INFO:tasks.workunit.client.1.smithi122.stdout:7/99: write d0/f2 [6337954,3834] 0 2024-03-22T02:28:45.872 INFO:tasks.workunit.client.1.smithi122.stdout:9/125: dread - db/d23/f11 zero size 2024-03-22T02:28:45.875 INFO:tasks.workunit.client.1.smithi122.stdout:3/84: dwrite d2/f15 [0,4194304] 0 2024-03-22T02:28:45.875 INFO:tasks.workunit.client.1.smithi122.stdout:7/100: write d0/f2 [7167685,104753] 0 2024-03-22T02:28:45.876 INFO:tasks.workunit.client.1.smithi122.stdout:0/121: truncate fb 2211004 0 2024-03-22T02:28:45.880 INFO:tasks.workunit.client.1.smithi122.stdout:7/101: write d0/fd [737783,334] 0 2024-03-22T02:28:45.880 INFO:tasks.workunit.client.1.smithi122.stdout:5/107: dwrite d8/da/f10 [0,4194304] 0 2024-03-22T02:28:45.880 INFO:tasks.workunit.client.1.smithi122.stdout:9/126: mkdir db/d23/d1f/d21/d29 0 2024-03-22T02:28:45.881 INFO:tasks.workunit.client.1.smithi122.stdout:7/102: chown d0 46 1 2024-03-22T02:28:45.882 INFO:tasks.workunit.client.1.smithi122.stdout:7/103: fsync d0/d4/f11 0 2024-03-22T02:28:45.884 INFO:tasks.workunit.client.1.smithi122.stdout:5/108: fsync d8/d1a/f1d 0 2024-03-22T02:28:45.885 INFO:tasks.workunit.client.1.smithi122.stdout:7/104: write d0/d4/f9 [491352,75588] 0 2024-03-22T02:28:45.885 INFO:tasks.workunit.client.1.smithi122.stdout:9/127: truncate db/d1b/f22 154192 0 2024-03-22T02:28:45.886 INFO:tasks.workunit.client.0.smithi103.stdout:5/733: creat d0/da2/ffe x:0 0 0 2024-03-22T02:28:45.887 INFO:tasks.workunit.client.1.smithi122.stdout:7/105: fdatasync d0/d4/fb 0 2024-03-22T02:28:45.887 INFO:tasks.workunit.client.1.smithi122.stdout:0/122: link d11/d14/d23/d24/f25 d11/d16/f2a 0 2024-03-22T02:28:45.887 INFO:tasks.workunit.client.1.smithi122.stdout:3/85: dread d2/f8 [0,4194304] 0 2024-03-22T02:28:45.887 INFO:tasks.workunit.client.1.smithi122.stdout:0/123: chown d11 3283 1 2024-03-22T02:28:45.887 INFO:tasks.workunit.client.1.smithi122.stdout:5/109: truncate d8/d1a/f1d 793554 0 2024-03-22T02:28:45.888 INFO:tasks.workunit.client.1.smithi122.stdout:9/128: write f7 [565749,88693] 0 2024-03-22T02:28:45.888 INFO:tasks.workunit.client.1.smithi122.stdout:5/110: chown d8/da 97118203 1 2024-03-22T02:28:45.891 INFO:tasks.workunit.client.0.smithi103.stdout:1/747: readlink db/d4a/d78/d92/da8/ddd/l7a 0 2024-03-22T02:28:45.893 INFO:tasks.workunit.client.0.smithi103.stdout:8/836: rename d4/d2a/cbd to d4/d48/d35/c111 0 2024-03-22T02:28:45.893 INFO:tasks.workunit.client.0.smithi103.stdout:3/768: write de/d1a/d25/f29 [4626998,125584] 0 2024-03-22T02:28:45.895 INFO:tasks.workunit.client.0.smithi103.stdout:7/719: dwrite d1/d10/d11/d19/d29/d61/d1d/f41 [4194304,4194304] 0 2024-03-22T02:28:45.898 INFO:tasks.workunit.client.0.smithi103.stdout:0/704: dwrite dd/f5b [0,4194304] 0 2024-03-22T02:28:45.898 INFO:tasks.workunit.client.0.smithi103.stdout:5/734: symlink d0/d14/d23/d36/de1/lff 0 2024-03-22T02:28:45.899 INFO:tasks.workunit.client.0.smithi103.stdout:1/748: readlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/le3 0 2024-03-22T02:28:45.902 INFO:tasks.workunit.client.0.smithi103.stdout:2/605: getdents d11/d19/d7e 0 2024-03-22T02:28:45.902 INFO:tasks.workunit.client.0.smithi103.stdout:1/749: chown db/d4a/d78/d92/da8/ddd/d29/d41/c5f 22586 1 2024-03-22T02:28:45.903 INFO:tasks.workunit.client.0.smithi103.stdout:3/769: mkdir de/d12/d5d/d102 0 2024-03-22T02:28:45.904 INFO:tasks.workunit.client.0.smithi103.stdout:8/837: mknod d4/d26/d27/d7e/d67/c112 0 2024-03-22T02:28:45.907 INFO:tasks.workunit.client.0.smithi103.stdout:0/705: creat dd/d16/d4b/d96/d29/d48/fec x:0 0 0 2024-03-22T02:28:45.910 INFO:tasks.workunit.client.0.smithi103.stdout:2/606: rename d11/d19/f8f to d11/da0/fc4 0 2024-03-22T02:28:45.911 INFO:tasks.workunit.client.0.smithi103.stdout:5/735: mknod d0/d14/d23/d2a/d1f/c100 0 2024-03-22T02:28:45.913 INFO:tasks.workunit.client.0.smithi103.stdout:1/750: unlink db/d4a/d78/d92/da8/ddd/d18/f3b 0 2024-03-22T02:28:45.915 INFO:tasks.workunit.client.0.smithi103.stdout:1/751: dread - db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fde zero size 2024-03-22T02:28:45.916 INFO:tasks.workunit.client.0.smithi103.stdout:2/607: rmdir d11/d19/d5d 39 2024-03-22T02:28:45.917 INFO:tasks.workunit.client.0.smithi103.stdout:1/752: write db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fcd [8233,27412] 0 2024-03-22T02:28:45.918 INFO:tasks.workunit.client.0.smithi103.stdout:5/736: mknod d0/da2/c101 0 2024-03-22T02:28:45.919 INFO:tasks.workunit.client.0.smithi103.stdout:1/753: chown db/d4a/d78/d92/da8/ddd/d29/f6e 5030 1 2024-03-22T02:28:45.924 INFO:tasks.workunit.client.0.smithi103.stdout:1/754: truncate db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ffa 260487 0 2024-03-22T02:28:45.925 INFO:tasks.workunit.client.0.smithi103.stdout:5/737: dwrite d0/d14/d23/fbc [0,4194304] 0 2024-03-22T02:28:45.928 INFO:tasks.workunit.client.0.smithi103.stdout:5/738: rmdir d0/d14/d23/d36/d95/d96 39 2024-03-22T02:28:45.931 INFO:tasks.workunit.client.0.smithi103.stdout:1/755: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/d101 0 2024-03-22T02:28:45.932 INFO:tasks.workunit.client.0.smithi103.stdout:5/739: dread d0/f1e [4194304,4194304] 0 2024-03-22T02:28:45.936 INFO:tasks.workunit.client.0.smithi103.stdout:1/756: creat db/d4a/d78/d92/da8/ddd/d28/d97/f102 x:0 0 0 2024-03-22T02:28:45.939 INFO:tasks.workunit.client.0.smithi103.stdout:0/706: dread dd/d16/d4b/d96/d35/d82/f7b [0,4194304] 0 2024-03-22T02:28:45.943 INFO:tasks.workunit.client.0.smithi103.stdout:2/608: dread d11/f16 [0,4194304] 0 2024-03-22T02:28:45.944 INFO:tasks.workunit.client.0.smithi103.stdout:5/740: rename d0/d14/d23/d2a/d1f/f32 to d0/d14/d23/d2a/f102 0 2024-03-22T02:28:45.945 INFO:tasks.workunit.client.0.smithi103.stdout:2/609: chown d11/d25/d46/d40/f6f 9 1 2024-03-22T02:28:45.946 INFO:tasks.workunit.client.0.smithi103.stdout:0/707: rmdir dd/d16/d4b/d96/d64/dab/dc6/d79 39 2024-03-22T02:28:45.946 INFO:tasks.workunit.client.0.smithi103.stdout:1/757: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/fb9 [4194304,4194304] 0 2024-03-22T02:28:45.948 INFO:tasks.workunit.client.0.smithi103.stdout:0/708: stat dd/f24 0 2024-03-22T02:28:45.949 INFO:tasks.workunit.client.0.smithi103.stdout:0/709: dread - dd/fc3 zero size 2024-03-22T02:28:45.950 INFO:tasks.workunit.client.0.smithi103.stdout:5/741: mkdir d0/d14/d23/d2a/d33/db5/dbd/ded/d103 0 2024-03-22T02:28:45.951 INFO:tasks.workunit.client.0.smithi103.stdout:2/610: readlink d11/d25/d63/d45/d47/d83/d78/db0/l3e 0 2024-03-22T02:28:45.952 INFO:tasks.workunit.client.0.smithi103.stdout:2/611: write d11/d25/d46/f9c [244343,28950] 0 2024-03-22T02:28:45.954 INFO:tasks.workunit.client.0.smithi103.stdout:5/742: readlink d0/d14/d23/d2a/d33/db5/dbd/lcc 0 2024-03-22T02:28:45.956 INFO:tasks.workunit.client.0.smithi103.stdout:0/710: dwrite f4 [0,4194304] 0 2024-03-22T02:28:45.956 INFO:tasks.workunit.client.0.smithi103.stdout:1/758: creat db/d4a/d78/d92/da8/ddd/d28/f103 x:0 0 0 2024-03-22T02:28:45.960 INFO:tasks.workunit.client.0.smithi103.stdout:0/711: dread dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 [0,4194304] 0 2024-03-22T02:28:45.961 INFO:tasks.workunit.client.0.smithi103.stdout:5/743: fdatasync d0/f97 0 2024-03-22T02:28:45.965 INFO:tasks.workunit.client.0.smithi103.stdout:0/712: rename dd/d34/dbd to dd/d16/d4b/d96/d29/d48/ded 0 2024-03-22T02:28:45.967 INFO:tasks.workunit.client.0.smithi103.stdout:1/759: fsync db/d4a/d78/d92/da8/ddd/f83 0 2024-03-22T02:28:45.970 INFO:tasks.workunit.client.0.smithi103.stdout:5/744: mkdir d0/d14/d23/d6f/d83/d104 0 2024-03-22T02:28:45.973 INFO:tasks.workunit.client.1.smithi122.stdout:0/124: symlink d11/d14/d1d/d27/l2b 0 2024-03-22T02:28:45.973 INFO:tasks.workunit.client.0.smithi103.stdout:0/713: rename dd/d16/d4b/d96/d29/f4d to dd/d16/d4b/d96/d29/d48/d5c/d9b/ddd/fee 0 2024-03-22T02:28:45.973 INFO:tasks.workunit.client.1.smithi122.stdout:5/111: dread - d8/da/f18 zero size 2024-03-22T02:28:45.974 INFO:tasks.workunit.client.0.smithi103.stdout:2/612: dread d11/d25/d63/d45/d47/f85 [0,4194304] 0 2024-03-22T02:28:45.974 INFO:tasks.workunit.client.1.smithi122.stdout:7/106: truncate d0/d4/f9 1042001 0 2024-03-22T02:28:45.975 INFO:tasks.workunit.client.1.smithi122.stdout:9/129: rename c9 to db/d23/c2a 0 2024-03-22T02:28:45.976 INFO:tasks.workunit.client.1.smithi122.stdout:3/86: dread d2/f8 [0,4194304] 0 2024-03-22T02:28:45.977 INFO:tasks.workunit.client.1.smithi122.stdout:9/130: fdatasync fa 0 2024-03-22T02:28:45.979 INFO:tasks.workunit.client.0.smithi103.stdout:0/714: rmdir dd/d16/d4b/d96/d64/dab 39 2024-03-22T02:28:45.980 INFO:tasks.workunit.client.1.smithi122.stdout:9/131: write db/f13 [1448663,40302] 0 2024-03-22T02:28:45.982 INFO:tasks.workunit.client.0.smithi103.stdout:5/745: mkdir d0/d14/d23/d2a/d33/db5/dbd/ded/d103/d105 0 2024-03-22T02:28:45.982 INFO:tasks.workunit.client.1.smithi122.stdout:5/112: dwrite d8/f17 [0,4194304] 0 2024-03-22T02:28:45.983 INFO:tasks.workunit.client.1.smithi122.stdout:9/132: write db/d23/f1a [407760,96986] 0 2024-03-22T02:28:45.984 INFO:tasks.workunit.client.0.smithi103.stdout:0/715: mknod dd/d34/d53/d8c/cef 0 2024-03-22T02:28:45.984 INFO:tasks.workunit.client.0.smithi103.stdout:5/746: chown d0/d14/d23/d2a/d33/db9/lce 376801 1 2024-03-22T02:28:45.984 INFO:tasks.workunit.client.1.smithi122.stdout:9/133: readlink db/d1b/l1c 0 2024-03-22T02:28:45.985 INFO:tasks.workunit.client.0.smithi103.stdout:2/613: mkdir d11/dc5 0 2024-03-22T02:28:45.985 INFO:tasks.workunit.client.0.smithi103.stdout:5/747: chown d0/d14/d23/d5c/d77/df2 804 1 2024-03-22T02:28:45.985 INFO:tasks.workunit.client.1.smithi122.stdout:3/87: rename d2/d9/dc/df/l11 to d2/d9/l1b 0 2024-03-22T02:28:45.986 INFO:tasks.workunit.client.0.smithi103.stdout:2/614: chown d11/d25 77529 1 2024-03-22T02:28:45.989 INFO:tasks.workunit.client.1.smithi122.stdout:3/88: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:45.991 INFO:tasks.workunit.client.0.smithi103.stdout:5/748: rename d0/d14/d23/d5c/fdb to d0/d14/d23/d2a/d33/db5/f106 0 2024-03-22T02:28:45.994 INFO:tasks.workunit.client.0.smithi103.stdout:5/749: creat d0/d3e/f107 x:0 0 0 2024-03-22T02:28:45.995 INFO:tasks.workunit.client.0.smithi103.stdout:5/750: chown d0/fe 215 1 2024-03-22T02:28:45.996 INFO:tasks.workunit.client.0.smithi103.stdout:2/615: link d11/d25/d63/d45/d47/d83/d78/db0/c50 d11/d25/d46/d48/da6/cc6 0 2024-03-22T02:28:45.997 INFO:tasks.workunit.client.0.smithi103.stdout:2/616: stat d11/d25/d46/d48/da6/cb8 0 2024-03-22T02:28:45.998 INFO:tasks.workunit.client.0.smithi103.stdout:1/760: dwrite db/d4a/d78/d92/da8/ddd/d29/f61 [0,4194304] 0 2024-03-22T02:28:45.998 INFO:tasks.workunit.client.0.smithi103.stdout:2/617: chown d11/f20 741 1 2024-03-22T02:28:46.000 INFO:tasks.workunit.client.0.smithi103.stdout:2/618: chown d11/d19/d5d/f71 1 1 2024-03-22T02:28:46.005 INFO:tasks.workunit.client.0.smithi103.stdout:1/761: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/dff/l104 0 2024-03-22T02:28:46.008 INFO:tasks.workunit.client.0.smithi103.stdout:1/762: creat db/d4a/d78/d92/da8/ddd/d29/d6c/f105 x:0 0 0 2024-03-22T02:28:46.013 INFO:tasks.workunit.client.0.smithi103.stdout:2/619: dwrite d11/d25/d63/d45/d47/f72 [0,4194304] 0 2024-03-22T02:28:46.015 INFO:tasks.workunit.client.0.smithi103.stdout:1/763: rename db/d4a/d78/d92/da8/ddd/d2e/l3a to db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/l106 0 2024-03-22T02:28:46.016 INFO:tasks.workunit.client.0.smithi103.stdout:2/620: mknod d11/d25/d46/d40/cc7 0 2024-03-22T02:28:46.020 INFO:tasks.workunit.client.0.smithi103.stdout:2/621: rename d11/d25/d46/d48/da6/cb8 to d11/da0/cc8 0 2024-03-22T02:28:46.029 INFO:tasks.workunit.client.0.smithi103.stdout:2/622: unlink d11/d25/d46/d40/f41 0 2024-03-22T02:28:46.031 INFO:tasks.workunit.client.0.smithi103.stdout:2/623: mknod d11/d25/d63/d45/cc9 0 2024-03-22T02:28:46.034 INFO:tasks.workunit.client.0.smithi103.stdout:2/624: creat d11/d25/d46/fca x:0 0 0 2024-03-22T02:28:46.035 INFO:tasks.workunit.client.0.smithi103.stdout:2/625: readlink d11/d25/d63/d45/d47/lc1 0 2024-03-22T02:28:46.035 INFO:tasks.workunit.client.0.smithi103.stdout:2/626: stat lb 0 2024-03-22T02:28:46.035 INFO:tasks.workunit.client.0.smithi103.stdout:2/627: chown d11/d25/d63/cbe 0 1 2024-03-22T02:28:46.038 INFO:tasks.workunit.client.0.smithi103.stdout:2/628: getdents d11/d25/d46/d40/d90 0 2024-03-22T02:28:46.039 INFO:tasks.workunit.client.0.smithi103.stdout:2/629: read - d11/d19/d5d/fba zero size 2024-03-22T02:28:46.042 INFO:tasks.workunit.client.0.smithi103.stdout:2/630: creat d11/d25/d46/fcb x:0 0 0 2024-03-22T02:28:46.043 INFO:tasks.workunit.client.0.smithi103.stdout:2/631: write d11/d25/d46/f37 [5412316,110144] 0 2024-03-22T02:28:46.045 INFO:tasks.workunit.client.0.smithi103.stdout:2/632: symlink d11/d25/d63/d45/d47/d83/d78/db0/lcc 0 2024-03-22T02:28:46.048 INFO:tasks.workunit.client.0.smithi103.stdout:2/633: creat d11/d25/d63/d45/d47/dae/fcd x:0 0 0 2024-03-22T02:28:46.050 INFO:tasks.workunit.client.0.smithi103.stdout:2/634: rename lc to d11/d25/d63/d45/d47/d83/d78/db0/lce 0 2024-03-22T02:28:46.057 INFO:tasks.workunit.client.1.smithi122.stdout:1/87: sync 2024-03-22T02:28:46.058 INFO:tasks.workunit.client.1.smithi122.stdout:7/107: link d0/c13 d0/d4/c14 0 2024-03-22T02:28:46.060 INFO:tasks.workunit.client.1.smithi122.stdout:9/134: creat db/d23/d1f/f2b x:0 0 0 2024-03-22T02:28:46.061 INFO:tasks.workunit.client.1.smithi122.stdout:3/89: mknod d2/d9/c1c 0 2024-03-22T02:28:46.063 INFO:tasks.workunit.client.0.smithi103.stdout:2/635: dread d11/f13 [0,4194304] 0 2024-03-22T02:28:46.064 INFO:tasks.workunit.client.0.smithi103.stdout:2/636: write d11/d25/d63/d45/f79 [3917717,93217] 0 2024-03-22T02:28:46.064 INFO:tasks.workunit.client.1.smithi122.stdout:1/88: creat dc/f1b x:0 0 0 2024-03-22T02:28:46.065 INFO:tasks.workunit.client.0.smithi103.stdout:2/637: write d11/f30 [10320765,125767] 0 2024-03-22T02:28:46.065 INFO:tasks.workunit.client.1.smithi122.stdout:7/108: rmdir d0 39 2024-03-22T02:28:46.066 INFO:tasks.workunit.client.0.smithi103.stdout:2/638: write d11/d19/da7/fb1 [1770169,129172] 0 2024-03-22T02:28:46.071 INFO:tasks.workunit.client.1.smithi122.stdout:3/90: symlink d2/d9/dc/df/l1d 0 2024-03-22T02:28:46.072 INFO:tasks.workunit.client.0.smithi103.stdout:2/639: chown d11/d25/lab 1848747731 1 2024-03-22T02:28:46.072 INFO:tasks.workunit.client.0.smithi103.stdout:2/640: creat d11/da0/fcf x:0 0 0 2024-03-22T02:28:46.073 INFO:tasks.workunit.client.1.smithi122.stdout:5/113: creat d8/da/f21 x:0 0 0 2024-03-22T02:28:46.073 INFO:tasks.workunit.client.0.smithi103.stdout:2/641: rename d11/d25/d63/d45/d47/d83/fac to d11/d25/d63/d45/d47/d83/da3/fd0 0 2024-03-22T02:28:46.074 INFO:tasks.workunit.client.1.smithi122.stdout:9/135: link f8 db/d23/d1f/d21/d29/f2c 0 2024-03-22T02:28:46.075 INFO:tasks.workunit.client.1.smithi122.stdout:9/136: chown db/d1b/l1c 2049391470 1 2024-03-22T02:28:46.076 INFO:tasks.workunit.client.1.smithi122.stdout:3/91: creat d2/d9/dc/df/f1e x:0 0 0 2024-03-22T02:28:46.077 INFO:tasks.workunit.client.0.smithi103.stdout:2/642: rmdir d11/d25/d46/d48 39 2024-03-22T02:28:46.077 INFO:tasks.workunit.client.1.smithi122.stdout:7/109: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:46.079 INFO:tasks.workunit.client.1.smithi122.stdout:9/137: unlink db/ld 0 2024-03-22T02:28:46.079 INFO:tasks.workunit.client.1.smithi122.stdout:1/89: write dc/f1a [194994,39833] 0 2024-03-22T02:28:46.080 INFO:tasks.workunit.client.0.smithi103.stdout:2/643: readlink d11/l2f 0 2024-03-22T02:28:46.083 INFO:tasks.workunit.client.1.smithi122.stdout:1/90: creat dc/f1c x:0 0 0 2024-03-22T02:28:46.084 INFO:tasks.workunit.client.0.smithi103.stdout:2/644: rename d11/d19/d5d/c6a to d11/dc5/cd1 0 2024-03-22T02:28:46.087 INFO:tasks.workunit.client.0.smithi103.stdout:2/645: mknod d11/d25/d63/d45/d47/cd2 0 2024-03-22T02:28:46.090 INFO:tasks.workunit.client.1.smithi122.stdout:5/114: getdents d8 0 2024-03-22T02:28:46.090 INFO:tasks.workunit.client.0.smithi103.stdout:2/646: rename d11/d25/d63/d45/d47/d83/d78/db0 to d11/d19/da7/dd3 0 2024-03-22T02:28:46.092 INFO:tasks.workunit.client.1.smithi122.stdout:1/91: dwrite dc/f1b [0,4194304] 0 2024-03-22T02:28:46.094 INFO:tasks.workunit.client.1.smithi122.stdout:5/115: write d8/da/f10 [1276938,44965] 0 2024-03-22T02:28:46.094 INFO:tasks.workunit.client.1.smithi122.stdout:1/92: fdatasync dc/f17 0 2024-03-22T02:28:46.094 INFO:tasks.workunit.client.1.smithi122.stdout:7/110: rename d0/d4/c14 to d0/c15 0 2024-03-22T02:28:46.096 INFO:tasks.workunit.client.1.smithi122.stdout:5/116: chown d8/da 2110913 1 2024-03-22T02:28:46.097 INFO:tasks.workunit.client.0.smithi103.stdout:2/647: dwrite d11/da0/fcf [0,4194304] 0 2024-03-22T02:28:46.098 INFO:tasks.workunit.client.1.smithi122.stdout:5/117: chown d8/da 1 1 2024-03-22T02:28:46.100 INFO:tasks.workunit.client.1.smithi122.stdout:5/118: fdatasync d8/f20 0 2024-03-22T02:28:46.100 INFO:tasks.workunit.client.1.smithi122.stdout:7/111: creat d0/f16 x:0 0 0 2024-03-22T02:28:46.101 INFO:tasks.workunit.client.1.smithi122.stdout:9/138: dwrite db/d23/d1f/d21/d29/f2c [0,4194304] 0 2024-03-22T02:28:46.101 INFO:tasks.workunit.client.1.smithi122.stdout:3/92: dwrite d2/f8 [0,4194304] 0 2024-03-22T02:28:46.103 INFO:tasks.workunit.client.1.smithi122.stdout:1/93: getdents dc 0 2024-03-22T02:28:46.103 INFO:tasks.workunit.client.1.smithi122.stdout:5/119: chown d8/d1a/l1b 12137 1 2024-03-22T02:28:46.104 INFO:tasks.workunit.client.1.smithi122.stdout:1/94: dread - dc/f17 zero size 2024-03-22T02:28:46.105 INFO:tasks.workunit.client.0.smithi103.stdout:2/648: symlink d11/ld4 0 2024-03-22T02:28:46.107 INFO:tasks.workunit.client.0.smithi103.stdout:2/649: fsync d11/d19/d7e/f80 0 2024-03-22T02:28:46.107 INFO:tasks.workunit.client.1.smithi122.stdout:7/112: mkdir d0/d17 0 2024-03-22T02:28:46.108 INFO:tasks.workunit.client.1.smithi122.stdout:3/93: readlink d2/d9/l1b 0 2024-03-22T02:28:46.108 INFO:tasks.workunit.client.0.smithi103.stdout:2/650: write d11/da0/fcf [2640164,101327] 0 2024-03-22T02:28:46.109 INFO:tasks.workunit.client.1.smithi122.stdout:3/94: chown d2/d9/dc 1 1 2024-03-22T02:28:46.110 INFO:tasks.workunit.client.0.smithi103.stdout:2/651: write d11/d25/d63/d45/d47/d83/da3/fb9 [29801,106347] 0 2024-03-22T02:28:46.111 INFO:tasks.workunit.client.1.smithi122.stdout:1/95: fsync dc/f11 0 2024-03-22T02:28:46.111 INFO:tasks.workunit.client.1.smithi122.stdout:9/139: write db/df/f19 [865603,115011] 0 2024-03-22T02:28:46.113 INFO:tasks.workunit.client.1.smithi122.stdout:5/120: mkdir d8/d1e/d22 0 2024-03-22T02:28:46.115 INFO:tasks.workunit.client.0.smithi103.stdout:2/652: getdents d11/d25/d63/d45/d47/d83/d78 0 2024-03-22T02:28:46.117 INFO:tasks.workunit.client.1.smithi122.stdout:1/96: rename dc/f18 to dc/f1d 0 2024-03-22T02:28:46.118 INFO:tasks.workunit.client.0.smithi103.stdout:2/653: mkdir d11/d19/d4e/dd5 0 2024-03-22T02:28:46.118 INFO:tasks.workunit.client.1.smithi122.stdout:3/95: dwrite d2/f15 [0,4194304] 0 2024-03-22T02:28:46.119 INFO:tasks.workunit.client.1.smithi122.stdout:5/121: chown d8/da/l14 309621854 1 2024-03-22T02:28:46.119 INFO:tasks.workunit.client.1.smithi122.stdout:9/140: mknod db/d1b/d28/c2d 0 2024-03-22T02:28:46.120 INFO:tasks.workunit.client.0.smithi103.stdout:2/654: symlink d11/d19/d5d/ld6 0 2024-03-22T02:28:46.120 INFO:tasks.workunit.client.1.smithi122.stdout:5/122: fdatasync d8/f20 0 2024-03-22T02:28:46.120 INFO:tasks.workunit.client.1.smithi122.stdout:1/97: symlink dc/l1e 0 2024-03-22T02:28:46.122 INFO:tasks.workunit.client.0.smithi103.stdout:2/655: rmdir d11/d25/d63/d45/d47 39 2024-03-22T02:28:46.123 INFO:tasks.workunit.client.1.smithi122.stdout:3/96: symlink d2/d9/dc/df/l1f 0 2024-03-22T02:28:46.124 INFO:tasks.workunit.client.1.smithi122.stdout:9/141: rmdir db/d23/d1f/d21/d29 39 2024-03-22T02:28:46.125 INFO:tasks.workunit.client.1.smithi122.stdout:5/123: dread d8/d1a/f1c [0,4194304] 0 2024-03-22T02:28:46.128 INFO:tasks.workunit.client.1.smithi122.stdout:9/142: dread db/d1b/f22 [0,4194304] 0 2024-03-22T02:28:46.129 INFO:tasks.workunit.client.0.smithi103.stdout:2/656: dwrite d11/d19/d5d/f75 [0,4194304] 0 2024-03-22T02:28:46.129 INFO:tasks.workunit.client.1.smithi122.stdout:3/97: rename d2/d9/dc/df/f1e to d2/d9/dc/d17/f20 0 2024-03-22T02:28:46.131 INFO:tasks.workunit.client.1.smithi122.stdout:9/143: mknod db/d1b/c2e 0 2024-03-22T02:28:46.134 INFO:tasks.workunit.client.1.smithi122.stdout:5/124: rename d8/da/f21 to d8/d1e/f23 0 2024-03-22T02:28:46.136 INFO:tasks.workunit.client.1.smithi122.stdout:5/125: readlink d8/d1a/l1b 0 2024-03-22T02:28:46.137 INFO:tasks.workunit.client.1.smithi122.stdout:3/98: mkdir d2/d9/dc/d21 0 2024-03-22T02:28:46.141 INFO:tasks.workunit.client.1.smithi122.stdout:9/144: dwrite db/d23/f1a [0,4194304] 0 2024-03-22T02:28:46.143 INFO:tasks.workunit.client.1.smithi122.stdout:5/126: symlink d8/d1a/l24 0 2024-03-22T02:28:46.145 INFO:tasks.workunit.client.1.smithi122.stdout:9/145: mkdir db/d2f 0 2024-03-22T02:28:46.146 INFO:tasks.workunit.client.1.smithi122.stdout:5/127: symlink d8/d1e/l25 0 2024-03-22T02:28:46.146 INFO:tasks.workunit.client.1.smithi122.stdout:5/128: readlink d8/d1e/l25 0 2024-03-22T02:28:46.147 INFO:tasks.workunit.client.1.smithi122.stdout:9/146: mkdir db/d23/d1e/d30 0 2024-03-22T02:28:46.148 INFO:tasks.workunit.client.1.smithi122.stdout:5/129: chown d8/d1a/l1b 5729 1 2024-03-22T02:28:46.150 INFO:tasks.workunit.client.1.smithi122.stdout:9/147: rmdir db/d1b 39 2024-03-22T02:28:46.152 INFO:tasks.workunit.client.1.smithi122.stdout:1/98: getdents dc 0 2024-03-22T02:28:46.153 INFO:tasks.workunit.client.1.smithi122.stdout:5/130: creat d8/f26 x:0 0 0 2024-03-22T02:28:46.154 INFO:tasks.workunit.client.1.smithi122.stdout:1/99: write dc/f1b [2549921,112791] 0 2024-03-22T02:28:46.155 INFO:tasks.workunit.client.0.smithi103.stdout:2/657: dread d11/d19/da7/fb1 [0,4194304] 0 2024-03-22T02:28:46.157 INFO:tasks.workunit.client.1.smithi122.stdout:9/148: dread f8 [0,4194304] 0 2024-03-22T02:28:46.158 INFO:tasks.workunit.client.1.smithi122.stdout:9/149: write fa [6722821,80493] 0 2024-03-22T02:28:46.160 INFO:tasks.workunit.client.0.smithi103.stdout:2/658: creat d11/d25/d63/d45/d47/d83/fd7 x:0 0 0 2024-03-22T02:28:46.161 INFO:tasks.workunit.client.1.smithi122.stdout:3/99: rmdir d2/d9 39 2024-03-22T02:28:46.162 INFO:tasks.workunit.client.0.smithi103.stdout:2/659: truncate d11/d25/d46/d48/f9d 892394 0 2024-03-22T02:28:46.162 INFO:tasks.workunit.client.0.smithi103.stdout:2/660: stat d11/f13 0 2024-03-22T02:28:46.162 INFO:tasks.workunit.client.0.smithi103.stdout:2/661: chown la 53441 1 2024-03-22T02:28:46.163 INFO:tasks.workunit.client.1.smithi122.stdout:1/100: dwrite dc/fd [4194304,4194304] 0 2024-03-22T02:28:46.164 INFO:tasks.workunit.client.1.smithi122.stdout:3/100: dread - d2/d9/dc/f16 zero size 2024-03-22T02:28:46.165 INFO:tasks.workunit.client.0.smithi103.stdout:2/662: rename d11/d25/d63/d45/cb4 to d11/d25/d63/d45/d47/d83/d78/cd8 0 2024-03-22T02:28:46.166 INFO:tasks.workunit.client.1.smithi122.stdout:9/150: dread db/d23/d1f/d21/d29/f2c [0,4194304] 0 2024-03-22T02:28:46.167 INFO:tasks.workunit.client.0.smithi103.stdout:2/663: mkdir d11/d19/da7/dd3/dd9 0 2024-03-22T02:28:46.167 INFO:tasks.workunit.client.1.smithi122.stdout:9/151: write f7 [247689,66831] 0 2024-03-22T02:28:46.168 INFO:tasks.workunit.client.1.smithi122.stdout:3/101: mknod d2/d9/dc/d17/c22 0 2024-03-22T02:28:46.168 INFO:tasks.workunit.client.1.smithi122.stdout:3/102: chown d2 341810016 1 2024-03-22T02:28:46.170 INFO:tasks.workunit.client.0.smithi103.stdout:2/664: dread d11/d25/d46/d48/f9d [0,4194304] 0 2024-03-22T02:28:46.170 INFO:tasks.workunit.client.0.smithi103.stdout:2/665: chown d11/d25/d46/d40 76074 1 2024-03-22T02:28:46.172 INFO:tasks.workunit.client.1.smithi122.stdout:5/131: write d8/da/f16 [1228904,114139] 0 2024-03-22T02:28:46.173 INFO:tasks.workunit.client.1.smithi122.stdout:9/152: symlink db/l31 0 2024-03-22T02:28:46.174 INFO:tasks.workunit.client.1.smithi122.stdout:3/103: creat d2/d9/dc/d21/f23 x:0 0 0 2024-03-22T02:28:46.175 INFO:tasks.workunit.client.0.smithi103.stdout:2/666: link d11/d25/d46/fc2 d11/d25/d63/d45/d47/d83/da3/fda 0 2024-03-22T02:28:46.176 INFO:tasks.workunit.client.1.smithi122.stdout:9/153: creat db/d23/d1f/d21/d29/f32 x:0 0 0 2024-03-22T02:28:46.177 INFO:tasks.workunit.client.1.smithi122.stdout:3/104: mkdir d2/d9/d24 0 2024-03-22T02:28:46.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.176+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e8410fa20 con 0x7f5e7006b8c0 2024-03-22T02:28:46.178 INFO:tasks.workunit.client.0.smithi103.stdout:2/667: symlink d11/d25/d63/d45/d47/dae/ldb 0 2024-03-22T02:28:46.178 INFO:tasks.workunit.client.0.smithi103.stdout:2/668: rename d11/d19 to d11/d19/d4e/ddc 22 2024-03-22T02:28:46.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.178+0000 7f5e7a7fc700 1 -- 172.21.15.103:0/3018643661 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f5e8410fa20 con 0x7f5e7006b8c0 2024-03-22T02:28:46.179 INFO:tasks.workunit.client.1.smithi122.stdout:5/132: mknod d8/d1e/c27 0 2024-03-22T02:28:46.180 INFO:tasks.workunit.client.1.smithi122.stdout:9/154: read - db/d23/d1f/d21/d29/f32 zero size 2024-03-22T02:28:46.180 INFO:tasks.workunit.client.1.smithi122.stdout:3/105: dread - d2/d9/dc/f16 zero size 2024-03-22T02:28:46.180 INFO:tasks.workunit.client.1.smithi122.stdout:5/133: write d8/f17 [2737853,22214] 0 2024-03-22T02:28:46.180 INFO:tasks.workunit.client.0.smithi103.stdout:2/669: mkdir d11/d25/d63/d45/ddd 0 2024-03-22T02:28:46.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 msgr2=0x7f5e7006dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 0x7f5e7006dd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b0d0 tx=0x7f5e74009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 msgr2=0x7f5e8410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000eb70 tx=0x7f5e8000b040 comp rx=0 tx=0).stop 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 shutdown_connections 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5e7006b8c0 0x7f5e7006dd70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e840fce40 0x7f5e8410d9c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 --2- 172.21.15.103:0/3018643661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e840fe040 0x7f5e8410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 >> 172.21.15.103:0/3018643661 conn(0x7f5e840f8570 msgr2=0x7f5e840f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:46.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 shutdown_connections 2024-03-22T02:28:46.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.181+0000 7f5e8c1f6700 1 -- 172.21.15.103:0/3018643661 wait complete. 2024-03-22T02:28:46.184 INFO:tasks.workunit.client.1.smithi122.stdout:3/106: write d2/d9/dc/d17/f20 [47350,105218] 0 2024-03-22T02:28:46.185 INFO:tasks.workunit.client.1.smithi122.stdout:5/134: fdatasync d8/d1e/f23 0 2024-03-22T02:28:46.185 INFO:tasks.workunit.client.1.smithi122.stdout:9/155: dread db/d23/d1f/d21/d29/f2c [0,4194304] 0 2024-03-22T02:28:46.186 INFO:tasks.workunit.client.1.smithi122.stdout:9/156: write fa [2117489,22547] 0 2024-03-22T02:28:46.187 INFO:tasks.workunit.client.1.smithi122.stdout:1/101: dread dc/f1d [0,4194304] 0 2024-03-22T02:28:46.188 INFO:tasks.workunit.client.1.smithi122.stdout:5/135: write d8/d1a/f1c [4347566,107074] 0 2024-03-22T02:28:46.190 INFO:tasks.workunit.client.1.smithi122.stdout:3/107: creat d2/f25 x:0 0 0 2024-03-22T02:28:46.191 INFO:tasks.workunit.client.1.smithi122.stdout:3/108: chown d2/d9/dc/d21/f23 0 1 2024-03-22T02:28:46.191 INFO:tasks.workunit.client.1.smithi122.stdout:1/102: symlink dc/l1f 0 2024-03-22T02:28:46.192 INFO:tasks.workunit.client.1.smithi122.stdout:1/103: readlink dc/l1e 0 2024-03-22T02:28:46.192 INFO:tasks.workunit.client.0.smithi103.stdout:2/670: write d11/d19/d7e/f80 [231318,23633] 0 2024-03-22T02:28:46.193 INFO:tasks.workunit.client.1.smithi122.stdout:9/157: symlink db/l33 0 2024-03-22T02:28:46.194 INFO:tasks.workunit.client.1.smithi122.stdout:3/109: creat d2/d9/dc/d17/f26 x:0 0 0 2024-03-22T02:28:46.195 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:28:46.195 INFO:tasks.workunit.client.0.smithi103.stdout:2/671: chown d11/d25/d63/d45/d47/d83/c4b 0 1 2024-03-22T02:28:46.195 INFO:tasks.workunit.client.1.smithi122.stdout:5/136: mkdir d8/da/d28 0 2024-03-22T02:28:46.197 INFO:tasks.workunit.client.0.smithi103.stdout:2/672: creat d11/d25/d46/d48/fde x:0 0 0 2024-03-22T02:28:46.198 INFO:tasks.workunit.client.1.smithi122.stdout:9/158: dread f7 [0,4194304] 0 2024-03-22T02:28:46.198 INFO:tasks.workunit.client.1.smithi122.stdout:3/110: unlink d2/d9/dc/d17/f26 0 2024-03-22T02:28:46.198 INFO:tasks.workunit.client.1.smithi122.stdout:2/109: sync 2024-03-22T02:28:46.198 INFO:tasks.workunit.client.0.smithi103.stdout:2/673: symlink d11/da0/ldf 0 2024-03-22T02:28:46.199 INFO:tasks.workunit.client.1.smithi122.stdout:3/111: rename d2/d9/dc to d2/d9/dc/d21/d27 22 2024-03-22T02:28:46.199 INFO:tasks.workunit.client.0.smithi103.stdout:2/674: write d11/d25/d46/f9c [1298707,53057] 0 2024-03-22T02:28:46.200 INFO:tasks.workunit.client.1.smithi122.stdout:9/159: creat db/d23/d1e/f34 x:0 0 0 2024-03-22T02:28:46.200 INFO:tasks.workunit.client.1.smithi122.stdout:1/104: link dc/c12 dc/c20 0 2024-03-22T02:28:46.203 INFO:tasks.workunit.client.0.smithi103.stdout:2/675: link d11/d25/d63/d45/d47/l54 d11/d25/d63/le0 0 2024-03-22T02:28:46.205 INFO:tasks.workunit.client.0.smithi103.stdout:2/676: rename d11/d25/l2c to d11/d19/da7/dd3/le1 0 2024-03-22T02:28:46.205 INFO:tasks.workunit.client.1.smithi122.stdout:9/160: creat db/d23/d1e/d30/f35 x:0 0 0 2024-03-22T02:28:46.205 INFO:tasks.workunit.client.1.smithi122.stdout:5/137: link d8/da/l14 d8/l29 0 2024-03-22T02:28:46.206 INFO:tasks.workunit.client.0.smithi103.stdout:2/677: write f10 [1598874,17273] 0 2024-03-22T02:28:46.207 INFO:tasks.workunit.client.1.smithi122.stdout:2/110: link df/d1f/c27 df/d1f/d2a/c2c 0 2024-03-22T02:28:46.208 INFO:tasks.workunit.client.1.smithi122.stdout:9/161: mknod db/c36 0 2024-03-22T02:28:46.210 INFO:tasks.workunit.client.1.smithi122.stdout:2/111: chown fd 1 1 2024-03-22T02:28:46.210 INFO:tasks.workunit.client.1.smithi122.stdout:2/112: fdatasync df/f2b 0 2024-03-22T02:28:46.211 INFO:tasks.workunit.client.1.smithi122.stdout:5/138: symlink d8/l2a 0 2024-03-22T02:28:46.211 INFO:tasks.workunit.client.1.smithi122.stdout:2/113: chown df/c1b 29 1 2024-03-22T02:28:46.213 INFO:tasks.workunit.client.1.smithi122.stdout:9/162: symlink db/d2f/l37 0 2024-03-22T02:28:46.215 INFO:tasks.workunit.client.0.smithi103.stdout:2/678: dwrite d11/d25/d63/d45/f8b [0,4194304] 0 2024-03-22T02:28:46.215 INFO:tasks.workunit.client.1.smithi122.stdout:2/114: creat df/d1f/f2d x:0 0 0 2024-03-22T02:28:46.216 INFO:tasks.workunit.client.1.smithi122.stdout:2/115: chown df/d1f/d23 173 1 2024-03-22T02:28:46.216 INFO:tasks.workunit.client.0.smithi103.stdout:2/679: dread - d11/d25/d46/d48/da6/fb3 zero size 2024-03-22T02:28:46.217 INFO:tasks.workunit.client.1.smithi122.stdout:5/139: dread d8/da/f10 [0,4194304] 0 2024-03-22T02:28:46.219 INFO:tasks.workunit.client.1.smithi122.stdout:9/163: unlink db/d23/d1f/d21/d25/l27 0 2024-03-22T02:28:46.222 INFO:tasks.workunit.client.0.smithi103.stdout:2/680: dwrite d11/d19/d5d/fba [0,4194304] 0 2024-03-22T02:28:46.222 INFO:tasks.workunit.client.1.smithi122.stdout:2/116: unlink df/f18 0 2024-03-22T02:28:46.225 INFO:tasks.workunit.client.0.smithi103.stdout:2/681: creat d11/d25/d63/d45/d47/d83/d78/fe2 x:0 0 0 2024-03-22T02:28:46.226 INFO:tasks.workunit.client.1.smithi122.stdout:2/117: dread fd [0,4194304] 0 2024-03-22T02:28:46.226 INFO:tasks.workunit.client.0.smithi103.stdout:2/682: dread - d11/d25/d46/fcb zero size 2024-03-22T02:28:46.228 INFO:tasks.workunit.client.0.smithi103.stdout:2/683: creat d11/d25/d63/d45/d47/dae/fe3 x:0 0 0 2024-03-22T02:28:46.229 INFO:tasks.workunit.client.0.smithi103.stdout:2/684: write d11/d19/d5d/f8d [6738773,13169] 0 2024-03-22T02:28:46.230 INFO:tasks.workunit.client.1.smithi122.stdout:5/140: creat d8/da/d28/f2b x:0 0 0 2024-03-22T02:28:46.230 INFO:tasks.workunit.client.0.smithi103.stdout:2/685: write d11/d25/d46/fca [866349,46094] 0 2024-03-22T02:28:46.231 INFO:tasks.workunit.client.1.smithi122.stdout:1/105: truncate dc/f1d 3205391 0 2024-03-22T02:28:46.231 INFO:tasks.workunit.client.1.smithi122.stdout:1/106: chown dc/f17 361786 1 2024-03-22T02:28:46.232 INFO:tasks.workunit.client.1.smithi122.stdout:1/107: dread - dc/f1c zero size 2024-03-22T02:28:46.233 INFO:tasks.workunit.client.0.smithi103.stdout:2/686: link d11/d25/d46/cbc d11/d25/d63/d45/d47/d83/da3/ce4 0 2024-03-22T02:28:46.234 INFO:tasks.workunit.client.1.smithi122.stdout:9/164: rename f5 to db/d23/d1f/d21/d29/f38 0 2024-03-22T02:28:46.234 INFO:tasks.workunit.client.1.smithi122.stdout:9/165: chown db/c36 114898 1 2024-03-22T02:28:46.235 INFO:tasks.workunit.client.1.smithi122.stdout:5/141: creat d8/da/d28/f2c x:0 0 0 2024-03-22T02:28:46.236 INFO:tasks.workunit.client.1.smithi122.stdout:2/118: mknod df/d1f/d2a/c2e 0 2024-03-22T02:28:46.237 INFO:tasks.workunit.client.1.smithi122.stdout:2/119: write f6 [509212,97049] 0 2024-03-22T02:28:46.237 INFO:tasks.workunit.client.1.smithi122.stdout:9/166: mkdir db/d23/d39 0 2024-03-22T02:28:46.239 INFO:tasks.workunit.client.1.smithi122.stdout:2/120: dread - df/d1f/f21 zero size 2024-03-22T02:28:46.239 INFO:tasks.workunit.client.1.smithi122.stdout:9/167: stat db/d1b/d28 0 2024-03-22T02:28:46.240 INFO:tasks.workunit.client.1.smithi122.stdout:1/108: dwrite dc/f11 [0,4194304] 0 2024-03-22T02:28:46.240 INFO:tasks.workunit.client.1.smithi122.stdout:5/142: mknod d8/d1e/d22/c2d 0 2024-03-22T02:28:46.244 INFO:tasks.workunit.client.1.smithi122.stdout:1/109: write dc/f10 [491278,36128] 0 2024-03-22T02:28:46.245 INFO:tasks.workunit.client.1.smithi122.stdout:9/168: creat db/d23/f3a x:0 0 0 2024-03-22T02:28:46.246 INFO:tasks.workunit.client.1.smithi122.stdout:5/143: mkdir d8/d1e/d2e 0 2024-03-22T02:28:46.247 INFO:tasks.workunit.client.0.smithi103.stdout:9/670: sync 2024-03-22T02:28:46.248 INFO:tasks.workunit.client.1.smithi122.stdout:1/110: creat dc/f21 x:0 0 0 2024-03-22T02:28:46.248 INFO:tasks.workunit.client.1.smithi122.stdout:1/111: stat l4 0 2024-03-22T02:28:46.249 INFO:tasks.workunit.client.0.smithi103.stdout:2/687: symlink d11/d25/d63/d45/d47/d83/le5 0 2024-03-22T02:28:46.250 INFO:tasks.workunit.client.0.smithi103.stdout:9/671: chown d2/da/d18/d1e/d34/l5f 19 1 2024-03-22T02:28:46.251 INFO:tasks.workunit.client.1.smithi122.stdout:1/112: creat dc/f22 x:0 0 0 2024-03-22T02:28:46.252 INFO:tasks.workunit.client.1.smithi122.stdout:5/144: creat d8/d1a/f2f x:0 0 0 2024-03-22T02:28:46.254 INFO:tasks.workunit.client.0.smithi103.stdout:9/672: mkdir d2/da/d18/dc4/dce 0 2024-03-22T02:28:46.256 INFO:tasks.workunit.client.0.smithi103.stdout:9/673: write d2/d10/d43/d7e/fc6 [812618,120304] 0 2024-03-22T02:28:46.256 INFO:tasks.workunit.client.1.smithi122.stdout:9/169: link db/d2f/l37 db/d23/l3b 0 2024-03-22T02:28:46.256 INFO:tasks.workunit.client.0.smithi103.stdout:2/688: link d11/d25/d63/d45/d47/d83/da3/fd0 d11/dc5/fe6 0 2024-03-22T02:28:46.257 INFO:tasks.workunit.client.0.smithi103.stdout:9/674: dread - d2/d5/d4c/d56/fbc zero size 2024-03-22T02:28:46.260 INFO:tasks.workunit.client.0.smithi103.stdout:9/675: truncate d2/da/d18/d1e/d34/d3d/d4d/f8f 716661 0 2024-03-22T02:28:46.261 INFO:tasks.workunit.client.1.smithi122.stdout:9/170: unlink f6 0 2024-03-22T02:28:46.261 INFO:tasks.workunit.client.0.smithi103.stdout:2/689: dread d11/d25/d63/d45/d47/f85 [0,4194304] 0 2024-03-22T02:28:46.262 INFO:tasks.workunit.client.1.smithi122.stdout:5/145: creat d8/f30 x:0 0 0 2024-03-22T02:28:46.262 INFO:tasks.workunit.client.0.smithi103.stdout:9/676: mknod d2/d10/d43/d7e/ccf 0 2024-03-22T02:28:46.263 INFO:tasks.workunit.client.1.smithi122.stdout:9/171: dread db/df/f19 [0,4194304] 0 2024-03-22T02:28:46.263 INFO:tasks.workunit.client.1.smithi122.stdout:9/172: stat db/d1b/f22 0 2024-03-22T02:28:46.264 INFO:tasks.workunit.client.0.smithi103.stdout:9/677: stat d2/d10/d43/d7e/d96/d9d/lb4 0 2024-03-22T02:28:46.264 INFO:tasks.workunit.client.0.smithi103.stdout:2/690: creat d11/d25/d63/fe7 x:0 0 0 2024-03-22T02:28:46.265 INFO:tasks.workunit.client.0.smithi103.stdout:2/691: mkdir d11/de8 0 2024-03-22T02:28:46.266 INFO:tasks.workunit.client.1.smithi122.stdout:5/146: mkdir d8/d1e/d31 0 2024-03-22T02:28:46.266 INFO:tasks.workunit.client.0.smithi103.stdout:2/692: fdatasync f10 0 2024-03-22T02:28:46.266 INFO:tasks.workunit.client.0.smithi103.stdout:9/678: getdents d2/da/d18/d1e/d77/d8b/dae/dcb 0 2024-03-22T02:28:46.267 INFO:tasks.workunit.client.0.smithi103.stdout:2/693: dread - d11/d25/d63/f65 zero size 2024-03-22T02:28:46.269 INFO:tasks.workunit.client.0.smithi103.stdout:2/694: mkdir d11/d19/d4e/de9 0 2024-03-22T02:28:46.269 INFO:tasks.workunit.client.0.smithi103.stdout:2/695: chown d11/d25/d46/d40/f6f 657147 1 2024-03-22T02:28:46.270 INFO:tasks.workunit.client.0.smithi103.stdout:9/679: getdents d2/da 0 2024-03-22T02:28:46.274 INFO:tasks.workunit.client.1.smithi122.stdout:2/121: dwrite df/f11 [4194304,4194304] 0 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.1.smithi122.stdout:9/173: link db/d2f/l37 db/d23/d1e/d30/l3c 0 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.1.smithi122.stdout:1/113: dread dc/ff [0,4194304] 0 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.0.smithi103.stdout:2/696: read d11/d25/d46/d48/f9d [133933,29900] 0 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.0.smithi103.stdout:9/680: rmdir d2/d10 39 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.0.smithi103.stdout:9/681: chown d2/d10/d43/d9b 1 1 2024-03-22T02:28:46.275 INFO:tasks.workunit.client.0.smithi103.stdout:2/697: creat d11/d25/d63/d45/d47/dae/fea x:0 0 0 2024-03-22T02:28:46.276 INFO:tasks.workunit.client.0.smithi103.stdout:2/698: write d11/d25/d63/d45/d47/dae/fe3 [815192,18555] 0 2024-03-22T02:28:46.277 INFO:tasks.workunit.client.0.smithi103.stdout:9/682: unlink d2/d5/c32 0 2024-03-22T02:28:46.278 INFO:tasks.workunit.client.1.smithi122.stdout:5/147: dwrite d8/da/d28/f2b [0,4194304] 0 2024-03-22T02:28:46.278 INFO:tasks.workunit.client.0.smithi103.stdout:2/699: write d11/d25/d46/fcb [363851,59039] 0 2024-03-22T02:28:46.279 INFO:tasks.workunit.client.0.smithi103.stdout:9/683: readlink d2/d5/d4c/l93 0 2024-03-22T02:28:46.280 INFO:tasks.workunit.client.1.smithi122.stdout:2/122: creat df/d1f/d2a/f2f x:0 0 0 2024-03-22T02:28:46.281 INFO:tasks.workunit.client.0.smithi103.stdout:2/700: dread - d11/d25/d63/d45/d47/dae/fcd zero size 2024-03-22T02:28:46.285 INFO:tasks.workunit.client.1.smithi122.stdout:9/174: mkdir db/d23/d1f/d21/d3d 0 2024-03-22T02:28:46.285 INFO:tasks.workunit.client.0.smithi103.stdout:2/701: creat d11/d25/d46/d40/feb x:0 0 0 2024-03-22T02:28:46.286 INFO:tasks.workunit.client.1.smithi122.stdout:5/148: creat d8/d1e/d2e/f32 x:0 0 0 2024-03-22T02:28:46.288 INFO:tasks.workunit.client.1.smithi122.stdout:2/123: dwrite df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:46.289 INFO:tasks.workunit.client.0.smithi103.stdout:9/684: rename d2/da/d18/d1e/d77/d8b/f26 to d2/da/d18/dc4/dce/fd0 0 2024-03-22T02:28:46.291 INFO:tasks.workunit.client.1.smithi122.stdout:9/175: mkdir db/df/d3e 0 2024-03-22T02:28:46.291 INFO:tasks.workunit.client.1.smithi122.stdout:9/176: fdatasync fa 0 2024-03-22T02:28:46.293 INFO:tasks.workunit.client.1.smithi122.stdout:2/124: mknod df/d1f/d23/c30 0 2024-03-22T02:28:46.293 INFO:tasks.workunit.client.1.smithi122.stdout:9/177: write db/df/f15 [342179,97779] 0 2024-03-22T02:28:46.295 INFO:tasks.workunit.client.1.smithi122.stdout:5/149: dread d8/f17 [0,4194304] 0 2024-03-22T02:28:46.296 INFO:tasks.workunit.client.0.smithi103.stdout:2/702: dwrite d11/d25/d63/d45/d47/d83/fd7 [0,4194304] 0 2024-03-22T02:28:46.296 INFO:tasks.workunit.client.1.smithi122.stdout:2/125: chown df/f10 799 1 2024-03-22T02:28:46.297 INFO:tasks.workunit.client.1.smithi122.stdout:9/178: creat db/d23/d1f/d21/d29/f3f x:0 0 0 2024-03-22T02:28:46.297 INFO:tasks.workunit.client.0.smithi103.stdout:2/703: stat d11/d25/d46/fcb 0 2024-03-22T02:28:46.300 INFO:tasks.workunit.client.1.smithi122.stdout:2/126: mknod df/d1f/d2a/c31 0 2024-03-22T02:28:46.300 INFO:tasks.workunit.client.0.smithi103.stdout:2/704: creat d11/d25/d46/db6/fec x:0 0 0 2024-03-22T02:28:46.300 INFO:tasks.workunit.client.0.smithi103.stdout:9/685: dwrite d2/da/d18/d1e/fcc [0,4194304] 0 2024-03-22T02:28:46.301 INFO:tasks.workunit.client.1.smithi122.stdout:2/127: write df/f14 [1886581,2171] 0 2024-03-22T02:28:46.303 INFO:tasks.workunit.client.1.smithi122.stdout:9/179: unlink db/df/f19 0 2024-03-22T02:28:46.306 INFO:tasks.workunit.client.1.smithi122.stdout:9/180: write db/df/f15 [2120826,7677] 0 2024-03-22T02:28:46.308 INFO:tasks.workunit.client.0.smithi103.stdout:2/705: rename d11/f18 to d11/d25/d46/d40/fed 0 2024-03-22T02:28:46.310 INFO:tasks.workunit.client.0.smithi103.stdout:9/686: dwrite d2/d10/d43/d7e/d96/d9d/f98 [0,4194304] 0 2024-03-22T02:28:46.310 INFO:tasks.workunit.client.1.smithi122.stdout:2/128: mkdir df/d32 0 2024-03-22T02:28:46.311 INFO:tasks.workunit.client.0.smithi103.stdout:2/706: readlink d11/d25/d63/lbf 0 2024-03-22T02:28:46.312 INFO:tasks.workunit.client.1.smithi122.stdout:1/114: dwrite dc/fe [0,4194304] 0 2024-03-22T02:28:46.314 INFO:tasks.workunit.client.1.smithi122.stdout:5/150: link d8/d1e/d2e/f32 d8/f33 0 2024-03-22T02:28:46.315 INFO:tasks.workunit.client.1.smithi122.stdout:1/115: read dc/f11 [1538501,26835] 0 2024-03-22T02:28:46.315 INFO:tasks.workunit.client.1.smithi122.stdout:9/181: mknod db/d1b/d28/c40 0 2024-03-22T02:28:46.317 INFO:tasks.workunit.client.1.smithi122.stdout:2/129: mkdir df/d1f/d33 0 2024-03-22T02:28:46.318 INFO:tasks.workunit.client.0.smithi103.stdout:2/707: dwrite d11/d25/d63/d45/d47/f72 [0,4194304] 0 2024-03-22T02:28:46.320 INFO:tasks.workunit.client.1.smithi122.stdout:1/116: symlink dc/l23 0 2024-03-22T02:28:46.323 INFO:tasks.workunit.client.1.smithi122.stdout:9/182: rename db/d23/d39 to db/d23/d1f/d21/d25/d41 0 2024-03-22T02:28:46.327 INFO:tasks.workunit.client.1.smithi122.stdout:1/117: creat dc/f24 x:0 0 0 2024-03-22T02:28:46.329 INFO:tasks.workunit.client.1.smithi122.stdout:2/130: mkdir df/d1f/d33/d34 0 2024-03-22T02:28:46.333 INFO:tasks.workunit.client.1.smithi122.stdout:9/183: dread db/f13 [0,4194304] 0 2024-03-22T02:28:46.334 INFO:tasks.workunit.client.1.smithi122.stdout:2/131: truncate df/f13 1395446 0 2024-03-22T02:28:46.334 INFO:tasks.workunit.client.1.smithi122.stdout:5/151: dwrite d8/da/f10 [4194304,4194304] 0 2024-03-22T02:28:46.336 INFO:tasks.workunit.client.1.smithi122.stdout:9/184: chown db/d23/l12 8262013 1 2024-03-22T02:28:46.337 INFO:tasks.workunit.client.1.smithi122.stdout:2/132: chown df/d1f/d23/c30 103 1 2024-03-22T02:28:46.337 INFO:tasks.workunit.client.1.smithi122.stdout:1/118: dread dc/f1d [0,4194304] 0 2024-03-22T02:28:46.338 INFO:tasks.workunit.client.1.smithi122.stdout:5/152: creat d8/d1a/f34 x:0 0 0 2024-03-22T02:28:46.339 INFO:tasks.workunit.client.1.smithi122.stdout:2/133: symlink df/l35 0 2024-03-22T02:28:46.339 INFO:tasks.workunit.client.1.smithi122.stdout:5/153: chown d8/d1a 45453979 1 2024-03-22T02:28:46.339 INFO:tasks.workunit.client.1.smithi122.stdout:1/119: creat dc/f25 x:0 0 0 2024-03-22T02:28:46.341 INFO:tasks.workunit.client.1.smithi122.stdout:2/134: creat df/f36 x:0 0 0 2024-03-22T02:28:46.341 INFO:tasks.workunit.client.1.smithi122.stdout:1/120: chown dc/f21 109782 1 2024-03-22T02:28:46.342 INFO:tasks.workunit.client.1.smithi122.stdout:1/121: chown dc/f21 2 1 2024-03-22T02:28:46.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 -- 172.21.15.103:0/2466783459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 msgr2=0x7f28a40fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 --2- 172.21.15.103:0/2466783459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a40fd400 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2898004940 tx=0x7f2898009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:46.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 -- 172.21.15.103:0/2466783459 shutdown_connections 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 --2- 172.21.15.103:0/2466783459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a40fd400 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 --2- 172.21.15.103:0/2466783459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a40fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 -- 172.21.15.103:0/2466783459 >> 172.21.15.103:0/2466783459 conn(0x7f28a40f7440 msgr2=0x7f28a40f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 -- 172.21.15.103:0/2466783459 shutdown_connections 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.340+0000 7f28a9436700 1 -- 172.21.15.103:0/2466783459 wait complete. 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.341+0000 7f28a9436700 1 Processor -- start 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.341+0000 7f28a9436700 1 -- start start 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a9436700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a4110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a9436700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a9436700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4066c40 con 0x7f28a40fcf90 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a9436700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4066d80 con 0x7f28a40fbce0 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39988/0 (socket says 172.21.15.103:39988) 2024-03-22T02:28:46.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 -- 172.21.15.103:0/2182045974 learned_addr learned my addr 172.21.15.103:0/2182045974 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a2ffd700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a4110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 -- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 msgr2=0x7f28a4110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a4110700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a27fc700 1 -- 172.21.15.103:0/2182045974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2898005040 con 0x7f28a40fcf90 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.342+0000 7f28a2ffd700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a4110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f28a27fc700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f289800b3b0 tx=0x7f2898009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289800e040 con 0x7f28a40fcf90 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2898004290 con 0x7f28a40fcf90 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f289801c630 con 0x7f28a40fcf90 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a41131f0 con 0x7f28a40fcf90 2024-03-22T02:28:46.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.343+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a410c8b0 con 0x7f28a40fcf90 2024-03-22T02:28:46.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.344+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a405f5f0 con 0x7f28a40fcf90 2024-03-22T02:28:46.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.345+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f289800bbf0 con 0x7f28a40fcf90 2024-03-22T02:28:46.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.345+0000 7f288b7fe700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 0x7f288c06dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.345+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f289808bb00 con 0x7f28a40fcf90 2024-03-22T02:28:46.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.346+0000 7f28a2ffd700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 0x7f288c06dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.346+0000 7f28a2ffd700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 0x7f288c06dd70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f28940041d0 tx=0x7f2894009040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:46.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.350+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f28980555d0 con 0x7f28a40fcf90 2024-03-22T02:28:46.372 INFO:tasks.workunit.client.1.smithi122.stdout:6/206: sync 2024-03-22T02:28:46.389 INFO:tasks.workunit.client.0.smithi103.stdout:2/708: getdents d11/d19/d4e/dd5 0 2024-03-22T02:28:46.394 INFO:tasks.workunit.client.0.smithi103.stdout:9/687: dwrite d2/d10/d43/d7e/fc8 [0,4194304] 0 2024-03-22T02:28:46.421 INFO:tasks.workunit.client.1.smithi122.stdout:9/185: creat db/d23/d1f/d21/f42 x:0 0 0 2024-03-22T02:28:46.422 INFO:tasks.workunit.client.1.smithi122.stdout:2/135: creat df/f37 x:0 0 0 2024-03-22T02:28:46.423 INFO:tasks.workunit.client.1.smithi122.stdout:1/122: chown dc/c12 96763 1 2024-03-22T02:28:46.424 INFO:tasks.workunit.client.1.smithi122.stdout:1/123: chown l4 635601 1 2024-03-22T02:28:46.425 INFO:tasks.workunit.client.1.smithi122.stdout:6/207: creat d7/de/dd/d23/f48 x:0 0 0 2024-03-22T02:28:46.428 INFO:tasks.workunit.client.1.smithi122.stdout:6/208: dread d7/f1b [0,4194304] 0 2024-03-22T02:28:46.449 INFO:tasks.workunit.client.1.smithi122.stdout:1/124: dwrite dc/f1c [0,4194304] 0 2024-03-22T02:28:46.451 INFO:tasks.workunit.client.1.smithi122.stdout:9/186: write f7 [4230581,2212] 0 2024-03-22T02:28:46.455 INFO:tasks.workunit.client.0.smithi103.stdout:2/709: dwrite d11/d25/d46/d48/f6e [0,4194304] 0 2024-03-22T02:28:46.458 INFO:tasks.workunit.client.0.smithi103.stdout:9/688: write d2/d10/d43/d7e/da4/dcd/f4f [326186,23315] 0 2024-03-22T02:28:46.458 INFO:tasks.workunit.client.1.smithi122.stdout:6/209: write f3 [3840466,130710] 0 2024-03-22T02:28:46.462 INFO:tasks.workunit.client.1.smithi122.stdout:9/187: dwrite db/d23/f3a [0,4194304] 0 2024-03-22T02:28:46.463 INFO:tasks.workunit.client.0.smithi103.stdout:9/689: creat d2/d10/d43/d7e/fd1 x:0 0 0 2024-03-22T02:28:46.465 INFO:tasks.workunit.client.0.smithi103.stdout:9/690: write d2/d5/d4c/d56/f95 [3263424,116670] 0 2024-03-22T02:28:46.466 INFO:tasks.workunit.client.0.smithi103.stdout:2/710: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:46.466 INFO:tasks.workunit.client.1.smithi122.stdout:6/210: mknod d7/de/d27/c49 0 2024-03-22T02:28:46.467 INFO:tasks.workunit.client.0.smithi103.stdout:9/691: dread - d2/d5/d4c/d56/fbc zero size 2024-03-22T02:28:46.469 INFO:tasks.workunit.client.0.smithi103.stdout:9/692: dread - d2/da/d18/d1e/d34/d3d/d61/f87 zero size 2024-03-22T02:28:46.469 INFO:tasks.workunit.client.1.smithi122.stdout:2/136: link l1 df/d1f/l38 0 2024-03-22T02:28:46.470 INFO:tasks.workunit.client.0.smithi103.stdout:2/711: mknod d11/d25/d63/d45/d47/cee 0 2024-03-22T02:28:46.471 INFO:tasks.workunit.client.1.smithi122.stdout:2/137: chown df/d1f/d2a/c2e 3 1 2024-03-22T02:28:46.472 INFO:tasks.workunit.client.0.smithi103.stdout:9/693: symlink d2/d10/ld2 0 2024-03-22T02:28:46.473 INFO:tasks.workunit.client.1.smithi122.stdout:9/188: mknod db/d23/d1f/d21/d25/c43 0 2024-03-22T02:28:46.474 INFO:tasks.workunit.client.0.smithi103.stdout:2/712: mknod d11/d25/d63/d45/d47/d83/cef 0 2024-03-22T02:28:46.479 INFO:tasks.workunit.client.1.smithi122.stdout:6/211: dwrite d7/de/dd/f42 [0,4194304] 0 2024-03-22T02:28:46.479 INFO:tasks.workunit.client.0.smithi103.stdout:9/694: truncate d2/f3b 608922 0 2024-03-22T02:28:46.482 INFO:tasks.workunit.client.0.smithi103.stdout:2/713: dwrite d11/d25/d46/fcb [0,4194304] 0 2024-03-22T02:28:46.483 INFO:tasks.workunit.client.0.smithi103.stdout:2/714: stat d11 0 2024-03-22T02:28:46.484 INFO:tasks.workunit.client.0.smithi103.stdout:2/715: read d11/dc5/fe6 [3380684,57860] 0 2024-03-22T02:28:46.485 INFO:tasks.workunit.client.0.smithi103.stdout:9/695: dread d2/da/f24 [0,4194304] 0 2024-03-22T02:28:46.486 INFO:tasks.workunit.client.1.smithi122.stdout:9/189: link db/l33 db/d23/d1e/d30/l44 0 2024-03-22T02:28:46.487 INFO:tasks.workunit.client.1.smithi122.stdout:2/138: dwrite df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:46.490 INFO:tasks.workunit.client.0.smithi103.stdout:9/696: mknod d2/da/d18/d1e/cd3 0 2024-03-22T02:28:46.491 INFO:tasks.workunit.client.1.smithi122.stdout:9/190: symlink db/d1b/d28/l45 0 2024-03-22T02:28:46.492 INFO:tasks.workunit.client.0.smithi103.stdout:9/697: write d2/da/d18/d1e/fcc [710622,9677] 0 2024-03-22T02:28:46.492 INFO:tasks.workunit.client.1.smithi122.stdout:6/212: dwrite d7/de/dd/d23/f48 [0,4194304] 0 2024-03-22T02:28:46.493 INFO:tasks.workunit.client.0.smithi103.stdout:9/698: dread - d2/d10/d59/f6d zero size 2024-03-22T02:28:46.494 INFO:tasks.workunit.client.1.smithi122.stdout:2/139: mkdir df/d1f/d2a/d39 0 2024-03-22T02:28:46.501 INFO:tasks.workunit.client.0.smithi103.stdout:9/699: mkdir d2/d5/d7b/d7d/dd4 0 2024-03-22T02:28:46.501 INFO:tasks.workunit.client.1.smithi122.stdout:9/191: symlink db/l46 0 2024-03-22T02:28:46.501 INFO:tasks.workunit.client.1.smithi122.stdout:6/213: rmdir d7/de/dd 39 2024-03-22T02:28:46.502 INFO:tasks.workunit.client.1.smithi122.stdout:9/192: dread - db/d23/d1f/f2b zero size 2024-03-22T02:28:46.503 INFO:tasks.workunit.client.1.smithi122.stdout:6/214: write f6 [1476551,73520] 0 2024-03-22T02:28:46.503 INFO:tasks.workunit.client.1.smithi122.stdout:9/193: dread - db/d23/f11 zero size 2024-03-22T02:28:46.505 INFO:tasks.workunit.client.1.smithi122.stdout:9/194: read - db/d23/d1f/d21/d29/f32 zero size 2024-03-22T02:28:46.507 INFO:tasks.workunit.client.1.smithi122.stdout:6/215: chown d7/de/dd/d20/l35 11661 1 2024-03-22T02:28:46.509 INFO:tasks.workunit.client.1.smithi122.stdout:6/216: readlink l1 0 2024-03-22T02:28:46.511 INFO:tasks.workunit.client.1.smithi122.stdout:9/195: symlink db/d23/d1f/d21/d3d/l47 0 2024-03-22T02:28:46.512 INFO:tasks.workunit.client.1.smithi122.stdout:6/217: rmdir d7/de/dd/df 39 2024-03-22T02:28:46.513 INFO:tasks.workunit.client.1.smithi122.stdout:6/218: chown d7/de/dd/d23 461030 1 2024-03-22T02:28:46.514 INFO:tasks.workunit.client.1.smithi122.stdout:1/125: dwrite dc/f1d [0,4194304] 0 2024-03-22T02:28:46.515 INFO:tasks.workunit.client.1.smithi122.stdout:9/196: mknod db/d23/d1f/d21/d25/d41/c48 0 2024-03-22T02:28:46.516 INFO:tasks.workunit.client.1.smithi122.stdout:9/197: chown db/df/c14 21155687 1 2024-03-22T02:28:46.519 INFO:tasks.workunit.client.0.smithi103.stdout:2/716: dwrite d11/d25/d63/d45/d47/d83/da3/fda [4194304,4194304] 0 2024-03-22T02:28:46.521 INFO:tasks.workunit.client.1.smithi122.stdout:9/198: dread db/fc [0,4194304] 0 2024-03-22T02:28:46.532 INFO:tasks.workunit.client.0.smithi103.stdout:9/700: dread d2/f3 [0,4194304] 0 2024-03-22T02:28:46.533 INFO:tasks.workunit.client.0.smithi103.stdout:9/701: chown d2/d10/d43/d7e/da4/dcd/cc2 2589 1 2024-03-22T02:28:46.535 INFO:tasks.workunit.client.1.smithi122.stdout:1/126: dwrite dc/f11 [0,4194304] 0 2024-03-22T02:28:46.535 INFO:tasks.workunit.client.1.smithi122.stdout:1/127: stat dc/fe 0 2024-03-22T02:28:46.538 INFO:tasks.workunit.client.1.smithi122.stdout:1/128: symlink dc/l26 0 2024-03-22T02:28:46.539 INFO:tasks.workunit.client.0.smithi103.stdout:9/702: rename d2/d10/d43/d7e/fd1 to d2/da/d18/d1e/d34/d3d/d61/db7/fd5 0 2024-03-22T02:28:46.540 INFO:tasks.workunit.client.1.smithi122.stdout:2/140: write df/f13 [373481,6437] 0 2024-03-22T02:28:46.541 INFO:tasks.workunit.client.1.smithi122.stdout:1/129: creat dc/f27 x:0 0 0 2024-03-22T02:28:46.541 INFO:tasks.workunit.client.1.smithi122.stdout:2/141: write df/f15 [689915,64719] 0 2024-03-22T02:28:46.541 INFO:tasks.workunit.client.1.smithi122.stdout:1/130: stat l5 0 2024-03-22T02:28:46.543 INFO:tasks.workunit.client.1.smithi122.stdout:1/131: stat dc/f1b 0 2024-03-22T02:28:46.544 INFO:tasks.workunit.client.1.smithi122.stdout:9/199: dwrite db/fc [0,4194304] 0 2024-03-22T02:28:46.545 INFO:tasks.workunit.client.1.smithi122.stdout:1/132: write dc/f27 [403002,40197] 0 2024-03-22T02:28:46.545 INFO:tasks.workunit.client.0.smithi103.stdout:9/703: dread d2/d10/d43/d7e/fc6 [0,4194304] 0 2024-03-22T02:28:46.546 INFO:tasks.workunit.client.0.smithi103.stdout:9/704: stat d2/d5/caa 0 2024-03-22T02:28:46.549 INFO:tasks.workunit.client.1.smithi122.stdout:1/133: creat dc/f28 x:0 0 0 2024-03-22T02:28:46.551 INFO:tasks.workunit.client.1.smithi122.stdout:1/134: write dc/f27 [774401,3363] 0 2024-03-22T02:28:46.553 INFO:tasks.workunit.client.0.smithi103.stdout:9/705: link d2/d10/d59/cc7 d2/da/d18/d1e/d34/d3d/d46/cd6 0 2024-03-22T02:28:46.554 INFO:tasks.workunit.client.1.smithi122.stdout:1/135: mknod dc/c29 0 2024-03-22T02:28:46.555 INFO:tasks.workunit.client.1.smithi122.stdout:9/200: dwrite db/f13 [0,4194304] 0 2024-03-22T02:28:46.557 INFO:tasks.workunit.client.0.smithi103.stdout:9/706: creat d2/d10/d43/d7e/d96/d9d/fd7 x:0 0 0 2024-03-22T02:28:46.558 INFO:tasks.workunit.client.0.smithi103.stdout:9/707: chown d2/d10/d43/d7e/d96/d9d/f7c 523834 1 2024-03-22T02:28:46.561 INFO:tasks.workunit.client.1.smithi122.stdout:1/136: dread dc/f1c [0,4194304] 0 2024-03-22T02:28:46.561 INFO:tasks.workunit.client.1.smithi122.stdout:6/219: dread d7/f3b [0,4194304] 0 2024-03-22T02:28:46.561 INFO:tasks.workunit.client.0.smithi103.stdout:2/717: dread d11/f58 [0,4194304] 0 2024-03-22T02:28:46.562 INFO:tasks.workunit.client.1.smithi122.stdout:1/137: dread - dc/f17 zero size 2024-03-22T02:28:46.562 INFO:tasks.workunit.client.0.smithi103.stdout:2/718: chown d11/d19/d5d/f66 500047667 1 2024-03-22T02:28:46.563 INFO:tasks.workunit.client.0.smithi103.stdout:9/708: getdents d2 0 2024-03-22T02:28:46.564 INFO:tasks.workunit.client.1.smithi122.stdout:2/142: fdatasync df/f15 0 2024-03-22T02:28:46.564 INFO:tasks.workunit.client.1.smithi122.stdout:1/138: unlink dc/c20 0 2024-03-22T02:28:46.564 INFO:tasks.workunit.client.0.smithi103.stdout:9/709: chown d2/da/d18/dc4 36978720 1 2024-03-22T02:28:46.566 INFO:tasks.workunit.client.1.smithi122.stdout:2/143: chown df/d1f/d33/d34 439686783 1 2024-03-22T02:28:46.566 INFO:tasks.workunit.client.1.smithi122.stdout:1/139: write dc/fd [4895693,101689] 0 2024-03-22T02:28:46.567 INFO:tasks.workunit.client.0.smithi103.stdout:2/719: getdents d11 0 2024-03-22T02:28:46.568 INFO:tasks.workunit.client.1.smithi122.stdout:2/144: mkdir df/d1f/d3a 0 2024-03-22T02:28:46.570 INFO:tasks.workunit.client.0.smithi103.stdout:2/720: symlink d11/d19/da7/dd3/dd9/lf0 0 2024-03-22T02:28:46.570 INFO:tasks.workunit.client.1.smithi122.stdout:1/140: rename dc/f27 to dc/f2a 0 2024-03-22T02:28:46.572 INFO:tasks.workunit.client.0.smithi103.stdout:2/721: chown d11/d25/d46/db6/lbb 30086189 1 2024-03-22T02:28:46.572 INFO:tasks.workunit.client.1.smithi122.stdout:2/145: creat df/d1f/d23/f3b x:0 0 0 2024-03-22T02:28:46.572 INFO:tasks.workunit.client.0.smithi103.stdout:9/710: dwrite d2/d5/f63 [0,4194304] 0 2024-03-22T02:28:46.574 INFO:tasks.workunit.client.0.smithi103.stdout:2/722: write d11/d25/d63/d45/d47/d83/fa5 [263330,87717] 0 2024-03-22T02:28:46.576 INFO:tasks.workunit.client.1.smithi122.stdout:2/146: rmdir df/d32 0 2024-03-22T02:28:46.577 INFO:tasks.workunit.client.1.smithi122.stdout:1/141: dwrite dc/fd [0,4194304] 0 2024-03-22T02:28:46.578 INFO:tasks.workunit.client.0.smithi103.stdout:2/723: fdatasync d11/d25/d63/d45/d47/d83/f76 0 2024-03-22T02:28:46.578 INFO:tasks.workunit.client.1.smithi122.stdout:2/147: fsync df/d1f/f21 0 2024-03-22T02:28:46.579 INFO:tasks.workunit.client.0.smithi103.stdout:2/724: mkdir d11/d19/da7/dd3/df1 0 2024-03-22T02:28:46.582 INFO:tasks.workunit.client.0.smithi103.stdout:2/725: rmdir d11/d25/d63/d45/d47/d83/d78 39 2024-03-22T02:28:46.584 INFO:tasks.workunit.client.0.smithi103.stdout:9/711: dread d2/da/d18/d1e/d34/f8d [0,4194304] 0 2024-03-22T02:28:46.584 INFO:tasks.workunit.client.0.smithi103.stdout:4/718: sync 2024-03-22T02:28:46.585 INFO:tasks.workunit.client.0.smithi103.stdout:6/504: sync 2024-03-22T02:28:46.585 INFO:tasks.workunit.client.1.smithi122.stdout:1/142: unlink dc/c16 0 2024-03-22T02:28:46.586 INFO:tasks.workunit.client.0.smithi103.stdout:6/505: dread - d0/d3/f74 zero size 2024-03-22T02:28:46.586 INFO:tasks.workunit.client.0.smithi103.stdout:9/712: fsync d2/da/d18/f6c 0 2024-03-22T02:28:46.586 INFO:tasks.workunit.client.0.smithi103.stdout:2/726: link d11/d25/d46/l9e d11/d25/d46/d48/lf2 0 2024-03-22T02:28:46.588 INFO:tasks.workunit.client.1.smithi122.stdout:1/143: chown dc/f1b 147257 1 2024-03-22T02:28:46.588 INFO:tasks.workunit.client.0.smithi103.stdout:6/506: chown d0/d3/f74 2 1 2024-03-22T02:28:46.591 INFO:tasks.workunit.client.1.smithi122.stdout:1/144: truncate dc/f2a 1473991 0 2024-03-22T02:28:46.591 INFO:tasks.workunit.client.1.smithi122.stdout:6/220: dwrite d7/f15 [0,4194304] 0 2024-03-22T02:28:46.591 INFO:tasks.workunit.client.1.smithi122.stdout:2/148: dwrite df/f37 [0,4194304] 0 2024-03-22T02:28:46.591 INFO:tasks.workunit.client.1.smithi122.stdout:1/145: dread - dc/f17 zero size 2024-03-22T02:28:46.594 INFO:tasks.workunit.client.1.smithi122.stdout:1/146: mkdir dc/d2b 0 2024-03-22T02:28:46.596 INFO:tasks.workunit.client.0.smithi103.stdout:6/507: link d0/d3/d33/c90 d0/d14/d57/c9e 0 2024-03-22T02:28:46.597 INFO:tasks.workunit.client.0.smithi103.stdout:2/727: dwrite d11/d25/d63/d45/d47/d83/da3/fda [4194304,4194304] 0 2024-03-22T02:28:46.597 INFO:tasks.workunit.client.1.smithi122.stdout:1/147: rename dc/f1c to dc/f2c 0 2024-03-22T02:28:46.602 INFO:tasks.workunit.client.1.smithi122.stdout:1/148: dwrite dc/f1d [0,4194304] 0 2024-03-22T02:28:46.603 INFO:tasks.workunit.client.1.smithi122.stdout:2/149: dread df/f15 [0,4194304] 0 2024-03-22T02:28:46.604 INFO:tasks.workunit.client.1.smithi122.stdout:2/150: chown df/c1a 0 1 2024-03-22T02:28:46.604 INFO:tasks.workunit.client.0.smithi103.stdout:2/728: link d11/d25/d63/l59 d11/d19/d4e/lf3 0 2024-03-22T02:28:46.607 INFO:tasks.workunit.client.0.smithi103.stdout:6/508: read d0/f1a [488501,4773] 0 2024-03-22T02:28:46.608 INFO:tasks.workunit.client.1.smithi122.stdout:2/151: link df/d1f/d23/f3b df/d1f/f3c 0 2024-03-22T02:28:46.609 INFO:tasks.workunit.client.1.smithi122.stdout:1/149: dwrite dc/fe [0,4194304] 0 2024-03-22T02:28:46.610 INFO:tasks.workunit.client.1.smithi122.stdout:2/152: truncate df/d1f/d23/f29 728922 0 2024-03-22T02:28:46.610 INFO:tasks.workunit.client.1.smithi122.stdout:2/153: fsync df/f14 0 2024-03-22T02:28:46.612 INFO:tasks.workunit.client.0.smithi103.stdout:2/729: rename d11/d25/d46/d48/f6d to d11/d25/d63/d45/ff4 0 2024-03-22T02:28:46.614 INFO:tasks.workunit.client.1.smithi122.stdout:1/150: dread dc/f1d [0,4194304] 0 2024-03-22T02:28:46.615 INFO:tasks.workunit.client.0.smithi103.stdout:4/719: dread d3/d1e/d25/d65/f6c [0,4194304] 0 2024-03-22T02:28:46.616 INFO:tasks.workunit.client.0.smithi103.stdout:6/509: dwrite d0/d14/f93 [0,4194304] 0 2024-03-22T02:28:46.618 INFO:tasks.workunit.client.1.smithi122.stdout:1/151: write dc/f11 [3392567,74140] 0 2024-03-22T02:28:46.621 INFO:tasks.workunit.client.0.smithi103.stdout:9/713: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:46.624 INFO:tasks.workunit.client.1.smithi122.stdout:2/154: dwrite df/d1f/f21 [0,4194304] 0 2024-03-22T02:28:46.624 INFO:tasks.workunit.client.1.smithi122.stdout:2/155: write df/d1f/f2d [1384241,41572] 0 2024-03-22T02:28:46.627 INFO:tasks.workunit.client.0.smithi103.stdout:6/510: dwrite d0/d3/f1b [4194304,4194304] 0 2024-03-22T02:28:46.629 INFO:tasks.workunit.client.1.smithi122.stdout:1/152: dread dc/f1a [0,4194304] 0 2024-03-22T02:28:46.630 INFO:tasks.workunit.client.1.smithi122.stdout:6/221: write d7/de/f43 [300509,3510] 0 2024-03-22T02:28:46.632 INFO:tasks.workunit.client.0.smithi103.stdout:6/511: creat d0/d88/f9f x:0 0 0 2024-03-22T02:28:46.641 INFO:tasks.workunit.client.1.smithi122.stdout:6/222: creat d7/de/dd/d39/f4a x:0 0 0 2024-03-22T02:28:46.642 INFO:tasks.workunit.client.1.smithi122.stdout:1/153: dwrite dc/f25 [0,4194304] 0 2024-03-22T02:28:46.644 INFO:tasks.workunit.client.1.smithi122.stdout:1/154: fsync dc/f24 0 2024-03-22T02:28:46.644 INFO:tasks.workunit.client.1.smithi122.stdout:0/125: sync 2024-03-22T02:28:46.645 INFO:tasks.workunit.client.1.smithi122.stdout:8/155: sync 2024-03-22T02:28:46.645 INFO:tasks.workunit.client.1.smithi122.stdout:1/155: write dc/f21 [45347,74710] 0 2024-03-22T02:28:46.647 INFO:tasks.workunit.client.1.smithi122.stdout:1/156: truncate dc/f24 745089 0 2024-03-22T02:28:46.648 INFO:tasks.workunit.client.1.smithi122.stdout:1/157: fdatasync dc/f17 0 2024-03-22T02:28:46.648 INFO:tasks.workunit.client.1.smithi122.stdout:8/156: dread - d1/d9/f28 zero size 2024-03-22T02:28:46.650 INFO:tasks.workunit.client.1.smithi122.stdout:1/158: write dc/f25 [2097046,63604] 0 2024-03-22T02:28:46.652 INFO:tasks.workunit.client.1.smithi122.stdout:0/126: creat d11/d14/d1b/f2c x:0 0 0 2024-03-22T02:28:46.656 INFO:tasks.workunit.client.1.smithi122.stdout:0/127: creat d11/d14/d23/f2d x:0 0 0 2024-03-22T02:28:46.658 INFO:tasks.workunit.client.1.smithi122.stdout:8/157: dwrite d1/d22/f2e [4194304,4194304] 0 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:5/751: sync 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:3/770: sync 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:8/838: sync 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:7/720: sync 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:0/716: sync 2024-03-22T02:28:46.659 INFO:tasks.workunit.client.0.smithi103.stdout:1/764: sync 2024-03-22T02:28:46.661 INFO:tasks.workunit.client.0.smithi103.stdout:0/717: stat dd/d16/d4b/d96/d29/d48/d5c/la5 0 2024-03-22T02:28:46.661 INFO:tasks.workunit.client.0.smithi103.stdout:3/771: chown de/d12/d5d/d92/fe1 1043591 1 2024-03-22T02:28:46.662 INFO:tasks.workunit.client.0.smithi103.stdout:5/752: dread - d0/d14/d23/d2a/d33/db5/f106 zero size 2024-03-22T02:28:46.662 INFO:tasks.workunit.client.1.smithi122.stdout:0/128: link d11/d16/f2a d11/d14/d1d/f2e 0 2024-03-22T02:28:46.662 INFO:tasks.workunit.client.1.smithi122.stdout:8/158: getdents d1/d8/d2f 0 2024-03-22T02:28:46.662 INFO:tasks.workunit.client.1.smithi122.stdout:8/159: readlink d1/d9/l16 0 2024-03-22T02:28:46.663 INFO:tasks.workunit.client.0.smithi103.stdout:7/721: chown d1/d10/d11/d19/d29/d61/d69/d5c/l81 42752 1 2024-03-22T02:28:46.663 INFO:tasks.workunit.client.0.smithi103.stdout:1/765: symlink db/d4a/d78/d92/da8/ddd/d28/d97/l107 0 2024-03-22T02:28:46.670 INFO:tasks.workunit.client.0.smithi103.stdout:3/772: rmdir de/d6b/d80/d4f 39 2024-03-22T02:28:46.671 INFO:tasks.workunit.client.1.smithi122.stdout:8/160: dwrite d1/d9/f11 [4194304,4194304] 0 2024-03-22T02:28:46.671 INFO:tasks.workunit.client.1.smithi122.stdout:8/161: readlink d1/l2a 0 2024-03-22T02:28:46.672 INFO:tasks.workunit.client.0.smithi103.stdout:1/766: write db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fde [503742,77262] 0 2024-03-22T02:28:46.673 INFO:tasks.workunit.client.1.smithi122.stdout:8/162: creat d1/d8/f33 x:0 0 0 2024-03-22T02:28:46.674 INFO:tasks.workunit.client.0.smithi103.stdout:3/773: chown de/d12/d5d/da9/db9/cdc 1758 1 2024-03-22T02:28:46.676 INFO:tasks.workunit.client.0.smithi103.stdout:5/753: dwrite d0/d14/f41 [8388608,4194304] 0 2024-03-22T02:28:46.677 INFO:tasks.workunit.client.0.smithi103.stdout:1/767: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ded 1776704 1 2024-03-22T02:28:46.678 INFO:tasks.workunit.client.1.smithi122.stdout:8/163: dread d1/d22/f2e [4194304,4194304] 0 2024-03-22T02:28:46.678 INFO:tasks.workunit.client.1.smithi122.stdout:8/164: chown d1/d9/f11 297674 1 2024-03-22T02:28:46.683 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.682+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28a41012c0 con 0x7f288c06b8c0 2024-03-22T02:28:46.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.684+0000 7f288b7fe700 1 -- 172.21.15.103:0/2182045974 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f28a41012c0 con 0x7f288c06b8c0 2024-03-22T02:28:46.685 INFO:tasks.workunit.client.1.smithi122.stdout:8/165: dwrite d1/d8/f33 [0,4194304] 0 2024-03-22T02:28:46.687 INFO:tasks.workunit.client.1.smithi122.stdout:8/166: read d1/d9/f24 [1639922,72693] 0 2024-03-22T02:28:46.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 msgr2=0x7f288c06dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 0x7f288c06dd70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f28940041d0 tx=0x7f2894009040 comp rx=0 tx=0).stop 2024-03-22T02:28:46.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 msgr2=0x7f28a4112c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f289800b3b0 tx=0x7f2898009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 shutdown_connections 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f288c06b8c0 0x7f288c06dd70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28a40fbce0 0x7f28a4110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 --2- 172.21.15.103:0/2182045974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28a40fcf90 0x7f28a4112c50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 >> 172.21.15.103:0/2182045974 conn(0x7f28a40f7440 msgr2=0x7f28a40f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 shutdown_connections 2024-03-22T02:28:46.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.688+0000 7f28a9436700 1 -- 172.21.15.103:0/2182045974 wait complete. 2024-03-22T02:28:46.695 INFO:tasks.workunit.client.1.smithi122.stdout:8/167: dwrite d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:46.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/168: chown d1/d9/c19 1 1 2024-03-22T02:28:46.704 INFO:tasks.workunit.client.1.smithi122.stdout:8/169: dwrite d1/f1a [0,4194304] 0 2024-03-22T02:28:46.705 INFO:tasks.workunit.client.1.smithi122.stdout:8/170: write d1/d22/f2e [5639168,78041] 0 2024-03-22T02:28:46.706 INFO:tasks.workunit.client.1.smithi122.stdout:8/171: stat d1/d9/l16 0 2024-03-22T02:28:46.707 INFO:tasks.workunit.client.0.smithi103.stdout:9/714: write d2/da/d18/d1e/d34/d3d/d4d/f58 [3896054,43744] 0 2024-03-22T02:28:46.709 INFO:tasks.workunit.client.0.smithi103.stdout:2/730: dwrite d11/d25/f35 [0,4194304] 0 2024-03-22T02:28:46.713 INFO:tasks.workunit.client.0.smithi103.stdout:2/731: dread - d11/d25/d63/d45/d47/d83/f76 zero size 2024-03-22T02:28:46.717 INFO:tasks.workunit.client.0.smithi103.stdout:1/768: chown db/d4a/d78/d92/da8/ddd/d18/d2d/l47 109965 1 2024-03-22T02:28:46.717 INFO:tasks.workunit.client.0.smithi103.stdout:3/774: readlink de/d6b/d80/d4f/l5e 0 2024-03-22T02:28:46.718 INFO:tasks.workunit.client.1.smithi122.stdout:8/172: dwrite d1/f2d [0,4194304] 0 2024-03-22T02:28:46.718 INFO:tasks.workunit.client.0.smithi103.stdout:7/722: getdents d1/d10/d11/d19/d29/d57 0 2024-03-22T02:28:46.719 INFO:tasks.workunit.client.0.smithi103.stdout:1/769: write db/f45 [1455515,80160] 0 2024-03-22T02:28:46.719 INFO:tasks.workunit.client.0.smithi103.stdout:5/754: write d0/d14/d23/d36/d95/d96/daa/daf/fab [218096,71494] 0 2024-03-22T02:28:46.720 INFO:tasks.workunit.client.1.smithi122.stdout:1/159: fdatasync dc/f25 0 2024-03-22T02:28:46.721 INFO:tasks.workunit.client.0.smithi103.stdout:9/715: mkdir d2/d5/d7b/d7d/dd4/dd8 0 2024-03-22T02:28:46.721 INFO:tasks.workunit.client.1.smithi122.stdout:2/156: write df/f10 [143186,42637] 0 2024-03-22T02:28:46.722 INFO:tasks.workunit.client.0.smithi103.stdout:6/512: write d0/d14/d57/d68/f71 [139471,103360] 0 2024-03-22T02:28:46.722 INFO:tasks.workunit.client.1.smithi122.stdout:6/223: rmdir d7/de/dd/d39 39 2024-03-22T02:28:46.724 INFO:tasks.workunit.client.1.smithi122.stdout:8/173: symlink d1/d9/l34 0 2024-03-22T02:28:46.725 INFO:tasks.workunit.client.0.smithi103.stdout:9/716: chown d2/d10/d43/f3e 2774 1 2024-03-22T02:28:46.726 INFO:tasks.workunit.client.1.smithi122.stdout:8/174: read d1/d9/f28 [485628,588] 0 2024-03-22T02:28:46.726 INFO:tasks.workunit.client.1.smithi122.stdout:2/157: mkdir df/d1f/d3d 0 2024-03-22T02:28:46.727 INFO:tasks.workunit.client.0.smithi103.stdout:1/770: chown db/d4a/d78/d92/da8/ddd/d29/d55/c9b 566365318 1 2024-03-22T02:28:46.728 INFO:tasks.workunit.client.1.smithi122.stdout:2/158: write df/d1f/f20 [578884,13696] 0 2024-03-22T02:28:46.728 INFO:tasks.workunit.client.0.smithi103.stdout:6/513: fsync d0/d14/d57/d68/f55 0 2024-03-22T02:28:46.730 INFO:tasks.workunit.client.1.smithi122.stdout:2/159: truncate df/f2b 118967 0 2024-03-22T02:28:46.732 INFO:tasks.workunit.client.1.smithi122.stdout:8/175: readlink d1/d8/l2c 0 2024-03-22T02:28:46.732 INFO:tasks.workunit.client.0.smithi103.stdout:6/514: truncate d0/d88/f9f 697424 0 2024-03-22T02:28:46.734 INFO:tasks.workunit.client.0.smithi103.stdout:5/755: getdents d0/d14/d23/d2a 0 2024-03-22T02:28:46.734 INFO:tasks.workunit.client.1.smithi122.stdout:6/224: creat d7/de/dd/df/f4b x:0 0 0 2024-03-22T02:28:46.735 INFO:tasks.workunit.client.0.smithi103.stdout:1/771: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d108 0 2024-03-22T02:28:46.735 INFO:tasks.workunit.client.0.smithi103.stdout:4/720: dread d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:46.738 INFO:tasks.workunit.client.1.smithi122.stdout:0/129: dwrite fd [0,4194304] 0 2024-03-22T02:28:46.739 INFO:tasks.workunit.client.1.smithi122.stdout:8/176: symlink d1/d22/l35 0 2024-03-22T02:28:46.740 INFO:tasks.workunit.client.0.smithi103.stdout:5/756: mknod d0/d14/d23/d5c/df3/c108 0 2024-03-22T02:28:46.741 INFO:tasks.workunit.client.1.smithi122.stdout:6/225: symlink d7/de/dd/df/l4c 0 2024-03-22T02:28:46.741 INFO:tasks.workunit.client.0.smithi103.stdout:9/717: dwrite d2/d10/d59/da5/fb1 [0,4194304] 0 2024-03-22T02:28:46.743 INFO:tasks.workunit.client.1.smithi122.stdout:0/130: fsync fe 0 2024-03-22T02:28:46.743 INFO:tasks.workunit.client.1.smithi122.stdout:0/131: stat d11/c20 0 2024-03-22T02:28:46.744 INFO:tasks.workunit.client.0.smithi103.stdout:5/757: write d0/d14/dc0/fea [1754433,112126] 0 2024-03-22T02:28:46.745 INFO:tasks.workunit.client.1.smithi122.stdout:8/177: creat d1/d8/f36 x:0 0 0 2024-03-22T02:28:46.747 INFO:tasks.workunit.client.1.smithi122.stdout:2/160: dread df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:46.749 INFO:tasks.workunit.client.0.smithi103.stdout:5/758: fdatasync d0/d14/d23/d36/fe3 0 2024-03-22T02:28:46.749 INFO:tasks.workunit.client.1.smithi122.stdout:2/161: readlink la 0 2024-03-22T02:28:46.750 INFO:tasks.workunit.client.0.smithi103.stdout:0/718: dwrite dd/d16/d4b/d96/d29/f3e [0,4194304] 0 2024-03-22T02:28:46.752 INFO:tasks.workunit.client.1.smithi122.stdout:2/162: chown df/f15 9712 1 2024-03-22T02:28:46.752 INFO:tasks.workunit.client.1.smithi122.stdout:1/160: truncate dc/fe 3775252 0 2024-03-22T02:28:46.752 INFO:tasks.workunit.client.0.smithi103.stdout:5/759: dread - d0/f73 zero size 2024-03-22T02:28:46.753 INFO:tasks.workunit.client.1.smithi122.stdout:0/132: dwrite d11/f29 [0,4194304] 0 2024-03-22T02:28:46.758 INFO:tasks.workunit.client.1.smithi122.stdout:8/178: dwrite d1/f1a [0,4194304] 0 2024-03-22T02:28:46.759 INFO:tasks.workunit.client.0.smithi103.stdout:4/721: dwrite d3/f8 [0,4194304] 0 2024-03-22T02:28:46.759 INFO:tasks.workunit.client.1.smithi122.stdout:1/161: creat dc/d2b/f2d x:0 0 0 2024-03-22T02:28:46.760 INFO:tasks.workunit.client.1.smithi122.stdout:8/179: chown d1/d9/l34 32993 1 2024-03-22T02:28:46.760 INFO:tasks.workunit.client.1.smithi122.stdout:1/162: readlink l5 0 2024-03-22T02:28:46.764 INFO:tasks.workunit.client.1.smithi122.stdout:0/133: dwrite fb [0,4194304] 0 2024-03-22T02:28:46.768 INFO:tasks.workunit.client.1.smithi122.stdout:0/134: mkdir d11/d14/d1d/d27/d2f 0 2024-03-22T02:28:46.769 INFO:tasks.workunit.client.1.smithi122.stdout:8/180: dread d1/f1a [0,4194304] 0 2024-03-22T02:28:46.770 INFO:tasks.workunit.client.0.smithi103.stdout:4/722: mknod d3/deb/cf4 0 2024-03-22T02:28:46.773 INFO:tasks.workunit.client.1.smithi122.stdout:0/135: fsync d11/d14/f15 0 2024-03-22T02:28:46.773 INFO:tasks.workunit.client.1.smithi122.stdout:1/163: dwrite dc/f1a [0,4194304] 0 2024-03-22T02:28:46.776 INFO:tasks.workunit.client.0.smithi103.stdout:4/723: creat d3/d1e/d53/d7a/da3/ff5 x:0 0 0 2024-03-22T02:28:46.777 INFO:tasks.workunit.client.0.smithi103.stdout:1/772: dread db/d4a/d78/d92/da8/ddd/d29/d41/f89 [0,4194304] 0 2024-03-22T02:28:46.780 INFO:tasks.workunit.client.1.smithi122.stdout:6/226: truncate d7/de/dd/df/f1a 3663505 0 2024-03-22T02:28:46.781 INFO:tasks.workunit.client.1.smithi122.stdout:6/227: read d7/de/dd/f3a [1238775,7286] 0 2024-03-22T02:28:46.784 INFO:tasks.workunit.client.0.smithi103.stdout:2/732: write d11/d19/d5d/f91 [1491843,83881] 0 2024-03-22T02:28:46.788 INFO:tasks.workunit.client.1.smithi122.stdout:6/228: symlink d7/de/dd/d39/d46/l4d 0 2024-03-22T02:28:46.789 INFO:tasks.workunit.client.0.smithi103.stdout:1/773: dwrite db/d4a/d78/d92/da8/ddd/d29/f61 [0,4194304] 0 2024-03-22T02:28:46.790 INFO:tasks.workunit.client.1.smithi122.stdout:6/229: creat d7/de/dd/d39/d46/f4e x:0 0 0 2024-03-22T02:28:46.791 INFO:tasks.workunit.client.1.smithi122.stdout:2/163: dwrite df/d1f/f3c [0,4194304] 0 2024-03-22T02:28:46.791 INFO:tasks.workunit.client.1.smithi122.stdout:6/230: readlink d7/de/l3d 0 2024-03-22T02:28:46.792 INFO:tasks.workunit.client.1.smithi122.stdout:2/164: fsync f6 0 2024-03-22T02:28:46.792 INFO:tasks.workunit.client.0.smithi103.stdout:2/733: rename d11/d19/da7/dd3/c3c to d11/d19/d5d/cf5 0 2024-03-22T02:28:46.794 INFO:tasks.workunit.client.0.smithi103.stdout:1/774: mknod db/c109 0 2024-03-22T02:28:46.795 INFO:tasks.workunit.client.1.smithi122.stdout:4/144: sync 2024-03-22T02:28:46.796 INFO:tasks.workunit.client.1.smithi122.stdout:2/165: dread - df/f36 zero size 2024-03-22T02:28:46.801 INFO:tasks.workunit.client.1.smithi122.stdout:6/231: dread d7/de/f3e [0,4194304] 0 2024-03-22T02:28:46.802 INFO:tasks.workunit.client.0.smithi103.stdout:5/760: dwrite d0/d14/f4e [4194304,4194304] 0 2024-03-22T02:28:46.803 INFO:tasks.workunit.client.0.smithi103.stdout:4/724: dread d3/d1e/d53/d8a/f9a [0,4194304] 0 2024-03-22T02:28:46.807 INFO:tasks.workunit.client.1.smithi122.stdout:4/145: dwrite d0/d12/d15/f29 [0,4194304] 0 2024-03-22T02:28:46.807 INFO:tasks.workunit.client.0.smithi103.stdout:0/719: dwrite dd/d16/d4b/d96/d64/fa3 [0,4194304] 0 2024-03-22T02:28:46.809 INFO:tasks.workunit.client.0.smithi103.stdout:1/775: rename db/d4a/d78/d92/da8/ddd/d6b/db5/fc3 to db/d4a/d78/d92/da8/ddd/d18/d84/f10a 0 2024-03-22T02:28:46.810 INFO:tasks.workunit.client.0.smithi103.stdout:0/720: stat dd/d34/fac 0 2024-03-22T02:28:46.811 INFO:tasks.workunit.client.0.smithi103.stdout:1/776: write db/d4a/d78/d92/da8/ddd/d18/d84/fc4 [277190,80424] 0 2024-03-22T02:28:46.813 INFO:tasks.workunit.client.0.smithi103.stdout:5/761: fsync d0/d14/d23/d36/d95/d96/daa/daf/f93 0 2024-03-22T02:28:46.816 INFO:tasks.workunit.client.0.smithi103.stdout:1/777: symlink db/d4a/d78/d92/da8/ddd/d2e/l10b 0 2024-03-22T02:28:46.817 INFO:tasks.workunit.client.1.smithi122.stdout:6/232: dwrite d7/de/dd/f42 [0,4194304] 0 2024-03-22T02:28:46.820 INFO:tasks.workunit.client.1.smithi122.stdout:4/146: dwrite d0/d12/f31 [0,4194304] 0 2024-03-22T02:28:46.825 INFO:tasks.workunit.client.1.smithi122.stdout:4/147: fsync d0/d12/f21 0 2024-03-22T02:28:46.826 INFO:tasks.workunit.client.1.smithi122.stdout:4/148: chown d0/d25/d7/f16 8 1 2024-03-22T02:28:46.827 INFO:tasks.workunit.client.1.smithi122.stdout:6/233: dwrite f3 [0,4194304] 0 2024-03-22T02:28:46.829 INFO:tasks.workunit.client.1.smithi122.stdout:4/149: rmdir d0/d12 39 2024-03-22T02:28:46.831 INFO:tasks.workunit.client.1.smithi122.stdout:6/234: creat d7/de/dd/d39/f4f x:0 0 0 2024-03-22T02:28:46.832 INFO:tasks.workunit.client.1.smithi122.stdout:6/235: chown d7/de/l25 480063 1 2024-03-22T02:28:46.833 INFO:tasks.workunit.client.1.smithi122.stdout:4/150: mkdir d0/d33 0 2024-03-22T02:28:46.833 INFO:tasks.workunit.client.0.smithi103.stdout:5/762: read d0/d14/d23/d5c/d77/df2/fdc [8694,108562] 0 2024-03-22T02:28:46.834 INFO:tasks.workunit.client.1.smithi122.stdout:6/236: write d7/de/dd/f13 [1084815,95293] 0 2024-03-22T02:28:46.835 INFO:tasks.workunit.client.1.smithi122.stdout:6/237: chown f5 106043492 1 2024-03-22T02:28:46.837 INFO:tasks.workunit.client.1.smithi122.stdout:6/238: fdatasync d7/de/dd/f3a 0 2024-03-22T02:28:46.839 INFO:tasks.workunit.client.1.smithi122.stdout:6/239: write f3 [975394,76145] 0 2024-03-22T02:28:46.840 INFO:tasks.workunit.client.1.smithi122.stdout:6/240: write f6 [1842310,64253] 0 2024-03-22T02:28:46.841 INFO:tasks.workunit.client.1.smithi122.stdout:2/166: truncate df/d1f/d23/f29 464226 0 2024-03-22T02:28:46.841 INFO:tasks.workunit.client.0.smithi103.stdout:2/734: dwrite d11/f16 [0,4194304] 0 2024-03-22T02:28:46.846 INFO:tasks.workunit.client.1.smithi122.stdout:2/167: fdatasync df/d1f/f20 0 2024-03-22T02:28:46.846 INFO:tasks.workunit.client.0.smithi103.stdout:0/721: dread dd/d34/fad [0,4194304] 0 2024-03-22T02:28:46.847 INFO:tasks.workunit.client.0.smithi103.stdout:2/735: fsync d11/d25/d46/d40/fed 0 2024-03-22T02:28:46.847 INFO:tasks.workunit.client.0.smithi103.stdout:4/725: truncate d3/d1e/f7f 4033361 0 2024-03-22T02:28:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:46 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi122 to 2581M 2024-03-22T02:28:46.849 INFO:tasks.workunit.client.0.smithi103.stdout:2/736: fsync d11/d25/d46/d48/fde 0 2024-03-22T02:28:46.849 INFO:tasks.workunit.client.1.smithi122.stdout:2/168: dread df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:46.851 INFO:tasks.workunit.client.0.smithi103.stdout:5/763: dwrite d0/d14/d23/d5c/fc9 [0,4194304] 0 2024-03-22T02:28:46.852 INFO:tasks.workunit.client.0.smithi103.stdout:1/778: write db/d4a/d78/d92/da8/ddd/d29/d41/f99 [57572,22171] 0 2024-03-22T02:28:46.852 INFO:tasks.workunit.client.1.smithi122.stdout:2/169: readlink df/l35 0 2024-03-22T02:28:46.854 INFO:tasks.workunit.client.1.smithi122.stdout:6/241: dwrite d7/de/f3e [0,4194304] 0 2024-03-22T02:28:46.854 INFO:tasks.workunit.client.0.smithi103.stdout:2/737: readlink d11/d25/d46/d48/l4d 0 2024-03-22T02:28:46.860 INFO:tasks.workunit.client.1.smithi122.stdout:2/170: rename df/f14 to df/d1f/f3e 0 2024-03-22T02:28:46.860 INFO:tasks.workunit.client.0.smithi103.stdout:1/779: dread - db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/fc1 zero size 2024-03-22T02:28:46.860 INFO:tasks.workunit.client.0.smithi103.stdout:5/764: rename d0/d14/d23/d6f/fa7 to d0/d14/d23/d36/de1/f109 0 2024-03-22T02:28:46.862 INFO:tasks.workunit.client.0.smithi103.stdout:2/738: unlink d11/d19/d7e/f93 0 2024-03-22T02:28:46.864 INFO:tasks.workunit.client.0.smithi103.stdout:5/765: write d0/d14/d23/d36/d95/f7d [1204752,97527] 0 2024-03-22T02:28:46.864 INFO:tasks.workunit.client.0.smithi103.stdout:1/780: chown db/d4a/d78/d92/da8/ddd/d29/d6c 78 1 2024-03-22T02:28:46.865 INFO:tasks.workunit.client.1.smithi122.stdout:4/151: dwrite d0/d25/f1d [0,4194304] 0 2024-03-22T02:28:46.866 INFO:tasks.workunit.client.0.smithi103.stdout:4/726: write d3/d1e/f6d [294826,59437] 0 2024-03-22T02:28:46.866 INFO:tasks.workunit.client.1.smithi122.stdout:0/136: dread d11/f13 [4194304,4194304] 0 2024-03-22T02:28:46.869 INFO:tasks.workunit.client.0.smithi103.stdout:4/727: truncate d3/d38/d5c/faa 115807 0 2024-03-22T02:28:46.871 INFO:tasks.workunit.client.1.smithi122.stdout:4/152: creat d0/d25/d7/f34 x:0 0 0 2024-03-22T02:28:46.871 INFO:tasks.workunit.client.0.smithi103.stdout:2/739: mkdir d11/d25/d63/d45/d47/d83/d78/df6 0 2024-03-22T02:28:46.873 INFO:tasks.workunit.client.1.smithi122.stdout:4/153: read - d0/d25/d7/f34 zero size 2024-03-22T02:28:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:46 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi122 to 2581M 2024-03-22T02:28:46.874 INFO:tasks.workunit.client.0.smithi103.stdout:4/728: mknod d3/d1e/d68/dbc/cf6 0 2024-03-22T02:28:46.877 INFO:tasks.workunit.client.0.smithi103.stdout:2/740: creat d11/d25/d63/d45/d47/d83/da3/ff7 x:0 0 0 2024-03-22T02:28:46.877 INFO:tasks.workunit.client.1.smithi122.stdout:2/171: dwrite df/d1f/d23/f3b [0,4194304] 0 2024-03-22T02:28:46.877 INFO:tasks.workunit.client.0.smithi103.stdout:1/781: fdatasync db/d4a/d78/d92/da8/ddd/d29/d41/f99 0 2024-03-22T02:28:46.877 INFO:tasks.workunit.client.0.smithi103.stdout:5/766: link d0/d14/dc0/le4 d0/d14/d23/d2a/d33/db5/dbd/ded/d103/d105/l10a 0 2024-03-22T02:28:46.878 INFO:tasks.workunit.client.1.smithi122.stdout:4/154: readlink d0/d12/l2f 0 2024-03-22T02:28:46.878 INFO:tasks.workunit.client.0.smithi103.stdout:4/729: mknod d3/cf7 0 2024-03-22T02:28:46.879 INFO:tasks.workunit.client.1.smithi122.stdout:0/137: dwrite fb [0,4194304] 0 2024-03-22T02:28:46.882 INFO:tasks.workunit.client.0.smithi103.stdout:2/741: unlink d11/d25/d46/d48/f7a 0 2024-03-22T02:28:46.883 INFO:tasks.workunit.client.1.smithi122.stdout:4/155: mknod d0/d12/d15/c35 0 2024-03-22T02:28:46.885 INFO:tasks.workunit.client.1.smithi122.stdout:4/156: fsync d0/d12/f21 0 2024-03-22T02:28:46.885 INFO:tasks.workunit.client.1.smithi122.stdout:1/164: truncate dc/fe 1159648 0 2024-03-22T02:28:46.886 INFO:tasks.workunit.client.0.smithi103.stdout:2/742: chown d11/d25/d63/lbf 593971 1 2024-03-22T02:28:46.886 INFO:tasks.workunit.client.0.smithi103.stdout:1/782: truncate db/d4a/d78/d92/da8/ddd/d28/d97/fb3 1383314 0 2024-03-22T02:28:46.890 INFO:tasks.workunit.client.1.smithi122.stdout:2/172: link df/d1f/l26 df/d1f/d2a/d39/l3f 0 2024-03-22T02:28:46.890 INFO:tasks.workunit.client.1.smithi122.stdout:6/242: dwrite d7/f1b [0,4194304] 0 2024-03-22T02:28:46.891 INFO:tasks.workunit.client.0.smithi103.stdout:0/722: dwrite dd/d16/d4b/d96/d29/d48/ded/fcd [0,4194304] 0 2024-03-22T02:28:46.891 INFO:tasks.workunit.client.1.smithi122.stdout:2/173: write df/d1f/f21 [2111815,108343] 0 2024-03-22T02:28:46.892 INFO:tasks.workunit.client.0.smithi103.stdout:4/730: mkdir d3/d38/d5c/d90/df8 0 2024-03-22T02:28:46.893 INFO:tasks.workunit.client.1.smithi122.stdout:1/165: write dc/f2a [79850,55185] 0 2024-03-22T02:28:46.893 INFO:tasks.workunit.client.0.smithi103.stdout:0/723: dread - dd/d16/d4b/d96/d29/d48/fec zero size 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.892+0000 7fa29aa27700 1 -- 172.21.15.103:0/1461038847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 msgr2=0x7fa28c093ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.892+0000 7fa29aa27700 1 --2- 172.21.15.103:0/1461038847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c093ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa294064460 tx=0x7fa29405f540 comp rx=0 tx=0).stop 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 -- 172.21.15.103:0/1461038847 shutdown_connections 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 --2- 172.21.15.103:0/1461038847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 --2- 172.21.15.103:0/1461038847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c093ee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 -- 172.21.15.103:0/1461038847 >> 172.21.15.103:0/1461038847 conn(0x7fa28c08f230 msgr2=0x7fa28c091660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 -- 172.21.15.103:0/1461038847 shutdown_connections 2024-03-22T02:28:46.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.893+0000 7fa29aa27700 1 -- 172.21.15.103:0/1461038847 wait complete. 2024-03-22T02:28:46.895 INFO:tasks.workunit.client.1.smithi122.stdout:6/243: chown d7/de/d27/f36 741259 1 2024-03-22T02:28:46.895 INFO:tasks.workunit.client.0.smithi103.stdout:0/724: chown dd/d16/d4b/d96/d35/l47 10428 1 2024-03-22T02:28:46.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.894+0000 7fa29aa27700 1 Processor -- start 2024-03-22T02:28:46.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.894+0000 7fa29aa27700 1 -- start start 2024-03-22T02:28:46.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa29aa27700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa29aa27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa29aa27700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a62d0 con 0x7fa28c094d80 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa29aa27700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a6440 con 0x7fa28c093ad0 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa299224700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa299224700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40014/0 (socket says 172.21.15.103:40014) 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.895+0000 7fa299224700 1 -- 172.21.15.103:0/3039196846 learned_addr learned my addr 172.21.15.103:0/3039196846 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.896+0000 7fa299a25700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.896+0000 7fa299224700 1 -- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 msgr2=0x7fa28c0a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.896+0000 7fa299224700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c0a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:46.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.896+0000 7fa299224700 1 -- 172.21.15.103:0/3039196846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29404bed0 con 0x7fa28c094d80 2024-03-22T02:28:46.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.896+0000 7fa299224700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa2900041d0 tx=0x7fa29000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:46.898 INFO:tasks.workunit.client.0.smithi103.stdout:1/783: unlink db/d4a/d78/d92/da8/ddd/d29/cce 0 2024-03-22T02:28:46.898 INFO:tasks.workunit.client.1.smithi122.stdout:4/157: read d0/d25/f1e [3850306,66761] 0 2024-03-22T02:28:46.898 INFO:tasks.workunit.client.1.smithi122.stdout:1/166: mknod dc/d2b/c2e 0 2024-03-22T02:28:46.899 INFO:tasks.workunit.client.1.smithi122.stdout:1/167: readlink dc/l23 0 2024-03-22T02:28:46.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.897+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa290007cc0 con 0x7fa28c094d80 2024-03-22T02:28:46.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.897+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c0aae40 con 0x7fa28c094d80 2024-03-22T02:28:46.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.897+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c0ab3d0 con 0x7fa28c094d80 2024-03-22T02:28:46.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.897+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29000f040 con 0x7fa28c094d80 2024-03-22T02:28:46.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.897+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa29000cb50 con 0x7fa28c094d80 2024-03-22T02:28:46.900 INFO:tasks.workunit.client.1.smithi122.stdout:2/174: rename df/d1f/d2a/c31 to df/d1f/d33/d34/c40 0 2024-03-22T02:28:46.900 INFO:tasks.workunit.client.1.smithi122.stdout:6/244: symlink d7/de/dd/d39/d46/l50 0 2024-03-22T02:28:46.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.899+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2780052e0 con 0x7fa28c094d80 2024-03-22T02:28:46.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.900+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7fa29000ccb0 con 0x7fa28c094d80 2024-03-22T02:28:46.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.901+0000 7fa28a7fc700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 0x7fa28006de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:46.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.901+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa29008a050 con 0x7fa28c094d80 2024-03-22T02:28:46.902 INFO:tasks.workunit.client.1.smithi122.stdout:2/175: fdatasync df/f2b 0 2024-03-22T02:28:46.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.902+0000 7fa299a25700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 0x7fa28006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:46.903 INFO:tasks.workunit.client.1.smithi122.stdout:4/158: fsync d0/d25/d7/f16 0 2024-03-22T02:28:46.904 INFO:tasks.workunit.client.0.smithi103.stdout:4/731: rename d3/d30/d45/d6e to d3/d1e/d78/df9 0 2024-03-22T02:28:46.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.904+0000 7fa299a25700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 0x7fa28006de90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa29404d500 tx=0x7fa294077040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:46.906 INFO:tasks.workunit.client.1.smithi122.stdout:2/176: chown f4 148647 1 2024-03-22T02:28:46.908 INFO:tasks.workunit.client.0.smithi103.stdout:1/784: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/c10c 0 2024-03-22T02:28:46.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:46.908+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fa290050040 con 0x7fa28c094d80 2024-03-22T02:28:46.910 INFO:tasks.workunit.client.1.smithi122.stdout:1/168: dwrite dc/f2a [0,4194304] 0 2024-03-22T02:28:46.911 INFO:tasks.workunit.client.0.smithi103.stdout:4/732: dread d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:46.915 INFO:tasks.workunit.client.0.smithi103.stdout:0/725: dwrite dd/d16/d4b/d96/d64/dab/dc6/d89/f8b [0,4194304] 0 2024-03-22T02:28:46.916 INFO:tasks.workunit.client.1.smithi122.stdout:6/245: dwrite f6 [4194304,4194304] 0 2024-03-22T02:28:46.916 INFO:tasks.workunit.client.1.smithi122.stdout:4/159: symlink d0/d25/l36 0 2024-03-22T02:28:46.918 INFO:tasks.workunit.client.1.smithi122.stdout:1/169: rename dc/f1b to dc/f2f 0 2024-03-22T02:28:46.918 INFO:tasks.workunit.client.0.smithi103.stdout:1/785: truncate db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/fc1 574078 0 2024-03-22T02:28:46.920 INFO:tasks.workunit.client.0.smithi103.stdout:0/726: mknod dd/d16/d4b/d96/d29/d48/cf0 0 2024-03-22T02:28:46.921 INFO:tasks.workunit.client.0.smithi103.stdout:4/733: dwrite d3/d38/f70 [0,4194304] 0 2024-03-22T02:28:46.921 INFO:tasks.workunit.client.1.smithi122.stdout:2/177: getdents df/d1f/d2a/d39 0 2024-03-22T02:28:46.922 INFO:tasks.workunit.client.1.smithi122.stdout:2/178: chown df/d1f/d2a/c2e 495 1 2024-03-22T02:28:46.923 INFO:tasks.workunit.client.1.smithi122.stdout:1/170: write dc/f1d [3575169,90038] 0 2024-03-22T02:28:46.924 INFO:tasks.workunit.client.1.smithi122.stdout:1/171: stat c1 0 2024-03-22T02:28:46.926 INFO:tasks.workunit.client.1.smithi122.stdout:2/179: mkdir df/d1f/d3a/d41 0 2024-03-22T02:28:46.926 INFO:tasks.workunit.client.0.smithi103.stdout:0/727: symlink dd/d16/d4b/d96/d35/lf1 0 2024-03-22T02:28:46.927 INFO:tasks.workunit.client.1.smithi122.stdout:6/246: creat d7/de/dd/d20/f51 x:0 0 0 2024-03-22T02:28:46.927 INFO:tasks.workunit.client.1.smithi122.stdout:2/180: chown fd 1062 1 2024-03-22T02:28:46.929 INFO:tasks.workunit.client.1.smithi122.stdout:6/247: chown d7/f15 58350 1 2024-03-22T02:28:46.930 INFO:tasks.workunit.client.1.smithi122.stdout:1/172: dread dc/f1d [0,4194304] 0 2024-03-22T02:28:46.931 INFO:tasks.workunit.client.0.smithi103.stdout:1/786: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/c10d 0 2024-03-22T02:28:46.933 INFO:tasks.workunit.client.0.smithi103.stdout:1/787: chown db/d4a/d78/d92/da8/ddd/d28/c4f 65411592 1 2024-03-22T02:28:46.934 INFO:tasks.workunit.client.1.smithi122.stdout:6/248: mkdir d7/de/d27/d52 0 2024-03-22T02:28:46.935 INFO:tasks.workunit.client.1.smithi122.stdout:4/160: dread d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:46.935 INFO:tasks.workunit.client.0.smithi103.stdout:2/743: truncate d11/d19/da7/dd3/f36 7286527 0 2024-03-22T02:28:46.936 INFO:tasks.workunit.client.1.smithi122.stdout:2/181: rename df/d1f/l26 to df/d1f/d3a/d41/l42 0 2024-03-22T02:28:46.938 INFO:tasks.workunit.client.1.smithi122.stdout:1/173: link dc/f21 dc/f30 0 2024-03-22T02:28:46.938 INFO:tasks.workunit.client.0.smithi103.stdout:0/728: mknod dd/d16/d4b/d96/cf2 0 2024-03-22T02:28:46.939 INFO:tasks.workunit.client.0.smithi103.stdout:1/788: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/l106 117282 1 2024-03-22T02:28:46.940 INFO:tasks.workunit.client.1.smithi122.stdout:6/249: mknod d7/de/dd/df/c53 0 2024-03-22T02:28:46.940 INFO:tasks.workunit.client.1.smithi122.stdout:6/250: readlink d7/l1c 0 2024-03-22T02:28:46.941 INFO:tasks.workunit.client.0.smithi103.stdout:0/729: write dd/d16/d4b/d96/d29/d48/ded/fd1 [763392,114561] 0 2024-03-22T02:28:46.941 INFO:tasks.workunit.client.1.smithi122.stdout:4/161: fsync d0/d25/f8 0 2024-03-22T02:28:46.943 INFO:tasks.workunit.client.0.smithi103.stdout:2/744: truncate d11/d25/d46/d40/f67 3353742 0 2024-03-22T02:28:46.943 INFO:tasks.workunit.client.1.smithi122.stdout:2/182: creat df/d1f/d33/d34/f43 x:0 0 0 2024-03-22T02:28:46.944 INFO:tasks.workunit.client.1.smithi122.stdout:1/174: creat dc/d2b/f31 x:0 0 0 2024-03-22T02:28:46.945 INFO:tasks.workunit.client.0.smithi103.stdout:0/730: write dd/d34/fac [570160,8422] 0 2024-03-22T02:28:46.947 INFO:tasks.workunit.client.1.smithi122.stdout:4/162: read d0/d25/f13 [3393445,35827] 0 2024-03-22T02:28:46.947 INFO:tasks.workunit.client.1.smithi122.stdout:6/251: dread d7/f15 [0,4194304] 0 2024-03-22T02:28:46.949 INFO:tasks.workunit.client.0.smithi103.stdout:2/745: getdents d11/d25/d63/d45/d47/d83/d78/df6 0 2024-03-22T02:28:46.949 INFO:tasks.workunit.client.1.smithi122.stdout:1/175: link dc/c19 dc/c32 0 2024-03-22T02:28:46.949 INFO:tasks.workunit.client.1.smithi122.stdout:2/183: dread df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:46.950 INFO:tasks.workunit.client.1.smithi122.stdout:1/176: chown c8 735 1 2024-03-22T02:28:46.950 INFO:tasks.workunit.client.0.smithi103.stdout:2/746: truncate d11/d19/d7e/fa9 60250 0 2024-03-22T02:28:46.951 INFO:tasks.workunit.client.1.smithi122.stdout:6/252: mkdir d7/de/dd/d20/d2d/d54 0 2024-03-22T02:28:46.952 INFO:tasks.workunit.client.0.smithi103.stdout:1/789: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d108/l10e 0 2024-03-22T02:28:46.952 INFO:tasks.workunit.client.1.smithi122.stdout:4/163: creat d0/f37 x:0 0 0 2024-03-22T02:28:46.952 INFO:tasks.workunit.client.1.smithi122.stdout:4/164: fsync d0/d12/f21 0 2024-03-22T02:28:46.954 INFO:tasks.workunit.client.0.smithi103.stdout:2/747: readlink d11/d19/da7/dd3/laf 0 2024-03-22T02:28:46.954 INFO:tasks.workunit.client.1.smithi122.stdout:2/184: write df/d1f/f2d [2260616,91657] 0 2024-03-22T02:28:46.956 INFO:tasks.workunit.client.0.smithi103.stdout:2/748: write d11/d25/d63/d45/d47/d83/fa5 [537685,53338] 0 2024-03-22T02:28:46.956 INFO:tasks.workunit.client.1.smithi122.stdout:6/253: mknod d7/de/d27/c55 0 2024-03-22T02:28:46.957 INFO:tasks.workunit.client.1.smithi122.stdout:2/185: write df/f10 [1687851,122424] 0 2024-03-22T02:28:46.961 INFO:tasks.workunit.client.0.smithi103.stdout:1/790: mkdir db/d4a/d78/d92/da8/ddd/d18/dca/de5/d10f 0 2024-03-22T02:28:46.962 INFO:tasks.workunit.client.0.smithi103.stdout:4/734: dwrite d3/f7 [0,4194304] 0 2024-03-22T02:28:46.963 INFO:tasks.workunit.client.1.smithi122.stdout:4/165: dwrite d0/f19 [0,4194304] 0 2024-03-22T02:28:46.963 INFO:tasks.workunit.client.0.smithi103.stdout:1/791: write db/f25 [4998703,15745] 0 2024-03-22T02:28:46.965 INFO:tasks.workunit.client.1.smithi122.stdout:2/186: mknod df/d1f/d23/c44 0 2024-03-22T02:28:46.966 INFO:tasks.workunit.client.1.smithi122.stdout:1/177: dwrite dc/f2f [4194304,4194304] 0 2024-03-22T02:28:46.968 INFO:tasks.workunit.client.0.smithi103.stdout:2/749: symlink d11/d19/da7/dd3/df1/lf8 0 2024-03-22T02:28:46.969 INFO:tasks.workunit.client.1.smithi122.stdout:6/254: creat d7/de/dd/f56 x:0 0 0 2024-03-22T02:28:46.972 INFO:tasks.workunit.client.0.smithi103.stdout:0/731: dwrite dd/d16/d4b/f50 [4194304,4194304] 0 2024-03-22T02:28:46.974 INFO:tasks.workunit.client.1.smithi122.stdout:2/187: mknod df/d1f/d23/c45 0 2024-03-22T02:28:46.974 INFO:tasks.workunit.client.1.smithi122.stdout:2/188: chown df/d1f 2242 1 2024-03-22T02:28:46.975 INFO:tasks.workunit.client.0.smithi103.stdout:1/792: mkdir db/d4a/d78/d92/da8/ddd/d18/dca/d110 0 2024-03-22T02:28:46.975 INFO:tasks.workunit.client.1.smithi122.stdout:2/189: chown f6 184 1 2024-03-22T02:28:46.978 INFO:tasks.workunit.client.1.smithi122.stdout:4/166: rmdir d0/d33 0 2024-03-22T02:28:46.979 INFO:tasks.workunit.client.1.smithi122.stdout:6/255: creat d7/de/d27/d52/f57 x:0 0 0 2024-03-22T02:28:46.979 INFO:tasks.workunit.client.0.smithi103.stdout:4/735: mknod d3/d38/d5c/cfa 0 2024-03-22T02:28:46.983 INFO:tasks.workunit.client.1.smithi122.stdout:4/167: write d0/d25/f1e [3781904,74761] 0 2024-03-22T02:28:46.985 INFO:tasks.workunit.client.0.smithi103.stdout:1/793: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d111 0 2024-03-22T02:28:46.985 INFO:tasks.workunit.client.1.smithi122.stdout:6/256: mknod d7/de/dd/df/c58 0 2024-03-22T02:28:46.987 INFO:tasks.workunit.client.1.smithi122.stdout:1/178: getdents dc/d2b 0 2024-03-22T02:28:46.987 INFO:tasks.workunit.client.0.smithi103.stdout:1/794: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f100 19063 1 2024-03-22T02:28:46.988 INFO:tasks.workunit.client.0.smithi103.stdout:0/732: rename dd/d16/d4b/d96/d64/dab/dc6/d79 to dd/d34/d53/df3 0 2024-03-22T02:28:46.989 INFO:tasks.workunit.client.1.smithi122.stdout:4/168: creat d0/d12/d15/d2d/f38 x:0 0 0 2024-03-22T02:28:46.991 INFO:tasks.workunit.client.0.smithi103.stdout:1/795: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/d112 0 2024-03-22T02:28:46.993 INFO:tasks.workunit.client.0.smithi103.stdout:0/733: dread dd/d16/d4b/d96/d29/d48/ded/fd1 [0,4194304] 0 2024-03-22T02:28:46.995 INFO:tasks.workunit.client.1.smithi122.stdout:1/179: dwrite dc/d2b/f31 [0,4194304] 0 2024-03-22T02:28:46.995 INFO:tasks.workunit.client.0.smithi103.stdout:1/796: unlink db/d4a/d78/d92/da8/ddd/d2e/f8a 0 2024-03-22T02:28:46.996 INFO:tasks.workunit.client.1.smithi122.stdout:1/180: fsync dc/f24 0 2024-03-22T02:28:47.000 INFO:tasks.workunit.client.1.smithi122.stdout:4/169: dwrite d0/d25/d7/f34 [0,4194304] 0 2024-03-22T02:28:47.001 INFO:tasks.workunit.client.1.smithi122.stdout:4/170: stat d0/d12/d15/d2d/f38 0 2024-03-22T02:28:47.002 INFO:tasks.workunit.client.0.smithi103.stdout:1/797: mkdir db/d4a/d78/d92/da8/ddd/d28/db2/dc8/d113 0 2024-03-22T02:28:47.003 INFO:tasks.workunit.client.0.smithi103.stdout:2/750: dwrite d11/d25/f7c [0,4194304] 0 2024-03-22T02:28:47.007 INFO:tasks.workunit.client.1.smithi122.stdout:1/181: dwrite dc/f24 [0,4194304] 0 2024-03-22T02:28:47.007 INFO:tasks.workunit.client.0.smithi103.stdout:4/736: truncate d3/f9b 3766706 0 2024-03-22T02:28:47.009 INFO:tasks.workunit.client.1.smithi122.stdout:4/171: rename d0/d25/l1a to d0/l39 0 2024-03-22T02:28:47.010 INFO:tasks.workunit.client.0.smithi103.stdout:1/798: creat db/d4a/d78/d92/da8/ddd/d18/dca/f114 x:0 0 0 2024-03-22T02:28:47.011 INFO:tasks.workunit.client.0.smithi103.stdout:2/751: symlink d11/d19/d4e/de9/lf9 0 2024-03-22T02:28:47.011 INFO:tasks.workunit.client.1.smithi122.stdout:4/172: write d0/d25/d7/f34 [3916028,98962] 0 2024-03-22T02:28:47.012 INFO:tasks.workunit.client.1.smithi122.stdout:1/182: chown dc/c13 6 1 2024-03-22T02:28:47.014 INFO:tasks.workunit.client.0.smithi103.stdout:0/734: write dd/f1d [2443680,19590] 0 2024-03-22T02:28:47.017 INFO:tasks.workunit.client.1.smithi122.stdout:2/190: write df/d1f/d23/f29 [624332,90480] 0 2024-03-22T02:28:47.019 INFO:tasks.workunit.client.1.smithi122.stdout:6/257: write d7/de/d27/f18 [1154164,60744] 0 2024-03-22T02:28:47.022 INFO:tasks.workunit.client.1.smithi122.stdout:6/258: rmdir d7/de/dd/d39/d46 39 2024-03-22T02:28:47.023 INFO:tasks.workunit.client.1.smithi122.stdout:9/201: sync 2024-03-22T02:28:47.023 INFO:tasks.workunit.client.1.smithi122.stdout:5/154: sync 2024-03-22T02:28:47.023 INFO:tasks.workunit.client.1.smithi122.stdout:7/113: sync 2024-03-22T02:28:47.024 INFO:tasks.workunit.client.1.smithi122.stdout:3/112: sync 2024-03-22T02:28:47.026 INFO:tasks.workunit.client.1.smithi122.stdout:6/259: fdatasync d7/de/dd/d20/f21 0 2024-03-22T02:28:47.028 INFO:tasks.workunit.client.1.smithi122.stdout:2/191: dwrite df/d1f/f3e [0,4194304] 0 2024-03-22T02:28:47.028 INFO:tasks.workunit.client.1.smithi122.stdout:9/202: truncate db/d23/f1a 4271249 0 2024-03-22T02:28:47.030 INFO:tasks.workunit.client.1.smithi122.stdout:9/203: chown db/d23/d1f/d21 377451364 1 2024-03-22T02:28:47.031 INFO:tasks.workunit.client.1.smithi122.stdout:5/155: mkdir d8/da/d28/d35 0 2024-03-22T02:28:47.031 INFO:tasks.workunit.client.0.smithi103.stdout:8/839: sync 2024-03-22T02:28:47.032 INFO:tasks.workunit.client.0.smithi103.stdout:4/737: creat d3/d38/d5c/d90/df8/ffb x:0 0 0 2024-03-22T02:28:47.033 INFO:tasks.workunit.client.0.smithi103.stdout:8/840: readlink d4/d2a/d3e/le9 0 2024-03-22T02:28:47.033 INFO:tasks.workunit.client.1.smithi122.stdout:9/204: truncate db/df/f15 2209359 0 2024-03-22T02:28:47.036 INFO:tasks.workunit.client.0.smithi103.stdout:8/841: fdatasync d4/d26/d27/f2c 0 2024-03-22T02:28:47.036 INFO:tasks.workunit.client.1.smithi122.stdout:9/205: chown db/d23/d1f/d21/d25/c43 510828 1 2024-03-22T02:28:47.036 INFO:tasks.workunit.client.1.smithi122.stdout:5/156: dread - d8/f33 zero size 2024-03-22T02:28:47.036 INFO:tasks.workunit.client.0.smithi103.stdout:2/752: rename d11/d19/d7e/cb5 to d11/cfa 0 2024-03-22T02:28:47.038 INFO:tasks.workunit.client.1.smithi122.stdout:5/157: readlink d8/l29 0 2024-03-22T02:28:47.039 INFO:tasks.workunit.client.0.smithi103.stdout:1/799: rename db/f17 to db/f115 0 2024-03-22T02:28:47.040 INFO:tasks.workunit.client.1.smithi122.stdout:6/260: dwrite d7/de/dd/d23/f48 [0,4194304] 0 2024-03-22T02:28:47.041 INFO:tasks.workunit.client.1.smithi122.stdout:2/192: dread df/d1f/d23/f29 [0,4194304] 0 2024-03-22T02:28:47.041 INFO:tasks.workunit.client.1.smithi122.stdout:5/158: truncate d8/da/d28/f2c 650461 0 2024-03-22T02:28:47.042 INFO:tasks.workunit.client.0.smithi103.stdout:0/735: rename dd/d16/d4b/d96/d64/d6a/d8d to dd/d16/d4b/d96/d64/d6a/dd5/deb/d9a/df4 0 2024-03-22T02:28:47.043 INFO:tasks.workunit.client.0.smithi103.stdout:8/842: mknod d4/d26/d27/d7e/d39/d49/d79/c113 0 2024-03-22T02:28:47.044 INFO:tasks.workunit.client.1.smithi122.stdout:7/114: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:47.045 INFO:tasks.workunit.client.1.smithi122.stdout:9/206: creat db/d23/d1f/d21/d25/f49 x:0 0 0 2024-03-22T02:28:47.046 INFO:tasks.workunit.client.1.smithi122.stdout:4/173: truncate d0/d25/f1e 985541 0 2024-03-22T02:28:47.048 INFO:tasks.workunit.client.1.smithi122.stdout:1/183: truncate dc/f1a 4129321 0 2024-03-22T02:28:47.049 INFO:tasks.workunit.client.0.smithi103.stdout:1/800: rmdir db/d4a/d78/d92/da8/ddd/d18/dca/d110 0 2024-03-22T02:28:47.050 INFO:tasks.workunit.client.0.smithi103.stdout:0/736: dwrite dd/f1d [0,4194304] 0 2024-03-22T02:28:47.050 INFO:tasks.workunit.client.1.smithi122.stdout:4/174: write d0/d25/d7/f22 [796025,127455] 0 2024-03-22T02:28:47.051 INFO:tasks.workunit.client.1.smithi122.stdout:7/115: chown d0/c13 31011319 1 2024-03-22T02:28:47.052 INFO:tasks.workunit.client.0.smithi103.stdout:1/801: stat db/d4a/d78/d92/fc5 0 2024-03-22T02:28:47.053 INFO:tasks.workunit.client.0.smithi103.stdout:0/737: stat dd/d16/d4b/d96/d29/d48/d5c/d9b/lcb 0 2024-03-22T02:28:47.054 INFO:tasks.workunit.client.1.smithi122.stdout:2/193: dread df/d1f/f3e [0,4194304] 0 2024-03-22T02:28:47.054 INFO:tasks.workunit.client.0.smithi103.stdout:8/843: rename d4/d26/d27/d7e/d39/d49/d79/l86 to d4/d26/d27/da3/df5/d105/l114 0 2024-03-22T02:28:47.055 INFO:tasks.workunit.client.1.smithi122.stdout:1/184: creat dc/d2b/f33 x:0 0 0 2024-03-22T02:28:47.057 INFO:tasks.workunit.client.1.smithi122.stdout:3/113: truncate d2/f7 2265883 0 2024-03-22T02:28:47.059 INFO:tasks.workunit.client.0.smithi103.stdout:1/802: chown db/d4a/d78/d92/da8/ddd/f1c 1 1 2024-03-22T02:28:47.059 INFO:tasks.workunit.client.0.smithi103.stdout:1/803: chown db/d4a/d78 631057 1 2024-03-22T02:28:47.061 INFO:tasks.workunit.client.0.smithi103.stdout:0/738: creat dd/d16/d4b/d96/d29/d48/d5c/d9b/ff5 x:0 0 0 2024-03-22T02:28:47.062 INFO:tasks.workunit.client.0.smithi103.stdout:8/844: mknod d4/d2a/c115 0 2024-03-22T02:28:47.063 INFO:tasks.workunit.client.0.smithi103.stdout:0/739: write dd/d16/d4b/d96/d64/fa3 [4512875,106888] 0 2024-03-22T02:28:47.063 INFO:tasks.workunit.client.0.smithi103.stdout:8/845: chown d4/d26 3646324 1 2024-03-22T02:28:47.063 INFO:tasks.workunit.client.1.smithi122.stdout:1/185: dwrite dc/d2b/f2d [0,4194304] 0 2024-03-22T02:28:47.063 INFO:tasks.workunit.client.1.smithi122.stdout:6/261: creat d7/de/dd/d20/d2d/d54/f59 x:0 0 0 2024-03-22T02:28:47.064 INFO:tasks.workunit.client.1.smithi122.stdout:8/181: sync 2024-03-22T02:28:47.065 INFO:tasks.workunit.client.1.smithi122.stdout:7/116: fsync d0/d4/f9 0 2024-03-22T02:28:47.066 INFO:tasks.workunit.client.0.smithi103.stdout:8/846: truncate d4/d26/d27/d7e/d39/d49/d51/f109 618568 0 2024-03-22T02:28:47.067 INFO:tasks.workunit.client.1.smithi122.stdout:2/194: creat df/d1f/d33/f46 x:0 0 0 2024-03-22T02:28:47.069 INFO:tasks.workunit.client.0.smithi103.stdout:4/738: truncate d3/d31/f47 2368472 0 2024-03-22T02:28:47.069 INFO:tasks.workunit.client.1.smithi122.stdout:5/159: rename d8/d1e/l25 to d8/l36 0 2024-03-22T02:28:47.072 INFO:tasks.workunit.client.1.smithi122.stdout:9/207: creat db/df/f4a x:0 0 0 2024-03-22T02:28:47.073 INFO:tasks.workunit.client.1.smithi122.stdout:1/186: unlink dc/f24 0 2024-03-22T02:28:47.074 INFO:tasks.workunit.client.1.smithi122.stdout:6/262: creat d7/de/dd/d20/d2d/d54/f5a x:0 0 0 2024-03-22T02:28:47.075 INFO:tasks.workunit.client.0.smithi103.stdout:2/753: mkdir d11/d25/d46/db6/dfb 0 2024-03-22T02:28:47.075 INFO:tasks.workunit.client.1.smithi122.stdout:6/263: readlink d7/de/l25 0 2024-03-22T02:28:47.075 INFO:tasks.workunit.client.1.smithi122.stdout:8/182: symlink d1/d9/l37 0 2024-03-22T02:28:47.076 INFO:tasks.workunit.client.0.smithi103.stdout:8/847: mknod d4/d2a/d3e/da4/c116 0 2024-03-22T02:28:47.077 INFO:tasks.workunit.client.0.smithi103.stdout:8/848: chown d4/d26/d27/d7e/d39/ld1 43 1 2024-03-22T02:28:47.077 INFO:tasks.workunit.client.1.smithi122.stdout:7/117: rmdir d0/d4 39 2024-03-22T02:28:47.078 INFO:tasks.workunit.client.0.smithi103.stdout:4/739: fdatasync d3/d69/f3b 0 2024-03-22T02:28:47.078 INFO:tasks.workunit.client.1.smithi122.stdout:3/114: stat d2/f7 0 2024-03-22T02:28:47.081 INFO:tasks.workunit.client.1.smithi122.stdout:5/160: write d8/f17 [1932877,125707] 0 2024-03-22T02:28:47.081 INFO:tasks.workunit.client.1.smithi122.stdout:5/161: stat c1 0 2024-03-22T02:28:47.082 INFO:tasks.workunit.client.1.smithi122.stdout:2/195: rename df/d1f/d23/f29 to df/d1f/d3a/d41/f47 0 2024-03-22T02:28:47.083 INFO:tasks.workunit.client.0.smithi103.stdout:8/849: mkdir d4/d26/d27/da3/d117 0 2024-03-22T02:28:47.084 INFO:tasks.workunit.client.1.smithi122.stdout:4/175: write d0/d25/d7/fb [5373,4693] 0 2024-03-22T02:28:47.085 INFO:tasks.workunit.client.1.smithi122.stdout:9/208: symlink db/d23/l4b 0 2024-03-22T02:28:47.086 INFO:tasks.workunit.client.0.smithi103.stdout:8/850: chown d4/d19/d8c/f8f 175226175 1 2024-03-22T02:28:47.087 INFO:tasks.workunit.client.1.smithi122.stdout:9/209: truncate db/d23/d1e/d30/f35 918696 0 2024-03-22T02:28:47.087 INFO:tasks.workunit.client.0.smithi103.stdout:1/804: dwrite db/d4a/d78/d92/da8/ddd/d18/d2d/fb0 [0,4194304] 0 2024-03-22T02:28:47.088 INFO:tasks.workunit.client.1.smithi122.stdout:9/210: readlink db/d23/l4b 0 2024-03-22T02:28:47.088 INFO:tasks.workunit.client.0.smithi103.stdout:0/740: creat dd/d16/d4b/ff6 x:0 0 0 2024-03-22T02:28:47.088 INFO:tasks.workunit.client.1.smithi122.stdout:6/264: readlink d7/de/dd/l3c 0 2024-03-22T02:28:47.089 INFO:tasks.workunit.client.0.smithi103.stdout:8/851: read f1 [3749085,79172] 0 2024-03-22T02:28:47.091 INFO:tasks.workunit.client.0.smithi103.stdout:1/805: readlink db/d4a/d78/d92/da8/ddd/d18/d2d/l47 0 2024-03-22T02:28:47.093 INFO:tasks.workunit.client.1.smithi122.stdout:2/196: symlink df/d1f/d33/l48 0 2024-03-22T02:28:47.094 INFO:tasks.workunit.client.0.smithi103.stdout:0/741: fsync dd/d34/d53/fa1 0 2024-03-22T02:28:47.098 INFO:tasks.workunit.client.1.smithi122.stdout:9/211: dwrite db/df/f15 [0,4194304] 0 2024-03-22T02:28:47.099 INFO:tasks.workunit.client.1.smithi122.stdout:2/197: dread df/d1f/d3a/d41/f47 [0,4194304] 0 2024-03-22T02:28:47.099 INFO:tasks.workunit.client.0.smithi103.stdout:4/740: creat d3/ffc x:0 0 0 2024-03-22T02:28:47.099 INFO:tasks.workunit.client.0.smithi103.stdout:2/754: dread d11/d25/d46/f29 [0,4194304] 0 2024-03-22T02:28:47.099 INFO:tasks.workunit.client.1.smithi122.stdout:2/198: read df/f37 [507664,57820] 0 2024-03-22T02:28:47.100 INFO:tasks.workunit.client.1.smithi122.stdout:8/183: creat d1/d8/d2f/f38 x:0 0 0 2024-03-22T02:28:47.103 INFO:tasks.workunit.client.1.smithi122.stdout:8/184: chown d1/d8/l29 5 1 2024-03-22T02:28:47.104 INFO:tasks.workunit.client.1.smithi122.stdout:5/162: creat d8/da/d28/d35/f37 x:0 0 0 2024-03-22T02:28:47.106 INFO:tasks.workunit.client.1.smithi122.stdout:9/212: chown db/d23/d1e/d30/l44 140670 1 2024-03-22T02:28:47.107 INFO:tasks.workunit.client.0.smithi103.stdout:4/741: mknod d3/d1e/d25/cfd 0 2024-03-22T02:28:47.107 INFO:tasks.workunit.client.1.smithi122.stdout:4/176: dwrite d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:47.110 INFO:tasks.workunit.client.0.smithi103.stdout:8/852: dwrite d4/d48/d35/fa0 [0,4194304] 0 2024-03-22T02:28:47.110 INFO:tasks.workunit.client.0.smithi103.stdout:2/755: mkdir d11/d19/d4e/dfc 0 2024-03-22T02:28:47.113 INFO:tasks.workunit.client.1.smithi122.stdout:7/118: creat d0/f18 x:0 0 0 2024-03-22T02:28:47.114 INFO:tasks.workunit.client.0.smithi103.stdout:0/742: link dd/d16/d4b/d96/d35/l47 dd/d16/d4b/d96/d64/d6a/dd5/deb/d9a/lf7 0 2024-03-22T02:28:47.116 INFO:tasks.workunit.client.0.smithi103.stdout:0/743: write dd/d34/d53/d8c/fbc [2007496,114609] 0 2024-03-22T02:28:47.117 INFO:tasks.workunit.client.0.smithi103.stdout:8/853: creat d4/d26/d27/d7e/d67/f118 x:0 0 0 2024-03-22T02:28:47.117 INFO:tasks.workunit.client.1.smithi122.stdout:8/185: dwrite d1/d8/f33 [4194304,4194304] 0 2024-03-22T02:28:47.118 INFO:tasks.workunit.client.0.smithi103.stdout:1/806: link db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/l106 db/d4a/d78/d92/da8/ddd/l116 0 2024-03-22T02:28:47.121 INFO:tasks.workunit.client.0.smithi103.stdout:2/756: getdents d11/d19/da7/dd3/dd9 0 2024-03-22T02:28:47.122 INFO:tasks.workunit.client.1.smithi122.stdout:5/163: mkdir d8/d1e/d2e/d38 0 2024-03-22T02:28:47.123 INFO:tasks.workunit.client.1.smithi122.stdout:9/213: mknod db/d1b/d28/c4c 0 2024-03-22T02:28:47.125 INFO:tasks.workunit.client.0.smithi103.stdout:2/757: creat d11/d25/d46/d40/d90/ffd x:0 0 0 2024-03-22T02:28:47.126 INFO:tasks.workunit.client.0.smithi103.stdout:1/807: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f100 [0,4194304] 0 2024-03-22T02:28:47.126 INFO:tasks.workunit.client.1.smithi122.stdout:1/187: dwrite dc/fe [0,4194304] 0 2024-03-22T02:28:47.130 INFO:tasks.workunit.client.1.smithi122.stdout:8/186: write d1/d8/d2f/f38 [533657,82253] 0 2024-03-22T02:28:47.130 INFO:tasks.workunit.client.1.smithi122.stdout:2/199: mknod df/d1f/d3d/c49 0 2024-03-22T02:28:47.132 INFO:tasks.workunit.client.1.smithi122.stdout:5/164: write d8/f17 [4468986,116861] 0 2024-03-22T02:28:47.137 INFO:tasks.workunit.client.1.smithi122.stdout:1/188: write dc/f25 [3687,23993] 0 2024-03-22T02:28:47.143 INFO:tasks.workunit.client.1.smithi122.stdout:4/177: chown d0/l39 21455 1 2024-03-22T02:28:47.143 INFO:tasks.workunit.client.1.smithi122.stdout:7/119: dwrite d0/f8 [0,4194304] 0 2024-03-22T02:28:47.146 INFO:tasks.workunit.client.1.smithi122.stdout:4/178: stat d0/d25/fa 0 2024-03-22T02:28:47.150 INFO:tasks.workunit.client.1.smithi122.stdout:7/120: mknod d0/d17/c19 0 2024-03-22T02:28:47.156 INFO:tasks.workunit.client.1.smithi122.stdout:7/121: dread - d0/f16 zero size 2024-03-22T02:28:47.157 INFO:tasks.workunit.client.1.smithi122.stdout:5/165: dwrite d8/d1e/f23 [0,4194304] 0 2024-03-22T02:28:47.157 INFO:tasks.workunit.client.1.smithi122.stdout:8/187: dwrite d1/d22/f32 [0,4194304] 0 2024-03-22T02:28:47.159 INFO:tasks.workunit.client.1.smithi122.stdout:1/189: dwrite dc/f2f [4194304,4194304] 0 2024-03-22T02:28:47.162 INFO:tasks.workunit.client.1.smithi122.stdout:7/122: symlink d0/l1a 0 2024-03-22T02:28:47.163 INFO:tasks.workunit.client.1.smithi122.stdout:5/166: chown d8/f17 1107 1 2024-03-22T02:28:47.164 INFO:tasks.workunit.client.1.smithi122.stdout:5/167: fsync d8/da/f10 0 2024-03-22T02:28:47.164 INFO:tasks.workunit.client.1.smithi122.stdout:8/188: symlink d1/d9/l39 0 2024-03-22T02:28:47.165 INFO:tasks.workunit.client.1.smithi122.stdout:1/190: mknod dc/d2b/c34 0 2024-03-22T02:28:47.167 INFO:tasks.workunit.client.1.smithi122.stdout:1/191: write dc/d2b/f2d [2027624,96345] 0 2024-03-22T02:28:47.168 INFO:tasks.workunit.client.1.smithi122.stdout:5/168: mkdir d8/d1e/d2e/d39 0 2024-03-22T02:28:47.170 INFO:tasks.workunit.client.1.smithi122.stdout:8/189: symlink d1/d22/l3a 0 2024-03-22T02:28:47.175 INFO:tasks.workunit.client.1.smithi122.stdout:3/115: write d2/f7 [2116916,57006] 0 2024-03-22T02:28:47.177 INFO:tasks.workunit.client.1.smithi122.stdout:1/192: link dc/c12 dc/c35 0 2024-03-22T02:28:47.178 INFO:tasks.workunit.client.0.smithi103.stdout:4/742: dread d3/d31/f37 [0,4194304] 0 2024-03-22T02:28:47.178 INFO:tasks.workunit.client.1.smithi122.stdout:1/193: readlink dc/l14 0 2024-03-22T02:28:47.178 INFO:tasks.workunit.client.1.smithi122.stdout:3/116: rmdir d2/d9/dc/d21 39 2024-03-22T02:28:47.181 INFO:tasks.workunit.client.1.smithi122.stdout:1/194: fdatasync dc/ff 0 2024-03-22T02:28:47.181 INFO:tasks.workunit.client.0.smithi103.stdout:1/808: dread db/d4a/d78/d92/da8/ddd/f5a [0,4194304] 0 2024-03-22T02:28:47.182 INFO:tasks.workunit.client.1.smithi122.stdout:8/190: link d1/c3 d1/d8/d2f/c3b 0 2024-03-22T02:28:47.182 INFO:tasks.workunit.client.0.smithi103.stdout:1/809: fsync db/f45 0 2024-03-22T02:28:47.183 INFO:tasks.workunit.client.1.smithi122.stdout:2/200: truncate fd 1023515 0 2024-03-22T02:28:47.186 INFO:tasks.workunit.client.0.smithi103.stdout:1/810: getdents db/d4a/d78/d92/da8/ddd/d18/d84 0 2024-03-22T02:28:47.187 INFO:tasks.workunit.client.1.smithi122.stdout:7/123: dread d0/fd [0,4194304] 0 2024-03-22T02:28:47.187 INFO:tasks.workunit.client.1.smithi122.stdout:2/201: creat df/d1f/d2a/d39/f4a x:0 0 0 2024-03-22T02:28:47.187 INFO:tasks.workunit.client.1.smithi122.stdout:7/124: stat d0/c15 0 2024-03-22T02:28:47.188 INFO:tasks.workunit.client.0.smithi103.stdout:1/811: truncate db/d4a/d78/d92/da8/ddd/d29/d41/d68/fda 960366 0 2024-03-22T02:28:47.188 INFO:tasks.workunit.client.1.smithi122.stdout:8/191: chown d1/c3 1614267 1 2024-03-22T02:28:47.189 INFO:tasks.workunit.client.1.smithi122.stdout:1/195: dread dc/f1a [0,4194304] 0 2024-03-22T02:28:47.190 INFO:tasks.workunit.client.0.smithi103.stdout:4/743: write d3/d1e/fc3 [1223698,82464] 0 2024-03-22T02:28:47.190 INFO:tasks.workunit.client.1.smithi122.stdout:9/214: truncate db/d23/d1f/d21/d29/f2c 1505542 0 2024-03-22T02:28:47.191 INFO:tasks.workunit.client.0.smithi103.stdout:1/812: creat db/d4a/d78/d92/da8/ddd/d18/d2d/f117 x:0 0 0 2024-03-22T02:28:47.192 INFO:tasks.workunit.client.1.smithi122.stdout:4/179: truncate d0/d25/fa 3786673 0 2024-03-22T02:28:47.192 INFO:tasks.workunit.client.0.smithi103.stdout:3/775: sync 2024-03-22T02:28:47.192 INFO:tasks.workunit.client.1.smithi122.stdout:8/192: write d1/d9/f1f [2749595,108465] 0 2024-03-22T02:28:47.195 INFO:tasks.workunit.client.0.smithi103.stdout:3/776: mkdir de/d12/d8a/d93/d103 0 2024-03-22T02:28:47.196 INFO:tasks.workunit.client.0.smithi103.stdout:3/777: stat de/d12/d4d/cb7 0 2024-03-22T02:28:47.197 INFO:tasks.workunit.client.1.smithi122.stdout:9/215: chown db/d23/d1f/d21/d29/f3f 0 1 2024-03-22T02:28:47.197 INFO:tasks.workunit.client.1.smithi122.stdout:5/169: rename d8/d1e/d2e to d8/d1e/d22/d3a 0 2024-03-22T02:28:47.198 INFO:tasks.workunit.client.0.smithi103.stdout:4/744: dwrite d3/da2/da6/fb2 [0,4194304] 0 2024-03-22T02:28:47.199 INFO:tasks.workunit.client.0.smithi103.stdout:3/778: creat de/d12/d8f/dfb/f104 x:0 0 0 2024-03-22T02:28:47.200 INFO:tasks.workunit.client.1.smithi122.stdout:1/196: creat dc/d2b/f36 x:0 0 0 2024-03-22T02:28:47.201 INFO:tasks.workunit.client.1.smithi122.stdout:1/197: stat l4 0 2024-03-22T02:28:47.201 INFO:tasks.workunit.client.1.smithi122.stdout:2/202: dwrite df/d1f/d2a/f2f [0,4194304] 0 2024-03-22T02:28:47.204 INFO:tasks.workunit.client.0.smithi103.stdout:3/779: symlink de/d12/d19/df2/l105 0 2024-03-22T02:28:47.205 INFO:tasks.workunit.client.1.smithi122.stdout:0/138: sync 2024-03-22T02:28:47.207 INFO:tasks.workunit.client.0.smithi103.stdout:4/745: dwrite d3/d38/fdc [0,4194304] 0 2024-03-22T02:28:47.209 INFO:tasks.workunit.client.1.smithi122.stdout:8/193: creat d1/f3c x:0 0 0 2024-03-22T02:28:47.209 INFO:tasks.workunit.client.1.smithi122.stdout:3/117: dread d2/f8 [0,4194304] 0 2024-03-22T02:28:47.210 INFO:tasks.workunit.client.0.smithi103.stdout:3/780: creat de/d6b/d80/f106 x:0 0 0 2024-03-22T02:28:47.212 INFO:tasks.workunit.client.1.smithi122.stdout:1/198: unlink dc/c13 0 2024-03-22T02:28:47.212 INFO:tasks.workunit.client.0.smithi103.stdout:4/746: symlink d3/d1e/d68/dbc/lfe 0 2024-03-22T02:28:47.215 INFO:tasks.workunit.client.0.smithi103.stdout:4/747: fsync d3/d1e/d53/f56 0 2024-03-22T02:28:47.216 INFO:tasks.workunit.client.1.smithi122.stdout:7/125: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:47.217 INFO:tasks.workunit.client.0.smithi103.stdout:1/813: dread db/d4a/d78/d92/da8/ddd/f1c [0,4194304] 0 2024-03-22T02:28:47.219 INFO:tasks.workunit.client.1.smithi122.stdout:9/216: symlink db/l4d 0 2024-03-22T02:28:47.220 INFO:tasks.workunit.client.0.smithi103.stdout:4/748: link d3/d1e/d25/d65/d83/da1/dbd/fe4 d3/da2/da6/fff 0 2024-03-22T02:28:47.220 INFO:tasks.workunit.client.1.smithi122.stdout:9/217: dread - db/d23/d1f/f2b zero size 2024-03-22T02:28:47.222 INFO:tasks.workunit.client.1.smithi122.stdout:0/139: chown c1 115 1 2024-03-22T02:28:47.223 INFO:tasks.workunit.client.0.smithi103.stdout:1/814: rename db/d4a/d78/d92/da8/ddd/d18/d84/fc4 to db/d4a/d78/d92/da8/ddd/d2e/dae/f118 0 2024-03-22T02:28:47.223 INFO:tasks.workunit.client.1.smithi122.stdout:3/118: dwrite d2/d9/dc/f16 [0,4194304] 0 2024-03-22T02:28:47.223 INFO:tasks.workunit.client.1.smithi122.stdout:9/218: stat db/d23/d1e/f34 0 2024-03-22T02:28:47.226 INFO:tasks.workunit.client.0.smithi103.stdout:4/749: dwrite d3/d69/fb0 [0,4194304] 0 2024-03-22T02:28:47.227 INFO:tasks.workunit.client.1.smithi122.stdout:4/180: creat d0/d25/f3a x:0 0 0 2024-03-22T02:28:47.229 INFO:tasks.workunit.client.1.smithi122.stdout:1/199: truncate dc/f1a 3915841 0 2024-03-22T02:28:47.229 INFO:tasks.workunit.client.1.smithi122.stdout:0/140: unlink d11/d14/d1d/c28 0 2024-03-22T02:28:47.230 INFO:tasks.workunit.client.1.smithi122.stdout:1/200: chown dc/d2b 152 1 2024-03-22T02:28:47.230 INFO:tasks.workunit.client.0.smithi103.stdout:1/815: creat db/d4a/d78/d92/da8/ddd/d29/f119 x:0 0 0 2024-03-22T02:28:47.231 INFO:tasks.workunit.client.0.smithi103.stdout:4/750: mkdir d3/d100 0 2024-03-22T02:28:47.232 INFO:tasks.workunit.client.1.smithi122.stdout:9/219: unlink db/l33 0 2024-03-22T02:28:47.232 INFO:tasks.workunit.client.1.smithi122.stdout:2/203: dwrite df/d1f/d3a/d41/f47 [0,4194304] 0 2024-03-22T02:28:47.232 INFO:tasks.workunit.client.1.smithi122.stdout:0/141: write fb [2595691,128999] 0 2024-03-22T02:28:47.233 INFO:tasks.workunit.client.1.smithi122.stdout:2/204: dread - df/f36 zero size 2024-03-22T02:28:47.233 INFO:tasks.workunit.client.0.smithi103.stdout:4/751: write d3/fb [3815199,60079] 0 2024-03-22T02:28:47.234 INFO:tasks.workunit.client.1.smithi122.stdout:2/205: fsync df/f36 0 2024-03-22T02:28:47.238 INFO:tasks.workunit.client.0.smithi103.stdout:1/816: rename db/d4a/d78/d92/da8/ddd/d18/d84/fe7 to db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/f11a 0 2024-03-22T02:28:47.239 INFO:tasks.workunit.client.1.smithi122.stdout:0/142: dread - d11/d14/d1b/f2c zero size 2024-03-22T02:28:47.240 INFO:tasks.workunit.client.0.smithi103.stdout:1/817: fdatasync db/d4a/d78/d92/da8/ddd/d18/d84/f96 0 2024-03-22T02:28:47.241 INFO:tasks.workunit.client.1.smithi122.stdout:1/201: creat dc/f37 x:0 0 0 2024-03-22T02:28:47.242 INFO:tasks.workunit.client.1.smithi122.stdout:9/220: readlink db/d23/l3b 0 2024-03-22T02:28:47.243 INFO:tasks.workunit.client.1.smithi122.stdout:8/194: creat d1/f3d x:0 0 0 2024-03-22T02:28:47.243 INFO:tasks.workunit.client.0.smithi103.stdout:1/818: chown db/d4a/d78/d92/da8/ddd/d29/d41/d68/fda 1967768 1 2024-03-22T02:28:47.246 INFO:tasks.workunit.client.0.smithi103.stdout:1/819: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d11b 0 2024-03-22T02:28:47.247 INFO:tasks.workunit.client.1.smithi122.stdout:7/126: creat d0/f1b x:0 0 0 2024-03-22T02:28:47.248 INFO:tasks.workunit.client.0.smithi103.stdout:1/820: creat db/d4a/d78/d92/da8/ddd/d6b/db5/f11c x:0 0 0 2024-03-22T02:28:47.248 INFO:tasks.workunit.client.1.smithi122.stdout:4/181: creat d0/d25/f3b x:0 0 0 2024-03-22T02:28:47.249 INFO:tasks.workunit.client.1.smithi122.stdout:9/221: mkdir db/d23/d1e/d4e 0 2024-03-22T02:28:47.250 INFO:tasks.workunit.client.1.smithi122.stdout:4/182: dread - d0/f37 zero size 2024-03-22T02:28:47.250 INFO:tasks.workunit.client.0.smithi103.stdout:1/821: creat db/d4a/d78/d92/da8/ddd/d28/d97/f11d x:0 0 0 2024-03-22T02:28:47.251 INFO:tasks.workunit.client.0.smithi103.stdout:1/822: write db/d4a/d78/d92/da8/ddd/d18/dca/f114 [215384,127416] 0 2024-03-22T02:28:47.252 INFO:tasks.workunit.client.1.smithi122.stdout:0/143: dwrite fb [4194304,4194304] 0 2024-03-22T02:28:47.253 INFO:tasks.workunit.client.0.smithi103.stdout:1/823: write db/d4a/d78/d92/da8/ddd/d21/fe4 [340606,20880] 0 2024-03-22T02:28:47.258 INFO:tasks.workunit.client.1.smithi122.stdout:9/222: truncate db/d23/d1f/d21/d29/f3f 1027458 0 2024-03-22T02:28:47.259 INFO:tasks.workunit.client.0.smithi103.stdout:1/824: dwrite db/d4a/f9c [0,4194304] 0 2024-03-22T02:28:47.260 INFO:tasks.workunit.client.1.smithi122.stdout:4/183: stat d0/d12/d15/c1f 0 2024-03-22T02:28:47.260 INFO:tasks.workunit.client.1.smithi122.stdout:8/195: dwrite d1/d9/f26 [0,4194304] 0 2024-03-22T02:28:47.260 INFO:tasks.workunit.client.1.smithi122.stdout:7/127: symlink d0/l1c 0 2024-03-22T02:28:47.263 INFO:tasks.workunit.client.0.smithi103.stdout:1/825: creat db/d4a/d78/d92/da8/ddd/d28/db2/dfd/f11e x:0 0 0 2024-03-22T02:28:47.263 INFO:tasks.workunit.client.1.smithi122.stdout:8/196: read d1/d22/f32 [1257848,63621] 0 2024-03-22T02:28:47.265 INFO:tasks.workunit.client.0.smithi103.stdout:1/826: fdatasync db/d4a/d78/d92/da8/ddd/d29/d6c/f71 0 2024-03-22T02:28:47.266 INFO:tasks.workunit.client.1.smithi122.stdout:4/184: read - d0/d25/f3a zero size 2024-03-22T02:28:47.268 INFO:tasks.workunit.client.0.smithi103.stdout:3/781: dread de/d12/d30/f63 [4194304,4194304] 0 2024-03-22T02:28:47.269 INFO:tasks.workunit.client.0.smithi103.stdout:3/782: chown de/d12/d30/d70/de5/fe9 4760 1 2024-03-22T02:28:47.269 INFO:tasks.workunit.client.1.smithi122.stdout:9/223: stat db/d23/d1f/d21/d29/f38 0 2024-03-22T02:28:47.269 INFO:tasks.workunit.client.1.smithi122.stdout:2/206: chown df/f13 78199 1 2024-03-22T02:28:47.271 INFO:tasks.workunit.client.0.smithi103.stdout:1/827: link db/d4a/d78/d92/da8/ddd/d21/fe4 db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/f11f 0 2024-03-22T02:28:47.271 INFO:tasks.workunit.client.0.smithi103.stdout:3/783: read de/d12/d5d/f75 [2483877,123125] 0 2024-03-22T02:28:47.272 INFO:tasks.workunit.client.1.smithi122.stdout:0/144: dwrite d11/f13 [0,4194304] 0 2024-03-22T02:28:47.274 INFO:tasks.workunit.client.0.smithi103.stdout:1/828: truncate db/d4a/d78/d92/da8/ddd/d29/d6c/f105 762420 0 2024-03-22T02:28:47.274 INFO:tasks.workunit.client.1.smithi122.stdout:7/128: readlink d0/d4/la 0 2024-03-22T02:28:47.274 INFO:tasks.workunit.client.1.smithi122.stdout:2/207: write df/d1f/d33/d34/f43 [726456,127955] 0 2024-03-22T02:28:47.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.276+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa278000bf0 con 0x7fa28006b9e0 2024-03-22T02:28:47.278 INFO:tasks.workunit.client.0.smithi103.stdout:1/829: mknod db/d4a/d78/d92/da8/deb/c120 0 2024-03-22T02:28:47.278 INFO:tasks.workunit.client.1.smithi122.stdout:8/197: creat d1/d8/d2f/f3e x:0 0 0 2024-03-22T02:28:47.280 INFO:tasks.workunit.client.0.smithi103.stdout:1/830: write db/d4a/d78/d92/da8/ddd/d28/d97/f11d [715377,31018] 0 2024-03-22T02:28:47.282 INFO:tasks.workunit.client.1.smithi122.stdout:3/119: truncate d2/f8 3546197 0 2024-03-22T02:28:47.283 INFO:tasks.workunit.client.1.smithi122.stdout:8/198: chown d1/d8/f36 40891 1 2024-03-22T02:28:47.283 INFO:tasks.workunit.client.1.smithi122.stdout:7/129: dwrite d0/f8 [0,4194304] 0 2024-03-22T02:28:47.285 INFO:tasks.workunit.client.0.smithi103.stdout:1/831: truncate db/d4a/d78/d92/da8/ddd/d29/d55/ff4 2360347 0 2024-03-22T02:28:47.285 INFO:tasks.workunit.client.1.smithi122.stdout:3/120: truncate d2/d9/dc/d17/f20 1062869 0 2024-03-22T02:28:47.287 INFO:tasks.workunit.client.0.smithi103.stdout:1/832: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d11b/l121 0 2024-03-22T02:28:47.288 INFO:tasks.workunit.client.1.smithi122.stdout:4/185: rename d0/d25/d7/l30 to d0/d12/d15/d2d/l3c 0 2024-03-22T02:28:47.288 INFO:tasks.workunit.client.0.smithi103.stdout:1/833: symlink db/d4a/l122 0 2024-03-22T02:28:47.289 INFO:tasks.workunit.client.1.smithi122.stdout:2/208: mkdir df/d1f/d4b 0 2024-03-22T02:28:47.289 INFO:tasks.workunit.client.1.smithi122.stdout:4/186: write d0/d25/f3a [105636,29337] 0 2024-03-22T02:28:47.290 INFO:tasks.workunit.client.0.smithi103.stdout:1/834: write db/d4a/d78/d92/da8/ddd/d18/d2d/f117 [494837,128766] 0 2024-03-22T02:28:47.291 INFO:tasks.workunit.client.1.smithi122.stdout:4/187: fsync d0/f19 0 2024-03-22T02:28:47.292 INFO:tasks.workunit.client.0.smithi103.stdout:3/784: truncate de/d6b/d80/d73/fe0 2787138 0 2024-03-22T02:28:47.294 INFO:tasks.workunit.client.1.smithi122.stdout:0/145: mknod d11/d14/d1d/d27/d2f/c30 0 2024-03-22T02:28:47.295 INFO:tasks.workunit.client.0.smithi103.stdout:1/835: unlink f5 0 2024-03-22T02:28:47.295 INFO:tasks.workunit.client.1.smithi122.stdout:3/121: mkdir d2/d9/dc/d17/d28 0 2024-03-22T02:28:47.296 INFO:tasks.workunit.client.0.smithi103.stdout:3/785: creat de/d12/d4d/f107 x:0 0 0 2024-03-22T02:28:47.296 INFO:tasks.workunit.client.1.smithi122.stdout:4/188: truncate d0/d12/d15/d2d/f38 248949 0 2024-03-22T02:28:47.297 INFO:tasks.workunit.client.1.smithi122.stdout:7/130: rename d0/d17 to d0/d4/d1d 0 2024-03-22T02:28:47.298 INFO:tasks.workunit.client.1.smithi122.stdout:8/199: mkdir d1/d3f 0 2024-03-22T02:28:47.302 INFO:tasks.workunit.client.0.smithi103.stdout:3/786: creat de/d12/d8a/dda/f108 x:0 0 0 2024-03-22T02:28:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.299+0000 7fa28a7fc700 1 -- 172.21.15.103:0/3039196846 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fa278000bf0 con 0x7fa28006b9e0 2024-03-22T02:28:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:28:47.302 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (7m) 2m ago 8m 24.8M - 0.25.0 c8568f914cd2 31b45ef21cbd 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (8m) 2m ago 8m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (7m) 17s ago 7m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (7m) 2m ago 8m 84.7M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (3m) 2m ago 3m 10.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (2m) 2m ago 2m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (2m) 17s ago 2m 13.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (2m) 17s ago 2m 100M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (9m) 2m ago 9m 481M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (55s) 17s ago 7m 489M - 19.0.0-2344-gc3a04c0e 9d92c22378ed c01e6142c155 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (10m) 2m ago 10m 48.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (7m) 17s ago 7m 39.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (8m) 2m ago 8m 16.2M - 1.5.0 0da6a335fe13 136d30f21984 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 17s ago 7m 16.3M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (6m) 2m ago 6m 45.4M 4288M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:28:47.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (6m) 2m ago 6m 45.3M 4288M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:28:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (5m) 2m ago 5m 47.6M 4288M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:28:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (5m) 17s ago 5m 143M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:28:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (4m) 17s ago 4m 135M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:28:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (4m) 17s ago 4m 125M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:28:47.304 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (6m) 2m ago 8m 54.3M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 msgr2=0x7fa28006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 0x7fa28006de90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa29404d500 tx=0x7fa294077040 comp rx=0 tx=0).stop 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 msgr2=0x7fa28c0a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa2900041d0 tx=0x7fa29000b040 comp rx=0 tx=0).stop 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 shutdown_connections 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa28006b9e0 0x7fa28006de90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c093ad0 0x7fa28c0a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 --2- 172.21.15.103:0/3039196846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa28c094d80 0x7fa28c0a5cd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 >> 172.21.15.103:0/3039196846 conn(0x7fa28c08f230 msgr2=0x7fa28c090950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.301+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 shutdown_connections 2024-03-22T02:28:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.302+0000 7fa29aa27700 1 -- 172.21.15.103:0/3039196846 wait complete. 2024-03-22T02:28:47.306 INFO:tasks.workunit.client.0.smithi103.stdout:1/836: read db/d4a/d78/d92/da8/ddd/d18/f20 [1080280,50042] 0 2024-03-22T02:28:47.306 INFO:tasks.workunit.client.0.smithi103.stdout:1/837: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/c123 0 2024-03-22T02:28:47.307 INFO:tasks.workunit.client.1.smithi122.stdout:1/202: dwrite dc/f1a [0,4194304] 0 2024-03-22T02:28:47.307 INFO:tasks.workunit.client.1.smithi122.stdout:8/200: dread - d1/d8/f36 zero size 2024-03-22T02:28:47.307 INFO:tasks.workunit.client.1.smithi122.stdout:2/209: symlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:47.307 INFO:tasks.workunit.client.1.smithi122.stdout:2/210: write df/d1f/d2a/f2f [3358306,30079] 0 2024-03-22T02:28:47.310 INFO:tasks.workunit.client.1.smithi122.stdout:7/131: symlink d0/d4/d1d/l1e 0 2024-03-22T02:28:47.310 INFO:tasks.workunit.client.1.smithi122.stdout:0/146: dwrite d11/d14/d23/f2d [0,4194304] 0 2024-03-22T02:28:47.311 INFO:tasks.workunit.client.1.smithi122.stdout:7/132: write d0/d4/f12 [683744,85095] 0 2024-03-22T02:28:47.312 INFO:tasks.workunit.client.1.smithi122.stdout:8/201: unlink d1/l20 0 2024-03-22T02:28:47.314 INFO:tasks.workunit.client.1.smithi122.stdout:7/133: chown d0/d4/la 28330 1 2024-03-22T02:28:47.316 INFO:tasks.workunit.client.1.smithi122.stdout:2/211: mkdir df/d1f/d33/d4d 0 2024-03-22T02:28:47.320 INFO:tasks.workunit.client.0.smithi103.stdout:1/838: rename db/d4a/d78/d92/da8/ddd/d29/fa5 to db/d4a/d78/d92/f124 0 2024-03-22T02:28:47.323 INFO:tasks.workunit.client.0.smithi103.stdout:1/839: symlink db/d4a/d78/d92/da8/ddd/d2e/dae/l125 0 2024-03-22T02:28:47.324 INFO:tasks.workunit.client.1.smithi122.stdout:7/134: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:47.325 INFO:tasks.workunit.client.1.smithi122.stdout:7/135: chown d0/l1a 1165475 1 2024-03-22T02:28:47.325 INFO:tasks.workunit.client.0.smithi103.stdout:1/840: write db/d4a/d78/d92/da8/ddd/d28/f43 [1063598,56666] 0 2024-03-22T02:28:47.326 INFO:tasks.workunit.client.1.smithi122.stdout:7/136: dread d0/fd [0,4194304] 0 2024-03-22T02:28:47.328 INFO:tasks.workunit.client.1.smithi122.stdout:8/202: mkdir d1/d40 0 2024-03-22T02:28:47.330 INFO:tasks.workunit.client.0.smithi103.stdout:1/841: getdents db 0 2024-03-22T02:28:47.330 INFO:tasks.workunit.client.1.smithi122.stdout:8/203: fsync d1/f3c 0 2024-03-22T02:28:47.331 INFO:tasks.workunit.client.1.smithi122.stdout:2/212: symlink df/d1f/d33/d34/l4e 0 2024-03-22T02:28:47.331 INFO:tasks.workunit.client.1.smithi122.stdout:9/224: dread db/f13 [0,4194304] 0 2024-03-22T02:28:47.333 INFO:tasks.workunit.client.1.smithi122.stdout:2/213: write df/d1f/d23/f3b [2267397,108940] 0 2024-03-22T02:28:47.333 INFO:tasks.workunit.client.1.smithi122.stdout:9/225: fsync db/d23/f11 0 2024-03-22T02:28:47.334 INFO:tasks.workunit.client.0.smithi103.stdout:1/842: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d111/f126 x:0 0 0 2024-03-22T02:28:47.334 INFO:tasks.workunit.client.1.smithi122.stdout:2/214: write df/d1f/f2d [4688512,61013] 0 2024-03-22T02:28:47.335 INFO:tasks.workunit.client.0.smithi103.stdout:3/787: dread fd [0,4194304] 0 2024-03-22T02:28:47.337 INFO:tasks.workunit.client.0.smithi103.stdout:1/843: fdatasync db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/f11f 0 2024-03-22T02:28:47.338 INFO:tasks.workunit.client.1.smithi122.stdout:3/122: chown d2/f8 13596 1 2024-03-22T02:28:47.338 INFO:tasks.workunit.client.0.smithi103.stdout:3/788: symlink de/d12/d8f/dfb/l109 0 2024-03-22T02:28:47.339 INFO:tasks.workunit.client.1.smithi122.stdout:3/123: dread - d2/f25 zero size 2024-03-22T02:28:47.340 INFO:tasks.workunit.client.1.smithi122.stdout:3/124: write d2/f6 [1544452,73347] 0 2024-03-22T02:28:47.340 INFO:tasks.workunit.client.0.smithi103.stdout:1/844: write db/d4a/d78/d92/da8/ddd/d18/f20 [3960668,101106] 0 2024-03-22T02:28:47.340 INFO:tasks.workunit.client.0.smithi103.stdout:3/789: creat de/d6b/d80/d73/f10a x:0 0 0 2024-03-22T02:28:47.342 INFO:tasks.workunit.client.0.smithi103.stdout:3/790: write de/d12/d19/df2/f58 [2896949,2512] 0 2024-03-22T02:28:47.344 INFO:tasks.workunit.client.1.smithi122.stdout:2/215: mkdir df/d1f/d3a/d4f 0 2024-03-22T02:28:47.344 INFO:tasks.workunit.client.1.smithi122.stdout:1/203: write dc/f2a [4348217,38302] 0 2024-03-22T02:28:47.346 INFO:tasks.workunit.client.1.smithi122.stdout:4/189: dread d0/d25/f27 [0,4194304] 0 2024-03-22T02:28:47.346 INFO:tasks.workunit.client.0.smithi103.stdout:1/845: symlink db/d4a/d78/d92/da8/ddd/d6b/l127 0 2024-03-22T02:28:47.350 INFO:tasks.workunit.client.0.smithi103.stdout:3/791: symlink de/d1a/d25/l10b 0 2024-03-22T02:28:47.351 INFO:tasks.workunit.client.1.smithi122.stdout:1/204: creat dc/f38 x:0 0 0 2024-03-22T02:28:47.351 INFO:tasks.workunit.client.1.smithi122.stdout:0/147: truncate d11/d14/d23/f2d 1520399 0 2024-03-22T02:28:47.351 INFO:tasks.workunit.client.1.smithi122.stdout:4/190: rmdir d0/d25/d7/de 39 2024-03-22T02:28:47.352 INFO:tasks.workunit.client.1.smithi122.stdout:8/204: rmdir d1 39 2024-03-22T02:28:47.354 INFO:tasks.workunit.client.1.smithi122.stdout:4/191: write d0/f37 [831355,73216] 0 2024-03-22T02:28:47.354 INFO:tasks.workunit.client.1.smithi122.stdout:7/137: dread d0/d4/fb [0,4194304] 0 2024-03-22T02:28:47.354 INFO:tasks.workunit.client.0.smithi103.stdout:1/846: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/fcb [4194304,4194304] 0 2024-03-22T02:28:47.355 INFO:tasks.workunit.client.1.smithi122.stdout:7/138: stat d0/d4/f11 0 2024-03-22T02:28:47.355 INFO:tasks.workunit.client.1.smithi122.stdout:2/216: rename lc to df/l50 0 2024-03-22T02:28:47.356 INFO:tasks.workunit.client.1.smithi122.stdout:2/217: read - df/f36 zero size 2024-03-22T02:28:47.357 INFO:tasks.workunit.client.0.smithi103.stdout:1/847: write db/d4a/d78/d92/da8/ddd/d29/d6c/f71 [995364,50242] 0 2024-03-22T02:28:47.357 INFO:tasks.workunit.client.1.smithi122.stdout:1/205: creat dc/f39 x:0 0 0 2024-03-22T02:28:47.360 INFO:tasks.workunit.client.1.smithi122.stdout:9/226: creat db/d23/d1e/d30/f4f x:0 0 0 2024-03-22T02:28:47.360 INFO:tasks.workunit.client.1.smithi122.stdout:0/148: dread d11/d14/d1d/f2e [0,4194304] 0 2024-03-22T02:28:47.361 INFO:tasks.workunit.client.0.smithi103.stdout:1/848: mkdir db/d4a/d78/d92/da8/ddd/d29/d55/d128 0 2024-03-22T02:28:47.363 INFO:tasks.workunit.client.0.smithi103.stdout:3/792: dwrite de/d12/d5d/d92/fe1 [0,4194304] 0 2024-03-22T02:28:47.363 INFO:tasks.workunit.client.1.smithi122.stdout:2/218: mkdir df/d1f/d2a/d51 0 2024-03-22T02:28:47.364 INFO:tasks.workunit.client.1.smithi122.stdout:1/206: symlink dc/d2b/l3a 0 2024-03-22T02:28:47.365 INFO:tasks.workunit.client.1.smithi122.stdout:7/139: symlink d0/l1f 0 2024-03-22T02:28:47.366 INFO:tasks.workunit.client.0.smithi103.stdout:3/793: symlink de/d12/d8f/dfb/l10c 0 2024-03-22T02:28:47.369 INFO:tasks.workunit.client.1.smithi122.stdout:2/219: mkdir df/d1f/d3d/d52 0 2024-03-22T02:28:47.370 INFO:tasks.workunit.client.1.smithi122.stdout:2/220: chown df/d1f/d3d 182 1 2024-03-22T02:28:47.370 INFO:tasks.workunit.client.1.smithi122.stdout:8/205: creat d1/d3f/f41 x:0 0 0 2024-03-22T02:28:47.371 INFO:tasks.workunit.client.0.smithi103.stdout:1/849: dwrite db/d4a/fa0 [0,4194304] 0 2024-03-22T02:28:47.372 INFO:tasks.workunit.client.1.smithi122.stdout:1/207: mkdir dc/d3b 0 2024-03-22T02:28:47.372 INFO:tasks.workunit.client.0.smithi103.stdout:3/794: creat de/d12/f10d x:0 0 0 2024-03-22T02:28:47.372 INFO:tasks.workunit.client.1.smithi122.stdout:7/140: dwrite d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:47.373 INFO:tasks.workunit.client.0.smithi103.stdout:1/850: chown db/d4a/d78/d92/da8/ddd/f83 6533 1 2024-03-22T02:28:47.373 INFO:tasks.workunit.client.1.smithi122.stdout:4/192: rmdir d0/d12/d15/d2d/d32 0 2024-03-22T02:28:47.374 INFO:tasks.workunit.client.0.smithi103.stdout:1/851: write db/d4a/d78/d92/da8/fe1 [2319423,114661] 0 2024-03-22T02:28:47.375 INFO:tasks.workunit.client.0.smithi103.stdout:3/795: truncate de/d1a/d25/f79 2016609 0 2024-03-22T02:28:47.381 INFO:tasks.workunit.client.1.smithi122.stdout:7/141: mkdir d0/d4/d20 0 2024-03-22T02:28:47.383 INFO:tasks.workunit.client.1.smithi122.stdout:4/193: mkdir d0/d12/d15/d3d 0 2024-03-22T02:28:47.383 INFO:tasks.workunit.client.1.smithi122.stdout:1/208: getdents dc/d3b 0 2024-03-22T02:28:47.385 INFO:tasks.workunit.client.1.smithi122.stdout:9/227: stat db/d23/d1e/d30/f4f 0 2024-03-22T02:28:47.385 INFO:tasks.workunit.client.1.smithi122.stdout:2/221: rmdir df/d1f/d2a 39 2024-03-22T02:28:47.386 INFO:tasks.workunit.client.0.smithi103.stdout:1/852: dwrite db/d4a/d78/d92/da8/ddd/d2e/dae/f118 [0,4194304] 0 2024-03-22T02:28:47.387 INFO:tasks.workunit.client.1.smithi122.stdout:4/194: dread d0/f37 [0,4194304] 0 2024-03-22T02:28:47.387 INFO:tasks.workunit.client.1.smithi122.stdout:9/228: chown db/d23/d1f/d21/d25/d41/c48 898 1 2024-03-22T02:28:47.389 INFO:tasks.workunit.client.1.smithi122.stdout:4/195: chown d0/d12 213 1 2024-03-22T02:28:47.395 INFO:tasks.workunit.client.0.smithi103.stdout:1/853: dwrite db/d4a/d78/d92/da8/ddd/d29/d6c/f105 [0,4194304] 0 2024-03-22T02:28:47.395 INFO:tasks.workunit.client.0.smithi103.stdout:3/796: truncate de/d12/d19/fbf 3977496 0 2024-03-22T02:28:47.397 INFO:tasks.workunit.client.1.smithi122.stdout:2/222: fdatasync f4 0 2024-03-22T02:28:47.398 INFO:tasks.workunit.client.0.smithi103.stdout:3/797: mkdir de/d12/d5d/da9/db9/d10e 0 2024-03-22T02:28:47.399 INFO:tasks.workunit.client.0.smithi103.stdout:3/798: dread - de/d12/d4d/f107 zero size 2024-03-22T02:28:47.399 INFO:tasks.workunit.client.1.smithi122.stdout:1/209: dread dc/fe [0,4194304] 0 2024-03-22T02:28:47.402 INFO:tasks.workunit.client.1.smithi122.stdout:7/142: dwrite d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:47.403 INFO:tasks.workunit.client.1.smithi122.stdout:1/210: truncate dc/d2b/f36 462779 0 2024-03-22T02:28:47.411 INFO:tasks.workunit.client.1.smithi122.stdout:4/196: dwrite d0/d25/d7/f34 [0,4194304] 0 2024-03-22T02:28:47.411 INFO:tasks.workunit.client.1.smithi122.stdout:0/149: dwrite d11/d14/d1d/f2e [0,4194304] 0 2024-03-22T02:28:47.415 INFO:tasks.workunit.client.1.smithi122.stdout:9/229: getdents db/d2f 0 2024-03-22T02:28:47.415 INFO:tasks.workunit.client.1.smithi122.stdout:2/223: creat df/d1f/d2a/d39/f53 x:0 0 0 2024-03-22T02:28:47.418 INFO:tasks.workunit.client.1.smithi122.stdout:7/143: unlink d0/d4/f11 0 2024-03-22T02:28:47.419 INFO:tasks.workunit.client.1.smithi122.stdout:7/144: readlink d0/l1f 0 2024-03-22T02:28:47.421 INFO:tasks.workunit.client.1.smithi122.stdout:8/206: dwrite d1/d9/f13 [0,4194304] 0 2024-03-22T02:28:47.422 INFO:tasks.workunit.client.1.smithi122.stdout:7/145: chown d0/l1f 1018397 1 2024-03-22T02:28:47.423 INFO:tasks.workunit.client.1.smithi122.stdout:4/197: rmdir d0 39 2024-03-22T02:28:47.426 INFO:tasks.workunit.client.1.smithi122.stdout:0/150: mkdir d11/d14/d23/d31 0 2024-03-22T02:28:47.429 INFO:tasks.workunit.client.1.smithi122.stdout:8/207: dwrite d1/d22/f2e [0,4194304] 0 2024-03-22T02:28:47.429 INFO:tasks.workunit.client.1.smithi122.stdout:9/230: symlink db/d23/d1f/d21/d29/l50 0 2024-03-22T02:28:47.433 INFO:tasks.workunit.client.1.smithi122.stdout:2/224: creat df/d1f/d33/f54 x:0 0 0 2024-03-22T02:28:47.434 INFO:tasks.workunit.client.1.smithi122.stdout:2/225: write df/f11 [1630239,47859] 0 2024-03-22T02:28:47.437 INFO:tasks.workunit.client.1.smithi122.stdout:1/211: link c1 dc/c3c 0 2024-03-22T02:28:47.437 INFO:tasks.workunit.client.1.smithi122.stdout:2/226: stat df/d1f/d23/f3b 0 2024-03-22T02:28:47.437 INFO:tasks.workunit.client.1.smithi122.stdout:0/151: mkdir d11/d14/d1d/d27/d32 0 2024-03-22T02:28:47.439 INFO:tasks.workunit.client.1.smithi122.stdout:9/231: mknod db/d23/d1e/c51 0 2024-03-22T02:28:47.440 INFO:tasks.workunit.client.1.smithi122.stdout:7/146: mkdir d0/d4/d20/d21 0 2024-03-22T02:28:47.442 INFO:tasks.workunit.client.1.smithi122.stdout:4/198: dread d0/d25/f1d [0,4194304] 0 2024-03-22T02:28:47.442 INFO:tasks.workunit.client.1.smithi122.stdout:2/227: truncate df/d1f/d2a/d39/f4a 108946 0 2024-03-22T02:28:47.444 INFO:tasks.workunit.client.1.smithi122.stdout:4/199: chown d0/d12/d15/d2d/l2e 1230031723 1 2024-03-22T02:28:47.444 INFO:tasks.workunit.client.1.smithi122.stdout:1/212: unlink c8 0 2024-03-22T02:28:47.444 INFO:tasks.workunit.client.1.smithi122.stdout:1/213: stat dc/d2b/f36 0 2024-03-22T02:28:47.445 INFO:tasks.workunit.client.1.smithi122.stdout:4/200: fdatasync d0/d25/f3a 0 2024-03-22T02:28:47.445 INFO:tasks.workunit.client.1.smithi122.stdout:0/152: mkdir d11/d14/d23/d24/d33 0 2024-03-22T02:28:47.445 INFO:tasks.workunit.client.0.smithi103.stdout:1/854: dwrite db/d4a/d78/d92/da8/ddd/d29/d55/fbd [0,4194304] 0 2024-03-22T02:28:47.446 INFO:tasks.workunit.client.1.smithi122.stdout:9/232: symlink db/d23/d1f/d21/d3d/l52 0 2024-03-22T02:28:47.447 INFO:tasks.workunit.client.1.smithi122.stdout:7/147: mkdir d0/d4/d22 0 2024-03-22T02:28:47.448 INFO:tasks.workunit.client.0.smithi103.stdout:1/855: mkdir db/d4a/d78/d92/da8/ddd/d2e/dae/d129 0 2024-03-22T02:28:47.449 INFO:tasks.workunit.client.1.smithi122.stdout:1/214: mknod dc/c3d 0 2024-03-22T02:28:47.450 INFO:tasks.workunit.client.1.smithi122.stdout:0/153: unlink d11/f13 0 2024-03-22T02:28:47.452 INFO:tasks.workunit.client.1.smithi122.stdout:4/201: mknod d0/d12/d15/d2d/c3e 0 2024-03-22T02:28:47.453 INFO:tasks.workunit.client.0.smithi103.stdout:1/856: link db/d4a/d78/d92/da8/ddd/d29/d41/d68/c93 db/d4a/d78/d92/da8/ddd/d29/d55/d128/c12a 0 2024-03-22T02:28:47.454 INFO:tasks.workunit.client.1.smithi122.stdout:7/148: symlink d0/d4/l23 0 2024-03-22T02:28:47.455 INFO:tasks.workunit.client.1.smithi122.stdout:8/208: link d1/d9/l37 d1/l42 0 2024-03-22T02:28:47.455 INFO:tasks.workunit.client.0.smithi103.stdout:1/857: mknod db/d4a/d78/d92/da8/ddd/d28/db2/c12b 0 2024-03-22T02:28:47.457 INFO:tasks.workunit.client.0.smithi103.stdout:1/858: write db/d4a/d78/d92/da8/ddd/d28/f103 [666722,48944] 0 2024-03-22T02:28:47.457 INFO:tasks.workunit.client.1.smithi122.stdout:4/202: dread d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:47.460 INFO:tasks.workunit.client.1.smithi122.stdout:1/215: mknod dc/d2b/c3e 0 2024-03-22T02:28:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 -- 172.21.15.103:0/92485261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 msgr2=0x7f12600fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 --2- 172.21.15.103:0/92485261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f12600fc120 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1254004940 tx=0x7f1254009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 -- 172.21.15.103:0/92485261 shutdown_connections 2024-03-22T02:28:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 --2- 172.21.15.103:0/92485261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 0x7f12600fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 --2- 172.21.15.103:0/92485261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f12600fc120 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 -- 172.21.15.103:0/92485261 >> 172.21.15.103:0/92485261 conn(0x7f12600f7490 msgr2=0x7f12600f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:47.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 -- 172.21.15.103:0/92485261 shutdown_connections 2024-03-22T02:28:47.462 INFO:tasks.workunit.client.1.smithi122.stdout:4/203: dread d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.461+0000 7f1266c0f700 1 -- 172.21.15.103:0/92485261 wait complete. 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 Processor -- start 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 -- start start 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 0x7f126006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:47.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126006fd80 con 0x7f12600fbd10 2024-03-22T02:28:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f1266c0f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126006fef0 con 0x7f12600fcfc0 2024-03-22T02:28:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.462+0000 7f12649ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40032/0 (socket says 172.21.15.103:40032) 2024-03-22T02:28:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 -- 172.21.15.103:0/3560893240 learned_addr learned my addr 172.21.15.103:0/3560893240 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:47.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f125ffff700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 0x7f126006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 -- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 msgr2=0x7f126006e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 0x7f126006e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 -- 172.21.15.103:0/3560893240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1254005040 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.463+0000 7f12649ab700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1254013870 tx=0x7f1254009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.464+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f125400e070 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.464+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1254013de0 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.464+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f125401cea0 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.464+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126006ee40 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.464+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12601001f0 con 0x7f12600fbd10 2024-03-22T02:28:47.465 INFO:tasks.workunit.client.0.smithi103.stdout:1/859: dwrite db/d4a/d78/d92/ff0 [0,4194304] 0 2024-03-22T02:28:47.467 INFO:tasks.workunit.client.0.smithi103.stdout:1/860: chown db/d4a/d78/d92/da8/ddd/d18/d2d/lc2 42997 1 2024-03-22T02:28:47.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.466+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1260106110 con 0x7f12600fbd10 2024-03-22T02:28:47.468 INFO:tasks.workunit.client.1.smithi122.stdout:8/209: mkdir d1/d8/d2f/d43 0 2024-03-22T02:28:47.469 INFO:tasks.workunit.client.1.smithi122.stdout:2/228: link df/d1f/d23/c30 df/d1f/d3d/c55 0 2024-03-22T02:28:47.470 INFO:tasks.workunit.client.0.smithi103.stdout:3/799: dread de/d12/d19/df2/f69 [0,4194304] 0 2024-03-22T02:28:47.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.469+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f125401c690 con 0x7f12600fbd10 2024-03-22T02:28:47.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.470+0000 7f125d7fa700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 0x7f125006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:47.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.470+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f125408b900 con 0x7f12600fbd10 2024-03-22T02:28:47.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.471+0000 7f125ffff700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 0x7f125006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:47.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.471+0000 7f125ffff700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 0x7f125006de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f124c009b80 tx=0x7f124c00b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:47.472 INFO:tasks.workunit.client.1.smithi122.stdout:7/149: dwrite d0/f2 [4194304,4194304] 0 2024-03-22T02:28:47.472 INFO:tasks.workunit.client.1.smithi122.stdout:1/216: mknod dc/d2b/c3f 0 2024-03-22T02:28:47.474 INFO:tasks.workunit.client.0.smithi103.stdout:1/861: unlink db/d4a/d78/d92/da8/ddd/d29/f119 0 2024-03-22T02:28:47.474 INFO:tasks.workunit.client.1.smithi122.stdout:7/150: fsync d0/f1b 0 2024-03-22T02:28:47.474 INFO:tasks.workunit.client.1.smithi122.stdout:1/217: chown dc/d2b/c2e 869419605 1 2024-03-22T02:28:47.476 INFO:tasks.workunit.client.0.smithi103.stdout:3/800: unlink de/d12/d19/l4c 0 2024-03-22T02:28:47.477 INFO:tasks.workunit.client.0.smithi103.stdout:3/801: chown de/d12/d5d/cc3 0 1 2024-03-22T02:28:47.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.478+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f1254058f50 con 0x7f12600fbd10 2024-03-22T02:28:47.480 INFO:tasks.workunit.client.0.smithi103.stdout:3/802: rmdir de/d1a/d25/d3f/d9c 39 2024-03-22T02:28:47.482 INFO:tasks.workunit.client.1.smithi122.stdout:8/210: creat d1/d8/d2f/f44 x:0 0 0 2024-03-22T02:28:47.483 INFO:tasks.workunit.client.0.smithi103.stdout:1/862: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f12c x:0 0 0 2024-03-22T02:28:47.485 INFO:tasks.workunit.client.0.smithi103.stdout:1/863: dread - db/d4a/d78/d92/da8/ddd/d29/d41/f90 zero size 2024-03-22T02:28:47.485 INFO:tasks.workunit.client.1.smithi122.stdout:2/229: write df/d1f/f3e [4754571,50868] 0 2024-03-22T02:28:47.485 INFO:tasks.workunit.client.1.smithi122.stdout:1/218: dwrite dc/f38 [0,4194304] 0 2024-03-22T02:28:47.485 INFO:tasks.workunit.client.1.smithi122.stdout:7/151: rename d0/d4/d1d/l1e to d0/d4/d22/l24 0 2024-03-22T02:28:47.486 INFO:tasks.workunit.client.1.smithi122.stdout:2/230: readlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:47.487 INFO:tasks.workunit.client.1.smithi122.stdout:1/219: write dc/f22 [246812,70980] 0 2024-03-22T02:28:47.487 INFO:tasks.workunit.client.1.smithi122.stdout:7/152: write d0/f8 [2680434,103896] 0 2024-03-22T02:28:47.489 INFO:tasks.workunit.client.0.smithi103.stdout:3/803: rename de/d1a/d25/d3f to de/d1a/d10f 0 2024-03-22T02:28:47.490 INFO:tasks.workunit.client.1.smithi122.stdout:4/204: mknod d0/d25/d7/de/c3f 0 2024-03-22T02:28:47.497 INFO:tasks.workunit.client.1.smithi122.stdout:1/220: dwrite dc/d2b/f33 [0,4194304] 0 2024-03-22T02:28:47.500 INFO:tasks.workunit.client.1.smithi122.stdout:2/231: symlink df/d1f/d23/l56 0 2024-03-22T02:28:47.510 INFO:tasks.workunit.client.0.smithi103.stdout:3/804: rmdir de/d1a/d10f/d9c/da7 39 2024-03-22T02:28:47.514 INFO:tasks.workunit.client.0.smithi103.stdout:1/864: dread db/d4a/d78/d92/da8/fe1 [0,4194304] 0 2024-03-22T02:28:47.515 INFO:tasks.workunit.client.0.smithi103.stdout:3/805: truncate de/d12/d5d/f99 6405875 0 2024-03-22T02:28:47.519 INFO:tasks.workunit.client.0.smithi103.stdout:1/865: dread db/d4a/d78/d92/da8/ddd/d29/d41/fcb [4194304,4194304] 0 2024-03-22T02:28:47.521 INFO:tasks.workunit.client.0.smithi103.stdout:1/866: write db/d4a/d78/d92/da8/ddd/d6b/db5/f11c [635644,35931] 0 2024-03-22T02:28:47.524 INFO:tasks.workunit.client.0.smithi103.stdout:3/806: dwrite de/d1a/d25/f6a [0,4194304] 0 2024-03-22T02:28:47.527 INFO:tasks.workunit.client.0.smithi103.stdout:1/867: fdatasync db/d4a/d78/d92/da8/ddd/d29/f6f 0 2024-03-22T02:28:47.529 INFO:tasks.workunit.client.0.smithi103.stdout:7/723: sync 2024-03-22T02:28:47.530 INFO:tasks.workunit.client.0.smithi103.stdout:3/807: read de/d1a/d10f/ff3 [183552,76218] 0 2024-03-22T02:28:47.530 INFO:tasks.workunit.client.0.smithi103.stdout:1/868: rmdir db/d4a/d78/d92/dea 39 2024-03-22T02:28:47.532 INFO:tasks.workunit.client.0.smithi103.stdout:3/808: write de/d1a/d10f/d9c/fca [1030109,40124] 0 2024-03-22T02:28:47.533 INFO:tasks.workunit.client.0.smithi103.stdout:7/724: dread - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f85 zero size 2024-03-22T02:28:47.538 INFO:tasks.workunit.client.0.smithi103.stdout:7/725: creat d1/d10/d11/d19/d29/d33/da8/dc5/fe4 x:0 0 0 2024-03-22T02:28:47.540 INFO:tasks.workunit.client.0.smithi103.stdout:3/809: rename de/d1a/d10f/d9c/da7 to de/d6b/d80/d73/d110 0 2024-03-22T02:28:47.541 INFO:tasks.workunit.client.0.smithi103.stdout:3/810: fdatasync de/d6b/d80/d73/f10a 0 2024-03-22T02:28:47.542 INFO:tasks.workunit.client.0.smithi103.stdout:1/869: getdents db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe 0 2024-03-22T02:28:47.543 INFO:tasks.workunit.client.0.smithi103.stdout:3/811: creat de/d12/d8a/dda/f111 x:0 0 0 2024-03-22T02:28:47.545 INFO:tasks.workunit.client.0.smithi103.stdout:3/812: stat de/d1a/lb3 0 2024-03-22T02:28:47.547 INFO:tasks.workunit.client.0.smithi103.stdout:3/813: mknod de/d12/d5d/c112 0 2024-03-22T02:28:47.548 INFO:tasks.workunit.client.0.smithi103.stdout:3/814: fdatasync de/d12/d19/ff7 0 2024-03-22T02:28:47.549 INFO:tasks.workunit.client.0.smithi103.stdout:1/870: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/d112/f12d x:0 0 0 2024-03-22T02:28:47.552 INFO:tasks.workunit.client.0.smithi103.stdout:3/815: rename de/d12/f2b to de/d12/f113 0 2024-03-22T02:28:47.553 INFO:tasks.workunit.client.0.smithi103.stdout:1/871: read db/d4a/d78/d92/da8/ddd/d18/d84/f96 [3207536,8605] 0 2024-03-22T02:28:47.555 INFO:tasks.workunit.client.0.smithi103.stdout:3/816: write de/d6b/d80/d73/fea [83104,46862] 0 2024-03-22T02:28:47.555 INFO:tasks.workunit.client.0.smithi103.stdout:7/726: dwrite d1/d10/d11/d19/d29/d61/d69/fa5 [0,4194304] 0 2024-03-22T02:28:47.556 INFO:tasks.workunit.client.0.smithi103.stdout:1/872: stat db/d4a/d78/d92/da8/ddd/d18/dca/de5 0 2024-03-22T02:28:47.558 INFO:tasks.workunit.client.0.smithi103.stdout:7/727: rmdir d1/d10/d11 39 2024-03-22T02:28:47.559 INFO:tasks.workunit.client.0.smithi103.stdout:1/873: mknod db/c12e 0 2024-03-22T02:28:47.561 INFO:tasks.workunit.client.0.smithi103.stdout:3/817: creat de/d12/d5d/da9/db9/d10e/f114 x:0 0 0 2024-03-22T02:28:47.561 INFO:tasks.workunit.client.0.smithi103.stdout:7/728: dread - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f85 zero size 2024-03-22T02:28:47.562 INFO:tasks.workunit.client.0.smithi103.stdout:7/729: stat d1/l6 0 2024-03-22T02:28:47.563 INFO:tasks.workunit.client.0.smithi103.stdout:3/818: rmdir de/d1a/d10f/d9c 39 2024-03-22T02:28:47.563 INFO:tasks.workunit.client.0.smithi103.stdout:7/730: chown d1/d10/d11/d19/d29/d61/d69/d5c 669 1 2024-03-22T02:28:47.567 INFO:tasks.workunit.client.0.smithi103.stdout:3/819: rename de/d12/d19/fd5 to de/d12/d5d/d92/f115 0 2024-03-22T02:28:47.570 INFO:tasks.workunit.client.0.smithi103.stdout:3/820: creat de/d12/d30/d70/de5/f116 x:0 0 0 2024-03-22T02:28:47.573 INFO:tasks.workunit.client.0.smithi103.stdout:3/821: symlink de/d6b/d80/d4f/d84/l117 0 2024-03-22T02:28:47.575 INFO:tasks.workunit.client.0.smithi103.stdout:1/874: dwrite db/d4a/d78/d92/da8/ddd/d29/d6c/f87 [0,4194304] 0 2024-03-22T02:28:47.575 INFO:tasks.workunit.client.0.smithi103.stdout:3/822: creat de/d12/d8a/f118 x:0 0 0 2024-03-22T02:28:47.579 INFO:tasks.workunit.client.0.smithi103.stdout:1/875: creat db/d4a/d78/d92/da8/ddd/d2e/f12f x:0 0 0 2024-03-22T02:28:47.581 INFO:tasks.workunit.client.0.smithi103.stdout:1/876: creat db/d4a/d78/d92/da8/f130 x:0 0 0 2024-03-22T02:28:47.584 INFO:tasks.workunit.client.0.smithi103.stdout:1/877: creat db/d4a/d78/d92/da8/dc9/f131 x:0 0 0 2024-03-22T02:28:47.587 INFO:tasks.workunit.client.0.smithi103.stdout:1/878: unlink db/d4a/d78/d92/da8/ddd/d28/fa7 0 2024-03-22T02:28:47.590 INFO:tasks.workunit.client.0.smithi103.stdout:3/823: dwrite de/d12/d19/df2/f54 [0,4194304] 0 2024-03-22T02:28:47.591 INFO:tasks.workunit.client.0.smithi103.stdout:3/824: chown de/d12/d30/f6e 167 1 2024-03-22T02:28:47.593 INFO:tasks.workunit.client.0.smithi103.stdout:1/879: mkdir db/d4a/d78/d92/da8/ddd/d29/d55/d128/d132 0 2024-03-22T02:28:47.593 INFO:tasks.workunit.client.1.smithi122.stdout:8/211: dwrite d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:47.594 INFO:tasks.workunit.client.1.smithi122.stdout:1/221: creat dc/f40 x:0 0 0 2024-03-22T02:28:47.596 INFO:tasks.workunit.client.0.smithi103.stdout:1/880: fdatasync db/d4a/d78/d92/da8/ddd/d6b/f8d 0 2024-03-22T02:28:47.596 INFO:tasks.workunit.client.1.smithi122.stdout:8/212: write d1/f3d [551828,102656] 0 2024-03-22T02:28:47.597 INFO:tasks.workunit.client.1.smithi122.stdout:0/154: truncate d11/d14/d23/d24/f25 4091046 0 2024-03-22T02:28:47.597 INFO:tasks.workunit.client.0.smithi103.stdout:3/825: dwrite de/f22 [4194304,4194304] 0 2024-03-22T02:28:47.599 INFO:tasks.workunit.client.1.smithi122.stdout:8/213: stat d1/d9/c19 0 2024-03-22T02:28:47.601 INFO:tasks.workunit.client.1.smithi122.stdout:1/222: symlink dc/d3b/l41 0 2024-03-22T02:28:47.604 INFO:tasks.workunit.client.1.smithi122.stdout:7/153: getdents d0/d4 0 2024-03-22T02:28:47.606 INFO:tasks.workunit.client.0.smithi103.stdout:3/826: creat de/d1a/d10f/f119 x:0 0 0 2024-03-22T02:28:47.608 INFO:tasks.workunit.client.1.smithi122.stdout:9/233: dwrite f8 [0,4194304] 0 2024-03-22T02:28:47.608 INFO:tasks.workunit.client.1.smithi122.stdout:7/154: mkdir d0/d25 0 2024-03-22T02:28:47.608 INFO:tasks.workunit.client.0.smithi103.stdout:3/827: creat de/d12/d5d/f11a x:0 0 0 2024-03-22T02:28:47.610 INFO:tasks.workunit.client.0.smithi103.stdout:1/881: dwrite db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:47.610 INFO:tasks.workunit.client.1.smithi122.stdout:7/155: chown d0/l1f 30 1 2024-03-22T02:28:47.611 INFO:tasks.workunit.client.1.smithi122.stdout:0/155: dwrite f10 [0,4194304] 0 2024-03-22T02:28:47.611 INFO:tasks.workunit.client.1.smithi122.stdout:9/234: fdatasync db/d23/d1f/d21/d29/f32 0 2024-03-22T02:28:47.612 INFO:tasks.workunit.client.0.smithi103.stdout:3/828: creat de/d12/d30/f11b x:0 0 0 2024-03-22T02:28:47.612 INFO:tasks.workunit.client.1.smithi122.stdout:7/156: write d0/f18 [514742,60394] 0 2024-03-22T02:28:47.614 INFO:tasks.workunit.client.1.smithi122.stdout:7/157: chown d0/d4/d1d/c19 3483981 1 2024-03-22T02:28:47.615 INFO:tasks.workunit.client.1.smithi122.stdout:9/235: creat db/d23/d1f/d21/d25/f53 x:0 0 0 2024-03-22T02:28:47.617 INFO:tasks.workunit.client.0.smithi103.stdout:1/882: link db/d4a/d78/d92/da8/ddd/d29/d6c/f71 db/d4a/d78/d92/da8/ddd/d28/db2/dc8/f133 0 2024-03-22T02:28:47.619 INFO:tasks.workunit.client.0.smithi103.stdout:1/883: write db/d4a/d78/d92/da8/ddd/d2e/dae/ff3 [309119,130519] 0 2024-03-22T02:28:47.620 INFO:tasks.workunit.client.1.smithi122.stdout:7/158: rename d0/d4/d1d/c19 to d0/d25/c26 0 2024-03-22T02:28:47.621 INFO:tasks.workunit.client.1.smithi122.stdout:1/223: dread dc/f10 [0,4194304] 0 2024-03-22T02:28:47.623 INFO:tasks.workunit.client.1.smithi122.stdout:7/159: write d0/f2 [2890181,35608] 0 2024-03-22T02:28:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:47 smithi103 ceph-mon[29258]: pgmap v17: 65 pgs: 65 active+clean; 1.6 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 60 MiB/s rd, 233 MiB/s wr, 511 op/s 2024-03-22T02:28:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:47 smithi103 ceph-mon[29258]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:47.624 INFO:tasks.workunit.client.1.smithi122.stdout:1/224: dread dc/f10 [0,4194304] 0 2024-03-22T02:28:47.629 INFO:tasks.workunit.client.0.smithi103.stdout:1/884: dread db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/f11f [0,4194304] 0 2024-03-22T02:28:47.629 INFO:tasks.workunit.client.1.smithi122.stdout:7/160: creat d0/d4/d20/f27 x:0 0 0 2024-03-22T02:28:47.632 INFO:tasks.workunit.client.1.smithi122.stdout:1/225: creat dc/f42 x:0 0 0 2024-03-22T02:28:47.632 INFO:tasks.workunit.client.1.smithi122.stdout:0/156: dread d11/d16/f2a [0,4194304] 0 2024-03-22T02:28:47.632 INFO:tasks.workunit.client.1.smithi122.stdout:7/161: rmdir d0/d25 39 2024-03-22T02:28:47.635 INFO:tasks.workunit.client.1.smithi122.stdout:0/157: symlink d11/d14/l34 0 2024-03-22T02:28:47.636 INFO:tasks.workunit.client.1.smithi122.stdout:2/232: dwrite df/f13 [0,4194304] 0 2024-03-22T02:28:47.637 INFO:tasks.workunit.client.1.smithi122.stdout:2/233: fdatasync df/f15 0 2024-03-22T02:28:47.638 INFO:tasks.workunit.client.1.smithi122.stdout:7/162: mkdir d0/d4/d20/d21/d28 0 2024-03-22T02:28:47.639 INFO:tasks.workunit.client.1.smithi122.stdout:0/158: mkdir d11/d14/d23/d35 0 2024-03-22T02:28:47.641 INFO:tasks.workunit.client.1.smithi122.stdout:2/234: symlink df/d1f/d3a/d41/l57 0 2024-03-22T02:28:47.643 INFO:tasks.workunit.client.1.smithi122.stdout:7/163: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:47.643 INFO:tasks.workunit.client.1.smithi122.stdout:0/159: mknod d11/d14/d23/d35/c36 0 2024-03-22T02:28:47.644 INFO:tasks.workunit.client.1.smithi122.stdout:0/160: write fd [5595556,82836] 0 2024-03-22T02:28:47.645 INFO:tasks.workunit.client.1.smithi122.stdout:2/235: creat df/d1f/d3a/f58 x:0 0 0 2024-03-22T02:28:47.646 INFO:tasks.workunit.client.1.smithi122.stdout:8/214: truncate d1/d9/f13 602431 0 2024-03-22T02:28:47.647 INFO:tasks.workunit.client.1.smithi122.stdout:7/164: mknod d0/d4/d20/d21/c29 0 2024-03-22T02:28:47.648 INFO:tasks.workunit.client.1.smithi122.stdout:8/215: mkdir d1/d3f/d45 0 2024-03-22T02:28:47.650 INFO:tasks.workunit.client.1.smithi122.stdout:2/236: rename df/d1f/d4b to df/d1f/d33/d59 0 2024-03-22T02:28:47.650 INFO:tasks.workunit.client.1.smithi122.stdout:2/237: readlink df/d1f/d33/l48 0 2024-03-22T02:28:47.652 INFO:tasks.workunit.client.1.smithi122.stdout:8/216: symlink d1/d8/d2f/l46 0 2024-03-22T02:28:47.652 INFO:tasks.workunit.client.1.smithi122.stdout:2/238: creat df/d1f/d33/d34/f5a x:0 0 0 2024-03-22T02:28:47.654 INFO:tasks.workunit.client.1.smithi122.stdout:8/217: write d1/f1a [3955846,71681] 0 2024-03-22T02:28:47.655 INFO:tasks.workunit.client.1.smithi122.stdout:9/236: fsync db/d23/d1f/d21/d29/f2c 0 2024-03-22T02:28:47.656 INFO:tasks.workunit.client.1.smithi122.stdout:2/239: readlink df/d1f/d2a/d39/l3f 0 2024-03-22T02:28:47.657 INFO:tasks.workunit.client.1.smithi122.stdout:8/218: fdatasync d1/d8/f36 0 2024-03-22T02:28:47.660 INFO:tasks.workunit.client.1.smithi122.stdout:7/165: dwrite d0/fd [0,4194304] 0 2024-03-22T02:28:47.662 INFO:tasks.workunit.client.1.smithi122.stdout:2/240: symlink df/d1f/d2a/d39/l5b 0 2024-03-22T02:28:47.663 INFO:tasks.workunit.client.1.smithi122.stdout:2/241: write df/f13 [4734350,40256] 0 2024-03-22T02:28:47.665 INFO:tasks.workunit.client.1.smithi122.stdout:8/219: creat d1/d40/f47 x:0 0 0 2024-03-22T02:28:47.667 INFO:tasks.workunit.client.1.smithi122.stdout:7/166: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:47.667 INFO:tasks.workunit.client.1.smithi122.stdout:2/242: dread df/d1f/d2a/d39/f4a [0,4194304] 0 2024-03-22T02:28:47.669 INFO:tasks.workunit.client.1.smithi122.stdout:7/167: fsync d0/d4/fb 0 2024-03-22T02:28:47.670 INFO:tasks.workunit.client.1.smithi122.stdout:2/243: stat df/d1f/d2a/d39/f53 0 2024-03-22T02:28:47.674 INFO:tasks.workunit.client.1.smithi122.stdout:2/244: write df/f13 [4980557,70143] 0 2024-03-22T02:28:47.675 INFO:tasks.workunit.client.1.smithi122.stdout:8/220: dwrite d1/d8/d2f/f44 [0,4194304] 0 2024-03-22T02:28:47.676 INFO:tasks.workunit.client.1.smithi122.stdout:2/245: fdatasync df/d1f/f21 0 2024-03-22T02:28:47.680 INFO:tasks.workunit.client.1.smithi122.stdout:9/237: write db/d1b/f22 [45999,79976] 0 2024-03-22T02:28:47.681 INFO:tasks.workunit.client.1.smithi122.stdout:7/168: dwrite d0/fd [0,4194304] 0 2024-03-22T02:28:47.682 INFO:tasks.workunit.client.1.smithi122.stdout:9/238: write fa [5482641,116150] 0 2024-03-22T02:28:47.685 INFO:tasks.workunit.client.1.smithi122.stdout:9/239: fsync db/d23/d1e/f34 0 2024-03-22T02:28:47.685 INFO:tasks.workunit.client.1.smithi122.stdout:6/265: sync 2024-03-22T02:28:47.686 INFO:tasks.workunit.client.1.smithi122.stdout:7/169: symlink d0/d4/d1d/l2a 0 2024-03-22T02:28:47.687 INFO:tasks.workunit.client.1.smithi122.stdout:2/246: dread df/f13 [0,4194304] 0 2024-03-22T02:28:47.688 INFO:tasks.workunit.client.1.smithi122.stdout:8/221: dread d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:47.689 INFO:tasks.workunit.client.1.smithi122.stdout:9/240: readlink db/d1b/l1c 0 2024-03-22T02:28:47.690 INFO:tasks.workunit.client.1.smithi122.stdout:8/222: write d1/f2d [1215712,58533] 0 2024-03-22T02:28:47.691 INFO:tasks.workunit.client.1.smithi122.stdout:8/223: stat d1/d40 0 2024-03-22T02:28:47.692 INFO:tasks.workunit.client.1.smithi122.stdout:2/247: unlink df/d1f/d2a/d39/l3f 0 2024-03-22T02:28:47.694 INFO:tasks.workunit.client.1.smithi122.stdout:7/170: dread d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:47.694 INFO:tasks.workunit.client.1.smithi122.stdout:9/241: creat db/d23/d1f/d21/d25/d41/f54 x:0 0 0 2024-03-22T02:28:47.695 INFO:tasks.workunit.client.1.smithi122.stdout:9/242: dread - db/d23/d1f/d21/d25/f49 zero size 2024-03-22T02:28:47.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/224: mkdir d1/d40/d48 0 2024-03-22T02:28:47.697 INFO:tasks.workunit.client.1.smithi122.stdout:6/266: creat d7/de/dd/d39/f5b x:0 0 0 2024-03-22T02:28:47.698 INFO:tasks.workunit.client.1.smithi122.stdout:9/243: write db/d23/d1f/d21/d29/f38 [3104531,118196] 0 2024-03-22T02:28:47.699 INFO:tasks.workunit.client.1.smithi122.stdout:7/171: dread d0/fd [0,4194304] 0 2024-03-22T02:28:47.700 INFO:tasks.workunit.client.1.smithi122.stdout:9/244: write db/d1b/f22 [555787,50702] 0 2024-03-22T02:28:47.702 INFO:tasks.workunit.client.1.smithi122.stdout:9/245: fdatasync f7 0 2024-03-22T02:28:47.703 INFO:tasks.workunit.client.1.smithi122.stdout:9/246: readlink db/d23/l4b 0 2024-03-22T02:28:47.704 INFO:tasks.workunit.client.1.smithi122.stdout:9/247: dread - db/df/f4a zero size 2024-03-22T02:28:47.704 INFO:tasks.workunit.client.1.smithi122.stdout:8/225: dread d1/d9/f13 [0,4194304] 0 2024-03-22T02:28:47.706 INFO:tasks.workunit.client.1.smithi122.stdout:8/226: read d1/f2d [991533,16116] 0 2024-03-22T02:28:47.707 INFO:tasks.workunit.client.1.smithi122.stdout:7/172: creat d0/d4/d20/d21/f2b x:0 0 0 2024-03-22T02:28:47.708 INFO:tasks.workunit.client.1.smithi122.stdout:7/173: truncate d0/f16 476806 0 2024-03-22T02:28:47.708 INFO:tasks.workunit.client.1.smithi122.stdout:2/248: rmdir df/d1f/d33/d59 0 2024-03-22T02:28:47.709 INFO:tasks.workunit.client.0.smithi103.stdout:9/718: sync 2024-03-22T02:28:47.709 INFO:tasks.workunit.client.0.smithi103.stdout:6/515: sync 2024-03-22T02:28:47.710 INFO:tasks.workunit.client.1.smithi122.stdout:9/248: creat db/d23/d1f/d21/d25/f55 x:0 0 0 2024-03-22T02:28:47.711 INFO:tasks.workunit.client.1.smithi122.stdout:8/227: read d1/d22/f32 [1205160,957] 0 2024-03-22T02:28:47.714 INFO:tasks.workunit.client.0.smithi103.stdout:6/516: unlink d0/d14/c17 0 2024-03-22T02:28:47.714 INFO:tasks.workunit.client.0.smithi103.stdout:9/719: mknod d2/d10/d43/d7e/cd9 0 2024-03-22T02:28:47.715 INFO:tasks.workunit.client.0.smithi103.stdout:9/720: write d2/da/d18/d1e/f88 [536531,64791] 0 2024-03-22T02:28:47.715 INFO:tasks.workunit.client.1.smithi122.stdout:6/267: dread d7/de/f3e [0,4194304] 0 2024-03-22T02:28:47.715 INFO:tasks.workunit.client.1.smithi122.stdout:7/174: mknod d0/c2c 0 2024-03-22T02:28:47.716 INFO:tasks.workunit.client.1.smithi122.stdout:7/175: stat d0/c13 0 2024-03-22T02:28:47.716 INFO:tasks.workunit.client.1.smithi122.stdout:7/176: dread - d0/d4/d20/d21/f2b zero size 2024-03-22T02:28:47.717 INFO:tasks.workunit.client.0.smithi103.stdout:9/721: write d2/d10/f8a [7161826,99939] 0 2024-03-22T02:28:47.717 INFO:tasks.workunit.client.1.smithi122.stdout:7/177: truncate d0/f16 577781 0 2024-03-22T02:28:47.719 INFO:tasks.workunit.client.0.smithi103.stdout:9/722: write d2/da/fd [2773458,121569] 0 2024-03-22T02:28:47.720 INFO:tasks.workunit.client.1.smithi122.stdout:8/228: creat d1/d3f/d45/f49 x:0 0 0 2024-03-22T02:28:47.720 INFO:tasks.workunit.client.0.smithi103.stdout:9/723: write d2/da/f23 [2921126,47549] 0 2024-03-22T02:28:47.720 INFO:tasks.workunit.client.1.smithi122.stdout:2/249: creat df/d1f/f5c x:0 0 0 2024-03-22T02:28:47.723 INFO:tasks.workunit.client.0.smithi103.stdout:9/724: read d2/d5/d4c/f5c [1303710,113561] 0 2024-03-22T02:28:47.723 INFO:tasks.workunit.client.1.smithi122.stdout:9/249: creat db/d23/d1f/d21/f56 x:0 0 0 2024-03-22T02:28:47.724 INFO:tasks.workunit.client.1.smithi122.stdout:7/178: dread d0/d4/fb [0,4194304] 0 2024-03-22T02:28:47.725 INFO:tasks.workunit.client.1.smithi122.stdout:9/250: fsync db/d23/d1f/d21/d25/d41/f54 0 2024-03-22T02:28:47.725 INFO:tasks.workunit.client.1.smithi122.stdout:6/268: rename d7/de/c1d to d7/de/dd/d39/c5c 0 2024-03-22T02:28:47.729 INFO:tasks.workunit.client.0.smithi103.stdout:6/517: dread d0/d3/d22/d58/f79 [0,4194304] 0 2024-03-22T02:28:47.729 INFO:tasks.workunit.client.1.smithi122.stdout:2/250: getdents df/d1f/d2a/d39 0 2024-03-22T02:28:47.730 INFO:tasks.workunit.client.1.smithi122.stdout:7/179: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:47.730 INFO:tasks.workunit.client.1.smithi122.stdout:2/251: stat df/d1f/d2a/d39/f4a 0 2024-03-22T02:28:47.731 INFO:tasks.workunit.client.1.smithi122.stdout:2/252: write df/d1f/d3a/d41/f47 [2436722,102674] 0 2024-03-22T02:28:47.732 INFO:tasks.workunit.client.1.smithi122.stdout:2/253: fdatasync df/d1f/d23/f3b 0 2024-03-22T02:28:47.732 INFO:tasks.workunit.client.1.smithi122.stdout:2/254: chown df/f11 12577 1 2024-03-22T02:28:47.733 INFO:tasks.workunit.client.1.smithi122.stdout:7/180: creat d0/d4/f2d x:0 0 0 2024-03-22T02:28:47.734 INFO:tasks.workunit.client.1.smithi122.stdout:8/229: truncate d1/f2d 727131 0 2024-03-22T02:28:47.736 INFO:tasks.workunit.client.1.smithi122.stdout:2/255: dread df/d1f/d33/d34/f43 [0,4194304] 0 2024-03-22T02:28:47.737 INFO:tasks.workunit.client.0.smithi103.stdout:9/725: dread d2/da/d18/d1e/d34/d3d/d4d/f58 [4194304,4194304] 0 2024-03-22T02:28:47.739 INFO:tasks.workunit.client.0.smithi103.stdout:9/726: write d2/d10/d59/f89 [3568032,89354] 0 2024-03-22T02:28:47.742 INFO:tasks.workunit.client.1.smithi122.stdout:9/251: truncate db/f13 2507220 0 2024-03-22T02:28:47.742 INFO:tasks.workunit.client.1.smithi122.stdout:6/269: link d7/de/dd/d20/f45 d7/de/dd/d20/f5d 0 2024-03-22T02:28:47.743 INFO:tasks.workunit.client.1.smithi122.stdout:6/270: readlink d7/de/dd/l26 0 2024-03-22T02:28:47.743 INFO:tasks.workunit.client.1.smithi122.stdout:9/252: write db/d23/f11 [727275,31905] 0 2024-03-22T02:28:47.745 INFO:tasks.workunit.client.0.smithi103.stdout:6/518: dread d0/f1a [0,4194304] 0 2024-03-22T02:28:47.746 INFO:tasks.workunit.client.0.smithi103.stdout:9/727: dwrite d2/d10/d43/d7e/fc8 [0,4194304] 0 2024-03-22T02:28:47.747 INFO:tasks.workunit.client.1.smithi122.stdout:7/181: symlink d0/l2e 0 2024-03-22T02:28:47.747 INFO:tasks.workunit.client.1.smithi122.stdout:2/256: creat df/d1f/d3d/d52/f5d x:0 0 0 2024-03-22T02:28:47.748 INFO:tasks.workunit.client.1.smithi122.stdout:2/257: chown df/d1f/d33/l48 1401431 1 2024-03-22T02:28:47.749 INFO:tasks.workunit.client.0.smithi103.stdout:6/519: symlink d0/la0 0 2024-03-22T02:28:47.749 INFO:tasks.workunit.client.1.smithi122.stdout:2/258: write df/f15 [82511,28131] 0 2024-03-22T02:28:47.753 INFO:tasks.workunit.client.0.smithi103.stdout:9/728: dread d2/f3b [0,4194304] 0 2024-03-22T02:28:47.754 INFO:tasks.workunit.client.0.smithi103.stdout:9/729: chown d2/d10/ld2 1893896553 1 2024-03-22T02:28:47.758 INFO:tasks.workunit.client.0.smithi103.stdout:6/520: mknod d0/d3/d22/d58/ca1 0 2024-03-22T02:28:47.759 INFO:tasks.workunit.client.1.smithi122.stdout:2/259: dwrite df/d1f/d2a/d39/f53 [0,4194304] 0 2024-03-22T02:28:47.759 INFO:tasks.workunit.client.0.smithi103.stdout:6/521: fdatasync d0/f64 0 2024-03-22T02:28:47.761 INFO:tasks.workunit.client.1.smithi122.stdout:8/230: dwrite d1/d22/f32 [0,4194304] 0 2024-03-22T02:28:47.761 INFO:tasks.workunit.client.0.smithi103.stdout:6/522: unlink d0/d14/d57/l59 0 2024-03-22T02:28:47.763 INFO:tasks.workunit.client.1.smithi122.stdout:2/260: rmdir df/d1f/d33 39 2024-03-22T02:28:47.764 INFO:tasks.workunit.client.0.smithi103.stdout:6/523: write d0/d3/d22/f50 [4211392,57201] 0 2024-03-22T02:28:47.765 INFO:tasks.workunit.client.1.smithi122.stdout:8/231: creat d1/d40/f4a x:0 0 0 2024-03-22T02:28:47.767 INFO:tasks.workunit.client.0.smithi103.stdout:6/524: creat d0/fa2 x:0 0 0 2024-03-22T02:28:47.769 INFO:tasks.workunit.client.1.smithi122.stdout:8/232: creat d1/d3f/d45/f4b x:0 0 0 2024-03-22T02:28:47.770 INFO:tasks.workunit.client.1.smithi122.stdout:6/271: truncate d7/de/d27/f18 737567 0 2024-03-22T02:28:47.770 INFO:tasks.workunit.client.1.smithi122.stdout:2/261: dread df/f15 [0,4194304] 0 2024-03-22T02:28:47.771 INFO:tasks.workunit.client.1.smithi122.stdout:8/233: mknod d1/d3f/c4c 0 2024-03-22T02:28:47.771 INFO:tasks.workunit.client.0.smithi103.stdout:6/525: mknod d0/d3/d22/d58/d6f/d75/ca3 0 2024-03-22T02:28:47.773 INFO:tasks.workunit.client.1.smithi122.stdout:7/182: truncate d0/d4/f5 1375846 0 2024-03-22T02:28:47.775 INFO:tasks.workunit.client.0.smithi103.stdout:6/526: symlink d0/d3/d22/d58/la4 0 2024-03-22T02:28:47.778 INFO:tasks.workunit.client.1.smithi122.stdout:2/262: creat df/d1f/d3a/d4f/f5e x:0 0 0 2024-03-22T02:28:47.779 INFO:tasks.workunit.client.0.smithi103.stdout:6/527: symlink d0/d3/d22/d58/d6f/d75/la5 0 2024-03-22T02:28:47.779 INFO:tasks.workunit.client.0.smithi103.stdout:9/730: dwrite d2/d10/d59/f5e [4194304,4194304] 0 2024-03-22T02:28:47.780 INFO:tasks.workunit.client.1.smithi122.stdout:8/234: mkdir d1/d4d 0 2024-03-22T02:28:47.780 INFO:tasks.workunit.client.0.smithi103.stdout:9/731: read - d2/da/f6b zero size 2024-03-22T02:28:47.781 INFO:tasks.workunit.client.1.smithi122.stdout:7/183: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:47.782 INFO:tasks.workunit.client.0.smithi103.stdout:6/528: symlink d0/d3/d33/la6 0 2024-03-22T02:28:47.784 INFO:tasks.workunit.client.1.smithi122.stdout:2/263: creat df/d1f/d33/d34/f5f x:0 0 0 2024-03-22T02:28:47.786 INFO:tasks.workunit.client.1.smithi122.stdout:2/264: write df/f10 [1429555,107280] 0 2024-03-22T02:28:47.788 INFO:tasks.workunit.client.0.smithi103.stdout:9/732: link d2/d10/d43/lb9 d2/da/d18/d1e/d77/d8b/dae/lda 0 2024-03-22T02:28:47.789 INFO:tasks.workunit.client.1.smithi122.stdout:7/184: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:47.789 INFO:tasks.workunit.client.0.smithi103.stdout:6/529: read d0/d3/d22/d58/f89 [6197247,130426] 0 2024-03-22T02:28:47.792 INFO:tasks.workunit.client.0.smithi103.stdout:6/530: chown d0/d14/d57/d68/f62 233 1 2024-03-22T02:28:47.795 INFO:tasks.workunit.client.1.smithi122.stdout:8/235: dwrite d1/d8/d2f/f3e [0,4194304] 0 2024-03-22T02:28:47.795 INFO:tasks.workunit.client.1.smithi122.stdout:7/185: dread - d0/d4/d20/d21/f2b zero size 2024-03-22T02:28:47.797 INFO:tasks.workunit.client.0.smithi103.stdout:9/733: getdents d2/d10/d43/d7e/d96 0 2024-03-22T02:28:47.797 INFO:tasks.workunit.client.0.smithi103.stdout:9/734: chown d2/da/d18/d1e/d34 238090 1 2024-03-22T02:28:47.798 INFO:tasks.workunit.client.1.smithi122.stdout:2/265: rename df/d1f/d2a/d39/f4a to df/d1f/d3d/f60 0 2024-03-22T02:28:47.798 INFO:tasks.workunit.client.1.smithi122.stdout:6/272: write d7/de/dd/df/f1a [3540396,106215] 0 2024-03-22T02:28:47.799 INFO:tasks.workunit.client.1.smithi122.stdout:2/266: chown df/d1f/d3a/f58 2250098 1 2024-03-22T02:28:47.800 INFO:tasks.workunit.client.1.smithi122.stdout:8/236: chown d1/d8/d2f/c3b 932 1 2024-03-22T02:28:47.801 INFO:tasks.workunit.client.1.smithi122.stdout:7/186: mknod d0/d4/d20/d21/c2f 0 2024-03-22T02:28:47.803 INFO:tasks.workunit.client.1.smithi122.stdout:2/267: symlink df/d1f/d2a/l61 0 2024-03-22T02:28:47.805 INFO:tasks.workunit.client.0.smithi103.stdout:9/735: dwrite d2/da/d18/dc4/dce/fd0 [0,4194304] 0 2024-03-22T02:28:47.805 INFO:tasks.workunit.client.1.smithi122.stdout:8/237: creat d1/d4d/f4e x:0 0 0 2024-03-22T02:28:47.806 INFO:tasks.workunit.client.1.smithi122.stdout:2/268: creat df/d1f/d2a/d39/f62 x:0 0 0 2024-03-22T02:28:47.807 INFO:tasks.workunit.client.1.smithi122.stdout:8/238: mknod d1/d3f/d45/c4f 0 2024-03-22T02:28:47.808 INFO:tasks.workunit.client.1.smithi122.stdout:2/269: symlink df/d1f/d33/d34/l63 0 2024-03-22T02:28:47.812 INFO:tasks.workunit.client.0.smithi103.stdout:9/736: dwrite d2/da/d18/dc4/dce/fd0 [4194304,4194304] 0 2024-03-22T02:28:47.817 INFO:tasks.workunit.client.0.smithi103.stdout:9/737: dread d2/d10/d59/f5e [4194304,4194304] 0 2024-03-22T02:28:47.820 INFO:tasks.workunit.client.0.smithi103.stdout:9/738: mkdir d2/d10/d43/d7e/ddb 0 2024-03-22T02:28:47.821 INFO:tasks.workunit.client.0.smithi103.stdout:6/531: write d0/f1e [6326110,98304] 0 2024-03-22T02:28:47.822 INFO:tasks.workunit.client.0.smithi103.stdout:9/739: write d2/d10/d43/d4b/f82 [322415,68465] 0 2024-03-22T02:28:47.823 INFO:tasks.workunit.client.0.smithi103.stdout:6/532: mknod d0/d3/ca7 0 2024-03-22T02:28:47.824 INFO:tasks.workunit.client.0.smithi103.stdout:9/740: write d2/d10/d43/f7a [2234652,61648] 0 2024-03-22T02:28:47.825 INFO:tasks.workunit.client.0.smithi103.stdout:6/533: write d0/d3/f1b [9250227,104165] 0 2024-03-22T02:28:47.830 INFO:tasks.workunit.client.0.smithi103.stdout:6/534: link d0/d3/d22/f50 d0/d14/d57/d68/fa8 0 2024-03-22T02:28:47.835 INFO:tasks.workunit.client.0.smithi103.stdout:6/535: link d0/d3/d22/d58/d6f/c87 d0/d3/d22/d58/ca9 0 2024-03-22T02:28:47.835 INFO:tasks.workunit.client.0.smithi103.stdout:6/536: stat d0/d14/f63 0 2024-03-22T02:28:47.843 INFO:tasks.workunit.client.0.smithi103.stdout:9/741: dread d2/f9 [4194304,4194304] 0 2024-03-22T02:28:47.844 INFO:tasks.workunit.client.0.smithi103.stdout:9/742: write d2/da/d18/d1e/fcc [3890666,34310] 0 2024-03-22T02:28:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:47 smithi122 ceph-mon[36403]: pgmap v17: 65 pgs: 65 active+clean; 1.6 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 60 MiB/s rd, 233 MiB/s wr, 511 op/s 2024-03-22T02:28:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:47 smithi122 ceph-mon[36403]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:47.849 INFO:tasks.workunit.client.0.smithi103.stdout:6/537: dread d0/d3/d22/f5c [0,4194304] 0 2024-03-22T02:28:47.851 INFO:tasks.workunit.client.0.smithi103.stdout:9/743: write d2/f3b [1356820,55418] 0 2024-03-22T02:28:47.852 INFO:tasks.workunit.client.0.smithi103.stdout:6/538: mkdir d0/d3/d22/d58/d6f/d75/daa 0 2024-03-22T02:28:47.856 INFO:tasks.workunit.client.0.smithi103.stdout:9/744: rmdir d2/d10/d43/d7e/ddb 0 2024-03-22T02:28:47.857 INFO:tasks.workunit.client.0.smithi103.stdout:6/539: creat d0/d3/d22/fab x:0 0 0 2024-03-22T02:28:47.857 INFO:tasks.workunit.client.0.smithi103.stdout:9/745: truncate d2/d5/d4c/f85 1721082 0 2024-03-22T02:28:47.860 INFO:tasks.workunit.client.0.smithi103.stdout:6/540: mknod d0/d3/d22/d58/cac 0 2024-03-22T02:28:47.862 INFO:tasks.workunit.client.0.smithi103.stdout:8/854: sync 2024-03-22T02:28:47.862 INFO:tasks.workunit.client.0.smithi103.stdout:0/744: sync 2024-03-22T02:28:47.862 INFO:tasks.workunit.client.0.smithi103.stdout:5/767: sync 2024-03-22T02:28:47.862 INFO:tasks.workunit.client.0.smithi103.stdout:2/758: sync 2024-03-22T02:28:47.862 INFO:tasks.workunit.client.0.smithi103.stdout:4/752: sync 2024-03-22T02:28:47.866 INFO:tasks.workunit.client.0.smithi103.stdout:5/768: write d0/d14/d23/d36/d95/fb1 [3506100,124234] 0 2024-03-22T02:28:47.869 INFO:tasks.workunit.client.0.smithi103.stdout:6/541: creat d0/d3/d33/fad x:0 0 0 2024-03-22T02:28:47.870 INFO:tasks.workunit.client.0.smithi103.stdout:4/753: dwrite d3/d1e/f43 [0,4194304] 0 2024-03-22T02:28:47.872 INFO:tasks.workunit.client.0.smithi103.stdout:4/754: chown d3/d30/lca 4492 1 2024-03-22T02:28:47.872 INFO:tasks.workunit.client.0.smithi103.stdout:2/759: rename d11/d25/d46/l9b to d11/d19/d5d/lfe 0 2024-03-22T02:28:47.873 INFO:tasks.workunit.client.0.smithi103.stdout:9/746: write d2/da/d18/d1e/d34/d3d/d46/fad [649070,65023] 0 2024-03-22T02:28:47.873 INFO:tasks.workunit.client.0.smithi103.stdout:2/760: chown d11/d25/d46/fca 2016254 1 2024-03-22T02:28:47.874 INFO:tasks.workunit.client.0.smithi103.stdout:5/769: mknod d0/d14/d23/d36/c10b 0 2024-03-22T02:28:47.875 INFO:tasks.workunit.client.0.smithi103.stdout:9/747: chown d2/d5/d4c/l55 8 1 2024-03-22T02:28:47.877 INFO:tasks.workunit.client.0.smithi103.stdout:6/542: mkdir d0/d88/d8e/dae 0 2024-03-22T02:28:47.878 INFO:tasks.workunit.client.0.smithi103.stdout:4/755: creat d3/d1e/d25/d65/d83/dc1/f101 x:0 0 0 2024-03-22T02:28:47.879 INFO:tasks.workunit.client.0.smithi103.stdout:4/756: chown d3/d1e/lc8 7674453 1 2024-03-22T02:28:47.880 INFO:tasks.workunit.client.0.smithi103.stdout:2/761: creat d11/d25/d63/fff x:0 0 0 2024-03-22T02:28:47.881 INFO:tasks.workunit.client.0.smithi103.stdout:5/770: write d0/f1e [2579511,29550] 0 2024-03-22T02:28:47.881 INFO:tasks.workunit.client.0.smithi103.stdout:4/757: write d3/d1e/d53/d8a/fd3 [187722,74374] 0 2024-03-22T02:28:47.884 INFO:tasks.workunit.client.0.smithi103.stdout:8/855: creat d4/d26/d27/d7e/d39/d49/f119 x:0 0 0 2024-03-22T02:28:47.885 INFO:tasks.workunit.client.0.smithi103.stdout:4/758: truncate d3/d1e/d25/d65/d83/da1/ff2 730260 0 2024-03-22T02:28:47.886 INFO:tasks.workunit.client.0.smithi103.stdout:4/759: readlink d3/d1e/d53/l59 0 2024-03-22T02:28:47.887 INFO:tasks.workunit.client.0.smithi103.stdout:4/760: fdatasync d3/d1e/d25/d65/d83/da1/ff2 0 2024-03-22T02:28:47.888 INFO:tasks.workunit.client.0.smithi103.stdout:8/856: dread - d4/d2a/d3e/fcf zero size 2024-03-22T02:28:47.888 INFO:tasks.workunit.client.0.smithi103.stdout:4/761: stat d3/d1e/d78/df9/l91 0 2024-03-22T02:28:47.889 INFO:tasks.workunit.client.1.smithi122.stdout:8/239: mkdir d1/d9/d50 0 2024-03-22T02:28:47.890 INFO:tasks.workunit.client.0.smithi103.stdout:9/748: getdents d2/d5/d4c 0 2024-03-22T02:28:47.890 INFO:tasks.workunit.client.0.smithi103.stdout:4/762: read - d3/d1e/d53/d7a/da3/fbe zero size 2024-03-22T02:28:47.890 INFO:tasks.workunit.client.1.smithi122.stdout:7/187: dwrite d0/f2 [4194304,4194304] 0 2024-03-22T02:28:47.891 INFO:tasks.workunit.client.1.smithi122.stdout:3/125: sync 2024-03-22T02:28:47.891 INFO:tasks.workunit.client.1.smithi122.stdout:5/170: sync 2024-03-22T02:28:47.891 INFO:tasks.workunit.client.1.smithi122.stdout:2/270: dread - df/d1f/d2a/d39/f62 zero size 2024-03-22T02:28:47.891 INFO:tasks.workunit.client.0.smithi103.stdout:2/762: rmdir d11/d25/d63/d45/d47/d83/d78/df6 0 2024-03-22T02:28:47.892 INFO:tasks.workunit.client.0.smithi103.stdout:8/857: mkdir d4/d19/d8c/d11a 0 2024-03-22T02:28:47.892 INFO:tasks.workunit.client.1.smithi122.stdout:6/273: write d7/de/dd/d20/f45 [688324,78774] 0 2024-03-22T02:28:47.894 INFO:tasks.workunit.client.1.smithi122.stdout:5/171: dread - d8/f19 zero size 2024-03-22T02:28:47.896 INFO:tasks.workunit.client.1.smithi122.stdout:2/271: mknod df/d1f/d23/c64 0 2024-03-22T02:28:47.897 INFO:tasks.workunit.client.1.smithi122.stdout:6/274: mknod d7/de/dd/df/c5e 0 2024-03-22T02:28:47.898 INFO:tasks.workunit.client.1.smithi122.stdout:3/126: mknod d2/d9/dc/df/c29 0 2024-03-22T02:28:47.899 INFO:tasks.workunit.client.1.smithi122.stdout:6/275: write d7/de/dd/d39/f5b [218535,75053] 0 2024-03-22T02:28:47.899 INFO:tasks.workunit.client.1.smithi122.stdout:7/188: mkdir d0/d30 0 2024-03-22T02:28:47.901 INFO:tasks.workunit.client.0.smithi103.stdout:4/763: link d3/d38/fdf d3/d1e/d78/f102 0 2024-03-22T02:28:47.903 INFO:tasks.workunit.client.0.smithi103.stdout:8/858: mknod d4/d2a/d3e/df9/c11b 0 2024-03-22T02:28:47.904 INFO:tasks.workunit.client.0.smithi103.stdout:6/543: dwrite d0/d14/d57/d68/f62 [0,4194304] 0 2024-03-22T02:28:47.905 INFO:tasks.workunit.client.1.smithi122.stdout:3/127: unlink d2/f6 0 2024-03-22T02:28:47.906 INFO:tasks.workunit.client.1.smithi122.stdout:6/276: creat d7/de/dd/d20/d2d/f5f x:0 0 0 2024-03-22T02:28:47.907 INFO:tasks.workunit.client.1.smithi122.stdout:7/189: symlink d0/d4/d20/d21/l31 0 2024-03-22T02:28:47.908 INFO:tasks.workunit.client.0.smithi103.stdout:9/749: write d2/d5/fb [161108,107579] 0 2024-03-22T02:28:47.909 INFO:tasks.workunit.client.0.smithi103.stdout:4/764: mkdir d3/da2/da6/d103 0 2024-03-22T02:28:47.909 INFO:tasks.workunit.client.0.smithi103.stdout:5/771: dwrite d0/f2d [0,4194304] 0 2024-03-22T02:28:47.911 INFO:tasks.workunit.client.1.smithi122.stdout:7/190: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:47.913 INFO:tasks.workunit.client.1.smithi122.stdout:3/128: rename d2/d9/dc/d17/c22 to d2/c2a 0 2024-03-22T02:28:47.913 INFO:tasks.workunit.client.0.smithi103.stdout:6/544: rmdir d0/d3/d22 39 2024-03-22T02:28:47.914 INFO:tasks.workunit.client.0.smithi103.stdout:4/765: rmdir d3/d1e/d53/d8a 39 2024-03-22T02:28:47.915 INFO:tasks.workunit.client.1.smithi122.stdout:7/191: creat d0/d4/d22/f32 x:0 0 0 2024-03-22T02:28:47.917 INFO:tasks.workunit.client.1.smithi122.stdout:6/277: rename d7/de/dd/d39/c44 to d7/de/dd/d20/d2d/c60 0 2024-03-22T02:28:47.917 INFO:tasks.workunit.client.0.smithi103.stdout:5/772: read d0/d14/d23/d6f/fbb [192684,99972] 0 2024-03-22T02:28:47.918 INFO:tasks.workunit.client.1.smithi122.stdout:2/272: rmdir df 39 2024-03-22T02:28:47.923 INFO:tasks.workunit.client.1.smithi122.stdout:6/278: creat d7/de/d27/d52/f61 x:0 0 0 2024-03-22T02:28:47.923 INFO:tasks.workunit.client.1.smithi122.stdout:2/273: fsync df/d1f/d33/d34/f5a 0 2024-03-22T02:28:47.923 INFO:tasks.workunit.client.0.smithi103.stdout:9/750: dwrite d2/d10/d43/d4b/f82 [0,4194304] 0 2024-03-22T02:28:47.923 INFO:tasks.workunit.client.1.smithi122.stdout:8/240: dwrite d1/d9/f24 [0,4194304] 0 2024-03-22T02:28:47.924 INFO:tasks.workunit.client.1.smithi122.stdout:7/192: mknod d0/d4/d20/d21/d28/c33 0 2024-03-22T02:28:47.925 INFO:tasks.workunit.client.0.smithi103.stdout:9/751: truncate d2/f3b 2011345 0 2024-03-22T02:28:47.927 INFO:tasks.workunit.client.1.smithi122.stdout:3/129: dwrite d2/f25 [0,4194304] 0 2024-03-22T02:28:47.927 INFO:tasks.workunit.client.0.smithi103.stdout:9/752: dread - d2/d10/d43/d7e/d96/d9d/fd7 zero size 2024-03-22T02:28:47.928 INFO:tasks.workunit.client.1.smithi122.stdout:8/241: chown d1/d9/f1f 724 1 2024-03-22T02:28:47.928 INFO:tasks.workunit.client.0.smithi103.stdout:6/545: mknod d0/d14/caf 0 2024-03-22T02:28:47.929 INFO:tasks.workunit.client.0.smithi103.stdout:5/773: rename d0/d14/d23/d2a/d33/db5/dbd/ded/d103/d105 to d0/d14/d23/d2a/d33/db5/dfb/d10c 0 2024-03-22T02:28:47.930 INFO:tasks.workunit.client.1.smithi122.stdout:7/193: mknod d0/d4/d20/c34 0 2024-03-22T02:28:47.931 INFO:tasks.workunit.client.1.smithi122.stdout:6/279: chown d7/de/dd/c1f 73 1 2024-03-22T02:28:47.931 INFO:tasks.workunit.client.1.smithi122.stdout:3/130: chown d2/d9/l18 91928 1 2024-03-22T02:28:47.931 INFO:tasks.workunit.client.1.smithi122.stdout:2/274: symlink df/d1f/d33/l65 0 2024-03-22T02:28:47.932 INFO:tasks.workunit.client.0.smithi103.stdout:4/766: mknod d3/d1e/d53/d8a/def/c104 0 2024-03-22T02:28:47.932 INFO:tasks.workunit.client.0.smithi103.stdout:6/546: truncate d0/d3/d33/fad 407244 0 2024-03-22T02:28:47.934 INFO:tasks.workunit.client.1.smithi122.stdout:5/172: dwrite d8/f33 [0,4194304] 0 2024-03-22T02:28:47.934 INFO:tasks.workunit.client.0.smithi103.stdout:9/753: mkdir d2/d5/d4c/d56/ddc 0 2024-03-22T02:28:47.934 INFO:tasks.workunit.client.0.smithi103.stdout:6/547: read d0/f5b [371929,40226] 0 2024-03-22T02:28:47.935 INFO:tasks.workunit.client.1.smithi122.stdout:8/242: mkdir d1/d40/d51 0 2024-03-22T02:28:47.937 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:28:47.937 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:28:47.937 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:28:47.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:28:47.938 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:28:47.939 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:28:47.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.934+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f126004c710 con 0x7f12600fbd10 2024-03-22T02:28:47.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.935+0000 7f125d7fa700 1 -- 172.21.15.103:0/3560893240 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f1254058ae0 con 0x7f12600fbd10 2024-03-22T02:28:47.939 INFO:tasks.workunit.client.0.smithi103.stdout:2/763: dread d11/f20 [0,4194304] 0 2024-03-22T02:28:47.939 INFO:tasks.workunit.client.0.smithi103.stdout:4/767: mknod d3/d1e/d25/d65/d83/c105 0 2024-03-22T02:28:47.940 INFO:tasks.workunit.client.1.smithi122.stdout:7/194: mknod d0/d4/d1d/c35 0 2024-03-22T02:28:47.940 INFO:tasks.workunit.client.0.smithi103.stdout:9/754: symlink d2/d5/d7b/ldd 0 2024-03-22T02:28:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.939+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 msgr2=0x7f125006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.939+0000 7f1266c0f700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 0x7f125006de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f124c009b80 tx=0x7f124c00b040 comp rx=0 tx=0).stop 2024-03-22T02:28:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.940+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 msgr2=0x7f126006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.940+0000 7f1266c0f700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1254013870 tx=0x7f1254009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:47.941 INFO:tasks.workunit.client.0.smithi103.stdout:4/768: chown d3/d1e/d78/df9/l91 30 1 2024-03-22T02:28:47.942 INFO:tasks.workunit.client.1.smithi122.stdout:3/131: mkdir d2/d9/dc/d17/d2b 0 2024-03-22T02:28:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.941+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 shutdown_connections 2024-03-22T02:28:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.941+0000 7f1266c0f700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f125006b9a0 0x7f125006de50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.941+0000 7f1266c0f700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12600fbd10 0x7f126006e3c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.941+0000 7f1266c0f700 1 --2- 172.21.15.103:0/3560893240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12600fcfc0 0x7f126006e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.941+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 >> 172.21.15.103:0/3560893240 conn(0x7f12600f7490 msgr2=0x7f12600f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:47.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.943+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 shutdown_connections 2024-03-22T02:28:47.944 INFO:tasks.workunit.client.0.smithi103.stdout:2/764: mkdir d11/d25/d63/d45/d100 0 2024-03-22T02:28:47.944 INFO:tasks.workunit.client.0.smithi103.stdout:9/755: write d2/da/d18/dc4/dce/fd0 [9083283,123951] 0 2024-03-22T02:28:47.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:47.945+0000 7f1266c0f700 1 -- 172.21.15.103:0/3560893240 wait complete. 2024-03-22T02:28:47.946 INFO:tasks.workunit.client.0.smithi103.stdout:6/548: dwrite d0/d14/f93 [0,4194304] 0 2024-03-22T02:28:47.948 INFO:tasks.workunit.client.1.smithi122.stdout:7/195: dwrite d0/d4/d20/f27 [0,4194304] 0 2024-03-22T02:28:47.950 INFO:tasks.workunit.client.0.smithi103.stdout:8/859: dread d4/d26/d27/d7e/f52 [0,4194304] 0 2024-03-22T02:28:47.953 INFO:tasks.workunit.client.0.smithi103.stdout:4/769: dwrite d3/fb [0,4194304] 0 2024-03-22T02:28:47.953 INFO:tasks.workunit.client.1.smithi122.stdout:2/275: rename cb to df/d1f/d33/c66 0 2024-03-22T02:28:47.956 INFO:tasks.workunit.client.0.smithi103.stdout:6/549: mkdir d0/d3/d22/d58/d6f/d75/db0 0 2024-03-22T02:28:47.956 INFO:tasks.workunit.client.0.smithi103.stdout:8/860: fsync d4/d19/f2e 0 2024-03-22T02:28:47.958 INFO:tasks.workunit.client.0.smithi103.stdout:8/861: fdatasync d4/d2a/d4f/fe3 0 2024-03-22T02:28:47.958 INFO:tasks.workunit.client.1.smithi122.stdout:5/173: mkdir d8/d3b 0 2024-03-22T02:28:47.958 INFO:tasks.workunit.client.0.smithi103.stdout:4/770: symlink d3/d1e/d53/d7a/l106 0 2024-03-22T02:28:47.960 INFO:tasks.workunit.client.1.smithi122.stdout:7/196: symlink d0/d4/d22/l36 0 2024-03-22T02:28:47.960 INFO:tasks.workunit.client.1.smithi122.stdout:2/276: truncate df/f37 5124175 0 2024-03-22T02:28:47.960 INFO:tasks.workunit.client.0.smithi103.stdout:8/862: chown d4/f89 6357591 1 2024-03-22T02:28:47.961 INFO:tasks.workunit.client.0.smithi103.stdout:6/550: mkdir d0/d3/d22/d58/d6f/d75/db0/db1 0 2024-03-22T02:28:47.963 INFO:tasks.workunit.client.0.smithi103.stdout:2/765: dwrite d11/d25/d63/d45/d47/f85 [0,4194304] 0 2024-03-22T02:28:47.964 INFO:tasks.workunit.client.0.smithi103.stdout:2/766: fsync d11/d19/d5d/fba 0 2024-03-22T02:28:47.966 INFO:tasks.workunit.client.1.smithi122.stdout:2/277: fsync df/d1f/d2a/d39/f53 0 2024-03-22T02:28:47.969 INFO:tasks.workunit.client.1.smithi122.stdout:5/174: dwrite d8/d1a/f34 [0,4194304] 0 2024-03-22T02:28:47.970 INFO:tasks.workunit.client.1.smithi122.stdout:5/175: write d8/da/f10 [5967761,56748] 0 2024-03-22T02:28:47.974 INFO:tasks.workunit.client.1.smithi122.stdout:5/176: dread - d8/f20 zero size 2024-03-22T02:28:47.976 INFO:tasks.workunit.client.1.smithi122.stdout:5/177: write d8/da/f16 [1739511,79702] 0 2024-03-22T02:28:47.977 INFO:tasks.workunit.client.0.smithi103.stdout:6/551: fsync d0/d14/d57/d68/f40 0 2024-03-22T02:28:47.978 INFO:tasks.workunit.client.1.smithi122.stdout:5/178: readlink d8/l29 0 2024-03-22T02:28:47.978 INFO:tasks.workunit.client.1.smithi122.stdout:2/278: dwrite df/d1f/d3d/d52/f5d [0,4194304] 0 2024-03-22T02:28:47.978 INFO:tasks.workunit.client.1.smithi122.stdout:5/179: dread - d8/d1a/f2f zero size 2024-03-22T02:28:47.980 INFO:tasks.workunit.client.0.smithi103.stdout:6/552: creat d0/d88/d8e/fb2 x:0 0 0 2024-03-22T02:28:47.980 INFO:tasks.workunit.client.0.smithi103.stdout:6/553: chown d0/d14/f56 39 1 2024-03-22T02:28:47.980 INFO:tasks.workunit.client.1.smithi122.stdout:5/180: write d8/f30 [695186,129075] 0 2024-03-22T02:28:47.981 INFO:tasks.workunit.client.0.smithi103.stdout:6/554: fsync d0/d14/d57/d68/f71 0 2024-03-22T02:28:47.982 INFO:tasks.workunit.client.0.smithi103.stdout:7/731: sync 2024-03-22T02:28:47.982 INFO:tasks.workunit.client.0.smithi103.stdout:3/829: sync 2024-03-22T02:28:47.983 INFO:tasks.workunit.client.0.smithi103.stdout:1/885: sync 2024-03-22T02:28:47.984 INFO:tasks.workunit.client.1.smithi122.stdout:5/181: mknod d8/d1e/d22/c3c 0 2024-03-22T02:28:47.986 INFO:tasks.workunit.client.0.smithi103.stdout:3/830: creat de/d12/d8a/d93/f11c x:0 0 0 2024-03-22T02:28:47.987 INFO:tasks.workunit.client.1.smithi122.stdout:5/182: write d8/d1a/f34 [3521278,56881] 0 2024-03-22T02:28:47.987 INFO:tasks.workunit.client.0.smithi103.stdout:7/732: mknod d1/da/d2f/ce5 0 2024-03-22T02:28:47.988 INFO:tasks.workunit.client.0.smithi103.stdout:6/555: dread d0/d3/d22/d58/d6f/f94 [0,4194304] 0 2024-03-22T02:28:47.989 INFO:tasks.workunit.client.0.smithi103.stdout:3/831: readlink de/d12/l15 0 2024-03-22T02:28:47.989 INFO:tasks.workunit.client.0.smithi103.stdout:7/733: fsync d1/d10/d11/d19/f55 0 2024-03-22T02:28:47.989 INFO:tasks.workunit.client.1.smithi122.stdout:2/279: dwrite df/d1f/d33/d34/f5f [0,4194304] 0 2024-03-22T02:28:47.991 INFO:tasks.workunit.client.0.smithi103.stdout:1/886: truncate db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fcd 408284 0 2024-03-22T02:28:47.993 INFO:tasks.workunit.client.1.smithi122.stdout:8/243: dread d1/d22/f2e [4194304,4194304] 0 2024-03-22T02:28:47.994 INFO:tasks.workunit.client.1.smithi122.stdout:5/183: creat d8/da/d28/f3d x:0 0 0 2024-03-22T02:28:47.994 INFO:tasks.workunit.client.0.smithi103.stdout:6/556: mknod d0/d88/d8e/dae/cb3 0 2024-03-22T02:28:47.996 INFO:tasks.workunit.client.0.smithi103.stdout:3/832: creat de/d12/d5d/d102/f11d x:0 0 0 2024-03-22T02:28:47.996 INFO:tasks.workunit.client.1.smithi122.stdout:6/280: truncate d7/de/f43 228368 0 2024-03-22T02:28:47.998 INFO:tasks.workunit.client.1.smithi122.stdout:2/280: mkdir df/d1f/d2a/d51/d67 0 2024-03-22T02:28:48.001 INFO:tasks.workunit.client.1.smithi122.stdout:8/244: rename d1/d9/f11 to d1/d22/f52 0 2024-03-22T02:28:48.003 INFO:tasks.workunit.client.0.smithi103.stdout:3/833: unlink de/d12/d19/ff7 0 2024-03-22T02:28:48.003 INFO:tasks.workunit.client.1.smithi122.stdout:4/205: sync 2024-03-22T02:28:48.004 INFO:tasks.workunit.client.0.smithi103.stdout:7/734: getdents d1/d10/d11/d19 0 2024-03-22T02:28:48.004 INFO:tasks.workunit.client.1.smithi122.stdout:6/281: rmdir d7/de/dd 39 2024-03-22T02:28:48.005 INFO:tasks.workunit.client.0.smithi103.stdout:1/887: rename db/d4a/d78/d92/da8/ddd/d29/d55 to db/d4a/d78/d92/dea/d134 0 2024-03-22T02:28:48.005 INFO:tasks.workunit.client.0.smithi103.stdout:7/735: chown d1/d10/d11/d19/d29/d61/d69/c25 1704828 1 2024-03-22T02:28:48.006 INFO:tasks.workunit.client.0.smithi103.stdout:6/557: mkdir d0/d3/d22/d58/db4 0 2024-03-22T02:28:48.006 INFO:tasks.workunit.client.1.smithi122.stdout:2/281: creat df/d1f/d3a/d4f/f68 x:0 0 0 2024-03-22T02:28:48.007 INFO:tasks.workunit.client.1.smithi122.stdout:2/282: stat df/f36 0 2024-03-22T02:28:48.009 INFO:tasks.workunit.client.1.smithi122.stdout:4/206: mkdir d0/d25/d7/de/d40 0 2024-03-22T02:28:48.009 INFO:tasks.workunit.client.0.smithi103.stdout:3/834: chown de/d6b/d80/d4f/d84/cc5 3948649 1 2024-03-22T02:28:48.010 INFO:tasks.workunit.client.0.smithi103.stdout:7/736: stat d1/d10/d11/d19/d29/d33/d4b/c65 0 2024-03-22T02:28:48.010 INFO:tasks.workunit.client.1.smithi122.stdout:3/132: creat d2/f2c x:0 0 0 2024-03-22T02:28:48.010 INFO:tasks.workunit.client.1.smithi122.stdout:5/184: link c5 d8/d1e/d22/d3a/d38/c3e 0 2024-03-22T02:28:48.011 INFO:tasks.workunit.client.1.smithi122.stdout:5/185: rename d8/da/d28 to d8/da/d28/d3f 22 2024-03-22T02:28:48.012 INFO:tasks.workunit.client.0.smithi103.stdout:1/888: dread db/d4a/d78/d92/da8/ddd/d18/d2d/f5d [0,4194304] 0 2024-03-22T02:28:48.012 INFO:tasks.workunit.client.0.smithi103.stdout:3/835: fdatasync de/d12/d8f/fbb 0 2024-03-22T02:28:48.014 INFO:tasks.workunit.client.1.smithi122.stdout:7/197: truncate d0/fd 2505189 0 2024-03-22T02:28:48.014 INFO:tasks.workunit.client.1.smithi122.stdout:7/198: stat d0/c13 0 2024-03-22T02:28:48.015 INFO:tasks.workunit.client.0.smithi103.stdout:2/767: truncate d11/f20 1232885 0 2024-03-22T02:28:48.016 INFO:tasks.workunit.client.1.smithi122.stdout:2/283: symlink df/d1f/d3d/d52/l69 0 2024-03-22T02:28:48.016 INFO:tasks.workunit.client.1.smithi122.stdout:5/186: read d8/da/f16 [345281,79695] 0 2024-03-22T02:28:48.020 INFO:tasks.workunit.client.1.smithi122.stdout:7/199: unlink d0/c3 0 2024-03-22T02:28:48.021 INFO:tasks.workunit.client.0.smithi103.stdout:2/768: symlink d11/d25/d63/l101 0 2024-03-22T02:28:48.022 INFO:tasks.workunit.client.1.smithi122.stdout:2/284: mkdir df/d1f/d33/d6a 0 2024-03-22T02:28:48.022 INFO:tasks.workunit.client.0.smithi103.stdout:4/771: dwrite d3/d1e/d78/f102 [0,4194304] 0 2024-03-22T02:28:48.023 INFO:tasks.workunit.client.1.smithi122.stdout:4/207: mkdir d0/d41 0 2024-03-22T02:28:48.023 INFO:tasks.workunit.client.1.smithi122.stdout:2/285: write df/d1f/d2a/d39/f62 [343126,99790] 0 2024-03-22T02:28:48.023 INFO:tasks.workunit.client.1.smithi122.stdout:5/187: rename d8/da to d8/da/d40 22 2024-03-22T02:28:48.024 INFO:tasks.workunit.client.0.smithi103.stdout:1/889: creat db/d4a/d78/d92/da8/ddd/d28/db2/dfd/f135 x:0 0 0 2024-03-22T02:28:48.024 INFO:tasks.workunit.client.1.smithi122.stdout:5/188: stat d8/da/f10 0 2024-03-22T02:28:48.028 INFO:tasks.workunit.client.1.smithi122.stdout:7/200: rmdir d0/d4/d20 39 2024-03-22T02:28:48.029 INFO:tasks.workunit.client.0.smithi103.stdout:2/769: chown d11/d25/cc3 41217 1 2024-03-22T02:28:48.030 INFO:tasks.workunit.client.1.smithi122.stdout:2/286: mknod df/d1f/d3d/c6b 0 2024-03-22T02:28:48.032 INFO:tasks.workunit.client.1.smithi122.stdout:4/208: mkdir d0/d12/d15/d42 0 2024-03-22T02:28:48.033 INFO:tasks.workunit.client.1.smithi122.stdout:5/189: dwrite d8/d1e/f23 [4194304,4194304] 0 2024-03-22T02:28:48.033 INFO:tasks.workunit.client.1.smithi122.stdout:5/190: read - d8/f26 zero size 2024-03-22T02:28:48.034 INFO:tasks.workunit.client.0.smithi103.stdout:8/863: dread d4/d2a/d3e/da4/f106 [0,4194304] 0 2024-03-22T02:28:48.034 INFO:tasks.workunit.client.0.smithi103.stdout:1/890: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/dfd/f11e [0,4194304] 0 2024-03-22T02:28:48.034 INFO:tasks.workunit.client.0.smithi103.stdout:3/836: link de/d6b/cf6 de/d12/d19/df2/c11e 0 2024-03-22T02:28:48.035 INFO:tasks.workunit.client.1.smithi122.stdout:4/209: write d0/f19 [4926574,41435] 0 2024-03-22T02:28:48.039 INFO:tasks.workunit.client.0.smithi103.stdout:8/864: truncate d4/d26/d27/d7e/d39/d7b/fe2 1365293 0 2024-03-22T02:28:48.039 INFO:tasks.workunit.client.1.smithi122.stdout:4/210: chown d0/d12/d15/d2d 1 1 2024-03-22T02:28:48.039 INFO:tasks.workunit.client.1.smithi122.stdout:5/191: rename d8/d1e/d22/d3a/d38 to d8/d1e/d22/d3a/d38/d41 22 2024-03-22T02:28:48.039 INFO:tasks.workunit.client.1.smithi122.stdout:8/245: truncate d1/d8/d2f/f44 1239315 0 2024-03-22T02:28:48.041 INFO:tasks.workunit.client.1.smithi122.stdout:7/201: mkdir d0/d4/d20/d21/d37 0 2024-03-22T02:28:48.042 INFO:tasks.workunit.client.1.smithi122.stdout:5/192: truncate d8/d1a/f2f 270687 0 2024-03-22T02:28:48.042 INFO:tasks.workunit.client.1.smithi122.stdout:8/246: read d1/d9/f26 [2523720,73531] 0 2024-03-22T02:28:48.042 INFO:tasks.workunit.client.1.smithi122.stdout:5/193: fsync d8/da/f10 0 2024-03-22T02:28:48.042 INFO:tasks.workunit.client.1.smithi122.stdout:8/247: readlink d1/d9/l34 0 2024-03-22T02:28:48.043 INFO:tasks.workunit.client.1.smithi122.stdout:2/287: dread df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:48.048 INFO:tasks.workunit.client.0.smithi103.stdout:7/737: dread d1/d10/d11/d19/d29/d61/d69/d6b/d7a/fdd [0,4194304] 0 2024-03-22T02:28:48.050 INFO:tasks.workunit.client.0.smithi103.stdout:1/891: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d11b/f136 x:0 0 0 2024-03-22T02:28:48.051 INFO:tasks.workunit.client.1.smithi122.stdout:5/194: mkdir d8/da/d28/d42 0 2024-03-22T02:28:48.052 INFO:tasks.workunit.client.0.smithi103.stdout:8/865: truncate d4/d48/d40/d68/f95 3453591 0 2024-03-22T02:28:48.053 INFO:tasks.workunit.client.0.smithi103.stdout:8/866: chown d4/d26/d27/d7e/d67/ccc 393263977 1 2024-03-22T02:28:48.060 INFO:tasks.workunit.client.0.smithi103.stdout:1/892: mkdir db/d4a/d78/d92/da8/ddd/d18/dca/de5/d137 0 2024-03-22T02:28:48.061 INFO:tasks.workunit.client.0.smithi103.stdout:8/867: rmdir d4/d2a/d4f 39 2024-03-22T02:28:48.061 INFO:tasks.workunit.client.1.smithi122.stdout:5/195: unlink d8/da/f18 0 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.1.smithi122.stdout:3/133: dread d2/f15 [0,4194304] 0 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.1.smithi122.stdout:5/196: read - d8/f19 zero size 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.1.smithi122.stdout:4/211: creat d0/d25/f43 x:0 0 0 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.1.smithi122.stdout:8/248: creat d1/f53 x:0 0 0 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.0.smithi103.stdout:2/770: dread d11/d25/d63/f96 [0,4194304] 0 2024-03-22T02:28:48.062 INFO:tasks.workunit.client.0.smithi103.stdout:4/772: getdents d3/d1e/d68/dbc 0 2024-03-22T02:28:48.063 INFO:tasks.workunit.client.0.smithi103.stdout:4/773: dread - d3/d1e/d53/d7a/da3/fbe zero size 2024-03-22T02:28:48.065 INFO:tasks.workunit.client.1.smithi122.stdout:3/134: write d2/d9/dc/f16 [943990,103234] 0 2024-03-22T02:28:48.065 INFO:tasks.workunit.client.0.smithi103.stdout:8/868: creat d4/d26/d27/d7e/d67/f11c x:0 0 0 2024-03-22T02:28:48.067 INFO:tasks.workunit.client.1.smithi122.stdout:5/197: symlink d8/da/d28/l43 0 2024-03-22T02:28:48.068 INFO:tasks.workunit.client.1.smithi122.stdout:5/198: read - d8/f19 zero size 2024-03-22T02:28:48.068 INFO:tasks.workunit.client.0.smithi103.stdout:7/738: creat d1/d10/d11/d19/d29/d33/fe6 x:0 0 0 2024-03-22T02:28:48.068 INFO:tasks.workunit.client.0.smithi103.stdout:8/869: stat d4/d19/d1a/d104 0 2024-03-22T02:28:48.070 INFO:tasks.workunit.client.1.smithi122.stdout:4/212: rename d0/f19 to d0/d25/d7/de/f44 0 2024-03-22T02:28:48.071 INFO:tasks.workunit.client.1.smithi122.stdout:8/249: truncate d1/d9/f13 539531 0 2024-03-22T02:28:48.071 INFO:tasks.workunit.client.1.smithi122.stdout:4/213: stat d0/d25/d7 0 2024-03-22T02:28:48.072 INFO:tasks.workunit.client.0.smithi103.stdout:4/774: mkdir d3/d30/d45/d107 0 2024-03-22T02:28:48.073 INFO:tasks.workunit.client.1.smithi122.stdout:2/288: link df/d1f/l38 df/d1f/d3d/l6c 0 2024-03-22T02:28:48.073 INFO:tasks.workunit.client.1.smithi122.stdout:8/250: truncate d1/d40/f47 155943 0 2024-03-22T02:28:48.074 INFO:tasks.workunit.client.0.smithi103.stdout:8/870: mknod d4/d19/d1a/d4a/c11d 0 2024-03-22T02:28:48.074 INFO:tasks.workunit.client.1.smithi122.stdout:8/251: dread - d1/f53 zero size 2024-03-22T02:28:48.075 INFO:tasks.workunit.client.1.smithi122.stdout:5/199: dwrite d8/f33 [0,4194304] 0 2024-03-22T02:28:48.075 INFO:tasks.workunit.client.0.smithi103.stdout:1/893: dwrite db/d4a/d78/d92/da8/ddd/d28/d97/fb3 [0,4194304] 0 2024-03-22T02:28:48.075 INFO:tasks.workunit.client.1.smithi122.stdout:3/135: unlink d2/d9/dc/d17/f20 0 2024-03-22T02:28:48.077 INFO:tasks.workunit.client.0.smithi103.stdout:3/837: dwrite de/d12/d4d/fc4 [0,4194304] 0 2024-03-22T02:28:48.077 INFO:tasks.workunit.client.1.smithi122.stdout:5/200: write d8/f26 [412734,91079] 0 2024-03-22T02:28:48.078 INFO:tasks.workunit.client.0.smithi103.stdout:7/739: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/fe7 x:0 0 0 2024-03-22T02:28:48.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/202: fsync d0/fd 0 2024-03-22T02:28:48.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/203: chown d0/d4/d20/f27 167895 1 2024-03-22T02:28:48.080 INFO:tasks.workunit.client.0.smithi103.stdout:4/775: creat d3/da2/f108 x:0 0 0 2024-03-22T02:28:48.080 INFO:tasks.workunit.client.1.smithi122.stdout:5/201: chown d8/d1e 180179 1 2024-03-22T02:28:48.083 INFO:tasks.workunit.client.0.smithi103.stdout:4/776: write d3/da2/f108 [93839,47067] 0 2024-03-22T02:28:48.083 INFO:tasks.workunit.client.0.smithi103.stdout:4/777: dread - d3/d31/fe6 zero size 2024-03-22T02:28:48.084 INFO:tasks.workunit.client.0.smithi103.stdout:4/778: dread - d3/d38/fda zero size 2024-03-22T02:28:48.084 INFO:tasks.workunit.client.0.smithi103.stdout:1/894: symlink db/d4a/d78/d92/da8/ddd/d28/db2/dc8/l138 0 2024-03-22T02:28:48.085 INFO:tasks.workunit.client.1.smithi122.stdout:4/214: symlink d0/d25/d7/de/l45 0 2024-03-22T02:28:48.085 INFO:tasks.workunit.client.0.smithi103.stdout:2/771: write d11/d25/d63/d45/d47/d83/f76 [346901,62455] 0 2024-03-22T02:28:48.088 INFO:tasks.workunit.client.0.smithi103.stdout:7/740: mkdir d1/d10/d11/d19/dc3/de8 0 2024-03-22T02:28:48.089 INFO:tasks.workunit.client.1.smithi122.stdout:7/204: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:48.092 INFO:tasks.workunit.client.0.smithi103.stdout:2/772: read d11/d19/d5d/f66 [2014857,39460] 0 2024-03-22T02:28:48.093 INFO:tasks.workunit.client.0.smithi103.stdout:8/871: rename d4/d48 to d4/d26/d27/d7e/d39/d49/d79/d11e 0 2024-03-22T02:28:48.093 INFO:tasks.workunit.client.1.smithi122.stdout:3/136: truncate d2/d9/dc/d21/f23 937089 0 2024-03-22T02:28:48.094 INFO:tasks.workunit.client.0.smithi103.stdout:4/779: dwrite d3/d38/fda [0,4194304] 0 2024-03-22T02:28:48.097 INFO:tasks.workunit.client.1.smithi122.stdout:2/289: mknod df/d1f/d33/d4d/c6d 0 2024-03-22T02:28:48.099 INFO:tasks.workunit.client.1.smithi122.stdout:7/205: unlink d0/f18 0 2024-03-22T02:28:48.100 INFO:tasks.workunit.client.1.smithi122.stdout:7/206: read - d0/d4/d22/f32 zero size 2024-03-22T02:28:48.101 INFO:tasks.workunit.client.1.smithi122.stdout:7/207: read d0/d4/fb [2916265,98495] 0 2024-03-22T02:28:48.102 INFO:tasks.workunit.client.0.smithi103.stdout:3/838: link de/d1a/d25/f8b de/d12/d4d/f11f 0 2024-03-22T02:28:48.103 INFO:tasks.workunit.client.0.smithi103.stdout:7/741: dwrite d1/d10/d11/d19/d29/d61/fc8 [0,4194304] 0 2024-03-22T02:28:48.103 INFO:tasks.workunit.client.0.smithi103.stdout:2/773: fdatasync f4 0 2024-03-22T02:28:48.104 INFO:tasks.workunit.client.1.smithi122.stdout:8/252: rename d1/l42 to d1/d8/d2f/l54 0 2024-03-22T02:28:48.108 INFO:tasks.workunit.client.1.smithi122.stdout:5/202: link d8/da/f10 d8/da/d28/f44 0 2024-03-22T02:28:48.109 INFO:tasks.workunit.client.1.smithi122.stdout:3/137: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:48.111 INFO:tasks.workunit.client.0.smithi103.stdout:2/774: creat d11/d25/d46/db6/f102 x:0 0 0 2024-03-22T02:28:48.112 INFO:tasks.workunit.client.1.smithi122.stdout:4/215: truncate d0/d25/f8 699217 0 2024-03-22T02:28:48.112 INFO:tasks.workunit.client.0.smithi103.stdout:8/872: dwrite d4/d2a/d4f/fdd [0,4194304] 0 2024-03-22T02:28:48.113 INFO:tasks.workunit.client.1.smithi122.stdout:4/216: truncate d0/d25/f28 1045474 0 2024-03-22T02:28:48.114 INFO:tasks.workunit.client.1.smithi122.stdout:7/208: symlink d0/d4/d20/l38 0 2024-03-22T02:28:48.114 INFO:tasks.workunit.client.1.smithi122.stdout:4/217: chown d0/d12/d15/d2d/f38 83 1 2024-03-22T02:28:48.115 INFO:tasks.workunit.client.0.smithi103.stdout:3/839: read fa [129448,47277] 0 2024-03-22T02:28:48.115 INFO:tasks.workunit.client.0.smithi103.stdout:7/742: getdents d1/d6f 0 2024-03-22T02:28:48.116 INFO:tasks.workunit.client.1.smithi122.stdout:4/218: write d0/d12/d15/d2d/f38 [173649,122862] 0 2024-03-22T02:28:48.117 INFO:tasks.workunit.client.1.smithi122.stdout:2/290: rename df/d1f/d23/f3b to df/d1f/d2a/d51/d67/f6e 0 2024-03-22T02:28:48.117 INFO:tasks.workunit.client.1.smithi122.stdout:4/219: write d0/d25/f3b [649104,45175] 0 2024-03-22T02:28:48.122 INFO:tasks.workunit.client.1.smithi122.stdout:8/253: creat d1/d22/f55 x:0 0 0 2024-03-22T02:28:48.123 INFO:tasks.workunit.client.1.smithi122.stdout:7/209: mkdir d0/d4/d20/d21/d39 0 2024-03-22T02:28:48.123 INFO:tasks.workunit.client.0.smithi103.stdout:2/775: creat d11/d19/da7/dd3/f103 x:0 0 0 2024-03-22T02:28:48.123 INFO:tasks.workunit.client.0.smithi103.stdout:3/840: mknod de/d12/c120 0 2024-03-22T02:28:48.125 INFO:tasks.workunit.client.1.smithi122.stdout:4/220: truncate d0/d12/f1c 1556203 0 2024-03-22T02:28:48.125 INFO:tasks.workunit.client.0.smithi103.stdout:8/873: mkdir d4/d26/d27/d7e/d39/d49/d79/d11e/d11f 0 2024-03-22T02:28:48.126 INFO:tasks.workunit.client.1.smithi122.stdout:5/203: creat d8/d3b/f45 x:0 0 0 2024-03-22T02:28:48.127 INFO:tasks.workunit.client.1.smithi122.stdout:5/204: chown d8/da/d28/f2c 159271 1 2024-03-22T02:28:48.127 INFO:tasks.workunit.client.1.smithi122.stdout:8/254: creat d1/d4d/f56 x:0 0 0 2024-03-22T02:28:48.129 INFO:tasks.workunit.client.1.smithi122.stdout:2/291: unlink df/d1f/d33/c66 0 2024-03-22T02:28:48.130 INFO:tasks.workunit.client.0.smithi103.stdout:1/895: dwrite db/d4a/d78/d92/da8/ddd/d21/f23 [0,4194304] 0 2024-03-22T02:28:48.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.129+0000 7f97d1258700 1 -- 172.21.15.103:0/3518896000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc0fdbe0 msgr2=0x7f97cc0fc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:48.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.129+0000 7f97d1258700 1 --2- 172.21.15.103:0/3518896000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc0fdbe0 0x7f97cc0fc000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f97bc004940 tx=0x7f97bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:48.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.129+0000 7f97d1258700 1 -- 172.21.15.103:0/3518896000 shutdown_connections 2024-03-22T02:28:48.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.129+0000 7f97d1258700 1 --2- 172.21.15.103:0/3518896000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc0fdbe0 0x7f97cc0fc000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.130+0000 7f97d1258700 1 --2- 172.21.15.103:0/3518896000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 0x7f97cc0fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.130+0000 7f97d1258700 1 -- 172.21.15.103:0/3518896000 >> 172.21.15.103:0/3518896000 conn(0x7f97cc0f6ba0 msgr2=0x7f97cc0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:48.131 INFO:tasks.workunit.client.1.smithi122.stdout:4/221: symlink d0/d12/l46 0 2024-03-22T02:28:48.131 INFO:tasks.workunit.client.0.smithi103.stdout:3/841: mkdir de/d1a/d25/d121 0 2024-03-22T02:28:48.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.131+0000 7f97d1258700 1 -- 172.21.15.103:0/3518896000 shutdown_connections 2024-03-22T02:28:48.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.132+0000 7f97d1258700 1 -- 172.21.15.103:0/3518896000 wait complete. 2024-03-22T02:28:48.133 INFO:tasks.workunit.client.0.smithi103.stdout:4/780: write d3/f11 [1412995,20999] 0 2024-03-22T02:28:48.133 INFO:tasks.workunit.client.0.smithi103.stdout:8/874: chown d4/d26/d27/d7e/d39/d49/d79/d11e/d40/d68/cb1 2 1 2024-03-22T02:28:48.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 Processor -- start 2024-03-22T02:28:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 -- start start 2024-03-22T02:28:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 0x7f97cc113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc114900 con 0x7f97cc114440 2024-03-22T02:28:48.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.133+0000 7f97d1258700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc114a40 con 0x7f97cc0fd290 2024-03-22T02:28:48.134 INFO:tasks.workunit.client.1.smithi122.stdout:5/205: mkdir d8/d1e/d22/d3a/d46 0 2024-03-22T02:28:48.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:48.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40056/0 (socket says 172.21.15.103:40056) 2024-03-22T02:28:48.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 -- 172.21.15.103:0/1718616165 learned_addr learned my addr 172.21.15.103:0/1718616165 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:28:48.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97cad9d700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 0x7f97cc113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:48.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 -- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 msgr2=0x7f97cc113f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:48.135 INFO:tasks.workunit.client.0.smithi103.stdout:4/781: chown d3/d1e/d25/d65/d83/c105 8260 1 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 0x7f97cc113f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.134+0000 7f97ca59c700 1 -- 172.21.15.103:0/1718616165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc005040 con 0x7f97cc114440 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97ca59c700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f97bc004890 tx=0x7f97bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97bc021070 con 0x7f97cc114440 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97bc013dd0 con 0x7f97cc114440 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc1157c0 con 0x7f97cc114440 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97bc00e070 con 0x7f97cc114440 2024-03-22T02:28:48.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.135+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc115be0 con 0x7f97cc114440 2024-03-22T02:28:48.137 INFO:tasks.workunit.client.0.smithi103.stdout:8/875: write d4/d2a/d4f/fb7 [304499,123543] 0 2024-03-22T02:28:48.137 INFO:tasks.workunit.client.1.smithi122.stdout:7/210: symlink d0/d25/l3a 0 2024-03-22T02:28:48.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.137+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc05f5f0 con 0x7f97cc114440 2024-03-22T02:28:48.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.138+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f97bc026410 con 0x7f97cc114440 2024-03-22T02:28:48.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.138+0000 7f97b37fe700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 0x7f97b406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:48.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.138+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f97bc08bbe0 con 0x7f97cc114440 2024-03-22T02:28:48.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.139+0000 7f97cad9d700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 0x7f97b406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:48.139 INFO:tasks.workunit.client.1.smithi122.stdout:4/222: symlink d0/d12/l47 0 2024-03-22T02:28:48.140 INFO:tasks.workunit.client.1.smithi122.stdout:8/255: mknod d1/d9/d50/c57 0 2024-03-22T02:28:48.141 INFO:tasks.workunit.client.0.smithi103.stdout:4/782: symlink d3/da2/da6/dc2/l109 0 2024-03-22T02:28:48.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.140+0000 7f97cad9d700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 0x7f97b406dd70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009b80 tx=0x7f97c000b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:48.142 INFO:tasks.workunit.client.1.smithi122.stdout:0/161: sync 2024-03-22T02:28:48.142 INFO:tasks.workunit.client.1.smithi122.stdout:9/253: sync 2024-03-22T02:28:48.142 INFO:tasks.workunit.client.1.smithi122.stdout:1/226: sync 2024-03-22T02:28:48.142 INFO:tasks.workunit.client.0.smithi103.stdout:4/783: fsync d3/d38/d5c/d86/fb6 0 2024-03-22T02:28:48.142 INFO:tasks.workunit.client.0.smithi103.stdout:8/876: rename d4/d26/d27/d7e/c41 to d4/d26/d27/d7e/c120 0 2024-03-22T02:28:48.143 INFO:tasks.workunit.client.1.smithi122.stdout:1/227: read - dc/f37 zero size 2024-03-22T02:28:48.144 INFO:tasks.workunit.client.0.smithi103.stdout:1/896: link db/d4a/d78/d92/da8/ddd/l7a db/d4a/d78/d92/dea/l139 0 2024-03-22T02:28:48.144 INFO:tasks.workunit.client.1.smithi122.stdout:7/211: creat d0/f3b x:0 0 0 2024-03-22T02:28:48.145 INFO:tasks.workunit.client.1.smithi122.stdout:0/162: mkdir d11/d16/d37 0 2024-03-22T02:28:48.146 INFO:tasks.workunit.client.0.smithi103.stdout:1/897: read db/d4a/d78/d92/da8/ddd/d29/d41/f89 [610964,4403] 0 2024-03-22T02:28:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.147+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f97bc059f50 con 0x7f97cc114440 2024-03-22T02:28:48.148 INFO:tasks.workunit.client.0.smithi103.stdout:3/842: write de/d6b/d80/d4f/fcd [43560,161] 0 2024-03-22T02:28:48.149 INFO:tasks.workunit.client.0.smithi103.stdout:3/843: chown de/d1a 265 1 2024-03-22T02:28:48.150 INFO:tasks.workunit.client.0.smithi103.stdout:8/877: write d4/d2a/d38/f10c [440450,128022] 0 2024-03-22T02:28:48.150 INFO:tasks.workunit.client.0.smithi103.stdout:1/898: mkdir db/d4a/d78/d92/da8/ddd/d18/d84/d13a 0 2024-03-22T02:28:48.151 INFO:tasks.workunit.client.1.smithi122.stdout:2/292: dread df/d1f/f3c [0,4194304] 0 2024-03-22T02:28:48.152 INFO:tasks.workunit.client.0.smithi103.stdout:8/878: dread - d4/d2a/d8b/f9f zero size 2024-03-22T02:28:48.152 INFO:tasks.workunit.client.1.smithi122.stdout:2/293: dread fd [4194304,4194304] 0 2024-03-22T02:28:48.154 INFO:tasks.workunit.client.0.smithi103.stdout:3/844: mkdir de/d12/d8a/db0/d122 0 2024-03-22T02:28:48.155 INFO:tasks.workunit.client.0.smithi103.stdout:1/899: dread db/d4a/f5e [0,4194304] 0 2024-03-22T02:28:48.155 INFO:tasks.workunit.client.0.smithi103.stdout:3/845: stat de/d6b/d80/d73/f10a 0 2024-03-22T02:28:48.155 INFO:tasks.workunit.client.0.smithi103.stdout:8/879: mknod d4/d2a/d3e/c121 0 2024-03-22T02:28:48.157 INFO:tasks.workunit.client.0.smithi103.stdout:8/880: write d4/d2a/d4f/fb7 [700826,79997] 0 2024-03-22T02:28:48.158 INFO:tasks.workunit.client.0.smithi103.stdout:3/846: chown de/d12/d4d/ccf 23 1 2024-03-22T02:28:48.159 INFO:tasks.workunit.client.0.smithi103.stdout:1/900: symlink db/d4a/d78/d92/da8/ddd/d29/d41/l13b 0 2024-03-22T02:28:48.160 INFO:tasks.workunit.client.0.smithi103.stdout:8/881: symlink d4/d26/d27/da3/l122 0 2024-03-22T02:28:48.161 INFO:tasks.workunit.client.0.smithi103.stdout:3/847: mknod de/d12/d5d/d102/c123 0 2024-03-22T02:28:48.165 INFO:tasks.workunit.client.0.smithi103.stdout:1/901: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/ded/f13c x:0 0 0 2024-03-22T02:28:48.168 INFO:tasks.workunit.client.0.smithi103.stdout:8/882: dwrite d4/d2a/d38/f9c [0,4194304] 0 2024-03-22T02:28:48.174 INFO:tasks.workunit.client.0.smithi103.stdout:3/848: dread de/d12/d5d/f75 [0,4194304] 0 2024-03-22T02:28:48.176 INFO:tasks.workunit.client.0.smithi103.stdout:3/849: chown de/d6b/d80/f7c 956191 1 2024-03-22T02:28:48.176 INFO:tasks.workunit.client.0.smithi103.stdout:1/902: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fde [0,4194304] 0 2024-03-22T02:28:48.183 INFO:tasks.workunit.client.0.smithi103.stdout:1/903: dwrite db/f35 [0,4194304] 0 2024-03-22T02:28:48.186 INFO:tasks.workunit.client.0.smithi103.stdout:1/904: rmdir db/d4a/d78/d92/da8/dc9 39 2024-03-22T02:28:48.201 INFO:tasks.workunit.client.1.smithi122.stdout:9/254: rename db/d2f to db/d23/d1f/d57 0 2024-03-22T02:28:48.205 INFO:tasks.workunit.client.1.smithi122.stdout:0/163: creat d11/d14/d1b/f38 x:0 0 0 2024-03-22T02:28:48.206 INFO:tasks.workunit.client.1.smithi122.stdout:1/228: unlink c1 0 2024-03-22T02:28:48.206 INFO:tasks.workunit.client.1.smithi122.stdout:0/164: readlink d11/d14/d1d/d27/l21 0 2024-03-22T02:28:48.207 INFO:tasks.workunit.client.1.smithi122.stdout:1/229: chown dc/d2b 4074 1 2024-03-22T02:28:48.208 INFO:tasks.workunit.client.1.smithi122.stdout:2/294: mknod df/d1f/d33/d4d/c6f 0 2024-03-22T02:28:48.208 INFO:tasks.workunit.client.0.smithi103.stdout:1/905: dread db/d4a/d78/d92/da8/ddd/f12 [0,4194304] 0 2024-03-22T02:28:48.209 INFO:tasks.workunit.client.1.smithi122.stdout:1/230: fsync dc/ff 0 2024-03-22T02:28:48.210 INFO:tasks.workunit.client.0.smithi103.stdout:3/850: dread de/f2d [4194304,4194304] 0 2024-03-22T02:28:48.211 INFO:tasks.workunit.client.1.smithi122.stdout:9/255: dwrite db/d23/d1f/d21/d25/f55 [0,4194304] 0 2024-03-22T02:28:48.212 INFO:tasks.workunit.client.0.smithi103.stdout:1/906: write db/d4a/d78/d92/da8/ddd/d28/d97/fb3 [5003691,30217] 0 2024-03-22T02:28:48.215 INFO:tasks.workunit.client.1.smithi122.stdout:2/295: creat df/d1f/d3a/d4f/f70 x:0 0 0 2024-03-22T02:28:48.216 INFO:tasks.workunit.client.0.smithi103.stdout:3/851: creat de/d12/f124 x:0 0 0 2024-03-22T02:28:48.217 INFO:tasks.workunit.client.0.smithi103.stdout:3/852: read - de/d12/d8a/f118 zero size 2024-03-22T02:28:48.218 INFO:tasks.workunit.client.1.smithi122.stdout:1/231: creat dc/d3b/f43 x:0 0 0 2024-03-22T02:28:48.218 INFO:tasks.workunit.client.0.smithi103.stdout:1/907: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/l13d 0 2024-03-22T02:28:48.219 INFO:tasks.workunit.client.1.smithi122.stdout:0/165: symlink d11/d16/d37/l39 0 2024-03-22T02:28:48.222 INFO:tasks.workunit.client.1.smithi122.stdout:9/256: unlink db/d1b/f22 0 2024-03-22T02:28:48.222 INFO:tasks.workunit.client.0.smithi103.stdout:0/745: sync 2024-03-22T02:28:48.223 INFO:tasks.workunit.client.1.smithi122.stdout:9/257: chown db/df/c17 28 1 2024-03-22T02:28:48.223 INFO:tasks.workunit.client.1.smithi122.stdout:9/258: chown db/d23/d1f/d57 2221 1 2024-03-22T02:28:48.223 INFO:tasks.workunit.client.0.smithi103.stdout:0/746: chown dd/d16/d4b/d96/d29/d48/d5c 45956142 1 2024-03-22T02:28:48.225 INFO:tasks.workunit.client.1.smithi122.stdout:7/212: rename d0/c15 to d0/d4/c3c 0 2024-03-22T02:28:48.225 INFO:tasks.workunit.client.1.smithi122.stdout:7/213: fdatasync d0/f1b 0 2024-03-22T02:28:48.227 INFO:tasks.workunit.client.1.smithi122.stdout:7/214: readlink d0/d25/l3a 0 2024-03-22T02:28:48.228 INFO:tasks.workunit.client.1.smithi122.stdout:1/232: mknod dc/d3b/c44 0 2024-03-22T02:28:48.229 INFO:tasks.workunit.client.1.smithi122.stdout:1/233: chown dc/f38 4409045 1 2024-03-22T02:28:48.230 INFO:tasks.workunit.client.1.smithi122.stdout:9/259: dwrite db/d23/d1f/f2b [0,4194304] 0 2024-03-22T02:28:48.231 INFO:tasks.workunit.client.0.smithi103.stdout:3/853: write de/d1a/d25/fd1 [2284308,74685] 0 2024-03-22T02:28:48.231 INFO:tasks.workunit.client.1.smithi122.stdout:0/166: creat d11/d14/d1d/d27/d2f/f3a x:0 0 0 2024-03-22T02:28:48.235 INFO:tasks.workunit.client.0.smithi103.stdout:0/747: dwrite dd/d16/d4b/d96/d29/d48/ded/fd1 [0,4194304] 0 2024-03-22T02:28:48.235 INFO:tasks.workunit.client.1.smithi122.stdout:7/215: mknod d0/d25/c3d 0 2024-03-22T02:28:48.236 INFO:tasks.workunit.client.1.smithi122.stdout:7/216: dread - d0/d4/d20/d21/f2b zero size 2024-03-22T02:28:48.238 INFO:tasks.workunit.client.1.smithi122.stdout:1/234: creat dc/d3b/f45 x:0 0 0 2024-03-22T02:28:48.238 INFO:tasks.workunit.client.0.smithi103.stdout:8/883: dread d4/fd [0,4194304] 0 2024-03-22T02:28:48.238 INFO:tasks.workunit.client.1.smithi122.stdout:4/223: truncate d0/d12/f31 1168378 0 2024-03-22T02:28:48.239 INFO:tasks.workunit.client.0.smithi103.stdout:3/854: unlink de/d12/d5d/da9/fff 0 2024-03-22T02:28:48.240 INFO:tasks.workunit.client.1.smithi122.stdout:8/256: truncate d1/d8/d2f/f3e 2512741 0 2024-03-22T02:28:48.241 INFO:tasks.workunit.client.0.smithi103.stdout:1/908: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/f57 [0,4194304] 0 2024-03-22T02:28:48.241 INFO:tasks.workunit.client.0.smithi103.stdout:3/855: chown de/d6b/d80/d73/lfa 2594 1 2024-03-22T02:28:48.242 INFO:tasks.workunit.client.1.smithi122.stdout:8/257: write d1/d3f/f41 [433733,64692] 0 2024-03-22T02:28:48.243 INFO:tasks.workunit.client.1.smithi122.stdout:0/167: rename d11/d14/d1d/f2e to d11/f3b 0 2024-03-22T02:28:48.245 INFO:tasks.workunit.client.1.smithi122.stdout:1/235: mkdir dc/d3b/d46 0 2024-03-22T02:28:48.245 INFO:tasks.workunit.client.0.smithi103.stdout:4/784: dread d3/f77 [0,4194304] 0 2024-03-22T02:28:48.245 INFO:tasks.workunit.client.0.smithi103.stdout:4/785: chown d3/d30/d45/l4b 5647558 1 2024-03-22T02:28:48.245 INFO:tasks.workunit.client.0.smithi103.stdout:0/748: rename dd/d16/d4b/d96/d64 to dd/d16/d4b/d96/d64/dab/df8 22 2024-03-22T02:28:48.246 INFO:tasks.workunit.client.0.smithi103.stdout:0/749: readlink dd/l75 0 2024-03-22T02:28:48.246 INFO:tasks.workunit.client.1.smithi122.stdout:7/217: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:48.246 INFO:tasks.workunit.client.1.smithi122.stdout:9/260: mkdir db/d23/d1e/d30/d58 0 2024-03-22T02:28:48.247 INFO:tasks.workunit.client.1.smithi122.stdout:1/236: chown dc/f22 114916 1 2024-03-22T02:28:48.248 INFO:tasks.workunit.client.0.smithi103.stdout:4/786: write d3/d1e/d53/d8a/fd3 [1006061,54356] 0 2024-03-22T02:28:48.250 INFO:tasks.workunit.client.0.smithi103.stdout:0/750: read - dd/d16/d4b/d96/fe1 zero size 2024-03-22T02:28:48.250 INFO:tasks.workunit.client.0.smithi103.stdout:3/856: dread de/d1a/d25/f6a [0,4194304] 0 2024-03-22T02:28:48.253 INFO:tasks.workunit.client.1.smithi122.stdout:0/168: unlink la 0 2024-03-22T02:28:48.253 INFO:tasks.workunit.client.1.smithi122.stdout:7/218: write d0/fd [1035756,115144] 0 2024-03-22T02:28:48.253 INFO:tasks.workunit.client.0.smithi103.stdout:1/909: unlink db/d4a/d78/d92/dea/l139 0 2024-03-22T02:28:48.255 INFO:tasks.workunit.client.0.smithi103.stdout:4/787: rmdir d3/d38 39 2024-03-22T02:28:48.256 INFO:tasks.workunit.client.1.smithi122.stdout:0/169: readlink d11/d16/l19 0 2024-03-22T02:28:48.256 INFO:tasks.workunit.client.1.smithi122.stdout:4/224: dwrite d0/d12/d15/f29 [0,4194304] 0 2024-03-22T02:28:48.257 INFO:tasks.workunit.client.0.smithi103.stdout:3/857: truncate de/f5f 1664525 0 2024-03-22T02:28:48.260 INFO:tasks.workunit.client.1.smithi122.stdout:8/258: fdatasync d1/f2d 0 2024-03-22T02:28:48.262 INFO:tasks.workunit.client.1.smithi122.stdout:9/261: dwrite f7 [0,4194304] 0 2024-03-22T02:28:48.263 INFO:tasks.workunit.client.0.smithi103.stdout:1/910: creat db/d4a/d78/d92/dea/d134/d128/f13e x:0 0 0 2024-03-22T02:28:48.264 INFO:tasks.workunit.client.1.smithi122.stdout:9/262: dread - db/df/f4a zero size 2024-03-22T02:28:48.264 INFO:tasks.workunit.client.1.smithi122.stdout:7/219: symlink d0/d4/d20/d21/l3e 0 2024-03-22T02:28:48.265 INFO:tasks.workunit.client.0.smithi103.stdout:0/751: getdents dd/d16/d4b/d96/d29/d48/d5c/d9b 0 2024-03-22T02:28:48.270 INFO:tasks.workunit.client.1.smithi122.stdout:4/225: write d0/d25/f27 [4387010,106239] 0 2024-03-22T02:28:48.270 INFO:tasks.workunit.client.0.smithi103.stdout:8/884: write d4/d2a/d8b/fbc [86086,27900] 0 2024-03-22T02:28:48.271 INFO:tasks.workunit.client.1.smithi122.stdout:4/226: stat d0/d12/d15/d2d 0 2024-03-22T02:28:48.276 INFO:tasks.workunit.client.1.smithi122.stdout:2/296: truncate fd 1339665 0 2024-03-22T02:28:48.277 INFO:tasks.workunit.client.1.smithi122.stdout:0/170: creat d11/d14/d1b/f3c x:0 0 0 2024-03-22T02:28:48.278 INFO:tasks.workunit.client.1.smithi122.stdout:8/259: creat d1/d4d/f58 x:0 0 0 2024-03-22T02:28:48.280 INFO:tasks.workunit.client.1.smithi122.stdout:8/260: write d1/d40/f4a [223268,38265] 0 2024-03-22T02:28:48.282 INFO:tasks.workunit.client.1.smithi122.stdout:7/220: write d0/f8 [3178047,107125] 0 2024-03-22T02:28:48.284 INFO:tasks.workunit.client.1.smithi122.stdout:7/221: write d0/d4/d20/f27 [2063363,60659] 0 2024-03-22T02:28:48.286 INFO:tasks.workunit.client.0.smithi103.stdout:1/911: write db/d4a/d78/d92/da8/ddd/d18/d84/fb6 [747659,4584] 0 2024-03-22T02:28:48.286 INFO:tasks.workunit.client.1.smithi122.stdout:2/297: rmdir df 39 2024-03-22T02:28:48.288 INFO:tasks.workunit.client.1.smithi122.stdout:0/171: unlink d11/d16/d37/l39 0 2024-03-22T02:28:48.289 INFO:tasks.workunit.client.1.smithi122.stdout:9/263: mkdir db/df/d3e/d59 0 2024-03-22T02:28:48.291 INFO:tasks.workunit.client.1.smithi122.stdout:8/261: creat d1/d9/d50/f59 x:0 0 0 2024-03-22T02:28:48.292 INFO:tasks.workunit.client.0.smithi103.stdout:0/752: dwrite dd/d16/d4b/d96/f68 [0,4194304] 0 2024-03-22T02:28:48.293 INFO:tasks.workunit.client.0.smithi103.stdout:8/885: dwrite d4/d26/d27/d7e/d67/f88 [0,4194304] 0 2024-03-22T02:28:48.296 INFO:tasks.workunit.client.1.smithi122.stdout:0/172: creat d11/f3d x:0 0 0 2024-03-22T02:28:48.297 INFO:tasks.workunit.client.0.smithi103.stdout:0/753: truncate dd/d16/d4b/d96/fe1 298603 0 2024-03-22T02:28:48.298 INFO:tasks.workunit.client.1.smithi122.stdout:9/264: mknod db/d1b/c5a 0 2024-03-22T02:28:48.299 INFO:tasks.workunit.client.0.smithi103.stdout:1/912: write db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/fc1 [1436907,54798] 0 2024-03-22T02:28:48.299 INFO:tasks.workunit.client.0.smithi103.stdout:8/886: dread d4/d19/f5c [0,4194304] 0 2024-03-22T02:28:48.301 INFO:tasks.workunit.client.0.smithi103.stdout:0/754: truncate dd/f6b 3300181 0 2024-03-22T02:28:48.301 INFO:tasks.workunit.client.1.smithi122.stdout:7/222: symlink d0/d4/d20/d21/d39/l3f 0 2024-03-22T02:28:48.301 INFO:tasks.workunit.client.1.smithi122.stdout:7/223: fdatasync d0/d4/fb 0 2024-03-22T02:28:48.303 INFO:tasks.workunit.client.0.smithi103.stdout:3/858: dread de/f5f [0,4194304] 0 2024-03-22T02:28:48.303 INFO:tasks.workunit.client.0.smithi103.stdout:8/887: fdatasync d4/d26/d27/f4d 0 2024-03-22T02:28:48.303 INFO:tasks.workunit.client.1.smithi122.stdout:2/298: chown f4 245674403 1 2024-03-22T02:28:48.305 INFO:tasks.workunit.client.1.smithi122.stdout:7/224: dread d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:48.305 INFO:tasks.workunit.client.1.smithi122.stdout:1/237: truncate dc/d2b/f33 1872412 0 2024-03-22T02:28:48.305 INFO:tasks.workunit.client.1.smithi122.stdout:8/262: mkdir d1/d40/d48/d5a 0 2024-03-22T02:28:48.306 INFO:tasks.workunit.client.1.smithi122.stdout:4/227: dread d0/f9 [0,4194304] 0 2024-03-22T02:28:48.308 INFO:tasks.workunit.client.0.smithi103.stdout:1/913: mknod db/d4a/d78/d92/da8/ddd/d29/c13f 0 2024-03-22T02:28:48.308 INFO:tasks.workunit.client.1.smithi122.stdout:9/265: mknod db/d23/c5b 0 2024-03-22T02:28:48.309 INFO:tasks.workunit.client.1.smithi122.stdout:9/266: readlink db/l4d 0 2024-03-22T02:28:48.309 INFO:tasks.workunit.client.0.smithi103.stdout:3/859: creat de/d12/d5d/da9/f125 x:0 0 0 2024-03-22T02:28:48.311 INFO:tasks.workunit.client.0.smithi103.stdout:0/755: dwrite fc [0,4194304] 0 2024-03-22T02:28:48.312 INFO:tasks.workunit.client.1.smithi122.stdout:2/299: fsync df/d1f/d3a/d4f/f70 0 2024-03-22T02:28:48.313 INFO:tasks.workunit.client.0.smithi103.stdout:3/860: readlink de/d6b/d80/d73/d110/lb5 0 2024-03-22T02:28:48.314 INFO:tasks.workunit.client.1.smithi122.stdout:2/300: write df/f10 [3271062,117657] 0 2024-03-22T02:28:48.314 INFO:tasks.workunit.client.1.smithi122.stdout:8/263: dwrite d1/d9/d50/f59 [0,4194304] 0 2024-03-22T02:28:48.315 INFO:tasks.workunit.client.1.smithi122.stdout:7/225: rename d0/d25/l3a to d0/d4/d20/l40 0 2024-03-22T02:28:48.315 INFO:tasks.workunit.client.0.smithi103.stdout:1/914: fsync db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d111/f126 0 2024-03-22T02:28:48.315 INFO:tasks.workunit.client.0.smithi103.stdout:0/756: truncate dd/d16/d4b/d96/f5d 1527828 0 2024-03-22T02:28:48.317 INFO:tasks.workunit.client.1.smithi122.stdout:7/226: read d0/f8 [2330776,85911] 0 2024-03-22T02:28:48.323 INFO:tasks.workunit.client.1.smithi122.stdout:9/267: unlink db/d23/d1f/d21/d25/f49 0 2024-03-22T02:28:48.323 INFO:tasks.workunit.client.0.smithi103.stdout:1/915: dwrite db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f100 [0,4194304] 0 2024-03-22T02:28:48.325 INFO:tasks.workunit.client.1.smithi122.stdout:9/268: chown db/df/c14 3760392 1 2024-03-22T02:28:48.326 INFO:tasks.workunit.client.1.smithi122.stdout:4/228: dwrite d0/d25/f1d [0,4194304] 0 2024-03-22T02:28:48.330 INFO:tasks.workunit.client.1.smithi122.stdout:9/269: write db/d23/f11 [1637650,99483] 0 2024-03-22T02:28:48.331 INFO:tasks.workunit.client.0.smithi103.stdout:1/916: creat db/d4a/d78/d92/da8/ddd/d18/d84/f140 x:0 0 0 2024-03-22T02:28:48.331 INFO:tasks.workunit.client.0.smithi103.stdout:1/917: chown db/d4a/d78/d92/da8/ddd/d18/f37 2645 1 2024-03-22T02:28:48.332 INFO:tasks.workunit.client.1.smithi122.stdout:2/301: dread f6 [0,4194304] 0 2024-03-22T02:28:48.332 INFO:tasks.workunit.client.1.smithi122.stdout:4/229: chown d0/d25/d7/de/l45 80423 1 2024-03-22T02:28:48.333 INFO:tasks.workunit.client.0.smithi103.stdout:3/861: truncate de/f22 277640 0 2024-03-22T02:28:48.335 INFO:tasks.workunit.client.0.smithi103.stdout:3/862: rmdir de/d1a 39 2024-03-22T02:28:48.336 INFO:tasks.workunit.client.1.smithi122.stdout:6/282: sync 2024-03-22T02:28:48.336 INFO:tasks.workunit.client.1.smithi122.stdout:8/264: link d1/d3f/d45/f4b d1/d40/f5b 0 2024-03-22T02:28:48.340 INFO:tasks.workunit.client.1.smithi122.stdout:7/227: rename d0/d4/d22/l24 to d0/d4/d20/l41 0 2024-03-22T02:28:48.340 INFO:tasks.workunit.client.0.smithi103.stdout:3/863: rename de/d6b/d80/d73/ff4 to de/d6b/f126 0 2024-03-22T02:28:48.342 INFO:tasks.workunit.client.0.smithi103.stdout:3/864: stat de/d1a/d10f 0 2024-03-22T02:28:48.342 INFO:tasks.workunit.client.1.smithi122.stdout:1/238: mkdir dc/d3b/d46/d47 0 2024-03-22T02:28:48.343 INFO:tasks.workunit.client.1.smithi122.stdout:2/302: creat df/d1f/d2a/d51/d67/f71 x:0 0 0 2024-03-22T02:28:48.346 INFO:tasks.workunit.client.1.smithi122.stdout:8/265: creat d1/d8/f5c x:0 0 0 2024-03-22T02:28:48.347 INFO:tasks.workunit.client.1.smithi122.stdout:8/266: readlink d1/l17 0 2024-03-22T02:28:48.347 INFO:tasks.workunit.client.1.smithi122.stdout:4/230: creat d0/d41/f48 x:0 0 0 2024-03-22T02:28:48.347 INFO:tasks.workunit.client.0.smithi103.stdout:0/757: dwrite dd/d16/d4b/d96/d64/dab/dc6/f70 [0,4194304] 0 2024-03-22T02:28:48.350 INFO:tasks.workunit.client.0.smithi103.stdout:8/888: dread d4/d26/d27/d7e/d39/d49/d5f/f60 [0,4194304] 0 2024-03-22T02:28:48.351 INFO:tasks.workunit.client.0.smithi103.stdout:0/758: mkdir dd/d16/d4b/d96/d35/dc4/df9 0 2024-03-22T02:28:48.352 INFO:tasks.workunit.client.0.smithi103.stdout:8/889: truncate d4/d26/d27/d7e/d67/f118 802821 0 2024-03-22T02:28:48.353 INFO:tasks.workunit.client.0.smithi103.stdout:8/890: fsync d4/f15 0 2024-03-22T02:28:48.353 INFO:tasks.workunit.client.1.smithi122.stdout:2/303: mkdir df/d1f/d2a/d72 0 2024-03-22T02:28:48.354 INFO:tasks.workunit.client.0.smithi103.stdout:0/759: rmdir dd/d16/d4b/d96/d64/d6a/dd5/deb 39 2024-03-22T02:28:48.354 INFO:tasks.workunit.client.1.smithi122.stdout:1/239: dread dc/f21 [0,4194304] 0 2024-03-22T02:28:48.356 INFO:tasks.workunit.client.0.smithi103.stdout:8/891: dread - d4/d26/d27/d7e/fa9 zero size 2024-03-22T02:28:48.359 INFO:tasks.workunit.client.1.smithi122.stdout:6/283: dwrite d7/de/f3e [0,4194304] 0 2024-03-22T02:28:48.360 INFO:tasks.workunit.client.1.smithi122.stdout:2/304: symlink df/d1f/d3d/d52/l73 0 2024-03-22T02:28:48.361 INFO:tasks.workunit.client.0.smithi103.stdout:0/760: truncate dd/d16/d4b/d96/d29/d48/d5c/fa8 320383 0 2024-03-22T02:28:48.361 INFO:tasks.workunit.client.0.smithi103.stdout:8/892: creat d4/d26/d27/d7e/d39/d49/d79/f123 x:0 0 0 2024-03-22T02:28:48.365 INFO:tasks.workunit.client.1.smithi122.stdout:1/240: chown dc/f42 6 1 2024-03-22T02:28:48.365 INFO:tasks.workunit.client.1.smithi122.stdout:4/231: dwrite d0/d41/f48 [0,4194304] 0 2024-03-22T02:28:48.366 INFO:tasks.workunit.client.1.smithi122.stdout:8/267: creat d1/f5d x:0 0 0 2024-03-22T02:28:48.366 INFO:tasks.workunit.client.0.smithi103.stdout:3/865: dwrite de/d12/d19/df2/f53 [0,4194304] 0 2024-03-22T02:28:48.367 INFO:tasks.workunit.client.1.smithi122.stdout:7/228: dwrite d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:48.369 INFO:tasks.workunit.client.1.smithi122.stdout:2/305: truncate df/d1f/d33/d34/f5a 847805 0 2024-03-22T02:28:48.369 INFO:tasks.workunit.client.0.smithi103.stdout:3/866: dread - de/d12/d30/d70/de5/fe9 zero size 2024-03-22T02:28:48.372 INFO:tasks.workunit.client.1.smithi122.stdout:7/229: write d0/d4/d20/d21/f2b [595978,9664] 0 2024-03-22T02:28:48.375 INFO:tasks.workunit.client.0.smithi103.stdout:3/867: creat de/d1a/d25/f127 x:0 0 0 2024-03-22T02:28:48.377 INFO:tasks.workunit.client.1.smithi122.stdout:1/241: creat dc/d3b/d46/d47/f48 x:0 0 0 2024-03-22T02:28:48.377 INFO:tasks.workunit.client.0.smithi103.stdout:3/868: creat de/d6b/d80/d73/f128 x:0 0 0 2024-03-22T02:28:48.380 INFO:tasks.workunit.client.1.smithi122.stdout:2/306: mknod df/c74 0 2024-03-22T02:28:48.380 INFO:tasks.workunit.client.1.smithi122.stdout:2/307: write df/f36 [392580,52567] 0 2024-03-22T02:28:48.381 INFO:tasks.workunit.client.1.smithi122.stdout:4/232: creat d0/d12/d15/d2d/f49 x:0 0 0 2024-03-22T02:28:48.382 INFO:tasks.workunit.client.1.smithi122.stdout:9/270: truncate db/d23/d1f/d21/d25/f55 1852351 0 2024-03-22T02:28:48.383 INFO:tasks.workunit.client.1.smithi122.stdout:2/308: write df/d1f/d33/d34/f5f [1663756,89487] 0 2024-03-22T02:28:48.383 INFO:tasks.workunit.client.0.smithi103.stdout:3/869: creat de/d12/d8a/d93/d103/f129 x:0 0 0 2024-03-22T02:28:48.384 INFO:tasks.workunit.client.1.smithi122.stdout:7/230: symlink d0/d4/d20/l42 0 2024-03-22T02:28:48.389 INFO:tasks.workunit.client.0.smithi103.stdout:3/870: unlink de/d1a/d10f/l40 0 2024-03-22T02:28:48.391 INFO:tasks.workunit.client.0.smithi103.stdout:0/761: dwrite dd/d16/d4b/d96/d64/f71 [0,4194304] 0 2024-03-22T02:28:48.392 INFO:tasks.workunit.client.1.smithi122.stdout:2/309: dread fd [0,4194304] 0 2024-03-22T02:28:48.393 INFO:tasks.workunit.client.1.smithi122.stdout:7/231: creat d0/d4/d22/f43 x:0 0 0 2024-03-22T02:28:48.393 INFO:tasks.workunit.client.1.smithi122.stdout:7/232: stat d0/c2c 0 2024-03-22T02:28:48.394 INFO:tasks.workunit.client.1.smithi122.stdout:7/233: write d0/d4/fb [4739953,91849] 0 2024-03-22T02:28:48.395 INFO:tasks.workunit.client.1.smithi122.stdout:8/268: rmdir d1/d40/d48/d5a 0 2024-03-22T02:28:48.397 INFO:tasks.workunit.client.0.smithi103.stdout:3/871: read de/d12/d30/f3d [707124,39679] 0 2024-03-22T02:28:48.397 INFO:tasks.workunit.client.0.smithi103.stdout:3/872: dread - de/d12/d19/fcb zero size 2024-03-22T02:28:48.397 INFO:tasks.workunit.client.1.smithi122.stdout:8/269: dread - d1/d40/f5b zero size 2024-03-22T02:28:48.397 INFO:tasks.workunit.client.0.smithi103.stdout:0/762: read dd/d34/d53/d8c/fbc [1526218,47475] 0 2024-03-22T02:28:48.398 INFO:tasks.workunit.client.1.smithi122.stdout:7/234: read d0/f16 [72622,1578] 0 2024-03-22T02:28:48.400 INFO:tasks.workunit.client.0.smithi103.stdout:8/893: dwrite d4/d2a/d38/dfe/f103 [0,4194304] 0 2024-03-22T02:28:48.401 INFO:tasks.workunit.client.0.smithi103.stdout:1/918: dread db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f82 [0,4194304] 0 2024-03-22T02:28:48.402 INFO:tasks.workunit.client.1.smithi122.stdout:6/284: getdents d7/de/dd/d20 0 2024-03-22T02:28:48.403 INFO:tasks.workunit.client.1.smithi122.stdout:2/310: dwrite df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:48.405 INFO:tasks.workunit.client.0.smithi103.stdout:9/756: sync 2024-03-22T02:28:48.405 INFO:tasks.workunit.client.0.smithi103.stdout:5/774: sync 2024-03-22T02:28:48.406 INFO:tasks.workunit.client.0.smithi103.stdout:1/919: write db/d4a/d78/d92/da8/ddd/d28/f103 [1664319,95961] 0 2024-03-22T02:28:48.407 INFO:tasks.workunit.client.1.smithi122.stdout:4/233: rename d0/d12/d15/d2d/f49 to d0/d25/d7/f4a 0 2024-03-22T02:28:48.407 INFO:tasks.workunit.client.1.smithi122.stdout:6/285: write d7/de/dd/df/f4b [905923,13353] 0 2024-03-22T02:28:48.408 INFO:tasks.workunit.client.1.smithi122.stdout:8/270: mkdir d1/d8/d5e 0 2024-03-22T02:28:48.409 INFO:tasks.workunit.client.0.smithi103.stdout:5/775: creat d0/d3e/f10d x:0 0 0 2024-03-22T02:28:48.410 INFO:tasks.workunit.client.0.smithi103.stdout:5/776: write d0/d14/d23/f24 [1373103,40668] 0 2024-03-22T02:28:48.411 INFO:tasks.workunit.client.1.smithi122.stdout:8/271: write d1/d8/d2f/f38 [1354134,73360] 0 2024-03-22T02:28:48.412 INFO:tasks.workunit.client.0.smithi103.stdout:9/757: read d2/da/d18/d1e/d34/d3d/d61/f9a [273857,103828] 0 2024-03-22T02:28:48.412 INFO:tasks.workunit.client.1.smithi122.stdout:2/311: dread df/d1f/d2a/d51/d67/f6e [0,4194304] 0 2024-03-22T02:28:48.413 INFO:tasks.workunit.client.0.smithi103.stdout:1/920: rmdir db/d4a/d78/d92/dea/d134/d128 39 2024-03-22T02:28:48.414 INFO:tasks.workunit.client.0.smithi103.stdout:1/921: stat db/d4a/d78/d92/da8/ddd/l85 0 2024-03-22T02:28:48.415 INFO:tasks.workunit.client.1.smithi122.stdout:7/235: mknod d0/d4/d22/c44 0 2024-03-22T02:28:48.419 INFO:tasks.workunit.client.1.smithi122.stdout:4/234: symlink d0/d12/l4b 0 2024-03-22T02:28:48.421 INFO:tasks.workunit.client.1.smithi122.stdout:6/286: rename d7/de/d27/d52/f61 to d7/de/dd/df/f62 0 2024-03-22T02:28:48.422 INFO:tasks.workunit.client.0.smithi103.stdout:9/758: symlink d2/d5/d7b/d7d/dd4/dd8/lde 0 2024-03-22T02:28:48.423 INFO:tasks.workunit.client.0.smithi103.stdout:9/759: chown d2/d5/d4c/c86 146417 1 2024-03-22T02:28:48.423 INFO:tasks.workunit.client.0.smithi103.stdout:3/873: dread de/d12/d19/df2/f2f [0,4194304] 0 2024-03-22T02:28:48.425 INFO:tasks.workunit.client.0.smithi103.stdout:5/777: getdents d0/da2 0 2024-03-22T02:28:48.426 INFO:tasks.workunit.client.0.smithi103.stdout:5/778: chown d0/f9a 655811246 1 2024-03-22T02:28:48.426 INFO:tasks.workunit.client.0.smithi103.stdout:9/760: mknod d2/da/d18/dc4/dce/cdf 0 2024-03-22T02:28:48.428 INFO:tasks.workunit.client.0.smithi103.stdout:3/874: creat de/d6b/d80/d73/d110/f12a x:0 0 0 2024-03-22T02:28:48.428 INFO:tasks.workunit.client.1.smithi122.stdout:1/242: dwrite dc/f21 [0,4194304] 0 2024-03-22T02:28:48.429 INFO:tasks.workunit.client.0.smithi103.stdout:5/779: creat d0/d14/d23/f10e x:0 0 0 2024-03-22T02:28:48.429 INFO:tasks.workunit.client.1.smithi122.stdout:4/235: link d0/d12/l46 d0/d12/l4c 0 2024-03-22T02:28:48.430 INFO:tasks.workunit.client.1.smithi122.stdout:6/287: dread d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:48.431 INFO:tasks.workunit.client.1.smithi122.stdout:4/236: fsync d0/d25/d7/f16 0 2024-03-22T02:28:48.431 INFO:tasks.workunit.client.0.smithi103.stdout:3/875: readlink de/d1a/lf1 0 2024-03-22T02:28:48.431 INFO:tasks.workunit.client.1.smithi122.stdout:6/288: chown d7/de/f37 1 1 2024-03-22T02:28:48.434 INFO:tasks.workunit.client.0.smithi103.stdout:0/763: write dd/d16/d4b/fd6 [595349,28915] 0 2024-03-22T02:28:48.439 INFO:tasks.workunit.client.1.smithi122.stdout:1/243: rename dc/fe to dc/f49 0 2024-03-22T02:28:48.440 INFO:tasks.workunit.client.0.smithi103.stdout:1/922: write db/d4a/d78/d92/da8/ddd/d18/f69 [75564,90464] 0 2024-03-22T02:28:48.441 INFO:tasks.workunit.client.0.smithi103.stdout:9/761: dwrite d2/d10/d43/d7e/d96/d9d/f7c [0,4194304] 0 2024-03-22T02:28:48.442 INFO:tasks.workunit.client.1.smithi122.stdout:8/272: write d1/d3f/d45/f4b [605658,47967] 0 2024-03-22T02:28:48.443 INFO:tasks.workunit.client.1.smithi122.stdout:9/271: dwrite db/f13 [0,4194304] 0 2024-03-22T02:28:48.444 INFO:tasks.workunit.client.0.smithi103.stdout:5/780: creat d0/d14/d23/d5c/d77/f10f x:0 0 0 2024-03-22T02:28:48.445 INFO:tasks.workunit.client.1.smithi122.stdout:8/273: write d1/d40/f47 [4327,108321] 0 2024-03-22T02:28:48.447 INFO:tasks.workunit.client.0.smithi103.stdout:1/923: truncate db/d4a/d78/d92/dea/d134/fd7 980810 0 2024-03-22T02:28:48.453 INFO:tasks.workunit.client.1.smithi122.stdout:9/272: dread db/f13 [0,4194304] 0 2024-03-22T02:28:48.453 INFO:tasks.workunit.client.0.smithi103.stdout:3/876: getdents de/d6b/d80/d4f 0 2024-03-22T02:28:48.454 INFO:tasks.workunit.client.1.smithi122.stdout:1/244: creat dc/f4a x:0 0 0 2024-03-22T02:28:48.455 INFO:tasks.workunit.client.1.smithi122.stdout:9/273: fdatasync db/fc 0 2024-03-22T02:28:48.455 INFO:tasks.workunit.client.0.smithi103.stdout:1/924: truncate db/d4a/d78/d92/da8/ddd/d29/d41/d68/fa3 1222569 0 2024-03-22T02:28:48.458 INFO:tasks.workunit.client.0.smithi103.stdout:3/877: creat de/d12/d8a/dda/f12b x:0 0 0 2024-03-22T02:28:48.461 INFO:tasks.workunit.client.0.smithi103.stdout:8/894: dread d4/d26/d27/f2c [0,4194304] 0 2024-03-22T02:28:48.462 INFO:tasks.workunit.client.1.smithi122.stdout:6/289: dread f3 [0,4194304] 0 2024-03-22T02:28:48.462 INFO:tasks.workunit.client.1.smithi122.stdout:1/245: dread - dc/f40 zero size 2024-03-22T02:28:48.464 INFO:tasks.workunit.client.1.smithi122.stdout:4/237: truncate d0/d25/d7/f23 3762625 0 2024-03-22T02:28:48.464 INFO:tasks.workunit.client.0.smithi103.stdout:8/895: write d4/d26/d27/d7e/d39/d49/d79/d11e/fff [821091,54635] 0 2024-03-22T02:28:48.466 INFO:tasks.workunit.client.0.smithi103.stdout:3/878: creat de/d1a/f12c x:0 0 0 2024-03-22T02:28:48.466 INFO:tasks.workunit.client.1.smithi122.stdout:9/274: dwrite f8 [0,4194304] 0 2024-03-22T02:28:48.467 INFO:tasks.workunit.client.1.smithi122.stdout:6/290: truncate f5 3211426 0 2024-03-22T02:28:48.467 INFO:tasks.workunit.client.1.smithi122.stdout:8/274: dwrite d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:48.468 INFO:tasks.workunit.client.0.smithi103.stdout:1/925: dwrite db/d4a/d78/d92/da8/ddd/d18/fa4 [0,4194304] 0 2024-03-22T02:28:48.469 INFO:tasks.workunit.client.1.smithi122.stdout:9/275: read - db/d23/d1e/d30/f4f zero size 2024-03-22T02:28:48.469 INFO:tasks.workunit.client.1.smithi122.stdout:4/238: truncate d0/d25/d7/f4a 165584 0 2024-03-22T02:28:48.470 INFO:tasks.workunit.client.0.smithi103.stdout:0/764: dread dd/d16/d4b/d96/d64/dab/dc6/f8e [0,4194304] 0 2024-03-22T02:28:48.471 INFO:tasks.workunit.client.1.smithi122.stdout:9/276: write db/d23/d1e/f34 [68158,44422] 0 2024-03-22T02:28:48.473 INFO:tasks.workunit.client.0.smithi103.stdout:3/879: mknod de/d12/d4d/c12d 0 2024-03-22T02:28:48.473 INFO:tasks.workunit.client.1.smithi122.stdout:8/275: rmdir d1/d22 39 2024-03-22T02:28:48.477 INFO:tasks.workunit.client.1.smithi122.stdout:6/291: write d7/de/dd/d20/f31 [893370,62847] 0 2024-03-22T02:28:48.479 INFO:tasks.workunit.client.1.smithi122.stdout:1/246: rename dc/f21 to dc/f4b 0 2024-03-22T02:28:48.479 INFO:tasks.workunit.client.1.smithi122.stdout:4/239: fdatasync d0/d25/d7/f4a 0 2024-03-22T02:28:48.480 INFO:tasks.workunit.client.0.smithi103.stdout:0/765: rename dd/d16/d4b/cdf to dd/d34/d53/cfa 0 2024-03-22T02:28:48.486 INFO:tasks.workunit.client.1.smithi122.stdout:9/277: dwrite db/d23/d1f/d21/f42 [0,4194304] 0 2024-03-22T02:28:48.486 INFO:tasks.workunit.client.1.smithi122.stdout:1/247: write dc/f38 [5109313,423] 0 2024-03-22T02:28:48.487 INFO:tasks.workunit.client.0.smithi103.stdout:8/896: dwrite d4/d26/d27/d7e/d39/d49/d51/f109 [0,4194304] 0 2024-03-22T02:28:48.489 INFO:tasks.workunit.client.1.smithi122.stdout:4/240: creat d0/d25/f4d x:0 0 0 2024-03-22T02:28:48.493 INFO:tasks.workunit.client.1.smithi122.stdout:1/248: mkdir dc/d4c 0 2024-03-22T02:28:48.493 INFO:tasks.workunit.client.0.smithi103.stdout:5/781: dwrite d0/d14/d23/d5c/d77/df2/dd9/fe2 [0,4194304] 0 2024-03-22T02:28:48.495 INFO:tasks.workunit.client.1.smithi122.stdout:8/276: dread d1/d8/d2f/f3e [0,4194304] 0 2024-03-22T02:28:48.495 INFO:tasks.workunit.client.1.smithi122.stdout:6/292: dwrite d7/de/dd/d39/f4f [0,4194304] 0 2024-03-22T02:28:48.495 INFO:tasks.workunit.client.1.smithi122.stdout:9/278: creat db/df/d3e/d59/f5c x:0 0 0 2024-03-22T02:28:48.498 INFO:tasks.workunit.client.1.smithi122.stdout:8/277: chown d1/d9/f28 5037492 1 2024-03-22T02:28:48.498 INFO:tasks.workunit.client.1.smithi122.stdout:1/249: mkdir dc/d3b/d46/d4d 0 2024-03-22T02:28:48.499 INFO:tasks.workunit.client.0.smithi103.stdout:8/897: symlink d4/d2a/d38/dfe/l124 0 2024-03-22T02:28:48.500 INFO:tasks.workunit.client.0.smithi103.stdout:1/926: rename db/c109 to db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/c141 0 2024-03-22T02:28:48.503 INFO:tasks.workunit.client.1.smithi122.stdout:6/293: rmdir d7/de 39 2024-03-22T02:28:48.507 INFO:tasks.workunit.client.0.smithi103.stdout:8/898: mknod d4/d2a/c125 0 2024-03-22T02:28:48.509 INFO:tasks.workunit.client.1.smithi122.stdout:4/241: rename d0/d25/d7 to d0/d12/d15/d42/d4e 0 2024-03-22T02:28:48.510 INFO:tasks.workunit.client.0.smithi103.stdout:3/880: dread fa [0,4194304] 0 2024-03-22T02:28:48.510 INFO:tasks.workunit.client.0.smithi103.stdout:8/899: write d4/d26/d27/d7e/d39/d49/f119 [1039536,32543] 0 2024-03-22T02:28:48.512 INFO:tasks.workunit.client.0.smithi103.stdout:5/782: rename d0/d14/d23/d5c/df3/c108 to d0/d14/c110 0 2024-03-22T02:28:48.513 INFO:tasks.workunit.client.0.smithi103.stdout:3/881: dread - de/f96 zero size 2024-03-22T02:28:48.514 INFO:tasks.workunit.client.0.smithi103.stdout:8/900: creat d4/d26/d27/da3/f126 x:0 0 0 2024-03-22T02:28:48.515 INFO:tasks.workunit.client.0.smithi103.stdout:3/882: mknod de/d12/d30/d70/c12e 0 2024-03-22T02:28:48.516 INFO:tasks.workunit.client.1.smithi122.stdout:6/294: unlink d7/de/dd/d20/f2c 0 2024-03-22T02:28:48.516 INFO:tasks.workunit.client.0.smithi103.stdout:8/901: write d4/fd [803270,25668] 0 2024-03-22T02:28:48.517 INFO:tasks.workunit.client.1.smithi122.stdout:1/250: creat dc/f4e x:0 0 0 2024-03-22T02:28:48.518 INFO:tasks.workunit.client.0.smithi103.stdout:3/883: mknod de/d12/d19/df2/c12f 0 2024-03-22T02:28:48.519 INFO:tasks.workunit.client.0.smithi103.stdout:5/783: getdents d0/d14/d23 0 2024-03-22T02:28:48.520 INFO:tasks.workunit.client.1.smithi122.stdout:4/242: link d0/d41/f48 d0/d12/d15/d2d/f4f 0 2024-03-22T02:28:48.521 INFO:tasks.workunit.client.0.smithi103.stdout:8/902: mkdir d4/d26/d27/d7e/d39/d49/d51/daf/d127 0 2024-03-22T02:28:48.521 INFO:tasks.workunit.client.1.smithi122.stdout:8/278: getdents d1/d3f/d45 0 2024-03-22T02:28:48.523 INFO:tasks.workunit.client.1.smithi122.stdout:4/243: mknod d0/d12/d15/d42/c50 0 2024-03-22T02:28:48.523 INFO:tasks.workunit.client.0.smithi103.stdout:8/903: dread d4/d19/f5c [0,4194304] 0 2024-03-22T02:28:48.524 INFO:tasks.workunit.client.0.smithi103.stdout:5/784: rename d0/d14/d23/d6f/fd7 to d0/d14/d23/d2a/df7/f111 0 2024-03-22T02:28:48.524 INFO:tasks.workunit.client.1.smithi122.stdout:1/251: mknod dc/d2b/c4f 0 2024-03-22T02:28:48.525 INFO:tasks.workunit.client.0.smithi103.stdout:8/904: chown d4/d26/d27/d7e/d39/d49/d79/d11e/d35/fa0 3744 1 2024-03-22T02:28:48.526 INFO:tasks.workunit.client.1.smithi122.stdout:9/279: fsync db/d23/d1f/d21/f42 0 2024-03-22T02:28:48.530 INFO:tasks.workunit.client.0.smithi103.stdout:5/785: mknod d0/d14/d23/d2a/d33/db5/c112 0 2024-03-22T02:28:48.530 INFO:tasks.workunit.client.0.smithi103.stdout:0/766: write dd/d34/d53/df3/f93 [3671626,34543] 0 2024-03-22T02:28:48.530 INFO:tasks.workunit.client.0.smithi103.stdout:0/767: fsync dd/d16/d4b/d96/d64/fa3 0 2024-03-22T02:28:48.531 INFO:tasks.workunit.client.1.smithi122.stdout:6/295: getdents d7/de/dd/df 0 2024-03-22T02:28:48.531 INFO:tasks.workunit.client.1.smithi122.stdout:9/280: write db/d23/f3a [1817412,28437] 0 2024-03-22T02:28:48.531 INFO:tasks.workunit.client.1.smithi122.stdout:1/252: rename dc/f4b to dc/d3b/d46/d47/f50 0 2024-03-22T02:28:48.531 INFO:tasks.workunit.client.1.smithi122.stdout:9/281: fsync db/d23/d1f/d21/d25/d41/f54 0 2024-03-22T02:28:48.532 INFO:tasks.workunit.client.0.smithi103.stdout:3/884: dread de/d12/d19/f71 [0,4194304] 0 2024-03-22T02:28:48.532 INFO:tasks.workunit.client.1.smithi122.stdout:4/244: truncate d0/d12/f1c 322485 0 2024-03-22T02:28:48.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.532+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97cc05f5f0 con 0x7f97b406b8c0 2024-03-22T02:28:48.532 INFO:tasks.workunit.client.1.smithi122.stdout:8/279: link d1/l2a d1/d9/d50/l5f 0 2024-03-22T02:28:48.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.534+0000 7f97b37fe700 1 -- 172.21.15.103:0/1718616165 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f97cc05f5f0 con 0x7f97b406b8c0 2024-03-22T02:28:48.534 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:28:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:28:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:28:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:28:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:28:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "1/2 daemons upgraded", 2024-03-22T02:28:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "", 2024-03-22T02:28:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:28:48.535 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:28:48.537 INFO:tasks.workunit.client.1.smithi122.stdout:6/296: rename d7/de/dd/df/f1a to d7/de/dd/d23/f63 0 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 msgr2=0x7f97b406dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 0x7f97b406dd70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009b80 tx=0x7f97c000b040 comp rx=0 tx=0).stop 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 msgr2=0x7f97cc115140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f97bc004890 tx=0x7f97bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 shutdown_connections 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f97b406b8c0 0x7f97b406dd70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97cc0fd290 0x7f97cc113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 --2- 172.21.15.103:0/1718616165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97cc114440 0x7f97cc115140 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:48.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 >> 172.21.15.103:0/1718616165 conn(0x7f97cc0f6ba0 msgr2=0x7f97cc0f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:48.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 shutdown_connections 2024-03-22T02:28:48.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:28:48.537+0000 7f97d1258700 1 -- 172.21.15.103:0/1718616165 wait complete. 2024-03-22T02:28:48.539 INFO:tasks.workunit.client.0.smithi103.stdout:5/786: link d0/d14/d23/d5c/d77/fa8 d0/d14/d23/d5c/f113 0 2024-03-22T02:28:48.540 INFO:tasks.workunit.client.1.smithi122.stdout:8/280: dwrite d1/f3c [0,4194304] 0 2024-03-22T02:28:48.543 INFO:tasks.workunit.client.0.smithi103.stdout:1/927: mkdir db/d4a/d78/d92/da8/ddd/d18/dca/d142 0 2024-03-22T02:28:48.543 INFO:tasks.workunit.client.0.smithi103.stdout:3/885: dwrite de/d12/d8f/dfb/f104 [0,4194304] 0 2024-03-22T02:28:48.543 INFO:tasks.workunit.client.0.smithi103.stdout:0/768: symlink dd/d16/d4b/d96/d64/dab/lfb 0 2024-03-22T02:28:48.547 INFO:tasks.workunit.client.0.smithi103.stdout:1/928: chown db/d4a/d78/d92/da8/ddd/d18/c24 500808 1 2024-03-22T02:28:48.548 INFO:tasks.workunit.client.1.smithi122.stdout:9/282: mknod db/c5d 0 2024-03-22T02:28:48.550 INFO:tasks.workunit.client.0.smithi103.stdout:5/787: creat d0/d14/d23/d46/de7/f114 x:0 0 0 2024-03-22T02:28:48.551 INFO:tasks.workunit.client.1.smithi122.stdout:5/206: sync 2024-03-22T02:28:48.552 INFO:tasks.workunit.client.1.smithi122.stdout:1/253: rmdir dc/d2b 39 2024-03-22T02:28:48.552 INFO:tasks.workunit.client.1.smithi122.stdout:3/138: sync 2024-03-22T02:28:48.552 INFO:tasks.workunit.client.0.smithi103.stdout:8/905: dread d4/f6e [0,4194304] 0 2024-03-22T02:28:48.553 INFO:tasks.workunit.client.1.smithi122.stdout:5/207: stat d8/f33 0 2024-03-22T02:28:48.554 INFO:tasks.workunit.client.0.smithi103.stdout:8/906: write d4/d2a/d4f/fdd [4405306,118602] 0 2024-03-22T02:28:48.555 INFO:tasks.workunit.client.0.smithi103.stdout:1/929: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/f143 x:0 0 0 2024-03-22T02:28:48.555 INFO:tasks.workunit.client.1.smithi122.stdout:5/208: read d8/d1a/f34 [3408594,108842] 0 2024-03-22T02:28:48.555 INFO:tasks.workunit.client.1.smithi122.stdout:9/283: rmdir db/d23/d1f/d57 39 2024-03-22T02:28:48.556 INFO:tasks.workunit.client.0.smithi103.stdout:8/907: write d4/d26/d27/d7e/d67/f88 [1945978,59127] 0 2024-03-22T02:28:48.558 INFO:tasks.workunit.client.1.smithi122.stdout:8/281: dwrite d1/d22/f32 [0,4194304] 0 2024-03-22T02:28:48.559 INFO:tasks.workunit.client.0.smithi103.stdout:1/930: rmdir db/d4a/d78/d92/da8/ddd/d18/d2d 39 2024-03-22T02:28:48.560 INFO:tasks.workunit.client.1.smithi122.stdout:8/282: dread - d1/f53 zero size 2024-03-22T02:28:48.560 INFO:tasks.workunit.client.0.smithi103.stdout:8/908: mknod d4/d26/d27/d7e/d39/d49/d51/daf/d127/c128 0 2024-03-22T02:28:48.562 INFO:tasks.workunit.client.1.smithi122.stdout:3/139: fdatasync d2/d9/dc/fd 0 2024-03-22T02:28:48.563 INFO:tasks.workunit.client.0.smithi103.stdout:8/909: write d4/d26/d27/d7e/d39/d49/d51/daf/ff4 [448402,83382] 0 2024-03-22T02:28:48.567 INFO:tasks.workunit.client.0.smithi103.stdout:1/931: dread db/d4a/d78/d92/da8/ddd/d18/d84/fb6 [0,4194304] 0 2024-03-22T02:28:48.568 INFO:tasks.workunit.client.0.smithi103.stdout:8/910: creat d4/d26/d27/d7e/d67/f129 x:0 0 0 2024-03-22T02:28:48.572 INFO:tasks.workunit.client.1.smithi122.stdout:9/284: mknod db/d23/d1f/d21/c5e 0 2024-03-22T02:28:48.574 INFO:tasks.workunit.client.1.smithi122.stdout:1/254: creat dc/f51 x:0 0 0 2024-03-22T02:28:48.575 INFO:tasks.workunit.client.1.smithi122.stdout:5/209: rmdir d8/d1e/d31 0 2024-03-22T02:28:48.575 INFO:tasks.workunit.client.1.smithi122.stdout:6/297: dwrite d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:48.577 INFO:tasks.workunit.client.0.smithi103.stdout:1/932: dread db/d4a/d78/d92/da8/ddd/d18/dca/f114 [0,4194304] 0 2024-03-22T02:28:48.579 INFO:tasks.workunit.client.0.smithi103.stdout:1/933: fsync db/f25 0 2024-03-22T02:28:48.580 INFO:tasks.workunit.client.1.smithi122.stdout:5/210: creat d8/d1e/d22/f47 x:0 0 0 2024-03-22T02:28:48.580 INFO:tasks.workunit.client.1.smithi122.stdout:9/285: creat db/d23/d1f/d21/d29/f5f x:0 0 0 2024-03-22T02:28:48.580 INFO:tasks.workunit.client.1.smithi122.stdout:1/255: write dc/f11 [1153039,70559] 0 2024-03-22T02:28:48.582 INFO:tasks.workunit.client.1.smithi122.stdout:1/256: write dc/f51 [589862,76602] 0 2024-03-22T02:28:48.582 INFO:tasks.workunit.client.1.smithi122.stdout:6/298: creat d7/de/d27/f64 x:0 0 0 2024-03-22T02:28:48.583 INFO:tasks.workunit.client.1.smithi122.stdout:5/211: creat d8/d1a/f48 x:0 0 0 2024-03-22T02:28:48.584 INFO:tasks.workunit.client.0.smithi103.stdout:1/934: getdents db/d4a/d78/d92/da8/ddd/d18/d98 0 2024-03-22T02:28:48.585 INFO:tasks.workunit.client.0.smithi103.stdout:0/769: dread dd/d16/d4b/d96/d64/d6a/f74 [0,4194304] 0 2024-03-22T02:28:48.586 INFO:tasks.workunit.client.0.smithi103.stdout:0/770: write dd/f12 [4028109,23601] 0 2024-03-22T02:28:48.588 INFO:tasks.workunit.client.1.smithi122.stdout:9/286: mknod db/d23/d1f/d57/c60 0 2024-03-22T02:28:48.589 INFO:tasks.workunit.client.1.smithi122.stdout:1/257: creat dc/d4c/f52 x:0 0 0 2024-03-22T02:28:48.591 INFO:tasks.workunit.client.1.smithi122.stdout:9/287: dread - db/d23/d1e/d30/f4f zero size 2024-03-22T02:28:48.593 INFO:tasks.workunit.client.1.smithi122.stdout:5/212: creat d8/d1e/d22/d3a/d39/f49 x:0 0 0 2024-03-22T02:28:48.594 INFO:tasks.workunit.client.1.smithi122.stdout:4/245: dwrite d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:48.598 INFO:tasks.workunit.client.1.smithi122.stdout:5/213: chown d8/d3b/f45 3 1 2024-03-22T02:28:48.599 INFO:tasks.workunit.client.1.smithi122.stdout:8/283: dwrite d1/f2d [0,4194304] 0 2024-03-22T02:28:48.600 INFO:tasks.workunit.client.0.smithi103.stdout:1/935: dwrite db/d4a/d78/d92/fc5 [0,4194304] 0 2024-03-22T02:28:48.601 INFO:tasks.workunit.client.0.smithi103.stdout:1/936: chown db/d4a/d78/d92/da8/ddd/d18/f20 103936766 1 2024-03-22T02:28:48.602 INFO:tasks.workunit.client.1.smithi122.stdout:9/288: creat db/d23/d1f/d21/d25/d41/f61 x:0 0 0 2024-03-22T02:28:48.603 INFO:tasks.workunit.client.1.smithi122.stdout:1/258: mknod dc/c53 0 2024-03-22T02:28:48.605 INFO:tasks.workunit.client.0.smithi103.stdout:6/558: sync 2024-03-22T02:28:48.605 INFO:tasks.workunit.client.0.smithi103.stdout:7/743: sync 2024-03-22T02:28:48.605 INFO:tasks.workunit.client.0.smithi103.stdout:5/788: dread d0/fe [0,4194304] 0 2024-03-22T02:28:48.606 INFO:tasks.workunit.client.1.smithi122.stdout:4/246: write d0/f9 [3416925,70568] 0 2024-03-22T02:28:48.608 INFO:tasks.workunit.client.0.smithi103.stdout:7/744: write d1/da/d2f/fa0 [1004511,2648] 0 2024-03-22T02:28:48.609 INFO:tasks.workunit.client.1.smithi122.stdout:9/289: read fa [1203354,25963] 0 2024-03-22T02:28:48.610 INFO:tasks.workunit.client.0.smithi103.stdout:5/789: write d0/da2/dca/fde [992119,108573] 0 2024-03-22T02:28:48.612 INFO:tasks.workunit.client.1.smithi122.stdout:5/214: dwrite d8/d3b/f45 [0,4194304] 0 2024-03-22T02:28:48.614 INFO:tasks.workunit.client.1.smithi122.stdout:5/215: write d8/d1e/d22/f47 [418528,74096] 0 2024-03-22T02:28:48.614 INFO:tasks.workunit.client.1.smithi122.stdout:8/284: dread d1/d9/d50/f59 [0,4194304] 0 2024-03-22T02:28:48.615 INFO:tasks.workunit.client.0.smithi103.stdout:1/937: dwrite db/f45 [0,4194304] 0 2024-03-22T02:28:48.616 INFO:tasks.workunit.client.0.smithi103.stdout:7/745: fdatasync d1/da/f74 0 2024-03-22T02:28:48.617 INFO:tasks.workunit.client.1.smithi122.stdout:1/259: symlink dc/d2b/l54 0 2024-03-22T02:28:48.618 INFO:tasks.workunit.client.0.smithi103.stdout:5/790: rename d0/d3e/l6b to d0/d3e/l115 0 2024-03-22T02:28:48.618 INFO:tasks.workunit.client.1.smithi122.stdout:6/299: write d7/de/dd/d23/f47 [1136402,12868] 0 2024-03-22T02:28:48.620 INFO:tasks.workunit.client.1.smithi122.stdout:5/216: mknod d8/d1e/d22/c4a 0 2024-03-22T02:28:48.620 INFO:tasks.workunit.client.0.smithi103.stdout:6/559: rename d0/d3/d22/c32 to d0/d3/d22/d58/d6f/d75/db0/cb5 0 2024-03-22T02:28:48.624 INFO:tasks.workunit.client.0.smithi103.stdout:1/938: unlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/le3 0 2024-03-22T02:28:48.625 INFO:tasks.workunit.client.1.smithi122.stdout:9/290: dwrite db/d23/d1f/d21/f56 [0,4194304] 0 2024-03-22T02:28:48.625 INFO:tasks.workunit.client.0.smithi103.stdout:1/939: chown db/d4a/d78/d92/da8/ddd/d28 393655843 1 2024-03-22T02:28:48.627 INFO:tasks.workunit.client.1.smithi122.stdout:1/260: fdatasync dc/f37 0 2024-03-22T02:28:48.627 INFO:tasks.workunit.client.0.smithi103.stdout:6/560: symlink d0/d3/d22/d58/d6f/d75/lb6 0 2024-03-22T02:28:48.628 INFO:tasks.workunit.client.0.smithi103.stdout:5/791: mkdir d0/d14/d116 0 2024-03-22T02:28:48.629 INFO:tasks.workunit.client.0.smithi103.stdout:6/561: chown d0/d14/f4c 459358257 1 2024-03-22T02:28:48.630 INFO:tasks.workunit.client.1.smithi122.stdout:5/217: creat d8/d1e/d22/d3a/d46/f4b x:0 0 0 2024-03-22T02:28:48.631 INFO:tasks.workunit.client.0.smithi103.stdout:7/746: rmdir d1/d10/d11/d19/d29/d33/d4b/db3 0 2024-03-22T02:28:48.632 INFO:tasks.workunit.client.0.smithi103.stdout:1/940: creat db/d4a/d78/d92/da8/ddd/d28/db2/dc8/d113/f144 x:0 0 0 2024-03-22T02:28:48.632 INFO:tasks.workunit.client.1.smithi122.stdout:9/291: unlink db/df/d3e/d59/f5c 0 2024-03-22T02:28:48.633 INFO:tasks.workunit.client.0.smithi103.stdout:5/792: creat d0/d14/d23/d36/de1/f117 x:0 0 0 2024-03-22T02:28:48.634 INFO:tasks.workunit.client.1.smithi122.stdout:1/261: dread dc/d2b/f36 [0,4194304] 0 2024-03-22T02:28:48.634 INFO:tasks.workunit.client.1.smithi122.stdout:5/218: chown d8/f30 14 1 2024-03-22T02:28:48.635 INFO:tasks.workunit.client.0.smithi103.stdout:6/562: chown d0/d14/d57/d68/c35 99 1 2024-03-22T02:28:48.635 INFO:tasks.workunit.client.1.smithi122.stdout:4/247: getdents d0/d12/d15/d42 0 2024-03-22T02:28:48.635 INFO:tasks.workunit.client.0.smithi103.stdout:7/747: creat d1/da/d2f/d36/fe9 x:0 0 0 2024-03-22T02:28:48.636 INFO:tasks.workunit.client.0.smithi103.stdout:7/748: readlink d1/da/d1c/l8d 0 2024-03-22T02:28:48.637 INFO:tasks.workunit.client.1.smithi122.stdout:8/285: dread d1/d9/f26 [0,4194304] 0 2024-03-22T02:28:48.637 INFO:tasks.workunit.client.0.smithi103.stdout:1/941: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d111/l145 0 2024-03-22T02:28:48.638 INFO:tasks.workunit.client.0.smithi103.stdout:5/793: write d0/d14/d23/d36/d95/ff1 [2140755,74155] 0 2024-03-22T02:28:48.640 INFO:tasks.workunit.client.0.smithi103.stdout:0/771: dread dd/d16/d4b/d96/f23 [0,4194304] 0 2024-03-22T02:28:48.641 INFO:tasks.workunit.client.0.smithi103.stdout:0/772: chown dd/d16/f36 50409309 1 2024-03-22T02:28:48.641 INFO:tasks.workunit.client.1.smithi122.stdout:5/219: unlink d8/d1e/d22/c2d 0 2024-03-22T02:28:48.643 INFO:tasks.workunit.client.0.smithi103.stdout:1/942: unlink db/c8b 0 2024-03-22T02:28:48.643 INFO:tasks.workunit.client.1.smithi122.stdout:4/248: creat d0/d12/f51 x:0 0 0 2024-03-22T02:28:48.644 INFO:tasks.workunit.client.1.smithi122.stdout:1/262: dwrite dc/f22 [0,4194304] 0 2024-03-22T02:28:48.645 INFO:tasks.workunit.client.0.smithi103.stdout:7/749: rename d1/da/d2f/fb7 to d1/d10/d11/d19/dc3/fea 0 2024-03-22T02:28:48.645 INFO:tasks.workunit.client.1.smithi122.stdout:5/220: chown d8/d1e/d22/d3a/f32 890 1 2024-03-22T02:28:48.646 INFO:tasks.workunit.client.0.smithi103.stdout:6/563: dwrite d0/d3/d22/f96 [0,4194304] 0 2024-03-22T02:28:48.647 INFO:tasks.workunit.client.1.smithi122.stdout:8/286: dwrite d1/f1a [0,4194304] 0 2024-03-22T02:28:48.649 INFO:tasks.workunit.client.0.smithi103.stdout:0/773: truncate dd/d16/d4b/d96/d29/f5a 963305 0 2024-03-22T02:28:48.650 INFO:tasks.workunit.client.1.smithi122.stdout:9/292: link db/l46 db/d23/d1f/d21/d25/l62 0 2024-03-22T02:28:48.651 INFO:tasks.workunit.client.1.smithi122.stdout:8/287: truncate d1/f3d 1415576 0 2024-03-22T02:28:48.651 INFO:tasks.workunit.client.1.smithi122.stdout:9/293: readlink db/d1b/d28/l45 0 2024-03-22T02:28:48.652 INFO:tasks.workunit.client.1.smithi122.stdout:1/263: mkdir dc/d55 0 2024-03-22T02:28:48.652 INFO:tasks.workunit.client.1.smithi122.stdout:5/221: creat d8/da/d28/d35/f4c x:0 0 0 2024-03-22T02:28:48.653 INFO:tasks.workunit.client.1.smithi122.stdout:5/222: chown c2 2944935 1 2024-03-22T02:28:48.655 INFO:tasks.workunit.client.1.smithi122.stdout:8/288: rmdir d1/d9/d50 39 2024-03-22T02:28:48.655 INFO:tasks.workunit.client.1.smithi122.stdout:8/289: chown d1/d40/f5b 0 1 2024-03-22T02:28:48.656 INFO:tasks.workunit.client.1.smithi122.stdout:8/290: fdatasync d1/d3f/d45/f49 0 2024-03-22T02:28:48.658 INFO:tasks.workunit.client.1.smithi122.stdout:9/294: rmdir db/d23/d1f/d57 39 2024-03-22T02:28:48.659 INFO:tasks.workunit.client.1.smithi122.stdout:4/249: getdents d0 0 2024-03-22T02:28:48.660 INFO:tasks.workunit.client.1.smithi122.stdout:5/223: chown d8/da/c13 206133 1 2024-03-22T02:28:48.662 INFO:tasks.workunit.client.0.smithi103.stdout:1/943: dread db/d4a/d78/d92/da8/ddd/d29/d6c/f71 [0,4194304] 0 2024-03-22T02:28:48.663 INFO:tasks.workunit.client.1.smithi122.stdout:8/291: mkdir d1/d60 0 2024-03-22T02:28:48.663 INFO:tasks.workunit.client.1.smithi122.stdout:9/295: mknod db/d23/d1f/d21/d3d/c63 0 2024-03-22T02:28:48.664 INFO:tasks.workunit.client.1.smithi122.stdout:4/250: write d0/d12/f21 [715178,89526] 0 2024-03-22T02:28:48.665 INFO:tasks.workunit.client.0.smithi103.stdout:1/944: readlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d108/l10e 0 2024-03-22T02:28:48.666 INFO:tasks.workunit.client.0.smithi103.stdout:1/945: write db/d4a/f65 [248243,17063] 0 2024-03-22T02:28:48.667 INFO:tasks.workunit.client.1.smithi122.stdout:1/264: dread dc/f30 [0,4194304] 0 2024-03-22T02:28:48.667 INFO:tasks.workunit.client.1.smithi122.stdout:5/224: symlink d8/d1e/d22/d3a/d46/l4d 0 2024-03-22T02:28:48.667 INFO:tasks.workunit.client.1.smithi122.stdout:6/300: dread d7/f3b [0,4194304] 0 2024-03-22T02:28:48.668 INFO:tasks.workunit.client.0.smithi103.stdout:1/946: chown db/d4a/d78/d92/da8/fe1 10902261 1 2024-03-22T02:28:48.672 INFO:tasks.workunit.client.1.smithi122.stdout:6/301: write d7/de/d27/f64 [63090,96013] 0 2024-03-22T02:28:48.676 INFO:tasks.workunit.client.1.smithi122.stdout:8/292: dwrite d1/d40/f4a [0,4194304] 0 2024-03-22T02:28:48.677 INFO:tasks.workunit.client.1.smithi122.stdout:5/225: mkdir d8/da/d28/d42/d4e 0 2024-03-22T02:28:48.678 INFO:tasks.workunit.client.1.smithi122.stdout:8/293: readlink d1/d8/d2f/l46 0 2024-03-22T02:28:48.679 INFO:tasks.workunit.client.1.smithi122.stdout:8/294: stat d1/d3f/d45/f49 0 2024-03-22T02:28:48.680 INFO:tasks.workunit.client.1.smithi122.stdout:1/265: getdents dc/d3b 0 2024-03-22T02:28:48.680 INFO:tasks.workunit.client.1.smithi122.stdout:9/296: getdents db/d23/d1f/d57 0 2024-03-22T02:28:48.682 INFO:tasks.workunit.client.1.smithi122.stdout:5/226: mkdir d8/d1e/d22/d3a/d46/d4f 0 2024-03-22T02:28:48.684 INFO:tasks.workunit.client.1.smithi122.stdout:8/295: creat d1/d22/f61 x:0 0 0 2024-03-22T02:28:48.685 INFO:tasks.workunit.client.1.smithi122.stdout:1/266: creat dc/d2b/f56 x:0 0 0 2024-03-22T02:28:48.686 INFO:tasks.workunit.client.1.smithi122.stdout:9/297: creat db/d23/d1f/d57/f64 x:0 0 0 2024-03-22T02:28:48.687 INFO:tasks.workunit.client.1.smithi122.stdout:4/251: write d0/d41/f48 [2064556,85940] 0 2024-03-22T02:28:48.688 INFO:tasks.workunit.client.1.smithi122.stdout:8/296: rmdir d1/d8 39 2024-03-22T02:28:48.690 INFO:tasks.workunit.client.1.smithi122.stdout:1/267: creat dc/d2b/f57 x:0 0 0 2024-03-22T02:28:48.690 INFO:tasks.workunit.client.1.smithi122.stdout:5/227: mknod d8/d1e/d22/d3a/d46/d4f/c50 0 2024-03-22T02:28:48.691 INFO:tasks.workunit.client.1.smithi122.stdout:9/298: mknod db/d23/d1f/d21/d3d/c65 0 2024-03-22T02:28:48.695 INFO:tasks.workunit.client.1.smithi122.stdout:1/268: rename dc/d2b/c3f to dc/d3b/d46/d47/c58 0 2024-03-22T02:28:48.696 INFO:tasks.workunit.client.1.smithi122.stdout:5/228: dread d8/d3b/f45 [0,4194304] 0 2024-03-22T02:28:48.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/297: creat d1/d8/d2f/f62 x:0 0 0 2024-03-22T02:28:48.699 INFO:tasks.workunit.client.1.smithi122.stdout:1/269: creat dc/d3b/d46/d47/f59 x:0 0 0 2024-03-22T02:28:48.701 INFO:tasks.workunit.client.1.smithi122.stdout:5/229: dread d8/d1e/d22/d3a/f32 [0,4194304] 0 2024-03-22T02:28:48.701 INFO:tasks.workunit.client.1.smithi122.stdout:8/298: mkdir d1/d8/d63 0 2024-03-22T02:28:48.704 INFO:tasks.workunit.client.1.smithi122.stdout:5/230: readlink d8/l1f 0 2024-03-22T02:28:48.705 INFO:tasks.workunit.client.1.smithi122.stdout:8/299: creat d1/d22/f64 x:0 0 0 2024-03-22T02:28:48.706 INFO:tasks.workunit.client.1.smithi122.stdout:8/300: chown d1/d3f/d45/c4f 241624043 1 2024-03-22T02:28:48.707 INFO:tasks.workunit.client.1.smithi122.stdout:5/231: mkdir d8/d1e/d51 0 2024-03-22T02:28:48.709 INFO:tasks.workunit.client.1.smithi122.stdout:5/232: creat d8/d3b/f52 x:0 0 0 2024-03-22T02:28:48.709 INFO:tasks.workunit.client.1.smithi122.stdout:1/270: link dc/l1e dc/d2b/l5a 0 2024-03-22T02:28:48.710 INFO:tasks.workunit.client.1.smithi122.stdout:8/301: creat d1/d8/d5e/f65 x:0 0 0 2024-03-22T02:28:48.711 INFO:tasks.workunit.client.1.smithi122.stdout:5/233: write d8/da/d28/d35/f37 [959862,48027] 0 2024-03-22T02:28:48.712 INFO:tasks.workunit.client.1.smithi122.stdout:8/302: write d1/d8/f5c [981173,107426] 0 2024-03-22T02:28:48.714 INFO:tasks.workunit.client.1.smithi122.stdout:1/271: rename dc/d2b/c4f to dc/d3b/d46/c5b 0 2024-03-22T02:28:48.715 INFO:tasks.workunit.client.1.smithi122.stdout:5/234: mknod d8/d1a/c53 0 2024-03-22T02:28:48.718 INFO:tasks.workunit.client.1.smithi122.stdout:1/272: creat dc/d3b/d46/d4d/f5c x:0 0 0 2024-03-22T02:28:48.719 INFO:tasks.workunit.client.1.smithi122.stdout:5/235: creat d8/d1e/d22/d3a/d46/d4f/f54 x:0 0 0 2024-03-22T02:28:48.721 INFO:tasks.workunit.client.1.smithi122.stdout:1/273: chown dc/c15 69620916 1 2024-03-22T02:28:48.723 INFO:tasks.workunit.client.1.smithi122.stdout:1/274: write dc/f2f [4542808,100783] 0 2024-03-22T02:28:48.725 INFO:tasks.workunit.client.1.smithi122.stdout:5/236: dwrite d8/d1a/f1c [0,4194304] 0 2024-03-22T02:28:48.727 INFO:tasks.workunit.client.1.smithi122.stdout:1/275: symlink dc/d55/l5d 0 2024-03-22T02:28:48.729 INFO:tasks.workunit.client.1.smithi122.stdout:5/237: mkdir d8/d1e/d22/d3a/d46/d4f/d55 0 2024-03-22T02:28:48.734 INFO:tasks.workunit.client.1.smithi122.stdout:1/276: write dc/f25 [83593,47838] 0 2024-03-22T02:28:48.735 INFO:tasks.workunit.client.1.smithi122.stdout:0/173: sync 2024-03-22T02:28:48.735 INFO:tasks.workunit.client.1.smithi122.stdout:8/303: truncate d1/d8/f33 7022467 0 2024-03-22T02:28:48.735 INFO:tasks.workunit.client.1.smithi122.stdout:8/304: write d1/d40/f47 [374311,96064] 0 2024-03-22T02:28:48.737 INFO:tasks.workunit.client.1.smithi122.stdout:5/238: link d8/da/c13 d8/d1a/c56 0 2024-03-22T02:28:48.740 INFO:tasks.workunit.client.1.smithi122.stdout:1/277: dwrite dc/d2b/f2d [0,4194304] 0 2024-03-22T02:28:48.741 INFO:tasks.workunit.client.0.smithi103.stdout:5/794: dwrite d0/da2/dca/ff8 [0,4194304] 0 2024-03-22T02:28:48.741 INFO:tasks.workunit.client.0.smithi103.stdout:5/795: chown d0/f73 2628148 1 2024-03-22T02:28:48.742 INFO:tasks.workunit.client.1.smithi122.stdout:8/305: rename d1/d8/cf to d1/d40/d48/c66 0 2024-03-22T02:28:48.744 INFO:tasks.workunit.client.1.smithi122.stdout:1/278: unlink dc/d2b/c34 0 2024-03-22T02:28:48.745 INFO:tasks.workunit.client.0.smithi103.stdout:0/774: write dd/d16/d4b/d96/d64/dab/dc6/f32 [9028618,99397] 0 2024-03-22T02:28:48.745 INFO:tasks.workunit.client.1.smithi122.stdout:1/279: fsync dc/f37 0 2024-03-22T02:28:48.747 INFO:tasks.workunit.client.1.smithi122.stdout:5/239: link d8/da/d28/f44 d8/d3b/f57 0 2024-03-22T02:28:48.749 INFO:tasks.workunit.client.0.smithi103.stdout:6/564: dwrite d0/d3/d33/f7c [0,4194304] 0 2024-03-22T02:28:48.750 INFO:tasks.workunit.client.1.smithi122.stdout:5/240: rmdir d8/da/d28/d35 39 2024-03-22T02:28:48.751 INFO:tasks.workunit.client.1.smithi122.stdout:8/306: dwrite d1/d3f/d45/f49 [0,4194304] 0 2024-03-22T02:28:48.755 INFO:tasks.workunit.client.1.smithi122.stdout:1/280: rename dc/d3b/d46/c5b to dc/d55/c5e 0 2024-03-22T02:28:48.761 INFO:tasks.workunit.client.1.smithi122.stdout:8/307: creat d1/d3f/d45/f67 x:0 0 0 2024-03-22T02:28:48.762 INFO:tasks.workunit.client.1.smithi122.stdout:8/308: truncate d1/d4d/f4e 691137 0 2024-03-22T02:28:48.763 INFO:tasks.workunit.client.1.smithi122.stdout:5/241: creat d8/da/d28/d42/d4e/f58 x:0 0 0 2024-03-22T02:28:48.763 INFO:tasks.workunit.client.1.smithi122.stdout:5/242: stat d8/d3b/f52 0 2024-03-22T02:28:48.766 INFO:tasks.workunit.client.1.smithi122.stdout:1/281: link dc/f51 dc/d55/f5f 0 2024-03-22T02:28:48.766 INFO:tasks.workunit.client.1.smithi122.stdout:0/174: truncate d11/d14/d23/f2d 49359 0 2024-03-22T02:28:48.769 INFO:tasks.workunit.client.1.smithi122.stdout:1/282: unlink dc/f17 0 2024-03-22T02:28:48.771 INFO:tasks.workunit.client.1.smithi122.stdout:5/243: dwrite d8/d1e/d22/d3a/d39/f49 [0,4194304] 0 2024-03-22T02:28:48.771 INFO:tasks.workunit.client.1.smithi122.stdout:0/175: rename fe to d11/d14/d23/d35/f3e 0 2024-03-22T02:28:48.774 INFO:tasks.workunit.client.1.smithi122.stdout:1/283: rmdir dc/d55 39 2024-03-22T02:28:48.777 INFO:tasks.workunit.client.1.smithi122.stdout:5/244: truncate d8/da/f16 4475549 0 2024-03-22T02:28:48.782 INFO:tasks.workunit.client.1.smithi122.stdout:0/176: link d11/d14/d1b/f38 d11/d14/d23/d24/d33/f3f 0 2024-03-22T02:28:48.785 INFO:tasks.workunit.client.1.smithi122.stdout:0/177: creat d11/f40 x:0 0 0 2024-03-22T02:28:48.785 INFO:tasks.workunit.client.1.smithi122.stdout:0/178: stat d11/c1a 0 2024-03-22T02:28:48.786 INFO:tasks.workunit.client.1.smithi122.stdout:1/284: dwrite dc/ff [0,4194304] 0 2024-03-22T02:28:48.791 INFO:tasks.workunit.client.1.smithi122.stdout:1/285: chown dc/fd 192 1 2024-03-22T02:28:48.794 INFO:tasks.workunit.client.1.smithi122.stdout:0/179: dwrite fb [4194304,4194304] 0 2024-03-22T02:28:48.794 INFO:tasks.workunit.client.1.smithi122.stdout:1/286: mknod dc/d2b/c60 0 2024-03-22T02:28:48.796 INFO:tasks.workunit.client.1.smithi122.stdout:1/287: fdatasync dc/d3b/f45 0 2024-03-22T02:28:48.797 INFO:tasks.workunit.client.1.smithi122.stdout:1/288: stat dc/f4a 0 2024-03-22T02:28:48.798 INFO:tasks.workunit.client.1.smithi122.stdout:0/180: creat d11/d14/d1d/f41 x:0 0 0 2024-03-22T02:28:48.800 INFO:tasks.workunit.client.1.smithi122.stdout:1/289: dread dc/d2b/f36 [0,4194304] 0 2024-03-22T02:28:48.802 INFO:tasks.workunit.client.1.smithi122.stdout:0/181: mknod d11/d14/d1d/d27/c42 0 2024-03-22T02:28:48.804 INFO:tasks.workunit.client.1.smithi122.stdout:1/290: mknod dc/c61 0 2024-03-22T02:28:48.805 INFO:tasks.workunit.client.0.smithi103.stdout:7/750: link d1/lf d1/d10/d11/d19/d29/d33/da8/leb 0 2024-03-22T02:28:48.807 INFO:tasks.workunit.client.0.smithi103.stdout:2/776: sync 2024-03-22T02:28:48.807 INFO:tasks.workunit.client.0.smithi103.stdout:4/788: sync 2024-03-22T02:28:48.809 INFO:tasks.workunit.client.0.smithi103.stdout:0/775: creat dd/d16/d4b/d96/d29/d48/d94/ffc x:0 0 0 2024-03-22T02:28:48.812 INFO:tasks.workunit.client.1.smithi122.stdout:0/182: truncate d11/f29 590275 0 2024-03-22T02:28:48.812 INFO:tasks.workunit.client.1.smithi122.stdout:1/291: dwrite dc/f2a [0,4194304] 0 2024-03-22T02:28:48.812 INFO:tasks.workunit.client.1.smithi122.stdout:1/292: chown dc/f49 3331612 1 2024-03-22T02:28:48.813 INFO:tasks.workunit.client.1.smithi122.stdout:0/183: write d11/f3d [181485,93215] 0 2024-03-22T02:28:48.814 INFO:tasks.workunit.client.0.smithi103.stdout:2/777: dwrite d11/d25/d63/d45/d47/dae/fe3 [0,4194304] 0 2024-03-22T02:28:48.817 INFO:tasks.workunit.client.0.smithi103.stdout:4/789: chown d3/c16 318204 1 2024-03-22T02:28:48.818 INFO:tasks.workunit.client.1.smithi122.stdout:0/184: getdents d11/d14/d1d/d27/d2f 0 2024-03-22T02:28:48.819 INFO:tasks.workunit.client.1.smithi122.stdout:1/293: dread dc/d3b/d46/d47/f50 [0,4194304] 0 2024-03-22T02:28:48.820 INFO:tasks.workunit.client.0.smithi103.stdout:0/776: write dd/d16/d4b/d96/d64/d6a/dd5/deb/d9a/d9d/fe5 [434089,70309] 0 2024-03-22T02:28:48.821 INFO:tasks.workunit.client.0.smithi103.stdout:7/751: dwrite d1/d10/d11/d70/d8a/fc9 [0,4194304] 0 2024-03-22T02:28:48.822 INFO:tasks.workunit.client.1.smithi122.stdout:0/185: symlink d11/d14/d23/l43 0 2024-03-22T02:28:48.822 INFO:tasks.workunit.client.1.smithi122.stdout:1/294: creat dc/d4c/f62 x:0 0 0 2024-03-22T02:28:48.824 INFO:tasks.workunit.client.1.smithi122.stdout:0/186: mkdir d11/d16/d44 0 2024-03-22T02:28:48.826 INFO:tasks.workunit.client.1.smithi122.stdout:1/295: rename dc/f42 to dc/d4c/f63 0 2024-03-22T02:28:48.827 INFO:tasks.workunit.client.1.smithi122.stdout:0/187: creat d11/d16/f45 x:0 0 0 2024-03-22T02:28:48.832 INFO:tasks.workunit.client.1.smithi122.stdout:1/296: fdatasync dc/f2a 0 2024-03-22T02:28:48.833 INFO:tasks.workunit.client.1.smithi122.stdout:0/188: dwrite f9 [4194304,4194304] 0 2024-03-22T02:28:48.834 INFO:tasks.workunit.client.0.smithi103.stdout:0/777: creat dd/d16/d4b/d96/d64/d6a/dd5/ffd x:0 0 0 2024-03-22T02:28:48.835 INFO:tasks.workunit.client.1.smithi122.stdout:1/297: write dc/d2b/f56 [845128,123201] 0 2024-03-22T02:28:48.838 INFO:tasks.workunit.client.1.smithi122.stdout:0/189: unlink d11/d14/d1d/d27/l21 0 2024-03-22T02:28:48.839 INFO:tasks.workunit.client.1.smithi122.stdout:1/298: creat dc/d4c/f64 x:0 0 0 2024-03-22T02:28:48.840 INFO:tasks.workunit.client.1.smithi122.stdout:1/299: write dc/f2f [8190586,107027] 0 2024-03-22T02:28:48.844 INFO:tasks.workunit.client.1.smithi122.stdout:1/300: unlink dc/d2b/f31 0 2024-03-22T02:28:48.847 INFO:tasks.workunit.client.1.smithi122.stdout:1/301: write dc/f30 [1903184,81683] 0 2024-03-22T02:28:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:48 smithi122 ceph-mon[36403]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:48 smithi122 ceph-mon[36403]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:48 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3560893240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:28:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:48 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:48.850 INFO:tasks.workunit.client.1.smithi122.stdout:1/302: readlink dc/l14 0 2024-03-22T02:28:48.851 INFO:tasks.workunit.client.1.smithi122.stdout:0/190: write d11/d14/d23/d35/f3e [1308391,69202] 0 2024-03-22T02:28:48.853 INFO:tasks.workunit.client.1.smithi122.stdout:0/191: unlink cf 0 2024-03-22T02:28:48.855 INFO:tasks.workunit.client.1.smithi122.stdout:0/192: fsync d11/d16/f2a 0 2024-03-22T02:28:48.858 INFO:tasks.workunit.client.1.smithi122.stdout:0/193: unlink fd 0 2024-03-22T02:28:48.860 INFO:tasks.workunit.client.1.smithi122.stdout:0/194: mknod d11/d14/d1b/c46 0 2024-03-22T02:28:48.862 INFO:tasks.workunit.client.1.smithi122.stdout:0/195: mkdir d11/d47 0 2024-03-22T02:28:48.866 INFO:tasks.workunit.client.1.smithi122.stdout:1/303: dread dc/f49 [0,4194304] 0 2024-03-22T02:28:48.866 INFO:tasks.workunit.client.0.smithi103.stdout:5/796: link d0/d14/d23/d36/d95/fa5 d0/d14/d23/d2a/d33/f118 0 2024-03-22T02:28:48.867 INFO:tasks.workunit.client.1.smithi122.stdout:1/304: creat dc/d2b/f65 x:0 0 0 2024-03-22T02:28:48.868 INFO:tasks.workunit.client.0.smithi103.stdout:5/797: rmdir d0/d14/d23/d2a/df7 39 2024-03-22T02:28:48.869 INFO:tasks.workunit.client.1.smithi122.stdout:1/305: symlink dc/d2b/l66 0 2024-03-22T02:28:48.870 INFO:tasks.workunit.client.1.smithi122.stdout:1/306: fdatasync dc/d4c/f52 0 2024-03-22T02:28:48.870 INFO:tasks.workunit.client.0.smithi103.stdout:2/778: rename d11/d25/d46/fa2 to d11/de8/f104 0 2024-03-22T02:28:48.871 INFO:tasks.workunit.client.1.smithi122.stdout:0/196: dwrite d11/d16/f2a [0,4194304] 0 2024-03-22T02:28:48.873 INFO:tasks.workunit.client.1.smithi122.stdout:0/197: chown d11/d14/d23/d24/f25 267009645 1 2024-03-22T02:28:48.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:48 smithi103 ceph-mon[29258]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:48 smithi103 ceph-mon[29258]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:48 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3560893240' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:28:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:48 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:28:48.874 INFO:tasks.workunit.client.0.smithi103.stdout:5/798: rename d0/d14/d23/d2a/d1f/l59 to d0/d14/d23/d2a/d33/db5/dfb/l119 0 2024-03-22T02:28:48.875 INFO:tasks.workunit.client.0.smithi103.stdout:5/799: chown d0/d14/d23/d36/d95/l69 103084 1 2024-03-22T02:28:48.876 INFO:tasks.workunit.client.0.smithi103.stdout:2/779: truncate f4 896205 0 2024-03-22T02:28:48.877 INFO:tasks.workunit.client.1.smithi122.stdout:1/307: rmdir dc 39 2024-03-22T02:28:48.879 INFO:tasks.workunit.client.1.smithi122.stdout:0/198: getdents d11/d14/d23/d24/d33 0 2024-03-22T02:28:48.879 INFO:tasks.workunit.client.1.smithi122.stdout:0/199: fdatasync d11/f40 0 2024-03-22T02:28:48.880 INFO:tasks.workunit.client.0.smithi103.stdout:1/947: rmdir db/d4a/d78/d92/da8/ddd/d21 39 2024-03-22T02:28:48.880 INFO:tasks.workunit.client.1.smithi122.stdout:1/308: mkdir dc/d3b/d46/d4d/d67 0 2024-03-22T02:28:48.881 INFO:tasks.workunit.client.1.smithi122.stdout:0/200: symlink d11/d14/l48 0 2024-03-22T02:28:48.881 INFO:tasks.workunit.client.0.smithi103.stdout:2/780: symlink d11/d25/d46/db6/dfb/l105 0 2024-03-22T02:28:48.882 INFO:tasks.workunit.client.1.smithi122.stdout:1/309: fdatasync dc/f2c 0 2024-03-22T02:28:48.883 INFO:tasks.workunit.client.1.smithi122.stdout:1/310: write dc/f11 [1105184,121688] 0 2024-03-22T02:28:48.883 INFO:tasks.workunit.client.1.smithi122.stdout:0/201: rename lc to d11/d47/l49 0 2024-03-22T02:28:48.884 INFO:tasks.workunit.client.1.smithi122.stdout:0/202: dread - d11/d14/d1b/f2c zero size 2024-03-22T02:28:48.885 INFO:tasks.workunit.client.0.smithi103.stdout:1/948: dread db/d4a/d78/d92/da8/ddd/d18/fa4 [0,4194304] 0 2024-03-22T02:28:48.887 INFO:tasks.workunit.client.1.smithi122.stdout:1/311: rename dc/f37 to dc/d3b/f68 0 2024-03-22T02:28:48.889 INFO:tasks.workunit.client.0.smithi103.stdout:7/752: write d1/d6f/f8c [393221,68017] 0 2024-03-22T02:28:48.889 INFO:tasks.workunit.client.0.smithi103.stdout:1/949: dread db/d4a/d78/d92/da8/ddd/d18/fad [0,4194304] 0 2024-03-22T02:28:48.892 INFO:tasks.workunit.client.1.smithi122.stdout:0/203: dwrite d11/f3b [0,4194304] 0 2024-03-22T02:28:48.893 INFO:tasks.workunit.client.0.smithi103.stdout:6/565: mknod d0/d3/d22/d58/d6f/cb7 0 2024-03-22T02:28:48.894 INFO:tasks.workunit.client.0.smithi103.stdout:6/566: write d0/d3/f1b [2691483,44369] 0 2024-03-22T02:28:48.895 INFO:tasks.workunit.client.0.smithi103.stdout:7/753: truncate d1/da/d2f/d36/f91 323146 0 2024-03-22T02:28:48.897 INFO:tasks.workunit.client.0.smithi103.stdout:7/754: write d1/d10/d11/d19/d29/d61/d1d/fcf [543349,101746] 0 2024-03-22T02:28:48.897 INFO:tasks.workunit.client.1.smithi122.stdout:1/312: dwrite dc/f28 [0,4194304] 0 2024-03-22T02:28:48.899 INFO:tasks.workunit.client.0.smithi103.stdout:1/950: getdents db/d4a/d78/d92/da8/ddd/d28/d97 0 2024-03-22T02:28:48.906 INFO:tasks.workunit.client.1.smithi122.stdout:1/313: dwrite dc/d4c/f52 [0,4194304] 0 2024-03-22T02:28:48.906 INFO:tasks.workunit.client.0.smithi103.stdout:0/778: unlink dd/d16/d4b/d96/d29/f2a 0 2024-03-22T02:28:48.907 INFO:tasks.workunit.client.1.smithi122.stdout:0/204: dwrite d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:48.908 INFO:tasks.workunit.client.0.smithi103.stdout:0/779: dread - dd/d34/f43 zero size 2024-03-22T02:28:48.909 INFO:tasks.workunit.client.0.smithi103.stdout:0/780: readlink dd/d16/d4b/d96/d35/lf1 0 2024-03-22T02:28:48.909 INFO:tasks.workunit.client.0.smithi103.stdout:1/951: mknod db/d4a/d78/d92/dea/d134/d128/c146 0 2024-03-22T02:28:48.910 INFO:tasks.workunit.client.1.smithi122.stdout:0/205: creat d11/f4a x:0 0 0 2024-03-22T02:28:48.910 INFO:tasks.workunit.client.1.smithi122.stdout:1/314: mknod dc/c69 0 2024-03-22T02:28:48.911 INFO:tasks.workunit.client.0.smithi103.stdout:0/781: fsync dd/d16/d4b/d96/d29/d48/fec 0 2024-03-22T02:28:48.913 INFO:tasks.workunit.client.0.smithi103.stdout:1/952: unlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/f143 0 2024-03-22T02:28:48.913 INFO:tasks.workunit.client.1.smithi122.stdout:0/206: creat d11/d14/d23/d35/f4b x:0 0 0 2024-03-22T02:28:48.914 INFO:tasks.workunit.client.1.smithi122.stdout:0/207: chown d11/d14/d1d/f41 695463534 1 2024-03-22T02:28:48.915 INFO:tasks.workunit.client.0.smithi103.stdout:0/782: symlink dd/d16/d4b/d96/d35/lfe 0 2024-03-22T02:28:48.915 INFO:tasks.workunit.client.1.smithi122.stdout:1/315: symlink dc/l6a 0 2024-03-22T02:28:48.918 INFO:tasks.workunit.client.0.smithi103.stdout:1/953: symlink db/d4a/d78/d92/da8/ddd/d6b/db5/l147 0 2024-03-22T02:28:48.919 INFO:tasks.workunit.client.1.smithi122.stdout:0/208: dread d11/f3d [0,4194304] 0 2024-03-22T02:28:48.919 INFO:tasks.workunit.client.0.smithi103.stdout:2/781: write d11/d19/d5d/f71 [1350371,78952] 0 2024-03-22T02:28:48.920 INFO:tasks.workunit.client.1.smithi122.stdout:1/316: creat dc/d3b/d46/f6b x:0 0 0 2024-03-22T02:28:48.921 INFO:tasks.workunit.client.0.smithi103.stdout:2/782: truncate d11/d25/f35 5013348 0 2024-03-22T02:28:48.921 INFO:tasks.workunit.client.0.smithi103.stdout:0/783: rename dd/d16/d4b/d96/d64/dab/dc6/fb4 to dd/d16/d4b/d96/d64/d6a/dd5/deb/d9a/df4/fff 0 2024-03-22T02:28:48.922 INFO:tasks.workunit.client.1.smithi122.stdout:1/317: rmdir dc/d3b 39 2024-03-22T02:28:48.923 INFO:tasks.workunit.client.1.smithi122.stdout:2/312: sync 2024-03-22T02:28:48.923 INFO:tasks.workunit.client.1.smithi122.stdout:7/236: sync 2024-03-22T02:28:48.923 INFO:tasks.workunit.client.1.smithi122.stdout:3/140: sync 2024-03-22T02:28:48.925 INFO:tasks.workunit.client.1.smithi122.stdout:3/141: write d2/f2c [643866,18607] 0 2024-03-22T02:28:48.926 INFO:tasks.workunit.client.1.smithi122.stdout:0/209: dwrite f10 [0,4194304] 0 2024-03-22T02:28:48.927 INFO:tasks.workunit.client.1.smithi122.stdout:1/318: creat dc/d3b/d46/d4d/d67/f6c x:0 0 0 2024-03-22T02:28:48.927 INFO:tasks.workunit.client.0.smithi103.stdout:7/755: write d1/da/fe [89833,69483] 0 2024-03-22T02:28:48.929 INFO:tasks.workunit.client.0.smithi103.stdout:6/567: truncate d0/d3/d22/f96 939831 0 2024-03-22T02:28:48.929 INFO:tasks.workunit.client.0.smithi103.stdout:7/756: fdatasync d1/da/d1c/f8b 0 2024-03-22T02:28:48.931 INFO:tasks.workunit.client.0.smithi103.stdout:1/954: dwrite db/d4a/d78/d92/da8/ddd/d18/d2d/f117 [0,4194304] 0 2024-03-22T02:28:48.939 INFO:tasks.workunit.client.0.smithi103.stdout:7/757: creat d1/d10/d11/dd9/fec x:0 0 0 2024-03-22T02:28:48.940 INFO:tasks.workunit.client.0.smithi103.stdout:7/758: rename d1/lf to d1/d10/d11/d19/d29/d61/d1d/da3/led 0 2024-03-22T02:28:48.940 INFO:tasks.workunit.client.0.smithi103.stdout:1/955: link db/d4a/d78/d92/dea/d134/f8f db/d4a/d78/d92/da8/ddd/d6b/f148 0 2024-03-22T02:28:48.944 INFO:tasks.workunit.client.0.smithi103.stdout:1/956: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f149 x:0 0 0 2024-03-22T02:28:48.949 INFO:tasks.workunit.client.0.smithi103.stdout:1/957: dwrite db/d4a/d78/d92/da8/deb/fec [0,4194304] 0 2024-03-22T02:28:48.953 INFO:tasks.workunit.client.1.smithi122.stdout:7/237: dread d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:48.953 INFO:tasks.workunit.client.1.smithi122.stdout:3/142: rename d2/f25 to d2/d9/dc/d17/d28/f2d 0 2024-03-22T02:28:48.955 INFO:tasks.workunit.client.1.smithi122.stdout:1/319: mknod dc/d3b/d46/d4d/d67/c6d 0 2024-03-22T02:28:48.956 INFO:tasks.workunit.client.1.smithi122.stdout:1/320: write dc/d3b/d46/d47/f59 [389754,1850] 0 2024-03-22T02:28:48.958 INFO:tasks.workunit.client.1.smithi122.stdout:7/238: dread d0/d4/d20/d21/f2b [0,4194304] 0 2024-03-22T02:28:48.961 INFO:tasks.workunit.client.1.smithi122.stdout:0/210: rename d11/d14/d23/d35 to d11/d16/d44/d4c 0 2024-03-22T02:28:48.965 INFO:tasks.workunit.client.1.smithi122.stdout:0/211: stat d11/d14/d1d/d27/c42 0 2024-03-22T02:28:48.965 INFO:tasks.workunit.client.1.smithi122.stdout:1/321: mkdir dc/d2b/d6e 0 2024-03-22T02:28:48.967 INFO:tasks.workunit.client.1.smithi122.stdout:7/239: dwrite d0/d4/d22/f32 [0,4194304] 0 2024-03-22T02:28:48.973 INFO:tasks.workunit.client.1.smithi122.stdout:1/322: dwrite dc/d2b/f56 [0,4194304] 0 2024-03-22T02:28:48.976 INFO:tasks.workunit.client.1.smithi122.stdout:0/212: link d11/d14/d1b/f38 d11/d14/d23/d24/f4d 0 2024-03-22T02:28:48.979 INFO:tasks.workunit.client.1.smithi122.stdout:3/143: symlink d2/d9/dc/d21/l2e 0 2024-03-22T02:28:48.982 INFO:tasks.workunit.client.1.smithi122.stdout:1/323: dwrite dc/f4e [0,4194304] 0 2024-03-22T02:28:48.984 INFO:tasks.workunit.client.1.smithi122.stdout:0/213: mkdir d11/d14/d1d/d27/d2f/d4e 0 2024-03-22T02:28:48.985 INFO:tasks.workunit.client.1.smithi122.stdout:0/214: chown d11/d14/d1d 191205 1 2024-03-22T02:28:48.986 INFO:tasks.workunit.client.1.smithi122.stdout:1/324: mkdir dc/d4c/d6f 0 2024-03-22T02:28:48.988 INFO:tasks.workunit.client.1.smithi122.stdout:1/325: read dc/d2b/f56 [1898996,986] 0 2024-03-22T02:28:48.988 INFO:tasks.workunit.client.1.smithi122.stdout:7/240: write d0/f16 [1001879,33438] 0 2024-03-22T02:28:48.990 INFO:tasks.workunit.client.1.smithi122.stdout:0/215: truncate d11/d14/d1d/f41 319034 0 2024-03-22T02:28:48.990 INFO:tasks.workunit.client.1.smithi122.stdout:7/241: readlink d0/d4/d20/l38 0 2024-03-22T02:28:48.997 INFO:tasks.workunit.client.1.smithi122.stdout:1/326: dwrite dc/d3b/d46/d47/f59 [0,4194304] 0 2024-03-22T02:28:48.998 INFO:tasks.workunit.client.0.smithi103.stdout:3/886: sync 2024-03-22T02:28:48.998 INFO:tasks.workunit.client.0.smithi103.stdout:8/911: sync 2024-03-22T02:28:48.999 INFO:tasks.workunit.client.0.smithi103.stdout:9/762: sync 2024-03-22T02:28:49.000 INFO:tasks.workunit.client.1.smithi122.stdout:3/144: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:49.001 INFO:tasks.workunit.client.1.smithi122.stdout:7/242: dwrite d0/d4/f2d [0,4194304] 0 2024-03-22T02:28:49.003 INFO:tasks.workunit.client.1.smithi122.stdout:3/145: write d2/f2c [1082754,85761] 0 2024-03-22T02:28:49.004 INFO:tasks.workunit.client.1.smithi122.stdout:1/327: unlink dc/d2b/f56 0 2024-03-22T02:28:49.007 INFO:tasks.workunit.client.1.smithi122.stdout:3/146: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:49.012 INFO:tasks.workunit.client.1.smithi122.stdout:1/328: rename dc/c35 to dc/d3b/d46/d47/c70 0 2024-03-22T02:28:49.012 INFO:tasks.workunit.client.1.smithi122.stdout:7/243: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:49.013 INFO:tasks.workunit.client.1.smithi122.stdout:1/329: dread - dc/d2b/f65 zero size 2024-03-22T02:28:49.013 INFO:tasks.workunit.client.1.smithi122.stdout:3/147: unlink d2/d9/dc/df/c12 0 2024-03-22T02:28:49.015 INFO:tasks.workunit.client.1.smithi122.stdout:7/244: mknod d0/d4/d20/d21/d28/c45 0 2024-03-22T02:28:49.016 INFO:tasks.workunit.client.1.smithi122.stdout:1/330: mkdir dc/d3b/d46/d4d/d71 0 2024-03-22T02:28:49.016 INFO:tasks.workunit.client.1.smithi122.stdout:3/148: rmdir d2/d9/dc 39 2024-03-22T02:28:49.018 INFO:tasks.workunit.client.1.smithi122.stdout:7/245: mkdir d0/d25/d46 0 2024-03-22T02:28:49.019 INFO:tasks.workunit.client.1.smithi122.stdout:0/216: dwrite fb [0,4194304] 0 2024-03-22T02:28:49.019 INFO:tasks.workunit.client.1.smithi122.stdout:1/331: mkdir dc/d2b/d6e/d72 0 2024-03-22T02:28:49.020 INFO:tasks.workunit.client.1.smithi122.stdout:7/246: chown d0/d4/d20/l38 0 1 2024-03-22T02:28:49.021 INFO:tasks.workunit.client.1.smithi122.stdout:0/217: readlink d11/d16/l19 0 2024-03-22T02:28:49.023 INFO:tasks.workunit.client.1.smithi122.stdout:1/332: mknod dc/c73 0 2024-03-22T02:28:49.023 INFO:tasks.workunit.client.1.smithi122.stdout:1/333: chown dc/d3b/d46/d4d 467031614 1 2024-03-22T02:28:49.024 INFO:tasks.workunit.client.1.smithi122.stdout:7/247: creat d0/d4/d20/d21/d28/f47 x:0 0 0 2024-03-22T02:28:49.025 INFO:tasks.workunit.client.1.smithi122.stdout:0/218: mkdir d11/d4f 0 2024-03-22T02:28:49.026 INFO:tasks.workunit.client.1.smithi122.stdout:1/334: mknod dc/d4c/c74 0 2024-03-22T02:28:49.028 INFO:tasks.workunit.client.1.smithi122.stdout:3/149: dwrite d2/d9/dc/d21/f23 [0,4194304] 0 2024-03-22T02:28:49.029 INFO:tasks.workunit.client.1.smithi122.stdout:0/219: creat d11/d14/d23/f50 x:0 0 0 2024-03-22T02:28:49.029 INFO:tasks.workunit.client.1.smithi122.stdout:0/220: readlink d11/d16/l19 0 2024-03-22T02:28:49.031 INFO:tasks.workunit.client.1.smithi122.stdout:7/248: mkdir d0/d25/d46/d48 0 2024-03-22T02:28:49.034 INFO:tasks.workunit.client.1.smithi122.stdout:0/221: unlink d11/d16/f2a 0 2024-03-22T02:28:49.037 INFO:tasks.workunit.client.1.smithi122.stdout:1/335: creat dc/d55/f75 x:0 0 0 2024-03-22T02:28:49.038 INFO:tasks.workunit.client.1.smithi122.stdout:3/150: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:49.045 INFO:tasks.workunit.client.1.smithi122.stdout:3/151: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:49.049 INFO:tasks.workunit.client.1.smithi122.stdout:1/336: fsync dc/d2b/f33 0 2024-03-22T02:28:49.050 INFO:tasks.workunit.client.1.smithi122.stdout:1/337: write dc/f40 [508128,79691] 0 2024-03-22T02:28:49.052 INFO:tasks.workunit.client.1.smithi122.stdout:3/152: write d2/d9/dc/d17/d28/f2d [258175,33538] 0 2024-03-22T02:28:49.053 INFO:tasks.workunit.client.1.smithi122.stdout:0/222: link d11/d14/l48 d11/d14/d1d/d27/d2f/d4e/l51 0 2024-03-22T02:28:49.054 INFO:tasks.workunit.client.1.smithi122.stdout:7/249: getdents d0/d4/d20/d21 0 2024-03-22T02:28:49.062 INFO:tasks.workunit.client.0.smithi103.stdout:1/958: dread db/d4a/f73 [0,4194304] 0 2024-03-22T02:28:49.062 INFO:tasks.workunit.client.0.smithi103.stdout:9/763: read d2/d10/d43/d4b/f82 [31563,18955] 0 2024-03-22T02:28:49.063 INFO:tasks.workunit.client.1.smithi122.stdout:7/250: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:49.063 INFO:tasks.workunit.client.0.smithi103.stdout:3/887: chown de/d6b/d80/d73/d110/ce7 2523 1 2024-03-22T02:28:49.064 INFO:tasks.workunit.client.0.smithi103.stdout:9/764: write d2/d5/d4c/d56/fbc [1014350,15099] 0 2024-03-22T02:28:49.064 INFO:tasks.workunit.client.0.smithi103.stdout:1/959: fsync db/d4a/d78/d92/da8/ddd/d28/db2/dc8/d113/f144 0 2024-03-22T02:28:49.064 INFO:tasks.workunit.client.1.smithi122.stdout:1/338: dwrite dc/f1a [0,4194304] 0 2024-03-22T02:28:49.065 INFO:tasks.workunit.client.1.smithi122.stdout:0/223: write d11/f29 [1165709,77835] 0 2024-03-22T02:28:49.067 INFO:tasks.workunit.client.1.smithi122.stdout:7/251: symlink d0/d4/d22/l49 0 2024-03-22T02:28:49.075 INFO:tasks.workunit.client.1.smithi122.stdout:0/224: dwrite d11/d16/d44/d4c/f4b [0,4194304] 0 2024-03-22T02:28:49.075 INFO:tasks.workunit.client.1.smithi122.stdout:0/225: chown d11/d16/d37 12 1 2024-03-22T02:28:49.077 INFO:tasks.workunit.client.1.smithi122.stdout:1/339: truncate dc/f30 3314692 0 2024-03-22T02:28:49.077 INFO:tasks.workunit.client.1.smithi122.stdout:1/340: fsync dc/d2b/f57 0 2024-03-22T02:28:49.079 INFO:tasks.workunit.client.1.smithi122.stdout:0/226: symlink d11/d14/d1d/d27/d32/l52 0 2024-03-22T02:28:49.080 INFO:tasks.workunit.client.1.smithi122.stdout:7/252: write d0/d4/f5 [1729939,40049] 0 2024-03-22T02:28:49.083 INFO:tasks.workunit.client.1.smithi122.stdout:0/227: fsync d11/d14/d23/d24/d33/f3f 0 2024-03-22T02:28:49.084 INFO:tasks.workunit.client.1.smithi122.stdout:7/253: unlink d0/d4/d20/d21/d28/f47 0 2024-03-22T02:28:49.087 INFO:tasks.workunit.client.1.smithi122.stdout:1/341: link dc/d2b/c3e dc/d3b/d46/d4d/d71/c76 0 2024-03-22T02:28:49.090 INFO:tasks.workunit.client.1.smithi122.stdout:1/342: chown dc/f10 70344 1 2024-03-22T02:28:49.092 INFO:tasks.workunit.client.1.smithi122.stdout:7/254: dwrite d0/d4/d22/f43 [0,4194304] 0 2024-03-22T02:28:49.096 INFO:tasks.workunit.client.1.smithi122.stdout:1/343: dread dc/ff [0,4194304] 0 2024-03-22T02:28:49.099 INFO:tasks.workunit.client.1.smithi122.stdout:1/344: creat dc/d55/f77 x:0 0 0 2024-03-22T02:28:49.101 INFO:tasks.workunit.client.1.smithi122.stdout:7/255: rmdir d0/d4/d20/d21/d37 0 2024-03-22T02:28:49.103 INFO:tasks.workunit.client.1.smithi122.stdout:7/256: symlink d0/d4/d20/l4a 0 2024-03-22T02:28:49.104 INFO:tasks.workunit.client.1.smithi122.stdout:0/228: truncate d11/f29 151947 0 2024-03-22T02:28:49.104 INFO:tasks.workunit.client.1.smithi122.stdout:7/257: read d0/f16 [605255,108604] 0 2024-03-22T02:28:49.105 INFO:tasks.workunit.client.1.smithi122.stdout:7/258: fsync d0/d4/fb 0 2024-03-22T02:28:49.107 INFO:tasks.workunit.client.1.smithi122.stdout:0/229: creat d11/d14/d1d/d27/d2f/f53 x:0 0 0 2024-03-22T02:28:49.108 INFO:tasks.workunit.client.1.smithi122.stdout:0/230: write d11/f40 [488574,30010] 0 2024-03-22T02:28:49.109 INFO:tasks.workunit.client.1.smithi122.stdout:7/259: rename d0/d4/la to d0/d4/d20/d21/d39/l4b 0 2024-03-22T02:28:49.112 INFO:tasks.workunit.client.1.smithi122.stdout:0/231: mkdir d11/d14/d23/d31/d54 0 2024-03-22T02:28:49.113 INFO:tasks.workunit.client.1.smithi122.stdout:7/260: dread d0/d4/d22/f32 [0,4194304] 0 2024-03-22T02:28:49.117 INFO:tasks.workunit.client.1.smithi122.stdout:7/261: symlink d0/d4/l4c 0 2024-03-22T02:28:49.121 INFO:tasks.workunit.client.1.smithi122.stdout:7/262: link d0/d4/l4c d0/d4/d20/d21/d28/l4d 0 2024-03-22T02:28:49.123 INFO:tasks.workunit.client.1.smithi122.stdout:7/263: mkdir d0/d4e 0 2024-03-22T02:28:49.124 INFO:tasks.workunit.client.1.smithi122.stdout:7/264: write d0/d4/d22/f43 [2277320,30927] 0 2024-03-22T02:28:49.127 INFO:tasks.workunit.client.1.smithi122.stdout:0/232: dwrite d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:49.127 INFO:tasks.workunit.client.1.smithi122.stdout:7/265: mknod d0/d4/d1d/c4f 0 2024-03-22T02:28:49.131 INFO:tasks.workunit.client.1.smithi122.stdout:7/266: rmdir d0/d4/d22 39 2024-03-22T02:28:49.131 INFO:tasks.workunit.client.0.smithi103.stdout:7/759: symlink d1/d10/d11/d19/dc3/lee 0 2024-03-22T02:28:49.134 INFO:tasks.workunit.client.0.smithi103.stdout:8/912: creat d4/d19/d1a/d104/f12a x:0 0 0 2024-03-22T02:28:49.134 INFO:tasks.workunit.client.0.smithi103.stdout:1/960: dread - db/d4a/d78/d92/ff6 zero size 2024-03-22T02:28:49.135 INFO:tasks.workunit.client.0.smithi103.stdout:0/784: dread dd/d16/d4b/d96/d64/dab/dc6/d89/f8b [0,4194304] 0 2024-03-22T02:28:49.135 INFO:tasks.workunit.client.0.smithi103.stdout:8/913: stat d4/c10 0 2024-03-22T02:28:49.135 INFO:tasks.workunit.client.1.smithi122.stdout:0/233: link d11/f4a d11/d16/f55 0 2024-03-22T02:28:49.136 INFO:tasks.workunit.client.1.smithi122.stdout:7/267: symlink d0/d4e/l50 0 2024-03-22T02:28:49.136 INFO:tasks.workunit.client.1.smithi122.stdout:7/268: readlink d0/d4/d20/d21/l31 0 2024-03-22T02:28:49.137 INFO:tasks.workunit.client.0.smithi103.stdout:3/888: truncate de/d12/d5d/d92/f115 617821 0 2024-03-22T02:28:49.138 INFO:tasks.workunit.client.0.smithi103.stdout:3/889: write de/d12/d4d/f107 [979328,112564] 0 2024-03-22T02:28:49.138 INFO:tasks.workunit.client.0.smithi103.stdout:9/765: rename d2/d10/d43/d4b/d65/lc5 to d2/d10/d43/d7e/le0 0 2024-03-22T02:28:49.139 INFO:tasks.workunit.client.1.smithi122.stdout:7/269: mknod d0/d4e/c51 0 2024-03-22T02:28:49.139 INFO:tasks.workunit.client.0.smithi103.stdout:1/961: mknod db/d4a/d78/d92/dea/c14a 0 2024-03-22T02:28:49.140 INFO:tasks.workunit.client.0.smithi103.stdout:0/785: read dd/d34/d53/fa1 [694307,89439] 0 2024-03-22T02:28:49.140 INFO:tasks.workunit.client.0.smithi103.stdout:9/766: readlink d2/da/d18/d1e/d77/d8b/l42 0 2024-03-22T02:28:49.144 INFO:tasks.workunit.client.0.smithi103.stdout:7/760: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/fef x:0 0 0 2024-03-22T02:28:49.144 INFO:tasks.workunit.client.1.smithi122.stdout:0/234: write d11/f3b [1083408,12574] 0 2024-03-22T02:28:49.148 INFO:tasks.workunit.client.1.smithi122.stdout:0/235: getdents d11/d16 0 2024-03-22T02:28:49.149 INFO:tasks.workunit.client.0.smithi103.stdout:1/962: dread db/d4a/d78/d92/da8/ddd/d29/f39 [0,4194304] 0 2024-03-22T02:28:49.150 INFO:tasks.workunit.client.1.smithi122.stdout:7/270: dwrite d0/f8 [0,4194304] 0 2024-03-22T02:28:49.150 INFO:tasks.workunit.client.0.smithi103.stdout:7/761: stat d1/da/l17 0 2024-03-22T02:28:49.151 INFO:tasks.workunit.client.1.smithi122.stdout:6/302: sync 2024-03-22T02:28:49.151 INFO:tasks.workunit.client.0.smithi103.stdout:0/786: dread dd/d16/d4b/d96/f5d [0,4194304] 0 2024-03-22T02:28:49.151 INFO:tasks.workunit.client.1.smithi122.stdout:0/236: unlink d11/d14/d1d/d27/d2f/c30 0 2024-03-22T02:28:49.153 INFO:tasks.workunit.client.1.smithi122.stdout:6/303: write d7/f3b [930071,30734] 0 2024-03-22T02:28:49.154 INFO:tasks.workunit.client.0.smithi103.stdout:3/890: write de/d1a/d25/f79 [2572407,84142] 0 2024-03-22T02:28:49.155 INFO:tasks.workunit.client.1.smithi122.stdout:0/237: dread d11/d14/d1d/f41 [0,4194304] 0 2024-03-22T02:28:49.155 INFO:tasks.workunit.client.1.smithi122.stdout:0/238: fdatasync f10 0 2024-03-22T02:28:49.157 INFO:tasks.workunit.client.1.smithi122.stdout:6/304: creat d7/f65 x:0 0 0 2024-03-22T02:28:49.157 INFO:tasks.workunit.client.0.smithi103.stdout:7/762: creat d1/d10/d11/d70/d8a/ff0 x:0 0 0 2024-03-22T02:28:49.158 INFO:tasks.workunit.client.1.smithi122.stdout:0/239: creat d11/d14/f56 x:0 0 0 2024-03-22T02:28:49.158 INFO:tasks.workunit.client.1.smithi122.stdout:0/240: chown d11/d47 115362 1 2024-03-22T02:28:49.159 INFO:tasks.workunit.client.0.smithi103.stdout:0/787: symlink dd/d16/d4b/d96/d29/l100 0 2024-03-22T02:28:49.163 INFO:tasks.workunit.client.0.smithi103.stdout:8/914: dwrite d4/f89 [0,4194304] 0 2024-03-22T02:28:49.164 INFO:tasks.workunit.client.1.smithi122.stdout:6/305: truncate d7/de/dd/d20/f31 1716195 0 2024-03-22T02:28:49.164 INFO:tasks.workunit.client.0.smithi103.stdout:9/767: dwrite d2/da/d18/d1e/d34/d3d/d61/f9a [0,4194304] 0 2024-03-22T02:28:49.165 INFO:tasks.workunit.client.1.smithi122.stdout:0/241: dwrite d11/f40 [0,4194304] 0 2024-03-22T02:28:49.167 INFO:tasks.workunit.client.0.smithi103.stdout:1/963: dwrite db/d4a/d78/ff9 [0,4194304] 0 2024-03-22T02:28:49.167 INFO:tasks.workunit.client.0.smithi103.stdout:3/891: rename de/d12/d4d/c12d to de/d12/d8a/db0/c130 0 2024-03-22T02:28:49.169 INFO:tasks.workunit.client.0.smithi103.stdout:3/892: chown de/d1a 6029915 1 2024-03-22T02:28:49.171 INFO:tasks.workunit.client.0.smithi103.stdout:0/788: creat dd/d16/d4b/d96/d64/dab/dc6/d89/f101 x:0 0 0 2024-03-22T02:28:49.172 INFO:tasks.workunit.client.0.smithi103.stdout:7/763: creat d1/d10/d11/d19/dc3/de8/ff1 x:0 0 0 2024-03-22T02:28:49.174 INFO:tasks.workunit.client.0.smithi103.stdout:9/768: read d2/da/d18/fb0 [3666937,61119] 0 2024-03-22T02:28:49.176 INFO:tasks.workunit.client.1.smithi122.stdout:6/306: dread d7/de/dd/d39/f5b [0,4194304] 0 2024-03-22T02:28:49.178 INFO:tasks.workunit.client.0.smithi103.stdout:7/764: creat d1/d6f/ff2 x:0 0 0 2024-03-22T02:28:49.181 INFO:tasks.workunit.client.0.smithi103.stdout:1/964: mkdir db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/d101/d14b 0 2024-03-22T02:28:49.183 INFO:tasks.workunit.client.0.smithi103.stdout:9/769: unlink d2/d10/c1d 0 2024-03-22T02:28:49.183 INFO:tasks.workunit.client.1.smithi122.stdout:6/307: dwrite d7/de/dd/d39/f4f [0,4194304] 0 2024-03-22T02:28:49.184 INFO:tasks.workunit.client.0.smithi103.stdout:1/965: write db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/fe2 [890833,12879] 0 2024-03-22T02:28:49.185 INFO:tasks.workunit.client.0.smithi103.stdout:8/915: rmdir d4/d26/d27/d7e/d39/d49/d51/daf/d101 0 2024-03-22T02:28:49.186 INFO:tasks.workunit.client.0.smithi103.stdout:0/789: mkdir dd/d16/d4b/d96/d29/d48/d5c/d102 0 2024-03-22T02:28:49.187 INFO:tasks.workunit.client.0.smithi103.stdout:1/966: chown db/d4a/d78/d92/da8/ddd/d2e/f51 2577 1 2024-03-22T02:28:49.188 INFO:tasks.workunit.client.0.smithi103.stdout:7/765: creat d1/d10/d11/d19/d29/d61/d1d/da3/db9/ff3 x:0 0 0 2024-03-22T02:28:49.191 INFO:tasks.workunit.client.0.smithi103.stdout:9/770: dread d2/d10/d43/d7e/d96/d9d/f7c [0,4194304] 0 2024-03-22T02:28:49.192 INFO:tasks.workunit.client.0.smithi103.stdout:1/967: dread - db/d4a/d78/d92/da8/ddd/d29/d6c/f7b zero size 2024-03-22T02:28:49.194 INFO:tasks.workunit.client.0.smithi103.stdout:7/766: write d1/d10/d11/d19/d29/d61/d69/d6b/d7a/fdd [680459,21478] 0 2024-03-22T02:28:49.195 INFO:tasks.workunit.client.0.smithi103.stdout:9/771: write d2/da/d18/d1e/d34/d3d/d61/db7/fd5 [104801,42245] 0 2024-03-22T02:28:49.195 INFO:tasks.workunit.client.0.smithi103.stdout:0/790: truncate dd/d16/d4b/d96/d64/d6a/dd5/deb/d9a/df4/fcf 457891 0 2024-03-22T02:28:49.196 INFO:tasks.workunit.client.0.smithi103.stdout:7/767: write d1/da/fe1 [186380,74073] 0 2024-03-22T02:28:49.197 INFO:tasks.workunit.client.0.smithi103.stdout:1/968: mkdir db/d4a/d78/d92/dea/d134/d128/d132/d14c 0 2024-03-22T02:28:49.200 INFO:tasks.workunit.client.0.smithi103.stdout:9/772: rename d2/d10/d43/d7e/cd9 to d2/da/d18/d1e/d34/ce1 0 2024-03-22T02:28:49.207 INFO:tasks.workunit.client.0.smithi103.stdout:9/773: creat d2/d5/d7b/d7d/dd4/dd8/fe2 x:0 0 0 2024-03-22T02:28:49.208 INFO:tasks.workunit.client.0.smithi103.stdout:7/768: creat d1/d10/d11/d19/d86/d92/dbf/ff4 x:0 0 0 2024-03-22T02:28:49.208 INFO:tasks.workunit.client.0.smithi103.stdout:9/774: read d2/d10/d43/d7e/fc6 [555199,71181] 0 2024-03-22T02:28:49.209 INFO:tasks.workunit.client.0.smithi103.stdout:7/769: stat d1/d10/d11/d19/d29/fbb 0 2024-03-22T02:28:49.210 INFO:tasks.workunit.client.0.smithi103.stdout:3/893: dread de/d12/d5d/f99 [4194304,4194304] 0 2024-03-22T02:28:49.211 INFO:tasks.workunit.client.0.smithi103.stdout:9/775: creat d2/d10/d43/d7e/da4/fe3 x:0 0 0 2024-03-22T02:28:49.213 INFO:tasks.workunit.client.0.smithi103.stdout:3/894: symlink de/d12/d30/d70/de5/l131 0 2024-03-22T02:28:49.217 INFO:tasks.workunit.client.0.smithi103.stdout:3/895: fdatasync de/f1f 0 2024-03-22T02:28:49.218 INFO:tasks.workunit.client.0.smithi103.stdout:9/776: dread d2/da/d18/d1e/d34/f97 [0,4194304] 0 2024-03-22T02:28:49.219 INFO:tasks.workunit.client.0.smithi103.stdout:3/896: chown de/d12/d5d/cc3 156 1 2024-03-22T02:28:49.220 INFO:tasks.workunit.client.0.smithi103.stdout:8/916: dwrite d4/d19/d1a/fb3 [0,4194304] 0 2024-03-22T02:28:49.221 INFO:tasks.workunit.client.0.smithi103.stdout:0/791: truncate dd/d16/d4b/d96/f23 5620512 0 2024-03-22T02:28:49.225 INFO:tasks.workunit.client.0.smithi103.stdout:9/777: mknod d2/d5/d7b/d7d/ce4 0 2024-03-22T02:28:49.226 INFO:tasks.workunit.client.0.smithi103.stdout:3/897: rmdir de/d12/d8a/d93/d103 39 2024-03-22T02:28:49.228 INFO:tasks.workunit.client.0.smithi103.stdout:8/917: fsync d4/d26/d27/d7e/d39/d49/d51/daf/fd6 0 2024-03-22T02:28:49.230 INFO:tasks.workunit.client.0.smithi103.stdout:1/969: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/dc8/fde [4194304,4194304] 0 2024-03-22T02:28:49.230 INFO:tasks.workunit.client.0.smithi103.stdout:7/770: write d1/da/d2f/d36/f91 [852729,53488] 0 2024-03-22T02:28:49.230 INFO:tasks.workunit.client.0.smithi103.stdout:9/778: truncate d2/d10/d43/d7e/f91 330571 0 2024-03-22T02:28:49.234 INFO:tasks.workunit.client.0.smithi103.stdout:9/779: write d2/da/d18/d1e/d34/d3d/d46/fb3 [339374,106112] 0 2024-03-22T02:28:49.235 INFO:tasks.workunit.client.0.smithi103.stdout:0/792: rename dd/d16/d4b/d96/d64/d6a/dd5/deb to dd/d16/d4b/d96/d29/d48/d5c/d103 0 2024-03-22T02:28:49.235 INFO:tasks.workunit.client.0.smithi103.stdout:4/790: sync 2024-03-22T02:28:49.235 INFO:tasks.workunit.client.0.smithi103.stdout:5/800: sync 2024-03-22T02:28:49.241 INFO:tasks.workunit.client.0.smithi103.stdout:5/801: stat d0/d14/dc0/le4 0 2024-03-22T02:28:49.243 INFO:tasks.workunit.client.0.smithi103.stdout:9/780: fdatasync d2/da/d18/d1e/d34/d3d/d4d/f58 0 2024-03-22T02:28:49.246 INFO:tasks.workunit.client.0.smithi103.stdout:4/791: dwrite d3/d1e/d53/d7a/fdb [0,4194304] 0 2024-03-22T02:28:49.246 INFO:tasks.workunit.client.0.smithi103.stdout:5/802: write d0/d14/d23/d5c/d77/df2/f5d [4368827,107621] 0 2024-03-22T02:28:49.248 INFO:tasks.workunit.client.0.smithi103.stdout:5/803: write d0/f9a [4649700,22097] 0 2024-03-22T02:28:49.248 INFO:tasks.workunit.client.0.smithi103.stdout:1/970: rename db/d4a/d78/d92/da8/ddd/d18/d84/f96 to db/d4a/d78/d92/da8/ddd/d21/f14d 0 2024-03-22T02:28:49.249 INFO:tasks.workunit.client.0.smithi103.stdout:0/793: unlink dd/c66 0 2024-03-22T02:28:49.253 INFO:tasks.workunit.client.0.smithi103.stdout:7/771: getdents d1/d10/d11/d19/d86 0 2024-03-22T02:28:49.255 INFO:tasks.workunit.client.0.smithi103.stdout:3/898: dread de/d6b/d80/d4f/d84/f88 [0,4194304] 0 2024-03-22T02:28:49.255 INFO:tasks.workunit.client.0.smithi103.stdout:8/918: write d4/d26/fa8 [207035,77241] 0 2024-03-22T02:28:49.261 INFO:tasks.workunit.client.0.smithi103.stdout:0/794: write dd/d16/d4b/d96/d64/dab/dc6/d89/f8b [8896,69619] 0 2024-03-22T02:28:49.264 INFO:tasks.workunit.client.0.smithi103.stdout:5/804: rename d0/d14/d23/d36/d95/fb1 to d0/d14/d23/d6f/d83/f11a 0 2024-03-22T02:28:49.264 INFO:tasks.workunit.client.0.smithi103.stdout:7/772: dwrite d1/d10/d11/d19/d29/d61/d69/d5c/fb8 [0,4194304] 0 2024-03-22T02:28:49.267 INFO:tasks.workunit.client.0.smithi103.stdout:7/773: chown d1/d10/d11/d19/d29/d33/d4b/c94 36667 1 2024-03-22T02:28:49.268 INFO:tasks.workunit.client.1.smithi122.stdout:0/242: unlink d11/f29 0 2024-03-22T02:28:49.269 INFO:tasks.workunit.client.0.smithi103.stdout:3/899: unlink de/d6b/d80/d73/fea 0 2024-03-22T02:28:49.270 INFO:tasks.workunit.client.1.smithi122.stdout:0/243: write d11/d14/d1d/d27/d2f/f3a [137596,78701] 0 2024-03-22T02:28:49.270 INFO:tasks.workunit.client.1.smithi122.stdout:6/308: symlink d7/de/d27/l66 0 2024-03-22T02:28:49.271 INFO:tasks.workunit.client.1.smithi122.stdout:0/244: chown d11/d14/f15 61956 1 2024-03-22T02:28:49.273 INFO:tasks.workunit.client.0.smithi103.stdout:9/781: link d2/da/d18/d1e/d34/d3d/d46/fad d2/d5/d7b/d7d/dd4/dd8/fe5 0 2024-03-22T02:28:49.275 INFO:tasks.workunit.client.1.smithi122.stdout:6/309: symlink d7/de/dd/d39/d46/l67 0 2024-03-22T02:28:49.276 INFO:tasks.workunit.client.1.smithi122.stdout:0/245: dread fb [0,4194304] 0 2024-03-22T02:28:49.277 INFO:tasks.workunit.client.0.smithi103.stdout:7/774: dwrite d1/d10/d11/d19/d29/d33/da8/dc5/fe4 [0,4194304] 0 2024-03-22T02:28:49.277 INFO:tasks.workunit.client.0.smithi103.stdout:1/971: symlink db/d4a/d78/d92/da8/ddd/d18/dca/de5/d137/l14e 0 2024-03-22T02:28:49.277 INFO:tasks.workunit.client.0.smithi103.stdout:9/782: fsync d2/d5/d4c/d56/f95 0 2024-03-22T02:28:49.280 INFO:tasks.workunit.client.1.smithi122.stdout:0/246: rename d11/c12 to d11/d14/d1b/c57 0 2024-03-22T02:28:49.282 INFO:tasks.workunit.client.1.smithi122.stdout:6/310: dwrite d7/de/f37 [0,4194304] 0 2024-03-22T02:28:49.282 INFO:tasks.workunit.client.0.smithi103.stdout:8/919: symlink d4/d19/l12b 0 2024-03-22T02:28:49.284 INFO:tasks.workunit.client.0.smithi103.stdout:3/900: dwrite de/d12/d4d/f107 [0,4194304] 0 2024-03-22T02:28:49.286 INFO:tasks.workunit.client.0.smithi103.stdout:4/792: dwrite d3/d30/f55 [0,4194304] 0 2024-03-22T02:28:49.287 INFO:tasks.workunit.client.0.smithi103.stdout:9/783: write d2/d5/d7b/d7d/dd4/dd8/fe2 [417578,42123] 0 2024-03-22T02:28:49.290 INFO:tasks.workunit.client.0.smithi103.stdout:9/784: stat d2/da/d18/d1e/d34/d3d/d46/f6a 0 2024-03-22T02:28:49.290 INFO:tasks.workunit.client.0.smithi103.stdout:5/805: fdatasync d0/d14/d23/d5c/d77/df2/f3f 0 2024-03-22T02:28:49.291 INFO:tasks.workunit.client.1.smithi122.stdout:0/247: dwrite d11/d16/d44/d4c/f4b [0,4194304] 0 2024-03-22T02:28:49.292 INFO:tasks.workunit.client.0.smithi103.stdout:9/785: stat d2/d10/d43/d7e/d96/d9d 0 2024-03-22T02:28:49.293 INFO:tasks.workunit.client.0.smithi103.stdout:7/775: rmdir d1/d10/d11/d19/d86/d92/dbf 39 2024-03-22T02:28:49.294 INFO:tasks.workunit.client.1.smithi122.stdout:0/248: creat d11/d14/d1d/d27/d2f/f58 x:0 0 0 2024-03-22T02:28:49.295 INFO:tasks.workunit.client.1.smithi122.stdout:0/249: truncate d11/d14/d23/f50 549856 0 2024-03-22T02:28:49.295 INFO:tasks.workunit.client.0.smithi103.stdout:7/776: write d1/d10/d11/d19/f9e [41547,118966] 0 2024-03-22T02:28:49.297 INFO:tasks.workunit.client.0.smithi103.stdout:1/972: creat db/d4a/d78/d92/da8/ddd/d18/dca/de5/f14f x:0 0 0 2024-03-22T02:28:49.299 INFO:tasks.workunit.client.1.smithi122.stdout:0/250: mkdir d11/d14/d1d/d27/d59 0 2024-03-22T02:28:49.301 INFO:tasks.workunit.client.0.smithi103.stdout:4/793: creat d3/d1e/d53/d7a/f10a x:0 0 0 2024-03-22T02:28:49.303 INFO:tasks.workunit.client.0.smithi103.stdout:5/806: unlink d0/d14/d23/d5c/d77/df2/dd9/lfd 0 2024-03-22T02:28:49.304 INFO:tasks.workunit.client.1.smithi122.stdout:0/251: dread d11/f40 [0,4194304] 0 2024-03-22T02:28:49.305 INFO:tasks.workunit.client.0.smithi103.stdout:8/920: mknod d4/d2a/d38/c12c 0 2024-03-22T02:28:49.305 INFO:tasks.workunit.client.1.smithi122.stdout:4/252: sync 2024-03-22T02:28:49.307 INFO:tasks.workunit.client.1.smithi122.stdout:0/252: mkdir d11/d4f/d5a 0 2024-03-22T02:28:49.307 INFO:tasks.workunit.client.0.smithi103.stdout:5/807: chown d0/d14/d23/d36/f82 4783 1 2024-03-22T02:28:49.307 INFO:tasks.workunit.client.0.smithi103.stdout:0/795: dwrite dd/d16/d4b/d96/d29/d48/fc7 [0,4194304] 0 2024-03-22T02:28:49.309 INFO:tasks.workunit.client.0.smithi103.stdout:8/921: chown d4/d2a/d38/c12c 196355806 1 2024-03-22T02:28:49.312 INFO:tasks.workunit.client.0.smithi103.stdout:1/973: chown db/l66 51337 1 2024-03-22T02:28:49.313 INFO:tasks.workunit.client.1.smithi122.stdout:4/253: dwrite d0/d41/f48 [4194304,4194304] 0 2024-03-22T02:28:49.314 INFO:tasks.workunit.client.0.smithi103.stdout:4/794: rmdir d3/d1e/d25/d65/d83/dc7 39 2024-03-22T02:28:49.314 INFO:tasks.workunit.client.0.smithi103.stdout:8/922: fdatasync d4/d26/d27/d7e/d39/d7b/dc7/ff3 0 2024-03-22T02:28:49.314 INFO:tasks.workunit.client.1.smithi122.stdout:0/253: dread f10 [0,4194304] 0 2024-03-22T02:28:49.316 INFO:tasks.workunit.client.0.smithi103.stdout:7/777: creat d1/d10/d11/d19/d86/d92/dbf/ff5 x:0 0 0 2024-03-22T02:28:49.317 INFO:tasks.workunit.client.1.smithi122.stdout:6/311: dread d7/de/dd/d20/f5d [0,4194304] 0 2024-03-22T02:28:49.317 INFO:tasks.workunit.client.1.smithi122.stdout:0/254: write d11/d14/d23/d24/d33/f3f [3848083,64577] 0 2024-03-22T02:28:49.317 INFO:tasks.workunit.client.0.smithi103.stdout:2/783: sync 2024-03-22T02:28:49.317 INFO:tasks.workunit.client.0.smithi103.stdout:6/568: sync 2024-03-22T02:28:49.317 INFO:tasks.workunit.client.1.smithi122.stdout:4/254: symlink d0/d12/d15/d42/l52 0 2024-03-22T02:28:49.319 INFO:tasks.workunit.client.1.smithi122.stdout:0/255: write d11/f3b [5145576,117382] 0 2024-03-22T02:28:49.319 INFO:tasks.workunit.client.1.smithi122.stdout:4/255: fdatasync d0/d25/f27 0 2024-03-22T02:28:49.320 INFO:tasks.workunit.client.0.smithi103.stdout:5/808: mknod d0/d14/d23/d2a/d33/c11b 0 2024-03-22T02:28:49.321 INFO:tasks.workunit.client.1.smithi122.stdout:0/256: fdatasync d11/f40 0 2024-03-22T02:28:49.323 INFO:tasks.workunit.client.1.smithi122.stdout:4/256: creat d0/d12/d15/d42/d4e/de/f53 x:0 0 0 2024-03-22T02:28:49.324 INFO:tasks.workunit.client.0.smithi103.stdout:6/569: read d0/d14/d57/d68/f40 [642770,31863] 0 2024-03-22T02:28:49.324 INFO:tasks.workunit.client.1.smithi122.stdout:4/257: write d0/d25/f3a [908523,6519] 0 2024-03-22T02:28:49.324 INFO:tasks.workunit.client.1.smithi122.stdout:0/257: mknod d11/d14/d1d/d27/d59/c5b 0 2024-03-22T02:28:49.326 INFO:tasks.workunit.client.1.smithi122.stdout:4/258: creat d0/d12/d15/d3d/f54 x:0 0 0 2024-03-22T02:28:49.327 INFO:tasks.workunit.client.0.smithi103.stdout:7/778: dwrite d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:49.329 INFO:tasks.workunit.client.1.smithi122.stdout:0/258: link d11/d14/d23/f50 d11/d4f/d5a/f5c 0 2024-03-22T02:28:49.330 INFO:tasks.workunit.client.1.smithi122.stdout:0/259: write f10 [3401230,42142] 0 2024-03-22T02:28:49.330 INFO:tasks.workunit.client.0.smithi103.stdout:4/795: rename d3/d1e/d53/le3 to d3/d100/l10b 0 2024-03-22T02:28:49.330 INFO:tasks.workunit.client.1.smithi122.stdout:0/260: stat c1 0 2024-03-22T02:28:49.333 INFO:tasks.workunit.client.0.smithi103.stdout:5/809: mkdir d0/d14/d23/d5c/d77/df2/dd9/d11c 0 2024-03-22T02:28:49.336 INFO:tasks.workunit.client.1.smithi122.stdout:0/261: dread d11/d14/d23/d24/f25 [0,4194304] 0 2024-03-22T02:28:49.336 INFO:tasks.workunit.client.0.smithi103.stdout:4/796: read - d3/d1e/d53/fe2 zero size 2024-03-22T02:28:49.340 INFO:tasks.workunit.client.0.smithi103.stdout:7/779: truncate d1/d10/d11/f28 5161964 0 2024-03-22T02:28:49.341 INFO:tasks.workunit.client.1.smithi122.stdout:0/262: fsync d11/d14/d23/d24/f4d 0 2024-03-22T02:28:49.341 INFO:tasks.workunit.client.0.smithi103.stdout:0/796: rename dd/d16/d4b/d96/fca to dd/d16/d4b/d96/d35/dc4/f104 0 2024-03-22T02:28:49.345 INFO:tasks.workunit.client.0.smithi103.stdout:8/923: getdents d4/d19 0 2024-03-22T02:28:49.347 INFO:tasks.workunit.client.1.smithi122.stdout:4/259: write d0/d25/f8 [428790,58025] 0 2024-03-22T02:28:49.348 INFO:tasks.workunit.client.0.smithi103.stdout:1/974: rename db/d4a/d78/d92/da8/ddd/d6b/db5/lf8 to db/d4a/d78/d92/dea/l150 0 2024-03-22T02:28:49.350 INFO:tasks.workunit.client.0.smithi103.stdout:3/901: dwrite de/fb4 [0,4194304] 0 2024-03-22T02:28:49.351 INFO:tasks.workunit.client.1.smithi122.stdout:4/260: creat d0/d12/d15/f55 x:0 0 0 2024-03-22T02:28:49.351 INFO:tasks.workunit.client.1.smithi122.stdout:6/312: dwrite d7/de/dd/f3a [0,4194304] 0 2024-03-22T02:28:49.352 INFO:tasks.workunit.client.1.smithi122.stdout:0/263: dwrite d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:49.352 INFO:tasks.workunit.client.0.smithi103.stdout:6/570: dread d0/d3/d22/f5c [0,4194304] 0 2024-03-22T02:28:49.354 INFO:tasks.workunit.client.0.smithi103.stdout:8/924: write d4/d19/d1a/fb3 [1307773,95011] 0 2024-03-22T02:28:49.354 INFO:tasks.workunit.client.1.smithi122.stdout:4/261: mknod d0/d41/c56 0 2024-03-22T02:28:49.355 INFO:tasks.workunit.client.0.smithi103.stdout:1/975: dread - db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/f12c zero size 2024-03-22T02:28:49.356 INFO:tasks.workunit.client.0.smithi103.stdout:0/797: truncate dd/d34/fad 1410810 0 2024-03-22T02:28:49.357 INFO:tasks.workunit.client.1.smithi122.stdout:6/313: rmdir d7/de/d27 39 2024-03-22T02:28:49.357 INFO:tasks.workunit.client.0.smithi103.stdout:7/780: rmdir d1/d10/d11/d19/d29 39 2024-03-22T02:28:49.358 INFO:tasks.workunit.client.1.smithi122.stdout:0/264: mkdir d11/d47/d5d 0 2024-03-22T02:28:49.359 INFO:tasks.workunit.client.1.smithi122.stdout:4/262: mkdir d0/d12/d15/d42/d57 0 2024-03-22T02:28:49.361 INFO:tasks.workunit.client.1.smithi122.stdout:4/263: read - d0/d12/d15/d42/d4e/de/f53 zero size 2024-03-22T02:28:49.361 INFO:tasks.workunit.client.0.smithi103.stdout:9/786: dwrite d2/da/d18/d1e/d34/d3d/d46/fad [0,4194304] 0 2024-03-22T02:28:49.364 INFO:tasks.workunit.client.1.smithi122.stdout:6/314: rename d7/de/f37 to d7/de/f68 0 2024-03-22T02:28:49.365 INFO:tasks.workunit.client.1.smithi122.stdout:6/315: chown l1 31 1 2024-03-22T02:28:49.366 INFO:tasks.workunit.client.1.smithi122.stdout:4/264: rmdir d0/d12/d15/d3d 39 2024-03-22T02:28:49.366 INFO:tasks.workunit.client.1.smithi122.stdout:6/316: chown d7/de/dd/d39 13395 1 2024-03-22T02:28:49.368 INFO:tasks.workunit.client.1.smithi122.stdout:0/265: dwrite d11/f3b [4194304,4194304] 0 2024-03-22T02:28:49.372 INFO:tasks.workunit.client.1.smithi122.stdout:6/317: chown d7/de/dd/d20/f45 0 1 2024-03-22T02:28:49.373 INFO:tasks.workunit.client.0.smithi103.stdout:7/781: mknod d1/d6f/cf6 0 2024-03-22T02:28:49.373 INFO:tasks.workunit.client.0.smithi103.stdout:9/787: mknod d2/da/d18/d1e/d77/d8b/ce6 0 2024-03-22T02:28:49.373 INFO:tasks.workunit.client.0.smithi103.stdout:4/797: dread d3/d69/f54 [0,4194304] 0 2024-03-22T02:28:49.374 INFO:tasks.workunit.client.1.smithi122.stdout:0/266: creat d11/d14/d1d/d27/f5e x:0 0 0 2024-03-22T02:28:49.375 INFO:tasks.workunit.client.0.smithi103.stdout:0/798: creat dd/d16/d4b/f105 x:0 0 0 2024-03-22T02:28:49.377 INFO:tasks.workunit.client.0.smithi103.stdout:4/798: chown d3/d1e/d78/df9/ld2 3 1 2024-03-22T02:28:49.378 INFO:tasks.workunit.client.1.smithi122.stdout:4/265: dwrite d0/d25/f43 [0,4194304] 0 2024-03-22T02:28:49.378 INFO:tasks.workunit.client.0.smithi103.stdout:0/799: chown dd/d16/d4b/d96/d29/c90 0 1 2024-03-22T02:28:49.381 INFO:tasks.workunit.client.0.smithi103.stdout:0/800: write dd/d16/d4b/d96/d29/d48/fc7 [5132815,86695] 0 2024-03-22T02:28:49.382 INFO:tasks.workunit.client.0.smithi103.stdout:9/788: fdatasync d2/da/d18/d1e/d34/d3d/d61/db7/fc3 0 2024-03-22T02:28:49.383 INFO:tasks.workunit.client.0.smithi103.stdout:7/782: dread d1/d10/d11/d70/d8a/fc9 [0,4194304] 0 2024-03-22T02:28:49.383 INFO:tasks.workunit.client.0.smithi103.stdout:8/925: getdents d4/d26/d27/d7e/d39/d49/d79/d11e/d40 0 2024-03-22T02:28:49.386 INFO:tasks.workunit.client.0.smithi103.stdout:8/926: dread - d4/d2a/d3e/fcf zero size 2024-03-22T02:28:49.393 INFO:tasks.workunit.client.0.smithi103.stdout:4/799: creat d3/d38/d5c/d90/df8/f10c x:0 0 0 2024-03-22T02:28:49.393 INFO:tasks.workunit.client.0.smithi103.stdout:7/783: read d1/da/fb [848953,79054] 0 2024-03-22T02:28:49.397 INFO:tasks.workunit.client.0.smithi103.stdout:0/801: dwrite dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d/fe5 [0,4194304] 0 2024-03-22T02:28:49.399 INFO:tasks.workunit.client.0.smithi103.stdout:7/784: dread - d1/da/d2f/d36/fe9 zero size 2024-03-22T02:28:49.400 INFO:tasks.workunit.client.0.smithi103.stdout:5/810: dread d0/d14/d23/d5c/d77/df2/f3f [0,4194304] 0 2024-03-22T02:28:49.401 INFO:tasks.workunit.client.0.smithi103.stdout:9/789: rename d2/d10/d43/d7e/le0 to d2/d5/d4c/d56/le7 0 2024-03-22T02:28:49.402 INFO:tasks.workunit.client.0.smithi103.stdout:8/927: truncate d4/d26/d27/f71 699411 0 2024-03-22T02:28:49.402 INFO:tasks.workunit.client.0.smithi103.stdout:0/802: fsync dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d/fe0 0 2024-03-22T02:28:49.405 INFO:tasks.workunit.client.1.smithi122.stdout:6/318: link d7/de/dd/d20/f51 d7/de/d27/f69 0 2024-03-22T02:28:49.407 INFO:tasks.workunit.client.1.smithi122.stdout:0/267: getdents d11/d14/d23/d24 0 2024-03-22T02:28:49.409 INFO:tasks.workunit.client.1.smithi122.stdout:4/266: getdents d0/d12/d15/d42/d4e/de 0 2024-03-22T02:28:49.409 INFO:tasks.workunit.client.0.smithi103.stdout:3/902: dwrite de/f1f [0,4194304] 0 2024-03-22T02:28:49.412 INFO:tasks.workunit.client.1.smithi122.stdout:9/299: sync 2024-03-22T02:28:49.413 INFO:tasks.workunit.client.0.smithi103.stdout:5/811: rename d0/d3e/f30 to d0/d14/d23/f11d 0 2024-03-22T02:28:49.413 INFO:tasks.workunit.client.1.smithi122.stdout:9/300: fdatasync db/d23/d1f/d21/f42 0 2024-03-22T02:28:49.415 INFO:tasks.workunit.client.0.smithi103.stdout:9/790: mkdir d2/da/d18/d1e/d34/d3d/d61/db7/de8 0 2024-03-22T02:28:49.416 INFO:tasks.workunit.client.1.smithi122.stdout:6/319: link d7/de/dd/df/f4b d7/de/d27/f6a 0 2024-03-22T02:28:49.418 INFO:tasks.workunit.client.1.smithi122.stdout:4/267: creat d0/d12/d15/d42/d57/f58 x:0 0 0 2024-03-22T02:28:49.419 INFO:tasks.workunit.client.0.smithi103.stdout:7/785: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/ff7 x:0 0 0 2024-03-22T02:28:49.419 INFO:tasks.workunit.client.0.smithi103.stdout:4/800: dwrite d3/d38/d5c/d86/fb6 [0,4194304] 0 2024-03-22T02:28:49.421 INFO:tasks.workunit.client.1.smithi122.stdout:9/301: mkdir db/d23/d1f/d57/d66 0 2024-03-22T02:28:49.423 INFO:tasks.workunit.client.1.smithi122.stdout:6/320: write d7/f15 [3231116,15755] 0 2024-03-22T02:28:49.423 INFO:tasks.workunit.client.1.smithi122.stdout:6/321: chown d7/c2a 7245 1 2024-03-22T02:28:49.425 INFO:tasks.workunit.client.1.smithi122.stdout:6/322: readlink d7/de/dd/d39/d46/l4d 0 2024-03-22T02:28:49.426 INFO:tasks.workunit.client.0.smithi103.stdout:4/801: chown d3/d1e/f7f 8121672 1 2024-03-22T02:28:49.426 INFO:tasks.workunit.client.0.smithi103.stdout:0/803: dwrite dd/f15 [0,4194304] 0 2024-03-22T02:28:49.427 INFO:tasks.workunit.client.0.smithi103.stdout:8/928: dwrite d4/d2a/d38/f9c [0,4194304] 0 2024-03-22T02:28:49.427 INFO:tasks.workunit.client.1.smithi122.stdout:4/268: mknod d0/d12/d15/c59 0 2024-03-22T02:28:49.430 INFO:tasks.workunit.client.1.smithi122.stdout:9/302: creat db/d23/d1e/d30/d58/f67 x:0 0 0 2024-03-22T02:28:49.434 INFO:tasks.workunit.client.1.smithi122.stdout:6/323: creat d7/de/dd/d39/d46/f6b x:0 0 0 2024-03-22T02:28:49.436 INFO:tasks.workunit.client.0.smithi103.stdout:3/903: creat de/d12/d5d/d92/f132 x:0 0 0 2024-03-22T02:28:49.436 INFO:tasks.workunit.client.0.smithi103.stdout:9/791: dwrite d2/d5/f63 [4194304,4194304] 0 2024-03-22T02:28:49.437 INFO:tasks.workunit.client.1.smithi122.stdout:4/269: dwrite d0/f9 [4194304,4194304] 0 2024-03-22T02:28:49.438 INFO:tasks.workunit.client.1.smithi122.stdout:4/270: fsync d0/d25/f27 0 2024-03-22T02:28:49.439 INFO:tasks.workunit.client.1.smithi122.stdout:0/268: dread d11/d14/f26 [0,4194304] 0 2024-03-22T02:28:49.442 INFO:tasks.workunit.client.0.smithi103.stdout:5/812: creat d0/d14/d23/d46/f11e x:0 0 0 2024-03-22T02:28:49.442 INFO:tasks.workunit.client.1.smithi122.stdout:4/271: truncate d0/d12/f51 154505 0 2024-03-22T02:28:49.444 INFO:tasks.workunit.client.0.smithi103.stdout:7/786: write d1/f7e [94332,47981] 0 2024-03-22T02:28:49.444 INFO:tasks.workunit.client.0.smithi103.stdout:9/792: truncate d2/da/f25 3324489 0 2024-03-22T02:28:49.445 INFO:tasks.workunit.client.1.smithi122.stdout:0/269: dread - d11/d14/d1b/f3c zero size 2024-03-22T02:28:49.445 INFO:tasks.workunit.client.1.smithi122.stdout:5/245: sync 2024-03-22T02:28:49.445 INFO:tasks.workunit.client.1.smithi122.stdout:2/313: sync 2024-03-22T02:28:49.445 INFO:tasks.workunit.client.1.smithi122.stdout:8/309: sync 2024-03-22T02:28:49.446 INFO:tasks.workunit.client.0.smithi103.stdout:0/804: creat dd/d16/d4b/d96/f106 x:0 0 0 2024-03-22T02:28:49.447 INFO:tasks.workunit.client.1.smithi122.stdout:4/272: write d0/d12/d15/f55 [571890,47567] 0 2024-03-22T02:28:49.449 INFO:tasks.workunit.client.1.smithi122.stdout:6/324: dwrite d7/f3b [0,4194304] 0 2024-03-22T02:28:49.450 INFO:tasks.workunit.client.0.smithi103.stdout:3/904: dwrite de/d1a/d25/f127 [0,4194304] 0 2024-03-22T02:28:49.450 INFO:tasks.workunit.client.0.smithi103.stdout:9/793: write d2/f3b [1349892,111386] 0 2024-03-22T02:28:49.453 INFO:tasks.workunit.client.0.smithi103.stdout:7/787: fdatasync d1/d10/d11/f28 0 2024-03-22T02:28:49.455 INFO:tasks.workunit.client.1.smithi122.stdout:2/314: unlink df/c74 0 2024-03-22T02:28:49.458 INFO:tasks.workunit.client.0.smithi103.stdout:4/802: truncate d3/fc 1366042 0 2024-03-22T02:28:49.459 INFO:tasks.workunit.client.1.smithi122.stdout:4/273: dwrite d0/d12/f21 [0,4194304] 0 2024-03-22T02:28:49.460 INFO:tasks.workunit.client.1.smithi122.stdout:8/310: write d1/d22/f2e [6454820,129019] 0 2024-03-22T02:28:49.462 INFO:tasks.workunit.client.0.smithi103.stdout:8/929: dread d4/d26/d27/d7e/d39/d49/d5f/f60 [0,4194304] 0 2024-03-22T02:28:49.463 INFO:tasks.workunit.client.0.smithi103.stdout:5/813: mkdir d0/d14/d23/d2a/df7/d11f 0 2024-03-22T02:28:49.463 INFO:tasks.workunit.client.0.smithi103.stdout:7/788: chown d1/d10/d11/d19/d86/d92/dbf/ff5 335 1 2024-03-22T02:28:49.463 INFO:tasks.workunit.client.0.smithi103.stdout:0/805: stat dd/d16/d4b/d96/d64/d6a/ca2 0 2024-03-22T02:28:49.465 INFO:tasks.workunit.client.1.smithi122.stdout:2/315: symlink df/d1f/d33/l75 0 2024-03-22T02:28:49.466 INFO:tasks.workunit.client.0.smithi103.stdout:8/930: chown d4/d26/d27/d7e/d39/d7b 11 1 2024-03-22T02:28:49.466 INFO:tasks.workunit.client.0.smithi103.stdout:3/905: rename de/d12/d30/l66 to de/d12/d30/d70/l133 0 2024-03-22T02:28:49.467 INFO:tasks.workunit.client.1.smithi122.stdout:6/325: rmdir d7/de/dd/df 39 2024-03-22T02:28:49.467 INFO:tasks.workunit.client.0.smithi103.stdout:7/789: truncate d1/da/d2f/f38 2293971 0 2024-03-22T02:28:49.469 INFO:tasks.workunit.client.0.smithi103.stdout:4/803: creat d3/da2/da6/f10d x:0 0 0 2024-03-22T02:28:49.470 INFO:tasks.workunit.client.0.smithi103.stdout:0/806: fdatasync dd/d34/fad 0 2024-03-22T02:28:49.471 INFO:tasks.workunit.client.1.smithi122.stdout:0/270: dwrite d11/d14/d1d/d27/d2f/f58 [0,4194304] 0 2024-03-22T02:28:49.473 INFO:tasks.workunit.client.1.smithi122.stdout:5/246: rmdir d8 39 2024-03-22T02:28:49.474 INFO:tasks.workunit.client.0.smithi103.stdout:4/804: truncate d3/d1e/d53/d7a/f10a 2185 0 2024-03-22T02:28:49.474 INFO:tasks.workunit.client.0.smithi103.stdout:8/931: symlink d4/d19/d8c/l12d 0 2024-03-22T02:28:49.475 INFO:tasks.workunit.client.0.smithi103.stdout:3/906: unlink de/d12/d8a/f118 0 2024-03-22T02:28:49.477 INFO:tasks.workunit.client.0.smithi103.stdout:4/805: truncate d3/d38/fda 4431349 0 2024-03-22T02:28:49.477 INFO:tasks.workunit.client.0.smithi103.stdout:9/794: creat d2/da/d18/fe9 x:0 0 0 2024-03-22T02:28:49.478 INFO:tasks.workunit.client.1.smithi122.stdout:4/274: dwrite d0/d25/f43 [0,4194304] 0 2024-03-22T02:28:49.479 INFO:tasks.workunit.client.0.smithi103.stdout:8/932: dread - d4/d2a/d3e/fcf zero size 2024-03-22T02:28:49.481 INFO:tasks.workunit.client.0.smithi103.stdout:3/907: rmdir de/d1a/d10f 39 2024-03-22T02:28:49.482 INFO:tasks.workunit.client.1.smithi122.stdout:8/311: dwrite d1/f1a [0,4194304] 0 2024-03-22T02:28:49.485 INFO:tasks.workunit.client.0.smithi103.stdout:4/806: rmdir d3/d38/d5c/d86 39 2024-03-22T02:28:49.485 INFO:tasks.workunit.client.1.smithi122.stdout:5/247: symlink d8/d3b/l59 0 2024-03-22T02:28:49.486 INFO:tasks.workunit.client.0.smithi103.stdout:5/814: dwrite d0/d14/d23/d36/d95/d96/daa/daf/ff0 [0,4194304] 0 2024-03-22T02:28:49.486 INFO:tasks.workunit.client.0.smithi103.stdout:7/790: getdents d1 0 2024-03-22T02:28:49.488 INFO:tasks.workunit.client.1.smithi122.stdout:0/271: creat d11/d14/d1d/d27/d32/f5f x:0 0 0 2024-03-22T02:28:49.489 INFO:tasks.workunit.client.1.smithi122.stdout:8/312: creat d1/d3f/d45/f68 x:0 0 0 2024-03-22T02:28:49.490 INFO:tasks.workunit.client.1.smithi122.stdout:5/248: creat d8/d3b/f5a x:0 0 0 2024-03-22T02:28:49.491 INFO:tasks.workunit.client.1.smithi122.stdout:2/316: dwrite df/f10 [0,4194304] 0 2024-03-22T02:28:49.493 INFO:tasks.workunit.client.1.smithi122.stdout:8/313: chown d1/d9/l39 337832 1 2024-03-22T02:28:49.495 INFO:tasks.workunit.client.1.smithi122.stdout:4/275: dwrite d0/d25/f27 [0,4194304] 0 2024-03-22T02:28:49.495 INFO:tasks.workunit.client.1.smithi122.stdout:1/345: sync 2024-03-22T02:28:49.495 INFO:tasks.workunit.client.1.smithi122.stdout:3/153: sync 2024-03-22T02:28:49.495 INFO:tasks.workunit.client.1.smithi122.stdout:7/271: sync 2024-03-22T02:28:49.496 INFO:tasks.workunit.client.0.smithi103.stdout:8/933: rename d4/d26/d27/d7e/d67 to d4/d26/d27/da3/df5/d12e 0 2024-03-22T02:28:49.498 INFO:tasks.workunit.client.1.smithi122.stdout:8/314: symlink d1/d8/l69 0 2024-03-22T02:28:49.499 INFO:tasks.workunit.client.1.smithi122.stdout:1/346: write dc/d3b/d46/f6b [902688,45164] 0 2024-03-22T02:28:49.504 INFO:tasks.workunit.client.0.smithi103.stdout:4/807: dwrite d3/d69/f3b [4194304,4194304] 0 2024-03-22T02:28:49.505 INFO:tasks.workunit.client.1.smithi122.stdout:5/249: link d8/f19 d8/da/f5b 0 2024-03-22T02:28:49.505 INFO:tasks.workunit.client.1.smithi122.stdout:0/272: dread d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:49.505 INFO:tasks.workunit.client.1.smithi122.stdout:5/250: chown d8/d3b/f5a 0 1 2024-03-22T02:28:49.507 INFO:tasks.workunit.client.1.smithi122.stdout:1/347: mknod dc/d3b/d46/d4d/c78 0 2024-03-22T02:28:49.508 INFO:tasks.workunit.client.1.smithi122.stdout:4/276: write d0/d12/d15/d3d/f54 [829620,4082] 0 2024-03-22T02:28:49.508 INFO:tasks.workunit.client.0.smithi103.stdout:7/791: dwrite d1/da/fc7 [0,4194304] 0 2024-03-22T02:28:49.509 INFO:tasks.workunit.client.1.smithi122.stdout:1/348: write dc/f38 [5551680,72062] 0 2024-03-22T02:28:49.511 INFO:tasks.workunit.client.1.smithi122.stdout:4/277: write d0/f9 [4225590,115706] 0 2024-03-22T02:28:49.511 INFO:tasks.workunit.client.1.smithi122.stdout:9/303: truncate db/d23/d1f/d21/d29/f2c 1325120 0 2024-03-22T02:28:49.512 INFO:tasks.workunit.client.0.smithi103.stdout:7/792: chown d1/d6f/f8c 19628411 1 2024-03-22T02:28:49.514 INFO:tasks.workunit.client.1.smithi122.stdout:4/278: chown d0/d25/f1d 13521047 1 2024-03-22T02:28:49.515 INFO:tasks.workunit.client.1.smithi122.stdout:7/272: creat d0/d30/f52 x:0 0 0 2024-03-22T02:28:49.519 INFO:tasks.workunit.client.1.smithi122.stdout:5/251: creat d8/d1a/f5c x:0 0 0 2024-03-22T02:28:49.520 INFO:tasks.workunit.client.1.smithi122.stdout:5/252: chown d8/d1e/d22/d3a/d46/d4f 0 1 2024-03-22T02:28:49.522 INFO:tasks.workunit.client.0.smithi103.stdout:3/908: rename de/d1a/d25/f29 to de/d1a/f134 0 2024-03-22T02:28:49.522 INFO:tasks.workunit.client.0.smithi103.stdout:8/934: getdents d4/d26/d27/d7e/d39/d49/d79/d11e/d40 0 2024-03-22T02:28:49.524 INFO:tasks.workunit.client.1.smithi122.stdout:1/349: rename dc/d3b/d46/d4d/c78 to dc/d2b/d6e/d72/c79 0 2024-03-22T02:28:49.526 INFO:tasks.workunit.client.0.smithi103.stdout:7/793: fdatasync d1/d10/d11/d19/d29/d61/d69/fa1 0 2024-03-22T02:28:49.527 INFO:tasks.workunit.client.1.smithi122.stdout:5/253: dread d8/d3b/f45 [0,4194304] 0 2024-03-22T02:28:49.528 INFO:tasks.workunit.client.0.smithi103.stdout:8/935: fdatasync d4/d19/d1a/fca 0 2024-03-22T02:28:49.530 INFO:tasks.workunit.client.0.smithi103.stdout:8/936: dread - d4/d26/d27/d7e/dc8/f10b zero size 2024-03-22T02:28:49.530 INFO:tasks.workunit.client.1.smithi122.stdout:7/273: creat d0/d30/f53 x:0 0 0 2024-03-22T02:28:49.531 INFO:tasks.workunit.client.0.smithi103.stdout:8/937: chown d4/d26/d27/da3/d117 25641050 1 2024-03-22T02:28:49.532 INFO:tasks.workunit.client.1.smithi122.stdout:9/304: rename db/d23/d1e/d30 to db/d23/d1f/d21/d29/d68 0 2024-03-22T02:28:49.533 INFO:tasks.workunit.client.0.smithi103.stdout:3/909: unlink de/d6b/d80/d4f/c50 0 2024-03-22T02:28:49.533 INFO:tasks.workunit.client.0.smithi103.stdout:8/938: fdatasync d4/d26/d27/d7e/d39/d49/d79/d11e/d35/fa0 0 2024-03-22T02:28:49.534 INFO:tasks.workunit.client.1.smithi122.stdout:5/254: write d8/da/f5b [779864,89529] 0 2024-03-22T02:28:49.535 INFO:tasks.workunit.client.1.smithi122.stdout:8/315: dread d1/d8/d2f/f38 [0,4194304] 0 2024-03-22T02:28:49.535 INFO:tasks.workunit.client.1.smithi122.stdout:5/255: chown d8/da/d28/d42 4133550 1 2024-03-22T02:28:49.536 INFO:tasks.workunit.client.1.smithi122.stdout:1/350: creat dc/d3b/d46/d4d/f7a x:0 0 0 2024-03-22T02:28:49.537 INFO:tasks.workunit.client.1.smithi122.stdout:4/279: creat d0/d12/d15/d42/d4e/f5a x:0 0 0 2024-03-22T02:28:49.538 INFO:tasks.workunit.client.0.smithi103.stdout:7/794: dread - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f zero size 2024-03-22T02:28:49.538 INFO:tasks.workunit.client.1.smithi122.stdout:8/316: mkdir d1/d22/d6a 0 2024-03-22T02:28:49.540 INFO:tasks.workunit.client.1.smithi122.stdout:5/256: rmdir d8/d1e/d22/d3a/d39 39 2024-03-22T02:28:49.541 INFO:tasks.workunit.client.0.smithi103.stdout:3/910: creat de/d6b/f135 x:0 0 0 2024-03-22T02:28:49.543 INFO:tasks.workunit.client.1.smithi122.stdout:9/305: dwrite db/d23/d1f/d21/d25/d41/f54 [0,4194304] 0 2024-03-22T02:28:49.544 INFO:tasks.workunit.client.1.smithi122.stdout:1/351: rename dc/d2b/f65 to dc/d3b/d46/d4d/d67/f7b 0 2024-03-22T02:28:49.544 INFO:tasks.workunit.client.0.smithi103.stdout:0/807: dwrite dd/d34/d53/d8c/fa0 [0,4194304] 0 2024-03-22T02:28:49.545 INFO:tasks.workunit.client.1.smithi122.stdout:1/352: chown dc/d55 746322 1 2024-03-22T02:28:49.545 INFO:tasks.workunit.client.0.smithi103.stdout:0/808: chown dd/d34/d53/d8c/fa0 197257 1 2024-03-22T02:28:49.546 INFO:tasks.workunit.client.0.smithi103.stdout:5/815: rename d0/d14/d23/d2a/d33/ff6 to d0/d14/d23/d2a/d1f/f120 0 2024-03-22T02:28:49.550 INFO:tasks.workunit.client.0.smithi103.stdout:7/795: rmdir d1/d10/d11/d19/d29/d33/dcd 39 2024-03-22T02:28:49.550 INFO:tasks.workunit.client.0.smithi103.stdout:3/911: creat de/d6b/f136 x:0 0 0 2024-03-22T02:28:49.552 INFO:tasks.workunit.client.1.smithi122.stdout:8/317: link d1/d8/d2f/f3e d1/d40/d51/f6b 0 2024-03-22T02:28:49.553 INFO:tasks.workunit.client.1.smithi122.stdout:1/353: symlink dc/d2b/d6e/l7c 0 2024-03-22T02:28:49.553 INFO:tasks.workunit.client.0.smithi103.stdout:3/912: write de/d12/d30/f11b [175254,93072] 0 2024-03-22T02:28:49.553 INFO:tasks.workunit.client.0.smithi103.stdout:0/809: rmdir dd/d16/d4b/d96/d64/d6a/dd5 39 2024-03-22T02:28:49.556 INFO:tasks.workunit.client.1.smithi122.stdout:5/257: dwrite d8/da/d28/f3d [0,4194304] 0 2024-03-22T02:28:49.558 INFO:tasks.workunit.client.1.smithi122.stdout:1/354: truncate dc/d4c/f64 891892 0 2024-03-22T02:28:49.558 INFO:tasks.workunit.client.0.smithi103.stdout:0/810: chown dd/d16/d4b/d96/d29/d48/d5c/d9b/lcb 1 1 2024-03-22T02:28:49.558 INFO:tasks.workunit.client.0.smithi103.stdout:0/811: fdatasync dd/d16/d4b/d96/d64/dab/dc6/fdc 0 2024-03-22T02:28:49.558 INFO:tasks.workunit.client.0.smithi103.stdout:7/796: mknod d1/d10/d11/d19/d29/d33/d4b/d5e/cf8 0 2024-03-22T02:28:49.561 INFO:tasks.workunit.client.1.smithi122.stdout:6/326: dwrite f5 [0,4194304] 0 2024-03-22T02:28:49.563 INFO:tasks.workunit.client.0.smithi103.stdout:5/816: rename d0/d14/d23/d36/l8b to d0/d14/d23/d36/d95/l121 0 2024-03-22T02:28:49.564 INFO:tasks.workunit.client.0.smithi103.stdout:4/808: write d3/d31/f32 [2440193,34591] 0 2024-03-22T02:28:49.565 INFO:tasks.workunit.client.1.smithi122.stdout:2/317: truncate df/f11 7425136 0 2024-03-22T02:28:49.565 INFO:tasks.workunit.client.1.smithi122.stdout:5/258: dread d8/da/d28/f3d [0,4194304] 0 2024-03-22T02:28:49.566 INFO:tasks.workunit.client.1.smithi122.stdout:5/259: fdatasync d8/f17 0 2024-03-22T02:28:49.568 INFO:tasks.workunit.client.1.smithi122.stdout:3/154: dread d2/d9/dc/d17/d28/f2d [0,4194304] 0 2024-03-22T02:28:49.569 INFO:tasks.workunit.client.1.smithi122.stdout:9/306: rmdir db/d23/d1e/d4e 0 2024-03-22T02:28:49.569 INFO:tasks.workunit.client.1.smithi122.stdout:3/155: chown d2/d9/c1c 24127 1 2024-03-22T02:28:49.569 INFO:tasks.workunit.client.1.smithi122.stdout:9/307: chown db/d23/c5b 1711754 1 2024-03-22T02:28:49.571 INFO:tasks.workunit.client.1.smithi122.stdout:9/308: dread - db/d23/d1f/d21/d29/f5f zero size 2024-03-22T02:28:49.572 INFO:tasks.workunit.client.0.smithi103.stdout:4/809: mknod d3/d1e/d53/d8a/c10e 0 2024-03-22T02:28:49.572 INFO:tasks.workunit.client.1.smithi122.stdout:0/273: truncate d11/f3b 4862084 0 2024-03-22T02:28:49.574 INFO:tasks.workunit.client.1.smithi122.stdout:2/318: stat df/d1f/d33/d34/c40 0 2024-03-22T02:28:49.574 INFO:tasks.workunit.client.0.smithi103.stdout:7/797: dwrite d1/d10/d11/d19/d29/d33/dcd/fde [0,4194304] 0 2024-03-22T02:28:49.574 INFO:tasks.workunit.client.0.smithi103.stdout:3/913: link de/d12/d8a/db0/fbe de/d12/d5d/d92/f137 0 2024-03-22T02:28:49.578 INFO:tasks.workunit.client.1.smithi122.stdout:6/327: dwrite d7/de/dd/d20/d2d/f30 [0,4194304] 0 2024-03-22T02:28:49.584 INFO:tasks.workunit.client.1.smithi122.stdout:9/309: chown db/ce 778161 1 2024-03-22T02:28:49.587 INFO:tasks.workunit.client.1.smithi122.stdout:0/274: creat d11/d14/d1b/f60 x:0 0 0 2024-03-22T02:28:49.587 INFO:tasks.workunit.client.0.smithi103.stdout:8/939: dwrite d4/d2a/d4f/fd9 [0,4194304] 0 2024-03-22T02:28:49.589 INFO:tasks.workunit.client.1.smithi122.stdout:1/355: rmdir dc/d4c/d6f 0 2024-03-22T02:28:49.590 INFO:tasks.workunit.client.0.smithi103.stdout:4/810: dwrite d3/d1e/d25/f52 [0,4194304] 0 2024-03-22T02:28:49.592 INFO:tasks.workunit.client.0.smithi103.stdout:7/798: fdatasync d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/fd1 0 2024-03-22T02:28:49.594 INFO:tasks.workunit.client.1.smithi122.stdout:8/318: write d1/d9/f26 [1160172,15790] 0 2024-03-22T02:28:49.595 INFO:tasks.workunit.client.1.smithi122.stdout:8/319: stat d1/d3f/d45/c4f 0 2024-03-22T02:28:49.596 INFO:tasks.workunit.client.0.smithi103.stdout:8/940: rename d4/d2a/d3e/da4 to d4/d26/d27/da3/d12f 0 2024-03-22T02:28:49.596 INFO:tasks.workunit.client.1.smithi122.stdout:9/310: dwrite db/d23/d1f/d21/d29/d68/f4f [0,4194304] 0 2024-03-22T02:28:49.597 INFO:tasks.workunit.client.0.smithi103.stdout:8/941: chown d4/f6e 320037037 1 2024-03-22T02:28:49.598 INFO:tasks.workunit.client.0.smithi103.stdout:0/812: dwrite dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/f95 [0,4194304] 0 2024-03-22T02:28:49.599 INFO:tasks.workunit.client.0.smithi103.stdout:4/811: truncate d3/fd 2590654 0 2024-03-22T02:28:49.599 INFO:tasks.workunit.client.0.smithi103.stdout:8/942: stat d4/d2a/d4f 0 2024-03-22T02:28:49.599 INFO:tasks.workunit.client.1.smithi122.stdout:5/260: creat d8/da/d28/d35/f5d x:0 0 0 2024-03-22T02:28:49.599 INFO:tasks.workunit.client.1.smithi122.stdout:8/320: dread - d1/d3f/d45/f67 zero size 2024-03-22T02:28:49.600 INFO:tasks.workunit.client.0.smithi103.stdout:5/817: getdents d0/d14/d23/d5c/d70 0 2024-03-22T02:28:49.600 INFO:tasks.workunit.client.0.smithi103.stdout:8/943: stat d4/ff6 0 2024-03-22T02:28:49.601 INFO:tasks.workunit.client.0.smithi103.stdout:7/799: creat d1/d10/d11/d19/d29/d33/da8/dc5/ff9 x:0 0 0 2024-03-22T02:28:49.604 INFO:tasks.workunit.client.0.smithi103.stdout:0/813: symlink dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/l107 0 2024-03-22T02:28:49.605 INFO:tasks.workunit.client.1.smithi122.stdout:5/261: truncate d8/d1a/f5c 305487 0 2024-03-22T02:28:49.605 INFO:tasks.workunit.client.1.smithi122.stdout:5/262: dread - d8/f20 zero size 2024-03-22T02:28:49.605 INFO:tasks.workunit.client.1.smithi122.stdout:3/156: fsync d2/f15 0 2024-03-22T02:28:49.606 INFO:tasks.workunit.client.1.smithi122.stdout:0/275: rmdir d11/d16 39 2024-03-22T02:28:49.609 INFO:tasks.workunit.client.0.smithi103.stdout:3/914: dwrite de/d12/d30/f3d [0,4194304] 0 2024-03-22T02:28:49.609 INFO:tasks.workunit.client.1.smithi122.stdout:2/319: rename df/d1f/d23/c30 to df/d1f/d2a/d72/c76 0 2024-03-22T02:28:49.610 INFO:tasks.workunit.client.0.smithi103.stdout:4/812: unlink d3/d1e/d53/d7a/da3/fbe 0 2024-03-22T02:28:49.610 INFO:tasks.workunit.client.0.smithi103.stdout:8/944: symlink d4/d19/d1a/l130 0 2024-03-22T02:28:49.611 INFO:tasks.workunit.client.1.smithi122.stdout:1/356: creat dc/d3b/d46/d4d/f7d x:0 0 0 2024-03-22T02:28:49.611 INFO:tasks.workunit.client.0.smithi103.stdout:0/814: readlink dd/d16/d4b/d96/l87 0 2024-03-22T02:28:49.612 INFO:tasks.workunit.client.0.smithi103.stdout:0/815: fdatasync dd/f12 0 2024-03-22T02:28:49.612 INFO:tasks.workunit.client.1.smithi122.stdout:6/328: fsync d7/de/dd/df/f62 0 2024-03-22T02:28:49.615 INFO:tasks.workunit.client.0.smithi103.stdout:7/800: rename d1/d10/d11/d19/d29/d57 to d1/dfa 0 2024-03-22T02:28:49.616 INFO:tasks.workunit.client.1.smithi122.stdout:1/357: write dc/f4e [2298713,45145] 0 2024-03-22T02:28:49.616 INFO:tasks.workunit.client.1.smithi122.stdout:9/311: mknod db/d23/d1f/d57/c69 0 2024-03-22T02:28:49.616 INFO:tasks.workunit.client.1.smithi122.stdout:8/321: dwrite d1/d8/f36 [0,4194304] 0 2024-03-22T02:28:49.618 INFO:tasks.workunit.client.1.smithi122.stdout:6/329: rmdir d7/de/d27 39 2024-03-22T02:28:49.619 INFO:tasks.workunit.client.0.smithi103.stdout:5/818: creat d0/d14/d23/d2a/d33/db5/dbd/ded/d103/f122 x:0 0 0 2024-03-22T02:28:49.621 INFO:tasks.workunit.client.0.smithi103.stdout:4/813: unlink d3/d38/l61 0 2024-03-22T02:28:49.621 INFO:tasks.workunit.client.0.smithi103.stdout:3/915: read de/d12/d8a/db0/fbe [127329,120565] 0 2024-03-22T02:28:49.625 INFO:tasks.workunit.client.0.smithi103.stdout:4/814: mkdir d3/d31/d10f 0 2024-03-22T02:28:49.626 INFO:tasks.workunit.client.0.smithi103.stdout:4/815: readlink d3/d69/lc0 0 2024-03-22T02:28:49.626 INFO:tasks.workunit.client.0.smithi103.stdout:5/819: mkdir d0/d14/d23/d36/d95/d123 0 2024-03-22T02:28:49.627 INFO:tasks.workunit.client.1.smithi122.stdout:0/276: creat d11/d4f/d5a/f61 x:0 0 0 2024-03-22T02:28:49.627 INFO:tasks.workunit.client.0.smithi103.stdout:5/820: fsync d0/fd4 0 2024-03-22T02:28:49.628 INFO:tasks.workunit.client.1.smithi122.stdout:9/312: creat db/d1b/f6a x:0 0 0 2024-03-22T02:28:49.629 INFO:tasks.workunit.client.0.smithi103.stdout:0/816: dwrite dd/d34/d53/d8c/fbc [0,4194304] 0 2024-03-22T02:28:49.630 INFO:tasks.workunit.client.0.smithi103.stdout:4/816: chown d3/d1e/d53/d8a/def/c104 2624465 1 2024-03-22T02:28:49.631 INFO:tasks.workunit.client.1.smithi122.stdout:5/263: dwrite d8/da/d28/d35/f37 [0,4194304] 0 2024-03-22T02:28:49.632 INFO:tasks.workunit.client.1.smithi122.stdout:9/313: fsync db/d23/f11 0 2024-03-22T02:28:49.632 INFO:tasks.workunit.client.1.smithi122.stdout:8/322: write d1/d3f/d45/f67 [1044732,62587] 0 2024-03-22T02:28:49.633 INFO:tasks.workunit.client.0.smithi103.stdout:4/817: stat d3/d31/f47 0 2024-03-22T02:28:49.633 INFO:tasks.workunit.client.0.smithi103.stdout:8/945: dread d4/f6e [0,4194304] 0 2024-03-22T02:28:49.634 INFO:tasks.workunit.client.1.smithi122.stdout:1/358: mknod dc/d3b/d46/d4d/c7e 0 2024-03-22T02:28:49.635 INFO:tasks.workunit.client.1.smithi122.stdout:2/320: dwrite df/d1f/f5c [0,4194304] 0 2024-03-22T02:28:49.636 INFO:tasks.workunit.client.0.smithi103.stdout:4/818: mknod d3/d38/d5c/c110 0 2024-03-22T02:28:49.636 INFO:tasks.workunit.client.0.smithi103.stdout:5/821: link d0/da2/cb4 d0/d14/d23/d36/d95/d96/c124 0 2024-03-22T02:28:49.640 INFO:tasks.workunit.client.0.smithi103.stdout:5/822: mkdir d0/d14/d23/d5c/d77/d125 0 2024-03-22T02:28:49.640 INFO:tasks.workunit.client.1.smithi122.stdout:0/277: dread - d11/d16/f55 zero size 2024-03-22T02:28:49.641 INFO:tasks.workunit.client.0.smithi103.stdout:5/823: chown d0/d14/d23/d2a/d33/db5/dfb/d10c 26 1 2024-03-22T02:28:49.641 INFO:tasks.workunit.client.0.smithi103.stdout:8/946: link d4/d26/d27/d7e/d39/d49/d79/d11e/d40/c47 d4/d19/d8c/d11a/c131 0 2024-03-22T02:28:49.641 INFO:tasks.workunit.client.1.smithi122.stdout:5/264: creat d8/d1e/f5e x:0 0 0 2024-03-22T02:28:49.642 INFO:tasks.workunit.client.0.smithi103.stdout:0/817: link dd/d16/d4b/d96/d29/d48/d5c/d9b/fa9 dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/f108 0 2024-03-22T02:28:49.642 INFO:tasks.workunit.client.0.smithi103.stdout:8/947: chown d4/d26/d27/d7e/d39/d49 1844561 1 2024-03-22T02:28:49.645 INFO:tasks.workunit.client.1.smithi122.stdout:1/359: rmdir dc/d2b/d6e 39 2024-03-22T02:28:49.645 INFO:tasks.workunit.client.1.smithi122.stdout:1/360: readlink dc/l14 0 2024-03-22T02:28:49.646 INFO:tasks.workunit.client.1.smithi122.stdout:6/330: creat d7/de/f6c x:0 0 0 2024-03-22T02:28:49.650 INFO:tasks.workunit.client.1.smithi122.stdout:3/157: write d2/f8 [1534568,114180] 0 2024-03-22T02:28:49.651 INFO:tasks.workunit.client.1.smithi122.stdout:3/158: write d2/f8 [926109,15216] 0 2024-03-22T02:28:49.651 INFO:tasks.workunit.client.1.smithi122.stdout:5/265: fdatasync d8/d1e/d22/d3a/d39/f49 0 2024-03-22T02:28:49.652 INFO:tasks.workunit.client.1.smithi122.stdout:5/266: dread - d8/d3b/f52 zero size 2024-03-22T02:28:49.653 INFO:tasks.workunit.client.1.smithi122.stdout:8/323: symlink d1/d9/d50/l6c 0 2024-03-22T02:28:49.654 INFO:tasks.workunit.client.0.smithi103.stdout:7/801: creat d1/d10/d11/dd9/ffb x:0 0 0 2024-03-22T02:28:49.655 INFO:tasks.workunit.client.1.smithi122.stdout:5/267: write d8/d1a/f48 [336866,122049] 0 2024-03-22T02:28:49.655 INFO:tasks.workunit.client.0.smithi103.stdout:7/802: chown d1/d10/d11/d19/d29/fe2 684 1 2024-03-22T02:28:49.656 INFO:tasks.workunit.client.1.smithi122.stdout:5/268: stat d8/d1e/d22/d3a/d46/f4b 0 2024-03-22T02:28:49.660 INFO:tasks.workunit.client.0.smithi103.stdout:7/803: dwrite d1/d10/d11/d19/f55 [4194304,4194304] 0 2024-03-22T02:28:49.661 INFO:tasks.workunit.client.1.smithi122.stdout:8/324: creat d1/d40/d48/f6d x:0 0 0 2024-03-22T02:28:49.662 INFO:tasks.workunit.client.1.smithi122.stdout:1/361: symlink dc/d2b/d6e/l7f 0 2024-03-22T02:28:49.665 INFO:tasks.workunit.client.1.smithi122.stdout:6/331: symlink d7/de/dd/d20/l6d 0 2024-03-22T02:28:49.670 INFO:tasks.workunit.client.1.smithi122.stdout:9/314: getdents db/d23/d1f/d21/d25 0 2024-03-22T02:28:49.673 INFO:tasks.workunit.client.1.smithi122.stdout:1/362: dwrite dc/f28 [0,4194304] 0 2024-03-22T02:28:49.675 INFO:tasks.workunit.client.1.smithi122.stdout:6/332: dread d7/de/dd/d20/f45 [0,4194304] 0 2024-03-22T02:28:49.675 INFO:tasks.workunit.client.1.smithi122.stdout:3/159: creat d2/d9/dc/f2f x:0 0 0 2024-03-22T02:28:49.675 INFO:tasks.workunit.client.1.smithi122.stdout:9/315: creat db/d1b/d28/f6b x:0 0 0 2024-03-22T02:28:49.678 INFO:tasks.workunit.client.1.smithi122.stdout:3/160: chown d2/d9/dc/d17/d28/f2d 54 1 2024-03-22T02:28:49.679 INFO:tasks.workunit.client.1.smithi122.stdout:1/363: creat dc/f80 x:0 0 0 2024-03-22T02:28:49.679 INFO:tasks.workunit.client.0.smithi103.stdout:2/784: sync 2024-03-22T02:28:49.680 INFO:tasks.workunit.client.1.smithi122.stdout:8/325: dwrite d1/d9/d50/f59 [0,4194304] 0 2024-03-22T02:28:49.687 INFO:tasks.workunit.client.1.smithi122.stdout:1/364: creat dc/d3b/f81 x:0 0 0 2024-03-22T02:28:49.689 INFO:tasks.workunit.client.1.smithi122.stdout:8/326: symlink d1/d60/l6e 0 2024-03-22T02:28:49.689 INFO:tasks.workunit.client.1.smithi122.stdout:3/161: creat d2/d9/dc/d17/d28/f30 x:0 0 0 2024-03-22T02:28:49.690 INFO:tasks.workunit.client.1.smithi122.stdout:8/327: stat d1/d8/d2f/f38 0 2024-03-22T02:28:49.692 INFO:tasks.workunit.client.1.smithi122.stdout:5/269: dread d8/d1e/f23 [4194304,4194304] 0 2024-03-22T02:28:49.692 INFO:tasks.workunit.client.1.smithi122.stdout:8/328: chown d1/d4d/f58 60613 1 2024-03-22T02:28:49.693 INFO:tasks.workunit.client.1.smithi122.stdout:1/365: creat dc/d3b/d46/f82 x:0 0 0 2024-03-22T02:28:49.694 INFO:tasks.workunit.client.1.smithi122.stdout:5/270: read d8/da/d28/d35/f37 [2890527,115676] 0 2024-03-22T02:28:49.695 INFO:tasks.workunit.client.1.smithi122.stdout:1/366: fsync dc/d2b/f57 0 2024-03-22T02:28:49.695 INFO:tasks.workunit.client.1.smithi122.stdout:5/271: chown d8/d3b/f5a 317994 1 2024-03-22T02:28:49.697 INFO:tasks.workunit.client.1.smithi122.stdout:9/316: dread db/d23/d1f/d21/d25/f55 [0,4194304] 0 2024-03-22T02:28:49.699 INFO:tasks.workunit.client.1.smithi122.stdout:9/317: write db/d23/d1f/d21/d29/f3f [1692428,118435] 0 2024-03-22T02:28:49.700 INFO:tasks.workunit.client.1.smithi122.stdout:9/318: chown db/df/l1d 7 1 2024-03-22T02:28:49.702 INFO:tasks.workunit.client.1.smithi122.stdout:5/272: rename d8/d1a/l1b to d8/d1e/d22/d3a/d46/l5f 0 2024-03-22T02:28:49.703 INFO:tasks.workunit.client.1.smithi122.stdout:3/162: getdents d2/d9/dc/d17 0 2024-03-22T02:28:49.707 INFO:tasks.workunit.client.1.smithi122.stdout:0/278: read d11/d14/d23/d24/f25 [2118156,8038] 0 2024-03-22T02:28:49.709 INFO:tasks.workunit.client.1.smithi122.stdout:1/367: dwrite dc/d3b/f68 [0,4194304] 0 2024-03-22T02:28:49.711 INFO:tasks.workunit.client.1.smithi122.stdout:0/279: chown d11/d14/f56 5331 1 2024-03-22T02:28:49.714 INFO:tasks.workunit.client.1.smithi122.stdout:0/280: chown l0 0 1 2024-03-22T02:28:49.714 INFO:tasks.workunit.client.1.smithi122.stdout:1/368: dread dc/d2b/f36 [0,4194304] 0 2024-03-22T02:28:49.715 INFO:tasks.workunit.client.1.smithi122.stdout:3/163: dread d2/f2c [0,4194304] 0 2024-03-22T02:28:49.716 INFO:tasks.workunit.client.1.smithi122.stdout:9/319: dwrite db/d23/d1f/d21/f56 [4194304,4194304] 0 2024-03-22T02:28:49.716 INFO:tasks.workunit.client.1.smithi122.stdout:0/281: dread - d11/d14/d1b/f60 zero size 2024-03-22T02:28:49.718 INFO:tasks.workunit.client.1.smithi122.stdout:1/369: chown dc/d55/l5d 1819 1 2024-03-22T02:28:49.718 INFO:tasks.workunit.client.1.smithi122.stdout:0/282: chown d11/d14/f56 627218379 1 2024-03-22T02:28:49.721 INFO:tasks.workunit.client.1.smithi122.stdout:5/273: rmdir d8/d1e/d22 39 2024-03-22T02:28:49.722 INFO:tasks.workunit.client.1.smithi122.stdout:1/370: write dc/d4c/f63 [365614,126101] 0 2024-03-22T02:28:49.724 INFO:tasks.workunit.client.1.smithi122.stdout:1/371: dread - dc/d3b/d46/d4d/d67/f6c zero size 2024-03-22T02:28:49.725 INFO:tasks.workunit.client.1.smithi122.stdout:9/320: unlink db/d23/d1f/d21/d25/l62 0 2024-03-22T02:28:49.726 INFO:tasks.workunit.client.1.smithi122.stdout:9/321: read - db/d23/d1f/d21/d29/f5f zero size 2024-03-22T02:28:49.727 INFO:tasks.workunit.client.1.smithi122.stdout:0/283: dread d11/d14/d23/f2d [0,4194304] 0 2024-03-22T02:28:49.730 INFO:tasks.workunit.client.1.smithi122.stdout:9/322: rename db/d23/d1f/d21/d29/d68 to db/d1b/d6c 0 2024-03-22T02:28:49.731 INFO:tasks.workunit.client.1.smithi122.stdout:9/323: dread - db/d1b/f6a zero size 2024-03-22T02:28:49.731 INFO:tasks.workunit.client.1.smithi122.stdout:0/284: mknod d11/d14/d1d/d27/c62 0 2024-03-22T02:28:49.732 INFO:tasks.workunit.client.1.smithi122.stdout:2/321: dread df/f11 [4194304,4194304] 0 2024-03-22T02:28:49.732 INFO:tasks.workunit.client.1.smithi122.stdout:3/164: write d2/d9/dc/fd [1625626,48880] 0 2024-03-22T02:28:49.733 INFO:tasks.workunit.client.1.smithi122.stdout:2/322: stat df/f36 0 2024-03-22T02:28:49.736 INFO:tasks.workunit.client.1.smithi122.stdout:9/324: getdents db 0 2024-03-22T02:28:49.738 INFO:tasks.workunit.client.1.smithi122.stdout:0/285: getdents d11 0 2024-03-22T02:28:49.740 INFO:tasks.workunit.client.1.smithi122.stdout:9/325: write fa [2561731,82260] 0 2024-03-22T02:28:49.741 INFO:tasks.workunit.client.1.smithi122.stdout:9/326: write db/d1b/d6c/f4f [3199932,39482] 0 2024-03-22T02:28:49.742 INFO:tasks.workunit.client.1.smithi122.stdout:3/165: link d2/d9/dc/f2f d2/d9/dc/d17/d28/f31 0 2024-03-22T02:28:49.744 INFO:tasks.workunit.client.1.smithi122.stdout:0/286: dread fb [0,4194304] 0 2024-03-22T02:28:49.745 INFO:tasks.workunit.client.1.smithi122.stdout:9/327: fdatasync db/d1b/d6c/f4f 0 2024-03-22T02:28:49.747 INFO:tasks.workunit.client.1.smithi122.stdout:0/287: symlink d11/d14/d1b/l63 0 2024-03-22T02:28:49.750 INFO:tasks.workunit.client.1.smithi122.stdout:3/166: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:49.754 INFO:tasks.workunit.client.1.smithi122.stdout:9/328: getdents db 0 2024-03-22T02:28:49.754 INFO:tasks.workunit.client.1.smithi122.stdout:1/372: dread dc/d2b/f2d [0,4194304] 0 2024-03-22T02:28:49.754 INFO:tasks.workunit.client.1.smithi122.stdout:9/329: dread - db/d23/d1f/d57/f64 zero size 2024-03-22T02:28:49.754 INFO:tasks.workunit.client.1.smithi122.stdout:9/330: chown db/d23/f1a 3 1 2024-03-22T02:28:49.754 INFO:tasks.workunit.client.1.smithi122.stdout:1/373: chown dc/d2b/l5a 13252000 1 2024-03-22T02:28:49.755 INFO:tasks.workunit.client.1.smithi122.stdout:3/167: symlink d2/d9/dc/d17/d28/l32 0 2024-03-22T02:28:49.755 INFO:tasks.workunit.client.1.smithi122.stdout:9/331: fdatasync db/f13 0 2024-03-22T02:28:49.757 INFO:tasks.workunit.client.1.smithi122.stdout:9/332: fdatasync db/d23/d1f/d21/d29/f5f 0 2024-03-22T02:28:49.757 INFO:tasks.workunit.client.1.smithi122.stdout:1/374: symlink dc/d2b/l83 0 2024-03-22T02:28:49.758 INFO:tasks.workunit.client.1.smithi122.stdout:3/168: unlink d2/d9/dc/d17/d28/f30 0 2024-03-22T02:28:49.763 INFO:tasks.workunit.client.1.smithi122.stdout:3/169: rename d2/d9/l1b to d2/d9/dc/l33 0 2024-03-22T02:28:49.768 INFO:tasks.workunit.client.1.smithi122.stdout:9/333: truncate db/d23/d1f/f2b 4152252 0 2024-03-22T02:28:49.770 INFO:tasks.workunit.client.1.smithi122.stdout:3/170: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:49.770 INFO:tasks.workunit.client.1.smithi122.stdout:9/334: chown db/d23/c2a 118454921 1 2024-03-22T02:28:49.774 INFO:tasks.workunit.client.1.smithi122.stdout:1/375: dwrite dc/f51 [0,4194304] 0 2024-03-22T02:28:49.774 INFO:tasks.workunit.client.1.smithi122.stdout:9/335: mknod db/df/d3e/d59/c6d 0 2024-03-22T02:28:49.817 INFO:tasks.workunit.client.0.smithi103.stdout:7/804: read - d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/ff7 zero size 2024-03-22T02:28:49.818 INFO:tasks.workunit.client.0.smithi103.stdout:2/785: mkdir d11/d25/d63/d45/d47/dae/d106 0 2024-03-22T02:28:49.819 INFO:tasks.workunit.client.0.smithi103.stdout:5/824: dwrite d0/d14/d23/d36/fb6 [0,4194304] 0 2024-03-22T02:28:49.820 INFO:tasks.workunit.client.0.smithi103.stdout:7/805: dread - d1/dfa/fd5 zero size 2024-03-22T02:28:49.824 INFO:tasks.workunit.client.0.smithi103.stdout:2/786: truncate d11/d19/da7/dd3/f52 938089 0 2024-03-22T02:28:49.826 INFO:tasks.workunit.client.0.smithi103.stdout:5/825: write d0/d14/d23/d36/d95/d96/fb0 [1107319,41986] 0 2024-03-22T02:28:49.827 INFO:tasks.workunit.client.0.smithi103.stdout:2/787: mknod d11/d25/d63/d45/d47/c107 0 2024-03-22T02:28:49.828 INFO:tasks.workunit.client.0.smithi103.stdout:5/826: fdatasync d0/d14/d23/d2a/fd8 0 2024-03-22T02:28:49.829 INFO:tasks.workunit.client.0.smithi103.stdout:0/818: dread dd/d16/d4b/d96/d29/f83 [0,4194304] 0 2024-03-22T02:28:49.832 INFO:tasks.workunit.client.0.smithi103.stdout:5/827: unlink d0/d14/d23/d2a/f102 0 2024-03-22T02:28:49.833 INFO:tasks.workunit.client.0.smithi103.stdout:5/828: stat d0/d14/d23/d2a/d33 0 2024-03-22T02:28:49.836 INFO:tasks.workunit.client.0.smithi103.stdout:0/819: write dd/d16/d4b/d96/d29/fb5 [7021331,20136] 0 2024-03-22T02:28:49.836 INFO:tasks.workunit.client.0.smithi103.stdout:5/829: mkdir d0/d14/d23/d6f/d83/d126 0 2024-03-22T02:28:49.838 INFO:tasks.workunit.client.0.smithi103.stdout:0/820: unlink dd/d16/d4b/la4 0 2024-03-22T02:28:49.838 INFO:tasks.workunit.client.0.smithi103.stdout:5/830: chown d0/d14/d23/d2a/d33/db5/dfb/d10c/l10a 21 1 2024-03-22T02:28:49.844 INFO:tasks.workunit.client.0.smithi103.stdout:0/821: write dd/d16/d4b/d96/d35/d82/f7b [3936551,91325] 0 2024-03-22T02:28:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:49 smithi122 ceph-mon[36403]: pgmap v18: 65 pgs: 65 active+clean; 1.6 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 50 MiB/s rd, 162 MiB/s wr, 317 op/s 2024-03-22T02:28:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:49 smithi122 ceph-mon[36403]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:49.851 INFO:tasks.workunit.client.0.smithi103.stdout:7/806: write d1/d10/d11/d19/d29/d61/fc8 [4225959,68991] 0 2024-03-22T02:28:49.853 INFO:tasks.workunit.client.0.smithi103.stdout:5/831: dread d0/da2/dca/fdd [0,4194304] 0 2024-03-22T02:28:49.853 INFO:tasks.workunit.client.0.smithi103.stdout:2/788: dread d11/f16 [0,4194304] 0 2024-03-22T02:28:49.854 INFO:tasks.workunit.client.1.smithi122.stdout:7/274: sync 2024-03-22T02:28:49.855 INFO:tasks.workunit.client.0.smithi103.stdout:5/832: write d0/d14/d23/d5c/d77/df2/f5d [3521543,110169] 0 2024-03-22T02:28:49.856 INFO:tasks.workunit.client.0.smithi103.stdout:5/833: readlink d0/d14/l60 0 2024-03-22T02:28:49.856 INFO:tasks.workunit.client.0.smithi103.stdout:7/807: creat d1/d10/d11/d19/d29/d61/d69/ffc x:0 0 0 2024-03-22T02:28:49.860 INFO:tasks.workunit.client.0.smithi103.stdout:0/822: truncate dd/d16/d4b/d96/fe1 64169 0 2024-03-22T02:28:49.862 INFO:tasks.workunit.client.0.smithi103.stdout:7/808: truncate d1/da/d2f/fb1 247897 0 2024-03-22T02:28:49.863 INFO:tasks.workunit.client.0.smithi103.stdout:0/823: symlink dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d/l109 0 2024-03-22T02:28:49.864 INFO:tasks.workunit.client.0.smithi103.stdout:7/809: write d1/da/d2f/fa0 [223940,85060] 0 2024-03-22T02:28:49.867 INFO:tasks.workunit.client.0.smithi103.stdout:2/789: truncate d11/d25/d46/fc2 4210809 0 2024-03-22T02:28:49.868 INFO:tasks.workunit.client.0.smithi103.stdout:0/824: dread dd/d16/d4b/d96/d29/d48/fc7 [4194304,4194304] 0 2024-03-22T02:28:49.869 INFO:tasks.workunit.client.0.smithi103.stdout:7/810: creat d1/d10/d11/d19/d29/d33/da8/ffd x:0 0 0 2024-03-22T02:28:49.869 INFO:tasks.workunit.client.0.smithi103.stdout:0/825: fdatasync dd/d34/fad 0 2024-03-22T02:28:49.873 INFO:tasks.workunit.client.0.smithi103.stdout:5/834: dread d0/d14/d23/d5c/d77/df2/f4d [0,4194304] 0 2024-03-22T02:28:49.874 INFO:tasks.workunit.client.0.smithi103.stdout:1/976: sync 2024-03-22T02:28:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:49 smithi103 ceph-mon[29258]: pgmap v18: 65 pgs: 65 active+clean; 1.6 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 50 MiB/s rd, 162 MiB/s wr, 317 op/s 2024-03-22T02:28:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:49 smithi103 ceph-mon[29258]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:28:49.874 INFO:tasks.workunit.client.0.smithi103.stdout:7/811: dread d1/d10/d11/d70/d8a/fc9 [0,4194304] 0 2024-03-22T02:28:49.874 INFO:tasks.workunit.client.0.smithi103.stdout:2/790: rename d11/d19/da7/dd3/f52 to d11/d19/d4e/dd5/f108 0 2024-03-22T02:28:49.876 INFO:tasks.workunit.client.0.smithi103.stdout:2/791: chown d11/d25/d46/d40/fed 330780948 1 2024-03-22T02:28:49.877 INFO:tasks.workunit.client.0.smithi103.stdout:7/812: unlink d1/d10/d11/d19/dc3/fea 0 2024-03-22T02:28:49.878 INFO:tasks.workunit.client.0.smithi103.stdout:2/792: rmdir d11/d25/d46/db6/dfb 39 2024-03-22T02:28:49.879 INFO:tasks.workunit.client.0.smithi103.stdout:5/835: dread d0/d14/d23/f11d [0,4194304] 0 2024-03-22T02:28:49.879 INFO:tasks.workunit.client.0.smithi103.stdout:2/793: readlink d11/d25/d63/d45/d47/dae/ldb 0 2024-03-22T02:28:49.880 INFO:tasks.workunit.client.0.smithi103.stdout:7/813: truncate d1/d10/d11/d70/d8a/ff0 412821 0 2024-03-22T02:28:49.880 INFO:tasks.workunit.client.1.smithi122.stdout:1/376: write dc/d55/f5f [3190631,101678] 0 2024-03-22T02:28:49.880 INFO:tasks.workunit.client.0.smithi103.stdout:1/977: rename db/d4a/d78/d92/dea/d134/d128/d132 to db/d4a/d78/d92/da8/ddd/d28/db2/d151 0 2024-03-22T02:28:49.882 INFO:tasks.workunit.client.1.smithi122.stdout:9/336: mknod db/df/d3e/c6e 0 2024-03-22T02:28:49.882 INFO:tasks.workunit.client.0.smithi103.stdout:7/814: chown d1/d10/d11/d19/d29/d33/d4b/c65 712408 1 2024-03-22T02:28:49.883 INFO:tasks.workunit.client.0.smithi103.stdout:5/836: truncate d0/d14/d23/d6f/f86 618132 0 2024-03-22T02:28:49.883 INFO:tasks.workunit.client.1.smithi122.stdout:1/377: symlink dc/d3b/d46/l84 0 2024-03-22T02:28:49.884 INFO:tasks.workunit.client.1.smithi122.stdout:9/337: read db/df/f15 [3760054,57811] 0 2024-03-22T02:28:49.884 INFO:tasks.workunit.client.0.smithi103.stdout:5/837: chown d0/d14/d23/d6f 155781766 1 2024-03-22T02:28:49.884 INFO:tasks.workunit.client.0.smithi103.stdout:2/794: read d11/d19/da7/dd3/f36 [2929340,58242] 0 2024-03-22T02:28:49.885 INFO:tasks.workunit.client.1.smithi122.stdout:7/275: symlink d0/d25/d46/d48/l54 0 2024-03-22T02:28:49.885 INFO:tasks.workunit.client.0.smithi103.stdout:1/978: mknod db/d4a/d78/d92/da8/ddd/d28/d97/c152 0 2024-03-22T02:28:49.886 INFO:tasks.workunit.client.0.smithi103.stdout:2/795: rmdir d11/d19/d5d 39 2024-03-22T02:28:49.886 INFO:tasks.workunit.client.1.smithi122.stdout:7/276: fsync d0/d4/fb 0 2024-03-22T02:28:49.887 INFO:tasks.workunit.client.1.smithi122.stdout:9/338: read db/d23/f3a [995405,105751] 0 2024-03-22T02:28:49.887 INFO:tasks.workunit.client.1.smithi122.stdout:7/277: read d0/f8 [809294,91504] 0 2024-03-22T02:28:49.887 INFO:tasks.workunit.client.0.smithi103.stdout:1/979: fsync db/d4a/d78/d92/dea/d134/fd7 0 2024-03-22T02:28:49.887 INFO:tasks.workunit.client.0.smithi103.stdout:7/815: creat d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/ffe x:0 0 0 2024-03-22T02:28:49.888 INFO:tasks.workunit.client.0.smithi103.stdout:2/796: chown d11/d25/d63/d45/d47/dae/fe3 1006754282 1 2024-03-22T02:28:49.888 INFO:tasks.workunit.client.1.smithi122.stdout:9/339: stat db/d23/d1f/d21/d25/d41 0 2024-03-22T02:28:49.889 INFO:tasks.workunit.client.0.smithi103.stdout:5/838: creat d0/d14/d23/d5c/d70/f127 x:0 0 0 2024-03-22T02:28:49.893 INFO:tasks.workunit.client.0.smithi103.stdout:7/816: unlink d1/d10/d11/d70/lbe 0 2024-03-22T02:28:49.893 INFO:tasks.workunit.client.0.smithi103.stdout:1/980: truncate db/d4a/d78/d92/da8/ddd/d18/d84/fc0 635773 0 2024-03-22T02:28:49.894 INFO:tasks.workunit.client.1.smithi122.stdout:7/278: creat d0/d4/d22/f55 x:0 0 0 2024-03-22T02:28:49.894 INFO:tasks.workunit.client.0.smithi103.stdout:1/981: fsync db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/fe2 0 2024-03-22T02:28:49.895 INFO:tasks.workunit.client.1.smithi122.stdout:3/171: dread d2/d9/dc/f16 [0,4194304] 0 2024-03-22T02:28:49.895 INFO:tasks.workunit.client.1.smithi122.stdout:7/279: stat d0/d4/c6 0 2024-03-22T02:28:49.895 INFO:tasks.workunit.client.0.smithi103.stdout:5/839: rename d0/f97 to d0/d14/d23/d2a/d33/db5/dfb/f128 0 2024-03-22T02:28:49.895 INFO:tasks.workunit.client.1.smithi122.stdout:9/340: mknod db/d23/d1f/c6f 0 2024-03-22T02:28:49.897 INFO:tasks.workunit.client.1.smithi122.stdout:3/172: chown d2/d9/dc/d21/l2e 1 1 2024-03-22T02:28:49.898 INFO:tasks.workunit.client.1.smithi122.stdout:7/280: creat d0/d25/f56 x:0 0 0 2024-03-22T02:28:49.898 INFO:tasks.workunit.client.1.smithi122.stdout:3/173: truncate d2/d9/f1a 54915 0 2024-03-22T02:28:49.898 INFO:tasks.workunit.client.1.smithi122.stdout:9/341: creat db/d23/d1e/f70 x:0 0 0 2024-03-22T02:28:49.900 INFO:tasks.workunit.client.1.smithi122.stdout:3/174: chown d2/d9/dc/d17/d28/l32 14558 1 2024-03-22T02:28:49.900 INFO:tasks.workunit.client.1.smithi122.stdout:9/342: fsync db/d23/d1f/d21/d29/f38 0 2024-03-22T02:28:49.901 INFO:tasks.workunit.client.1.smithi122.stdout:7/281: mkdir d0/d4/d20/d57 0 2024-03-22T02:28:49.901 INFO:tasks.workunit.client.1.smithi122.stdout:1/378: dread dc/f22 [0,4194304] 0 2024-03-22T02:28:49.902 INFO:tasks.workunit.client.1.smithi122.stdout:9/343: readlink db/l46 0 2024-03-22T02:28:49.902 INFO:tasks.workunit.client.1.smithi122.stdout:1/379: dread - dc/f80 zero size 2024-03-22T02:28:49.905 INFO:tasks.workunit.client.1.smithi122.stdout:3/175: rename d2/d9/dc/d21/f23 to d2/f34 0 2024-03-22T02:28:49.905 INFO:tasks.workunit.client.1.smithi122.stdout:7/282: dread d0/d4/d22/f32 [0,4194304] 0 2024-03-22T02:28:49.906 INFO:tasks.workunit.client.1.smithi122.stdout:9/344: symlink db/d23/d1f/d21/d3d/l71 0 2024-03-22T02:28:49.909 INFO:tasks.workunit.client.1.smithi122.stdout:7/283: mknod d0/d4/d20/d21/d28/c58 0 2024-03-22T02:28:49.912 INFO:tasks.workunit.client.0.smithi103.stdout:7/817: write d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f [868160,42673] 0 2024-03-22T02:28:49.912 INFO:tasks.workunit.client.1.smithi122.stdout:9/345: symlink db/d1b/d6c/l72 0 2024-03-22T02:28:49.915 INFO:tasks.workunit.client.0.smithi103.stdout:2/797: dwrite d11/f20 [0,4194304] 0 2024-03-22T02:28:49.915 INFO:tasks.workunit.client.0.smithi103.stdout:1/982: write db/d4a/d78/d92/da8/ddd/d18/d2d/f48 [2106176,64211] 0 2024-03-22T02:28:49.915 INFO:tasks.workunit.client.1.smithi122.stdout:1/380: dwrite dc/f1d [0,4194304] 0 2024-03-22T02:28:49.916 INFO:tasks.workunit.client.0.smithi103.stdout:7/818: write d1/d10/d11/d19/d29/d33/da8/ffd [803819,54292] 0 2024-03-22T02:28:49.918 INFO:tasks.workunit.client.0.smithi103.stdout:1/983: read - db/d4a/d78/d92/da8/ddd/d28/db2/dfd/f135 zero size 2024-03-22T02:28:49.919 INFO:tasks.workunit.client.0.smithi103.stdout:5/840: dread d0/d14/d23/d2a/d33/f118 [0,4194304] 0 2024-03-22T02:28:49.920 INFO:tasks.workunit.client.1.smithi122.stdout:7/284: dwrite d0/f2 [4194304,4194304] 0 2024-03-22T02:28:49.920 INFO:tasks.workunit.client.0.smithi103.stdout:7/819: mkdir d1/d10/d11/d70/d8a/d9d/dff 0 2024-03-22T02:28:49.922 INFO:tasks.workunit.client.1.smithi122.stdout:9/346: mkdir db/d23/d1f/d57/d66/d73 0 2024-03-22T02:28:49.922 INFO:tasks.workunit.client.0.smithi103.stdout:1/984: mkdir db/d4a/d78/d92/da8/ddd/d28/db2/dc8/d153 0 2024-03-22T02:28:49.924 INFO:tasks.workunit.client.0.smithi103.stdout:1/985: fdatasync db/f35 0 2024-03-22T02:28:49.925 INFO:tasks.workunit.client.0.smithi103.stdout:7/820: mkdir d1/d10/d11/d19/d29/d61/d69/d6b/d100 0 2024-03-22T02:28:49.927 INFO:tasks.workunit.client.1.smithi122.stdout:7/285: unlink d0/d4/d22/f43 0 2024-03-22T02:28:49.929 INFO:tasks.workunit.client.0.smithi103.stdout:1/986: symlink db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dd5/dff/l154 0 2024-03-22T02:28:49.930 INFO:tasks.workunit.client.1.smithi122.stdout:9/347: mkdir db/d1b/d6c/d74 0 2024-03-22T02:28:49.931 INFO:tasks.workunit.client.1.smithi122.stdout:7/286: mknod d0/d4/d1d/c59 0 2024-03-22T02:28:49.932 INFO:tasks.workunit.client.0.smithi103.stdout:7/821: rename d1/d10/d11/d19/d29/d61/fc0 to d1/d10/d11/d19/d29/d61/d1d/f101 0 2024-03-22T02:28:49.933 INFO:tasks.workunit.client.1.smithi122.stdout:3/176: dwrite d2/f15 [0,4194304] 0 2024-03-22T02:28:49.933 INFO:tasks.workunit.client.1.smithi122.stdout:9/348: symlink db/d1b/d6c/l75 0 2024-03-22T02:28:49.934 INFO:tasks.workunit.client.1.smithi122.stdout:7/287: symlink d0/d25/l5a 0 2024-03-22T02:28:49.935 INFO:tasks.workunit.client.1.smithi122.stdout:9/349: dread - db/d23/d1e/f70 zero size 2024-03-22T02:28:49.935 INFO:tasks.workunit.client.0.smithi103.stdout:1/987: rename db/d4a/d78/d92/da8/ddd/d2e/l10b to db/d4a/d78/d92/da8/ddd/d2e/dae/d129/l155 0 2024-03-22T02:28:49.936 INFO:tasks.workunit.client.1.smithi122.stdout:3/177: stat d2/d9/dc/fd 0 2024-03-22T02:28:49.938 INFO:tasks.workunit.client.1.smithi122.stdout:3/178: dread - d2/d9/dc/d17/d28/f31 zero size 2024-03-22T02:28:49.938 INFO:tasks.workunit.client.0.smithi103.stdout:2/798: dread d11/d25/d46/fc2 [0,4194304] 0 2024-03-22T02:28:49.941 INFO:tasks.workunit.client.1.smithi122.stdout:9/350: rmdir db/d1b/d6c/d74 0 2024-03-22T02:28:49.942 INFO:tasks.workunit.client.1.smithi122.stdout:3/179: fdatasync d2/d9/dc/d17/d28/f2d 0 2024-03-22T02:28:49.942 INFO:tasks.workunit.client.0.smithi103.stdout:7/822: dwrite d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/ff7 [0,4194304] 0 2024-03-22T02:28:49.943 INFO:tasks.workunit.client.0.smithi103.stdout:2/799: mknod d11/d19/da7/dd3/dd9/c109 0 2024-03-22T02:28:49.943 INFO:tasks.workunit.client.1.smithi122.stdout:1/381: dread dc/f11 [0,4194304] 0 2024-03-22T02:28:49.944 INFO:tasks.workunit.client.1.smithi122.stdout:3/180: write d2/f7 [39050,103847] 0 2024-03-22T02:28:49.947 INFO:tasks.workunit.client.1.smithi122.stdout:1/382: mknod dc/d3b/d46/d4d/d67/c85 0 2024-03-22T02:28:49.948 INFO:tasks.workunit.client.1.smithi122.stdout:3/181: dread d2/d9/f1a [0,4194304] 0 2024-03-22T02:28:49.948 INFO:tasks.workunit.client.1.smithi122.stdout:9/351: dread db/d23/d1f/d21/d25/d41/f54 [0,4194304] 0 2024-03-22T02:28:49.948 INFO:tasks.workunit.client.0.smithi103.stdout:1/988: dwrite db/d4a/d78/d92/da8/ddd/d28/db2/dfd/f11e [0,4194304] 0 2024-03-22T02:28:49.949 INFO:tasks.workunit.client.0.smithi103.stdout:7/823: truncate d1/d10/d11/d19/d29/d33/da8/dc5/ff9 675316 0 2024-03-22T02:28:49.949 INFO:tasks.workunit.client.0.smithi103.stdout:2/800: mkdir d11/d25/d63/d45/d10a 0 2024-03-22T02:28:49.951 INFO:tasks.workunit.client.0.smithi103.stdout:2/801: chown d11/d25/d46/f9c 26826 1 2024-03-22T02:28:49.951 INFO:tasks.workunit.client.0.smithi103.stdout:1/989: stat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/c123 0 2024-03-22T02:28:49.951 INFO:tasks.workunit.client.0.smithi103.stdout:7/824: fsync d1/d10/d11/d19/d29/d61/d1d/fcf 0 2024-03-22T02:28:49.954 INFO:tasks.workunit.client.1.smithi122.stdout:1/383: dread dc/d3b/d46/f6b [0,4194304] 0 2024-03-22T02:28:49.954 INFO:tasks.workunit.client.0.smithi103.stdout:2/802: symlink d11/d25/d63/l10b 0 2024-03-22T02:28:49.955 INFO:tasks.workunit.client.0.smithi103.stdout:7/825: creat d1/d10/d11/d19/dc3/de8/f102 x:0 0 0 2024-03-22T02:28:49.955 INFO:tasks.workunit.client.0.smithi103.stdout:2/803: readlink d11/d19/da7/dd3/l5f 0 2024-03-22T02:28:49.955 INFO:tasks.workunit.client.1.smithi122.stdout:9/352: rename db/d23/d1f/d21/f56 to db/d23/d1f/d57/d66/d73/f76 0 2024-03-22T02:28:49.958 INFO:tasks.workunit.client.1.smithi122.stdout:1/384: write dc/d3b/d46/f6b [584086,120301] 0 2024-03-22T02:28:49.959 INFO:tasks.workunit.client.1.smithi122.stdout:3/182: dwrite d2/f7 [0,4194304] 0 2024-03-22T02:28:49.960 INFO:tasks.workunit.client.1.smithi122.stdout:3/183: dread - d2/d9/dc/d17/d28/f31 zero size 2024-03-22T02:28:49.966 INFO:tasks.workunit.client.1.smithi122.stdout:3/184: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:49.967 INFO:tasks.workunit.client.1.smithi122.stdout:1/385: unlink dc/f30 0 2024-03-22T02:28:49.969 INFO:tasks.workunit.client.0.smithi103.stdout:1/990: write db/d4a/d78/d92/da8/ddd/d21/f33 [268631,35510] 0 2024-03-22T02:28:49.969 INFO:tasks.workunit.client.0.smithi103.stdout:2/804: write d11/d25/d46/fc2 [1217223,59617] 0 2024-03-22T02:28:49.971 INFO:tasks.workunit.client.1.smithi122.stdout:9/353: truncate db/d23/d1f/d21/d29/f38 3751267 0 2024-03-22T02:28:49.973 INFO:tasks.workunit.client.1.smithi122.stdout:3/185: fdatasync d2/f34 0 2024-03-22T02:28:49.973 INFO:tasks.workunit.client.0.smithi103.stdout:2/805: creat d11/f10c x:0 0 0 2024-03-22T02:28:49.973 INFO:tasks.workunit.client.1.smithi122.stdout:1/386: dwrite dc/f1d [0,4194304] 0 2024-03-22T02:28:49.974 INFO:tasks.workunit.client.0.smithi103.stdout:7/826: dwrite d1/d10/d11/d19/d29/d33/f5f [0,4194304] 0 2024-03-22T02:28:49.976 INFO:tasks.workunit.client.1.smithi122.stdout:9/354: mknod db/df/d3e/d59/c77 0 2024-03-22T02:28:49.976 INFO:tasks.workunit.client.0.smithi103.stdout:7/827: truncate d1/d10/d11/d19/dc3/de8/ff1 286516 0 2024-03-22T02:28:49.980 INFO:tasks.workunit.client.1.smithi122.stdout:1/387: mkdir dc/d2b/d86 0 2024-03-22T02:28:49.982 INFO:tasks.workunit.client.1.smithi122.stdout:9/355: mknod db/d23/d1f/d57/c78 0 2024-03-22T02:28:49.983 INFO:tasks.workunit.client.0.smithi103.stdout:9/795: sync 2024-03-22T02:28:49.983 INFO:tasks.workunit.client.0.smithi103.stdout:6/571: sync 2024-03-22T02:28:49.984 INFO:tasks.workunit.client.1.smithi122.stdout:3/186: link d2/d9/dc/d17/d28/f2d d2/d9/dc/d17/d28/f35 0 2024-03-22T02:28:49.985 INFO:tasks.workunit.client.0.smithi103.stdout:1/991: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/d11b/f156 x:0 0 0 2024-03-22T02:28:49.986 INFO:tasks.workunit.client.1.smithi122.stdout:1/388: symlink dc/d3b/d46/d4d/l87 0 2024-03-22T02:28:49.987 INFO:tasks.workunit.client.1.smithi122.stdout:9/356: mkdir db/d23/d1f/d21/d3d/d79 0 2024-03-22T02:28:49.987 INFO:tasks.workunit.client.1.smithi122.stdout:9/357: dread - db/d1b/d28/f6b zero size 2024-03-22T02:28:49.988 INFO:tasks.workunit.client.0.smithi103.stdout:9/796: truncate d2/da/d18/d1e/d34/d3d/d61/db7/fc3 738077 0 2024-03-22T02:28:49.988 INFO:tasks.workunit.client.0.smithi103.stdout:7/828: write d1/d10/d11/d19/d29/fbb [1695968,115741] 0 2024-03-22T02:28:49.988 INFO:tasks.workunit.client.1.smithi122.stdout:1/389: fsync dc/ff 0 2024-03-22T02:28:49.990 INFO:tasks.workunit.client.0.smithi103.stdout:6/572: creat d0/d3/fb8 x:0 0 0 2024-03-22T02:28:49.993 INFO:tasks.workunit.client.0.smithi103.stdout:7/829: creat d1/d10/f103 x:0 0 0 2024-03-22T02:28:49.993 INFO:tasks.workunit.client.1.smithi122.stdout:3/187: rename d2/f34 to d2/d9/dc/f36 0 2024-03-22T02:28:49.995 INFO:tasks.workunit.client.0.smithi103.stdout:9/797: dwrite d2/d5/d7b/d7d/dd4/dd8/fe2 [0,4194304] 0 2024-03-22T02:28:49.996 INFO:tasks.workunit.client.1.smithi122.stdout:1/390: creat dc/f88 x:0 0 0 2024-03-22T02:28:49.996 INFO:tasks.workunit.client.0.smithi103.stdout:9/798: chown d2/d5/d7b/d7d 348013421 1 2024-03-22T02:28:49.996 INFO:tasks.workunit.client.1.smithi122.stdout:1/391: readlink dc/d2b/l3a 0 2024-03-22T02:28:49.997 INFO:tasks.workunit.client.1.smithi122.stdout:3/188: symlink d2/d9/dc/l37 0 2024-03-22T02:28:49.998 INFO:tasks.workunit.client.0.smithi103.stdout:9/799: write d2/da/d18/d1e/d34/d3d/d61/db7/fd5 [91516,111861] 0 2024-03-22T02:28:50.000 INFO:tasks.workunit.client.1.smithi122.stdout:3/189: write d2/d9/dc/f16 [3321343,106960] 0 2024-03-22T02:28:50.001 INFO:tasks.workunit.client.1.smithi122.stdout:1/392: dread dc/d4c/f63 [0,4194304] 0 2024-03-22T02:28:50.002 INFO:tasks.workunit.client.1.smithi122.stdout:3/190: write d2/f15 [346577,7111] 0 2024-03-22T02:28:50.007 INFO:tasks.workunit.client.0.smithi103.stdout:6/573: dwrite d0/d3/f9 [4194304,4194304] 0 2024-03-22T02:28:50.007 INFO:tasks.workunit.client.0.smithi103.stdout:9/800: dwrite d2/da/d18/f6c [0,4194304] 0 2024-03-22T02:28:50.009 INFO:tasks.workunit.client.1.smithi122.stdout:9/358: dread db/d23/d1f/d21/f42 [0,4194304] 0 2024-03-22T02:28:50.009 INFO:tasks.workunit.client.1.smithi122.stdout:1/393: dwrite dc/f80 [0,4194304] 0 2024-03-22T02:28:50.010 INFO:tasks.workunit.client.1.smithi122.stdout:1/394: stat dc/d2b/d6e/l7f 0 2024-03-22T02:28:50.012 INFO:tasks.workunit.client.0.smithi103.stdout:9/801: rmdir d2/da/d18/d1e/d77/d8b 39 2024-03-22T02:28:50.013 INFO:tasks.workunit.client.0.smithi103.stdout:9/802: chown d2/da 1479 1 2024-03-22T02:28:50.018 INFO:tasks.workunit.client.1.smithi122.stdout:9/359: dwrite db/d1b/d28/f6b [0,4194304] 0 2024-03-22T02:28:50.019 INFO:tasks.workunit.client.0.smithi103.stdout:6/574: dwrite d0/fa2 [0,4194304] 0 2024-03-22T02:28:50.019 INFO:tasks.workunit.client.1.smithi122.stdout:9/360: stat db/d23/d1f/d21 0 2024-03-22T02:28:50.020 INFO:tasks.workunit.client.1.smithi122.stdout:9/361: chown db/d23/d1f/d21/d3d/c63 1 1 2024-03-22T02:28:50.021 INFO:tasks.workunit.client.1.smithi122.stdout:9/362: chown db/d23/d1f/d21/d29/l50 126828 1 2024-03-22T02:28:50.023 INFO:tasks.workunit.client.0.smithi103.stdout:6/575: fsync d0/d14/d57/d68/f2d 0 2024-03-22T02:28:50.028 INFO:tasks.workunit.client.0.smithi103.stdout:9/803: dwrite d2/d10/d59/f6d [0,4194304] 0 2024-03-22T02:28:50.029 INFO:tasks.workunit.client.0.smithi103.stdout:2/806: dread d11/d19/da7/dd3/f36 [0,4194304] 0 2024-03-22T02:28:50.031 INFO:tasks.workunit.client.1.smithi122.stdout:3/191: truncate d2/d9/dc/f36 4080006 0 2024-03-22T02:28:50.032 INFO:tasks.workunit.client.1.smithi122.stdout:9/363: read f7 [3418342,62465] 0 2024-03-22T02:28:50.034 INFO:tasks.workunit.client.1.smithi122.stdout:9/364: read - db/d23/d1f/d21/d25/f53 zero size 2024-03-22T02:28:50.036 INFO:tasks.workunit.client.0.smithi103.stdout:2/807: creat d11/d19/d5d/f10d x:0 0 0 2024-03-22T02:28:50.036 INFO:tasks.workunit.client.0.smithi103.stdout:9/804: read d2/f3b [374955,75606] 0 2024-03-22T02:28:50.038 INFO:tasks.workunit.client.0.smithi103.stdout:2/808: chown d11/d25/d63/d45/ff4 2 1 2024-03-22T02:28:50.040 INFO:tasks.workunit.client.0.smithi103.stdout:9/805: symlink d2/da/d18/d1e/d34/d3d/d61/da0/lea 0 2024-03-22T02:28:50.042 INFO:tasks.workunit.client.0.smithi103.stdout:2/809: creat d11/d25/d63/d45/ddd/f10e x:0 0 0 2024-03-22T02:28:50.044 INFO:tasks.workunit.client.0.smithi103.stdout:2/810: write d11/d25/d63/d45/ddd/f10e [904366,3770] 0 2024-03-22T02:28:50.046 INFO:tasks.workunit.client.1.smithi122.stdout:6/333: sync 2024-03-22T02:28:50.046 INFO:tasks.workunit.client.1.smithi122.stdout:4/280: sync 2024-03-22T02:28:50.047 INFO:tasks.workunit.client.0.smithi103.stdout:9/806: dwrite d2/d10/d59/da5/fb1 [4194304,4194304] 0 2024-03-22T02:28:50.049 INFO:tasks.workunit.client.0.smithi103.stdout:9/807: readlink d2/da/d18/d1e/d34/d3d/d61/da0/lea 0 2024-03-22T02:28:50.049 INFO:tasks.workunit.client.0.smithi103.stdout:2/811: creat d11/d25/d46/db6/dfb/f10f x:0 0 0 2024-03-22T02:28:50.051 INFO:tasks.workunit.client.0.smithi103.stdout:2/812: mkdir d11/d19/da7/dd3/dd9/d110 0 2024-03-22T02:28:50.052 INFO:tasks.workunit.client.0.smithi103.stdout:9/808: creat d2/d10/d43/d4b/d65/feb x:0 0 0 2024-03-22T02:28:50.053 INFO:tasks.workunit.client.1.smithi122.stdout:4/281: dwrite d0/f9 [4194304,4194304] 0 2024-03-22T02:28:50.059 INFO:tasks.workunit.client.0.smithi103.stdout:6/576: write d0/d3/d22/f15 [4650813,91670] 0 2024-03-22T02:28:50.059 INFO:tasks.workunit.client.1.smithi122.stdout:4/282: creat d0/d12/d15/d42/d4e/f5b x:0 0 0 2024-03-22T02:28:50.060 INFO:tasks.workunit.client.0.smithi103.stdout:6/577: dread - d0/d3/f92 zero size 2024-03-22T02:28:50.063 INFO:tasks.workunit.client.0.smithi103.stdout:6/578: mknod d0/d14/cb9 0 2024-03-22T02:28:50.064 INFO:tasks.workunit.client.0.smithi103.stdout:2/813: write d11/f13 [2932905,28878] 0 2024-03-22T02:28:50.065 INFO:tasks.workunit.client.0.smithi103.stdout:6/579: symlink d0/d14/d9a/lba 0 2024-03-22T02:28:50.066 INFO:tasks.workunit.client.0.smithi103.stdout:2/814: creat d11/d25/f111 x:0 0 0 2024-03-22T02:28:50.067 INFO:tasks.workunit.client.1.smithi122.stdout:9/365: dwrite db/d23/d1f/d21/d25/f55 [0,4194304] 0 2024-03-22T02:28:50.068 INFO:tasks.workunit.client.0.smithi103.stdout:6/580: write d0/d3/d22/f47 [4866580,56617] 0 2024-03-22T02:28:50.068 INFO:tasks.workunit.client.1.smithi122.stdout:4/283: dwrite d0/d12/d15/d42/d4e/f34 [0,4194304] 0 2024-03-22T02:28:50.069 INFO:tasks.workunit.client.0.smithi103.stdout:2/815: write d11/d25/d63/d45/ff4 [569668,64300] 0 2024-03-22T02:28:50.073 INFO:tasks.workunit.client.1.smithi122.stdout:9/366: creat db/d23/d1f/d57/d66/f7a x:0 0 0 2024-03-22T02:28:50.076 INFO:tasks.workunit.client.1.smithi122.stdout:9/367: readlink db/d1b/d6c/l44 0 2024-03-22T02:28:50.078 INFO:tasks.workunit.client.0.smithi103.stdout:6/581: dwrite d0/d14/d57/d68/f6b [0,4194304] 0 2024-03-22T02:28:50.078 INFO:tasks.workunit.client.1.smithi122.stdout:4/284: dwrite d0/d25/f43 [4194304,4194304] 0 2024-03-22T02:28:50.082 INFO:tasks.workunit.client.1.smithi122.stdout:4/285: rmdir d0/d12/d15/d42 39 2024-03-22T02:28:50.082 INFO:tasks.workunit.client.0.smithi103.stdout:6/582: truncate d0/d3/d33/f60 77042 0 2024-03-22T02:28:50.085 INFO:tasks.workunit.client.1.smithi122.stdout:4/286: creat d0/d12/d15/d3d/f5c x:0 0 0 2024-03-22T02:28:50.088 INFO:tasks.workunit.client.0.smithi103.stdout:6/583: mknod d0/d3/d33/cbb 0 2024-03-22T02:28:50.088 INFO:tasks.workunit.client.1.smithi122.stdout:6/334: dread d7/de/dd/d39/f4f [0,4194304] 0 2024-03-22T02:28:50.089 INFO:tasks.workunit.client.1.smithi122.stdout:4/287: write d0/d25/f1d [2206859,112782] 0 2024-03-22T02:28:50.089 INFO:tasks.workunit.client.1.smithi122.stdout:3/192: dwrite d2/d9/dc/f36 [4194304,4194304] 0 2024-03-22T02:28:50.090 INFO:tasks.workunit.client.0.smithi103.stdout:6/584: dread - d0/d3/d22/d58/d6f/f8b zero size 2024-03-22T02:28:50.090 INFO:tasks.workunit.client.0.smithi103.stdout:6/585: dread - d0/d3/d22/fab zero size 2024-03-22T02:28:50.095 INFO:tasks.workunit.client.0.smithi103.stdout:6/586: rename d0/d14/d57/d68/f71 to d0/d3/fbc 0 2024-03-22T02:28:50.096 INFO:tasks.workunit.client.1.smithi122.stdout:6/335: write d7/de/dd/d39/f4f [5125853,7119] 0 2024-03-22T02:28:50.097 INFO:tasks.workunit.client.0.smithi103.stdout:6/587: mkdir d0/dbd 0 2024-03-22T02:28:50.099 INFO:tasks.workunit.client.1.smithi122.stdout:4/288: dwrite d0/d25/f43 [4194304,4194304] 0 2024-03-22T02:28:50.100 INFO:tasks.workunit.client.1.smithi122.stdout:3/193: dread d2/f15 [0,4194304] 0 2024-03-22T02:28:50.100 INFO:tasks.workunit.client.1.smithi122.stdout:6/336: dread - d7/de/d27/d52/f57 zero size 2024-03-22T02:28:50.101 INFO:tasks.workunit.client.0.smithi103.stdout:9/809: dread d2/d10/d43/d7e/fc8 [0,4194304] 0 2024-03-22T02:28:50.101 INFO:tasks.workunit.client.0.smithi103.stdout:6/588: truncate d0/d3/d22/f5c 3399020 0 2024-03-22T02:28:50.104 INFO:tasks.workunit.client.1.smithi122.stdout:6/337: creat d7/de/dd/df/f6e x:0 0 0 2024-03-22T02:28:50.105 INFO:tasks.workunit.client.0.smithi103.stdout:9/810: unlink d2/da/d18/d1e/d34/l54 0 2024-03-22T02:28:50.105 INFO:tasks.workunit.client.1.smithi122.stdout:3/194: rename d2/d9/d24 to d2/d9/dc/df/d38 0 2024-03-22T02:28:50.106 INFO:tasks.workunit.client.1.smithi122.stdout:3/195: read d2/f15 [1906281,44361] 0 2024-03-22T02:28:50.106 INFO:tasks.workunit.client.1.smithi122.stdout:4/289: creat d0/d12/d15/d42/d4e/de/f5d x:0 0 0 2024-03-22T02:28:50.108 INFO:tasks.workunit.client.0.smithi103.stdout:9/811: fdatasync d2/d10/d43/d7e/da4/fe3 0 2024-03-22T02:28:50.109 INFO:tasks.workunit.client.0.smithi103.stdout:6/589: dwrite d0/d3/d22/d58/d6f/f94 [0,4194304] 0 2024-03-22T02:28:50.110 INFO:tasks.workunit.client.1.smithi122.stdout:6/338: getdents d7/de/dd/d23 0 2024-03-22T02:28:50.112 INFO:tasks.workunit.client.1.smithi122.stdout:3/196: creat d2/f39 x:0 0 0 2024-03-22T02:28:50.113 INFO:tasks.workunit.client.1.smithi122.stdout:4/290: creat d0/d25/f5e x:0 0 0 2024-03-22T02:28:50.114 INFO:tasks.workunit.client.1.smithi122.stdout:9/368: dread db/d23/d1f/d21/d29/f2c [0,4194304] 0 2024-03-22T02:28:50.115 INFO:tasks.workunit.client.0.smithi103.stdout:6/590: mkdir d0/d3/d22/d58/d6f/dbe 0 2024-03-22T02:28:50.115 INFO:tasks.workunit.client.1.smithi122.stdout:3/197: write d2/f7 [2462492,6983] 0 2024-03-22T02:28:50.115 INFO:tasks.workunit.client.1.smithi122.stdout:3/198: readlink d2/l13 0 2024-03-22T02:28:50.117 INFO:tasks.workunit.client.1.smithi122.stdout:4/291: symlink d0/d41/l5f 0 2024-03-22T02:28:50.117 INFO:tasks.workunit.client.0.smithi103.stdout:9/812: dwrite d2/da/d18/d1e/d34/d3d/d61/f9a [0,4194304] 0 2024-03-22T02:28:50.118 INFO:tasks.workunit.client.1.smithi122.stdout:4/292: truncate d0/d12/d15/f55 741294 0 2024-03-22T02:28:50.119 INFO:tasks.workunit.client.1.smithi122.stdout:9/369: mknod db/d23/d1f/d21/d25/c7b 0 2024-03-22T02:28:50.120 INFO:tasks.workunit.client.1.smithi122.stdout:9/370: fsync db/d1b/f6a 0 2024-03-22T02:28:50.122 INFO:tasks.workunit.client.1.smithi122.stdout:3/199: mknod d2/d9/dc/d17/d2b/c3a 0 2024-03-22T02:28:50.123 INFO:tasks.workunit.client.1.smithi122.stdout:3/200: write d2/f39 [759653,28484] 0 2024-03-22T02:28:50.124 INFO:tasks.workunit.client.1.smithi122.stdout:3/201: chown d2/f15 0 1 2024-03-22T02:28:50.125 INFO:tasks.workunit.client.0.smithi103.stdout:2/816: dread d11/d25/d63/d45/f79 [8388608,4194304] 0 2024-03-22T02:28:50.125 INFO:tasks.workunit.client.1.smithi122.stdout:3/202: write d2/f39 [684798,12932] 0 2024-03-22T02:28:50.125 INFO:tasks.workunit.client.0.smithi103.stdout:2/817: chown d11/d19/d4e/de9 1900079 1 2024-03-22T02:28:50.127 INFO:tasks.workunit.client.0.smithi103.stdout:2/818: chown d11/f58 247805061 1 2024-03-22T02:28:50.128 INFO:tasks.workunit.client.0.smithi103.stdout:2/819: chown d11/d25/d46/l2d 31446300 1 2024-03-22T02:28:50.129 INFO:tasks.workunit.client.1.smithi122.stdout:3/203: rename d2/d9/f1a to d2/d9/dc/df/d38/f3b 0 2024-03-22T02:28:50.130 INFO:tasks.workunit.client.1.smithi122.stdout:3/204: rmdir d2/d9/dc/d21 39 2024-03-22T02:28:50.131 INFO:tasks.workunit.client.0.smithi103.stdout:2/820: truncate d11/d25/d46/d40/f67 200823 0 2024-03-22T02:28:50.135 INFO:tasks.workunit.client.0.smithi103.stdout:3/916: sync 2024-03-22T02:28:50.138 INFO:tasks.workunit.client.1.smithi122.stdout:4/293: getdents d0/d25 0 2024-03-22T02:28:50.139 INFO:tasks.workunit.client.1.smithi122.stdout:6/339: truncate d7/de/dd/d20/f45 494111 0 2024-03-22T02:28:50.141 INFO:tasks.workunit.client.1.smithi122.stdout:3/205: getdents d2/d9/dc/d17/d2b 0 2024-03-22T02:28:50.142 INFO:tasks.workunit.client.0.smithi103.stdout:9/813: dread d2/d10/f8a [0,4194304] 0 2024-03-22T02:28:50.145 INFO:tasks.workunit.client.1.smithi122.stdout:6/340: creat d7/de/dd/d39/f6f x:0 0 0 2024-03-22T02:28:50.149 INFO:tasks.workunit.client.1.smithi122.stdout:4/294: dwrite d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:50.150 INFO:tasks.workunit.client.1.smithi122.stdout:3/206: dwrite d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:50.151 INFO:tasks.workunit.client.1.smithi122.stdout:6/341: symlink d7/de/dd/d39/l70 0 2024-03-22T02:28:50.151 INFO:tasks.workunit.client.1.smithi122.stdout:4/295: stat d0/d12/d15/d42/d4e/de/c20 0 2024-03-22T02:28:50.152 INFO:tasks.workunit.client.1.smithi122.stdout:3/207: truncate d2/f8 3986815 0 2024-03-22T02:28:50.153 INFO:tasks.workunit.client.1.smithi122.stdout:4/296: creat d0/d12/d15/d42/d4e/de/d40/f60 x:0 0 0 2024-03-22T02:28:50.154 INFO:tasks.workunit.client.1.smithi122.stdout:3/208: mkdir d2/d9/dc/df/d3c 0 2024-03-22T02:28:50.155 INFO:tasks.workunit.client.1.smithi122.stdout:3/209: fdatasync d2/f8 0 2024-03-22T02:28:50.155 INFO:tasks.workunit.client.1.smithi122.stdout:4/297: mkdir d0/d12/d61 0 2024-03-22T02:28:50.156 INFO:tasks.workunit.client.1.smithi122.stdout:4/298: write d0/d25/f27 [250064,98922] 0 2024-03-22T02:28:50.157 INFO:tasks.workunit.client.1.smithi122.stdout:4/299: chown d0/d12/d15/d42/d57 74144 1 2024-03-22T02:28:50.157 INFO:tasks.workunit.client.1.smithi122.stdout:3/210: symlink d2/l3d 0 2024-03-22T02:28:50.160 INFO:tasks.workunit.client.1.smithi122.stdout:3/211: symlink d2/d9/dc/df/d3c/l3e 0 2024-03-22T02:28:50.164 INFO:tasks.workunit.client.1.smithi122.stdout:3/212: rename d2/d9/dc/d21 to d2/d9/dc/d17/d2b/d3f 0 2024-03-22T02:28:50.171 INFO:tasks.workunit.client.0.smithi103.stdout:2/821: write d11/dc5/fe6 [1726129,16693] 0 2024-03-22T02:28:50.172 INFO:tasks.workunit.client.0.smithi103.stdout:3/917: dread de/d6b/d80/d73/d110/fdf [0,4194304] 0 2024-03-22T02:28:50.172 INFO:tasks.workunit.client.0.smithi103.stdout:6/591: dread d0/d3/d33/f5d [0,4194304] 0 2024-03-22T02:28:50.174 INFO:tasks.workunit.client.0.smithi103.stdout:6/592: stat d0/d3/c52 0 2024-03-22T02:28:50.176 INFO:tasks.workunit.client.0.smithi103.stdout:6/593: chown d0/d88/d8e/fb2 47492118 1 2024-03-22T02:28:50.176 INFO:tasks.workunit.client.1.smithi122.stdout:3/213: write d2/d9/dc/d17/d28/f35 [3962087,21073] 0 2024-03-22T02:28:50.177 INFO:tasks.workunit.client.0.smithi103.stdout:9/814: truncate d2/d5/d4c/f5c 3987301 0 2024-03-22T02:28:50.178 INFO:tasks.workunit.client.0.smithi103.stdout:3/918: write de/d12/d19/df2/f69 [3064631,51696] 0 2024-03-22T02:28:50.180 INFO:tasks.workunit.client.0.smithi103.stdout:2/822: truncate d11/d25/d46/d48/f9d 1361813 0 2024-03-22T02:28:50.181 INFO:tasks.workunit.client.1.smithi122.stdout:3/214: link d2/l3d d2/d9/dc/d17/l40 0 2024-03-22T02:28:50.181 INFO:tasks.workunit.client.0.smithi103.stdout:3/919: write de/d1a/d25/f79 [2258307,94871] 0 2024-03-22T02:28:50.182 INFO:tasks.workunit.client.0.smithi103.stdout:2/823: chown d11/d25/d63/d45/d47/dae/fea 249937 1 2024-03-22T02:28:50.184 INFO:tasks.workunit.client.1.smithi122.stdout:4/300: dread d0/d12/f31 [0,4194304] 0 2024-03-22T02:28:50.185 INFO:tasks.workunit.client.1.smithi122.stdout:3/215: dread d2/f39 [0,4194304] 0 2024-03-22T02:28:50.186 INFO:tasks.workunit.client.1.smithi122.stdout:3/216: stat d2/d9/c1c 0 2024-03-22T02:28:50.187 INFO:tasks.workunit.client.0.smithi103.stdout:3/920: mkdir de/d1a/d10f/d138 0 2024-03-22T02:28:50.187 INFO:tasks.workunit.client.1.smithi122.stdout:3/217: write d2/d9/dc/f16 [976824,125696] 0 2024-03-22T02:28:50.190 INFO:tasks.workunit.client.1.smithi122.stdout:3/218: creat d2/d9/dc/df/d3c/f41 x:0 0 0 2024-03-22T02:28:50.190 INFO:tasks.workunit.client.0.smithi103.stdout:3/921: creat de/d12/d5d/da9/db9/f139 x:0 0 0 2024-03-22T02:28:50.191 INFO:tasks.workunit.client.0.smithi103.stdout:3/922: write de/d12/d8a/d93/f11c [306546,49418] 0 2024-03-22T02:28:50.192 INFO:tasks.workunit.client.0.smithi103.stdout:3/923: chown de/d1a/d10f/f119 209 1 2024-03-22T02:28:50.194 INFO:tasks.workunit.client.1.smithi122.stdout:6/342: dread d7/de/dd/f16 [0,4194304] 0 2024-03-22T02:28:50.194 INFO:tasks.workunit.client.0.smithi103.stdout:3/924: fdatasync de/d6b/d80/d4f/f64 0 2024-03-22T02:28:50.194 INFO:tasks.workunit.client.1.smithi122.stdout:3/219: dread d2/f15 [0,4194304] 0 2024-03-22T02:28:50.196 INFO:tasks.workunit.client.0.smithi103.stdout:3/925: chown de/d12/l18 4296005 1 2024-03-22T02:28:50.198 INFO:tasks.workunit.client.1.smithi122.stdout:4/301: truncate d0/d12/f21 3330877 0 2024-03-22T02:28:50.199 INFO:tasks.workunit.client.1.smithi122.stdout:3/220: fdatasync d2/d9/dc/df/d38/f3b 0 2024-03-22T02:28:50.200 INFO:tasks.workunit.client.0.smithi103.stdout:2/824: write f2 [1893069,113102] 0 2024-03-22T02:28:50.201 INFO:tasks.workunit.client.1.smithi122.stdout:4/302: creat d0/d12/d15/d3d/f62 x:0 0 0 2024-03-22T02:28:50.201 INFO:tasks.workunit.client.0.smithi103.stdout:3/926: link de/d12/d19/df2/lc0 de/d6b/d80/d4f/d84/l13a 0 2024-03-22T02:28:50.202 INFO:tasks.workunit.client.1.smithi122.stdout:4/303: stat d0/d12/d15/d42/d4e/f22 0 2024-03-22T02:28:50.202 INFO:tasks.workunit.client.0.smithi103.stdout:3/927: dread - de/d12/d5d/d92/f132 zero size 2024-03-22T02:28:50.203 INFO:tasks.workunit.client.0.smithi103.stdout:2/825: creat d11/d25/d46/d40/f112 x:0 0 0 2024-03-22T02:28:50.204 INFO:tasks.workunit.client.0.smithi103.stdout:9/815: write d2/d5/d4c/f5c [3153798,75333] 0 2024-03-22T02:28:50.205 INFO:tasks.workunit.client.0.smithi103.stdout:9/816: chown d2/da/d18/dc4 529243943 1 2024-03-22T02:28:50.207 INFO:tasks.workunit.client.1.smithi122.stdout:4/304: symlink d0/d12/d15/d42/d4e/l63 0 2024-03-22T02:28:50.207 INFO:tasks.workunit.client.0.smithi103.stdout:9/817: unlink d2/d5/d4c/d56/l57 0 2024-03-22T02:28:50.208 INFO:tasks.workunit.client.1.smithi122.stdout:3/221: write d2/d9/dc/d17/d28/f31 [592121,68989] 0 2024-03-22T02:28:50.212 INFO:tasks.workunit.client.0.smithi103.stdout:9/818: rename d2/d10/d43/d7e/da4/dcd/f51 to d2/da/d18/d1e/d77/fec 0 2024-03-22T02:28:50.212 INFO:tasks.workunit.client.1.smithi122.stdout:3/222: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:50.213 INFO:tasks.workunit.client.1.smithi122.stdout:6/343: dread d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:50.213 INFO:tasks.workunit.client.0.smithi103.stdout:9/819: readlink d2/d10/d43/d7e/d96/d9d/la8 0 2024-03-22T02:28:50.214 INFO:tasks.workunit.client.0.smithi103.stdout:9/820: truncate d2/d5/d4c/f9f 670351 0 2024-03-22T02:28:50.215 INFO:tasks.workunit.client.1.smithi122.stdout:3/223: rmdir d2/d9/dc/df 39 2024-03-22T02:28:50.215 INFO:tasks.workunit.client.0.smithi103.stdout:3/928: dread de/d1a/d25/f42 [0,4194304] 0 2024-03-22T02:28:50.217 INFO:tasks.workunit.client.0.smithi103.stdout:3/929: write de/d12/d5d/d92/fce [29846,119670] 0 2024-03-22T02:28:50.218 INFO:tasks.workunit.client.1.smithi122.stdout:3/224: creat d2/d9/dc/d17/f42 x:0 0 0 2024-03-22T02:28:50.220 INFO:tasks.workunit.client.0.smithi103.stdout:9/821: dwrite d2/d10/d43/d7e/da4/dcd/f4f [0,4194304] 0 2024-03-22T02:28:50.220 INFO:tasks.workunit.client.1.smithi122.stdout:3/225: symlink d2/d9/dc/d17/l43 0 2024-03-22T02:28:50.221 INFO:tasks.workunit.client.1.smithi122.stdout:4/305: dread d0/d12/d15/d42/d4e/f23 [0,4194304] 0 2024-03-22T02:28:50.222 INFO:tasks.workunit.client.0.smithi103.stdout:3/930: creat de/d12/d30/d70/de5/f13b x:0 0 0 2024-03-22T02:28:50.223 INFO:tasks.workunit.client.0.smithi103.stdout:3/931: write de/d6b/f136 [537113,27181] 0 2024-03-22T02:28:50.224 INFO:tasks.workunit.client.1.smithi122.stdout:4/306: creat d0/d12/d15/d42/d57/f64 x:0 0 0 2024-03-22T02:28:50.225 INFO:tasks.workunit.client.1.smithi122.stdout:4/307: readlink d0/d12/d15/d42/d4e/de/l45 0 2024-03-22T02:28:50.226 INFO:tasks.workunit.client.1.smithi122.stdout:3/226: rename d2/l13 to d2/d9/dc/df/l44 0 2024-03-22T02:28:50.227 INFO:tasks.workunit.client.1.smithi122.stdout:4/308: symlink d0/d25/l65 0 2024-03-22T02:28:50.228 INFO:tasks.workunit.client.1.smithi122.stdout:4/309: write d0/d12/d15/f55 [871050,108185] 0 2024-03-22T02:28:50.229 INFO:tasks.workunit.client.1.smithi122.stdout:3/227: mkdir d2/d9/dc/d17/d45 0 2024-03-22T02:28:50.232 INFO:tasks.workunit.client.1.smithi122.stdout:4/310: creat d0/d12/d15/d42/d4e/de/f66 x:0 0 0 2024-03-22T02:28:50.233 INFO:tasks.workunit.client.1.smithi122.stdout:3/228: creat d2/d9/dc/d17/d45/f46 x:0 0 0 2024-03-22T02:28:50.233 INFO:tasks.workunit.client.1.smithi122.stdout:4/311: mknod d0/d12/d15/d42/c67 0 2024-03-22T02:28:50.235 INFO:tasks.workunit.client.1.smithi122.stdout:4/312: fsync d0/d12/d15/d42/d4e/de/f44 0 2024-03-22T02:28:50.236 INFO:tasks.workunit.client.1.smithi122.stdout:4/313: creat d0/d12/d15/d3d/f68 x:0 0 0 2024-03-22T02:28:50.240 INFO:tasks.workunit.client.1.smithi122.stdout:4/314: rename d0/d25/f43 to d0/d12/f69 0 2024-03-22T02:28:50.243 INFO:tasks.workunit.client.1.smithi122.stdout:8/329: sync 2024-03-22T02:28:50.244 INFO:tasks.workunit.client.1.smithi122.stdout:8/330: dread - d1/d4d/f58 zero size 2024-03-22T02:28:50.246 INFO:tasks.workunit.client.1.smithi122.stdout:8/331: dread d1/d8/d2f/f38 [0,4194304] 0 2024-03-22T02:28:50.247 INFO:tasks.workunit.client.1.smithi122.stdout:8/332: write d1/d22/f32 [2772620,68793] 0 2024-03-22T02:28:50.249 INFO:tasks.workunit.client.1.smithi122.stdout:8/333: chown d1/d9/l16 167339 1 2024-03-22T02:28:50.249 INFO:tasks.workunit.client.1.smithi122.stdout:4/315: getdents d0/d12/d15/d42/d4e 0 2024-03-22T02:28:50.252 INFO:tasks.workunit.client.1.smithi122.stdout:4/316: creat d0/f6a x:0 0 0 2024-03-22T02:28:50.253 INFO:tasks.workunit.client.1.smithi122.stdout:6/344: truncate d7/de/dd/f3a 1234321 0 2024-03-22T02:28:50.254 INFO:tasks.workunit.client.1.smithi122.stdout:6/345: chown d7/de/dd/df/f62 3434553 1 2024-03-22T02:28:50.260 INFO:tasks.workunit.client.1.smithi122.stdout:3/229: getdents d2 0 2024-03-22T02:28:50.260 INFO:tasks.workunit.client.0.smithi103.stdout:3/932: dread de/d1a/fde [0,4194304] 0 2024-03-22T02:28:50.260 INFO:tasks.workunit.client.1.smithi122.stdout:8/334: dread d1/d8/d2f/f44 [0,4194304] 0 2024-03-22T02:28:50.263 INFO:tasks.workunit.client.1.smithi122.stdout:4/317: dread d0/d25/f3a [0,4194304] 0 2024-03-22T02:28:50.264 INFO:tasks.workunit.client.1.smithi122.stdout:8/335: symlink d1/d9/d50/l6f 0 2024-03-22T02:28:50.264 INFO:tasks.workunit.client.1.smithi122.stdout:4/318: chown d0/d12/d15/d2d/l2e 3212 1 2024-03-22T02:28:50.265 INFO:tasks.workunit.client.1.smithi122.stdout:3/230: mkdir d2/d9/dc/df/d47 0 2024-03-22T02:28:50.265 INFO:tasks.workunit.client.1.smithi122.stdout:8/336: write d1/d3f/d45/f49 [3893730,56623] 0 2024-03-22T02:28:50.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/933: dread de/d12/d19/f71 [0,4194304] 0 2024-03-22T02:28:50.266 INFO:tasks.workunit.client.1.smithi122.stdout:6/346: truncate d7/de/dd/d20/f45 876577 0 2024-03-22T02:28:50.267 INFO:tasks.workunit.client.1.smithi122.stdout:8/337: fdatasync d1/d8/d5e/f65 0 2024-03-22T02:28:50.268 INFO:tasks.workunit.client.1.smithi122.stdout:8/338: readlink d1/l17 0 2024-03-22T02:28:50.268 INFO:tasks.workunit.client.1.smithi122.stdout:3/231: write d2/d9/dc/f36 [856373,71617] 0 2024-03-22T02:28:50.269 INFO:tasks.workunit.client.1.smithi122.stdout:4/319: mknod d0/d12/d15/d2d/c6b 0 2024-03-22T02:28:50.269 INFO:tasks.workunit.client.0.smithi103.stdout:3/934: creat de/d12/d8f/dfb/f13c x:0 0 0 2024-03-22T02:28:50.270 INFO:tasks.workunit.client.1.smithi122.stdout:4/320: fdatasync d0/d12/d15/d42/d57/f58 0 2024-03-22T02:28:50.270 INFO:tasks.workunit.client.0.smithi103.stdout:3/935: symlink de/d12/d8a/db0/d122/l13d 0 2024-03-22T02:28:50.271 INFO:tasks.workunit.client.1.smithi122.stdout:3/232: chown d2/d9/dc/d17/d28 140604 1 2024-03-22T02:28:50.271 INFO:tasks.workunit.client.1.smithi122.stdout:6/347: mknod d7/de/dd/d20/c71 0 2024-03-22T02:28:50.271 INFO:tasks.workunit.client.1.smithi122.stdout:3/233: truncate d2/f8 4701983 0 2024-03-22T02:28:50.272 INFO:tasks.workunit.client.1.smithi122.stdout:8/339: symlink d1/d3f/d45/l70 0 2024-03-22T02:28:50.272 INFO:tasks.workunit.client.0.smithi103.stdout:3/936: dread de/d12/d5d/d92/fce [0,4194304] 0 2024-03-22T02:28:50.273 INFO:tasks.workunit.client.0.smithi103.stdout:3/937: dread de/d12/d5d/d92/fce [0,4194304] 0 2024-03-22T02:28:50.274 INFO:tasks.workunit.client.0.smithi103.stdout:3/938: fsync de/d12/d30/f3d 0 2024-03-22T02:28:50.277 INFO:tasks.workunit.client.1.smithi122.stdout:8/340: readlink d1/l2a 0 2024-03-22T02:28:50.279 INFO:tasks.workunit.client.1.smithi122.stdout:6/348: creat d7/de/dd/d20/d2d/d54/f72 x:0 0 0 2024-03-22T02:28:50.281 INFO:tasks.workunit.client.1.smithi122.stdout:3/234: link d2/d9/dc/df/d3c/f41 d2/d9/dc/df/f48 0 2024-03-22T02:28:50.282 INFO:tasks.workunit.client.1.smithi122.stdout:3/235: chown d2/d9/dc/d17/d28 12970687 1 2024-03-22T02:28:50.284 INFO:tasks.workunit.client.1.smithi122.stdout:8/341: unlink d1/d22/c23 0 2024-03-22T02:28:50.288 INFO:tasks.workunit.client.1.smithi122.stdout:3/236: write d2/d9/dc/df/d3c/f41 [1029918,74040] 0 2024-03-22T02:28:50.288 INFO:tasks.workunit.client.1.smithi122.stdout:4/321: dread d0/d25/f13 [0,4194304] 0 2024-03-22T02:28:50.289 INFO:tasks.workunit.client.1.smithi122.stdout:8/342: mkdir d1/d40/d71 0 2024-03-22T02:28:50.290 INFO:tasks.workunit.client.1.smithi122.stdout:3/237: write d2/d9/dc/d17/d28/f31 [1476455,40849] 0 2024-03-22T02:28:50.294 INFO:tasks.workunit.client.1.smithi122.stdout:8/343: rmdir d1/d3f/d45 39 2024-03-22T02:28:50.295 INFO:tasks.workunit.client.1.smithi122.stdout:6/349: dread d7/de/dd/f42 [0,4194304] 0 2024-03-22T02:28:50.296 INFO:tasks.workunit.client.1.smithi122.stdout:3/238: mkdir d2/d9/dc/df/d49 0 2024-03-22T02:28:50.297 INFO:tasks.workunit.client.1.smithi122.stdout:8/344: creat d1/d8/d5e/f72 x:0 0 0 2024-03-22T02:28:50.297 INFO:tasks.workunit.client.1.smithi122.stdout:8/345: read d1/d8/d2f/f38 [810305,22175] 0 2024-03-22T02:28:50.298 INFO:tasks.workunit.client.1.smithi122.stdout:3/239: write d2/d9/dc/df/d3c/f41 [1415199,116101] 0 2024-03-22T02:28:50.298 INFO:tasks.workunit.client.1.smithi122.stdout:6/350: dread - d7/de/dd/d20/f51 zero size 2024-03-22T02:28:50.299 INFO:tasks.workunit.client.1.smithi122.stdout:3/240: chown d2/d9/dc/f36 252348631 1 2024-03-22T02:28:50.301 INFO:tasks.workunit.client.1.smithi122.stdout:6/351: write f3 [4958774,28083] 0 2024-03-22T02:28:50.303 INFO:tasks.workunit.client.1.smithi122.stdout:8/346: unlink d1/d22/f2e 0 2024-03-22T02:28:50.307 INFO:tasks.workunit.client.1.smithi122.stdout:8/347: mkdir d1/d60/d73 0 2024-03-22T02:28:50.308 INFO:tasks.workunit.client.1.smithi122.stdout:6/352: dwrite d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:50.309 INFO:tasks.workunit.client.1.smithi122.stdout:6/353: truncate d7/de/f43 1879470 0 2024-03-22T02:28:50.309 INFO:tasks.workunit.client.1.smithi122.stdout:8/348: mknod d1/d8/d63/c74 0 2024-03-22T02:28:50.311 INFO:tasks.workunit.client.1.smithi122.stdout:6/354: read - d7/de/dd/d20/d2d/d54/f59 zero size 2024-03-22T02:28:50.313 INFO:tasks.workunit.client.1.smithi122.stdout:8/349: creat d1/d8/f75 x:0 0 0 2024-03-22T02:28:50.313 INFO:tasks.workunit.client.1.smithi122.stdout:8/350: fdatasync d1/d8/f5c 0 2024-03-22T02:28:50.314 INFO:tasks.workunit.client.1.smithi122.stdout:8/351: stat d1/d8/d2f/c3b 0 2024-03-22T02:28:50.315 INFO:tasks.workunit.client.1.smithi122.stdout:8/352: chown d1/d9/d50 13 1 2024-03-22T02:28:50.316 INFO:tasks.workunit.client.1.smithi122.stdout:3/241: truncate d2/d9/dc/f36 7634619 0 2024-03-22T02:28:50.317 INFO:tasks.workunit.client.1.smithi122.stdout:8/353: creat d1/d40/d48/f76 x:0 0 0 2024-03-22T02:28:50.317 INFO:tasks.workunit.client.1.smithi122.stdout:6/355: creat d7/de/dd/d39/f73 x:0 0 0 2024-03-22T02:28:50.318 INFO:tasks.workunit.client.1.smithi122.stdout:8/354: chown d1/d40/f47 144815060 1 2024-03-22T02:28:50.321 INFO:tasks.workunit.client.1.smithi122.stdout:3/242: getdents d2/d9/dc/d17 0 2024-03-22T02:28:50.322 INFO:tasks.workunit.client.1.smithi122.stdout:8/355: creat d1/d9/f77 x:0 0 0 2024-03-22T02:28:50.324 INFO:tasks.workunit.client.1.smithi122.stdout:6/356: getdents d7/de/dd 0 2024-03-22T02:28:50.327 INFO:tasks.workunit.client.1.smithi122.stdout:6/357: mknod d7/de/dd/c74 0 2024-03-22T02:28:50.329 INFO:tasks.workunit.client.1.smithi122.stdout:8/356: dread d1/d8/f36 [0,4194304] 0 2024-03-22T02:28:50.329 INFO:tasks.workunit.client.1.smithi122.stdout:3/243: dwrite d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:50.333 INFO:tasks.workunit.client.1.smithi122.stdout:8/357: mkdir d1/d8/d2f/d43/d78 0 2024-03-22T02:28:50.336 INFO:tasks.workunit.client.1.smithi122.stdout:6/358: dwrite d7/de/f6c [0,4194304] 0 2024-03-22T02:28:50.339 INFO:tasks.workunit.client.1.smithi122.stdout:6/359: symlink d7/de/d27/d52/l75 0 2024-03-22T02:28:50.341 INFO:tasks.workunit.client.1.smithi122.stdout:3/244: dwrite d2/d9/dc/df/d3c/f41 [0,4194304] 0 2024-03-22T02:28:50.344 INFO:tasks.workunit.client.1.smithi122.stdout:3/245: truncate d2/d9/dc/d17/d45/f46 385460 0 2024-03-22T02:28:50.346 INFO:tasks.workunit.client.1.smithi122.stdout:3/246: mknod d2/d9/dc/df/d49/c4a 0 2024-03-22T02:28:50.349 INFO:tasks.workunit.client.1.smithi122.stdout:3/247: link d2/d9/dc/df/c29 d2/d9/dc/d17/c4b 0 2024-03-22T02:28:50.350 INFO:tasks.workunit.client.1.smithi122.stdout:3/248: mkdir d2/d9/dc/d17/d28/d4c 0 2024-03-22T02:28:50.352 INFO:tasks.workunit.client.1.smithi122.stdout:3/249: creat d2/d9/dc/d17/d28/f4d x:0 0 0 2024-03-22T02:28:50.354 INFO:tasks.workunit.client.1.smithi122.stdout:3/250: mknod d2/d9/dc/df/d47/c4e 0 2024-03-22T02:28:50.356 INFO:tasks.workunit.client.1.smithi122.stdout:3/251: creat d2/d9/dc/df/d49/f4f x:0 0 0 2024-03-22T02:28:50.361 INFO:tasks.workunit.client.1.smithi122.stdout:3/252: dwrite d2/d9/dc/df/d3c/f41 [0,4194304] 0 2024-03-22T02:28:50.388 INFO:tasks.workunit.client.1.smithi122.stdout:6/360: read d7/de/dd/f42 [926887,12300] 0 2024-03-22T02:28:50.392 INFO:tasks.workunit.client.1.smithi122.stdout:6/361: rename d7/de/dd/d39/d46 to d7/de/dd/d20/d2d/d54/d76 0 2024-03-22T02:28:50.392 INFO:tasks.workunit.client.1.smithi122.stdout:6/362: chown d7/de/d27/d52 0 1 2024-03-22T02:28:50.393 INFO:tasks.workunit.client.1.smithi122.stdout:6/363: write d7/de/dd/d20/d2d/d54/f72 [1047687,48683] 0 2024-03-22T02:28:50.444 INFO:tasks.workunit.client.1.smithi122.stdout:5/274: sync 2024-03-22T02:28:50.445 INFO:tasks.workunit.client.1.smithi122.stdout:5/275: symlink d8/da/d28/d42/d4e/l60 0 2024-03-22T02:28:50.446 INFO:tasks.workunit.client.1.smithi122.stdout:5/276: stat d8/d3b 0 2024-03-22T02:28:50.454 INFO:tasks.workunit.client.1.smithi122.stdout:8/358: truncate d1/d22/f52 6367384 0 2024-03-22T02:28:50.455 INFO:tasks.workunit.client.1.smithi122.stdout:5/277: link d8/d1e/d22/d3a/d46/l4d d8/d1e/d22/d3a/d39/l61 0 2024-03-22T02:28:50.462 INFO:tasks.workunit.client.1.smithi122.stdout:8/359: dwrite d1/d40/f5b [0,4194304] 0 2024-03-22T02:28:50.465 INFO:tasks.workunit.client.1.smithi122.stdout:8/360: symlink d1/d40/d48/l79 0 2024-03-22T02:28:50.469 INFO:tasks.workunit.client.1.smithi122.stdout:8/361: link d1/d3f/f41 d1/d40/f7a 0 2024-03-22T02:28:50.470 INFO:tasks.workunit.client.1.smithi122.stdout:6/364: dread d7/de/f3e [0,4194304] 0 2024-03-22T02:28:50.474 INFO:tasks.workunit.client.1.smithi122.stdout:8/362: rename d1/f2d to d1/d22/f7b 0 2024-03-22T02:28:50.475 INFO:tasks.workunit.client.1.smithi122.stdout:8/363: fsync d1/d22/f55 0 2024-03-22T02:28:50.475 INFO:tasks.workunit.client.1.smithi122.stdout:5/278: write d8/da/f16 [1888605,63043] 0 2024-03-22T02:28:50.479 INFO:tasks.workunit.client.1.smithi122.stdout:5/279: creat d8/d3b/f62 x:0 0 0 2024-03-22T02:28:50.485 INFO:tasks.workunit.client.1.smithi122.stdout:6/365: truncate d7/de/dd/d20/f3f 867367 0 2024-03-22T02:28:50.485 INFO:tasks.workunit.client.1.smithi122.stdout:5/280: dwrite d8/d1e/d22/f47 [0,4194304] 0 2024-03-22T02:28:50.487 INFO:tasks.workunit.client.1.smithi122.stdout:8/364: link d1/d9/l16 d1/d60/d73/l7c 0 2024-03-22T02:28:50.491 INFO:tasks.workunit.client.0.smithi103.stdout:0/826: sync 2024-03-22T02:28:50.491 INFO:tasks.workunit.client.0.smithi103.stdout:4/819: sync 2024-03-22T02:28:50.492 INFO:tasks.workunit.client.0.smithi103.stdout:8/948: sync 2024-03-22T02:28:50.492 INFO:tasks.workunit.client.1.smithi122.stdout:5/281: getdents d8/d3b 0 2024-03-22T02:28:50.492 INFO:tasks.workunit.client.1.smithi122.stdout:6/366: getdents d7/de/dd/d20/d2d/d54 0 2024-03-22T02:28:50.494 INFO:tasks.workunit.client.0.smithi103.stdout:0/827: chown dd/d16/d4b/d96/d64/dab/dc6/l3d 5019181 1 2024-03-22T02:28:50.495 INFO:tasks.workunit.client.0.smithi103.stdout:0/828: chown dd/d34/d53/d8c/fae 4494 1 2024-03-22T02:28:50.495 INFO:tasks.workunit.client.0.smithi103.stdout:4/820: mkdir d3/d1e/d25/d65/dc9/d111 0 2024-03-22T02:28:50.496 INFO:tasks.workunit.client.0.smithi103.stdout:0/829: stat dd/d34/d53/d8c 0 2024-03-22T02:28:50.496 INFO:tasks.workunit.client.1.smithi122.stdout:5/282: creat d8/d1e/d22/d3a/d46/d4f/f63 x:0 0 0 2024-03-22T02:28:50.496 INFO:tasks.workunit.client.0.smithi103.stdout:8/949: mkdir d4/d19/d1a/dc3/d132 0 2024-03-22T02:28:50.497 INFO:tasks.workunit.client.0.smithi103.stdout:8/950: readlink d4/d19/d1a/l5b 0 2024-03-22T02:28:50.498 INFO:tasks.workunit.client.0.smithi103.stdout:4/821: creat d3/d38/d5c/d90/f112 x:0 0 0 2024-03-22T02:28:50.498 INFO:tasks.workunit.client.1.smithi122.stdout:6/367: unlink d7/de/dd/d39/f4a 0 2024-03-22T02:28:50.499 INFO:tasks.workunit.client.0.smithi103.stdout:0/830: mkdir dd/d16/d10a 0 2024-03-22T02:28:50.500 INFO:tasks.workunit.client.0.smithi103.stdout:8/951: symlink d4/d26/d27/da3/d117/l133 0 2024-03-22T02:28:50.501 INFO:tasks.workunit.client.0.smithi103.stdout:8/952: chown d4/d2a/d3e 10777223 1 2024-03-22T02:28:50.501 INFO:tasks.workunit.client.0.smithi103.stdout:4/822: read d3/d38/fdf [1420940,48069] 0 2024-03-22T02:28:50.503 INFO:tasks.workunit.client.0.smithi103.stdout:4/823: write d3/d38/d5c/d90/df8/ffb [895283,29603] 0 2024-03-22T02:28:50.503 INFO:tasks.workunit.client.1.smithi122.stdout:8/365: truncate d1/f3d 1280155 0 2024-03-22T02:28:50.504 INFO:tasks.workunit.client.0.smithi103.stdout:0/831: truncate dd/d16/d4b/d96/d29/f5a 230220 0 2024-03-22T02:28:50.506 INFO:tasks.workunit.client.1.smithi122.stdout:8/366: mknod d1/d22/d6a/c7d 0 2024-03-22T02:28:50.506 INFO:tasks.workunit.client.0.smithi103.stdout:4/824: creat d3/d1e/d25/d65/dc9/f113 x:0 0 0 2024-03-22T02:28:50.507 INFO:tasks.workunit.client.1.smithi122.stdout:6/368: dwrite d7/de/dd/d20/d2d/d54/d76/f4e [0,4194304] 0 2024-03-22T02:28:50.507 INFO:tasks.workunit.client.0.smithi103.stdout:0/832: mknod dd/d16/d4b/d96/d29/d48/d5c/d102/c10b 0 2024-03-22T02:28:50.508 INFO:tasks.workunit.client.0.smithi103.stdout:4/825: fsync d3/d1e/d25/d65/d83/da1/fad 0 2024-03-22T02:28:50.509 INFO:tasks.workunit.client.0.smithi103.stdout:0/833: fsync dd/d34/f43 0 2024-03-22T02:28:50.511 INFO:tasks.workunit.client.0.smithi103.stdout:8/953: dread d4/d26/d27/d7e/d39/d7b/dc7/ff3 [0,4194304] 0 2024-03-22T02:28:50.512 INFO:tasks.workunit.client.0.smithi103.stdout:4/826: mknod d3/d1e/d53/d7a/da3/c114 0 2024-03-22T02:28:50.513 INFO:tasks.workunit.client.1.smithi122.stdout:8/367: symlink d1/d3f/l7e 0 2024-03-22T02:28:50.513 INFO:tasks.workunit.client.0.smithi103.stdout:4/827: fsync d3/d38/fda 0 2024-03-22T02:28:50.517 INFO:tasks.workunit.client.1.smithi122.stdout:6/369: creat d7/de/dd/d39/f77 x:0 0 0 2024-03-22T02:28:50.518 INFO:tasks.workunit.client.1.smithi122.stdout:5/283: dwrite d8/d1a/f34 [0,4194304] 0 2024-03-22T02:28:50.518 INFO:tasks.workunit.client.1.smithi122.stdout:8/368: mknod d1/d40/d71/c7f 0 2024-03-22T02:28:50.519 INFO:tasks.workunit.client.0.smithi103.stdout:0/834: rename dd/d16/c7f to dd/d16/d4b/d96/d29/d48/d5c/d103/c10c 0 2024-03-22T02:28:50.520 INFO:tasks.workunit.client.0.smithi103.stdout:0/835: stat dd/d16/d4b/d96/d29/d48/d5c/d102 0 2024-03-22T02:28:50.527 INFO:tasks.workunit.client.1.smithi122.stdout:8/369: dwrite d1/d22/f55 [0,4194304] 0 2024-03-22T02:28:50.529 INFO:tasks.workunit.client.0.smithi103.stdout:0/836: dread dd/d34/d53/d8c/fae [0,4194304] 0 2024-03-22T02:28:50.530 INFO:tasks.workunit.client.0.smithi103.stdout:0/837: chown dd/d16/d4b/d96/d35/d82/caa 835643 1 2024-03-22T02:28:50.532 INFO:tasks.workunit.client.1.smithi122.stdout:8/370: rename d1/d40/d48/f76 to d1/d40/d71/f80 0 2024-03-22T02:28:50.533 INFO:tasks.workunit.client.1.smithi122.stdout:8/371: dread - d1/d4d/f56 zero size 2024-03-22T02:28:50.533 INFO:tasks.workunit.client.0.smithi103.stdout:0/838: stat dd/d16/d4b/d96/f23 0 2024-03-22T02:28:50.534 INFO:tasks.workunit.client.0.smithi103.stdout:0/839: unlink dd/c52 0 2024-03-22T02:28:50.537 INFO:tasks.workunit.client.0.smithi103.stdout:0/840: mknod dd/d16/d4b/d96/d64/c10d 0 2024-03-22T02:28:50.538 INFO:tasks.workunit.client.1.smithi122.stdout:5/284: dwrite d8/da/d28/f44 [4194304,4194304] 0 2024-03-22T02:28:50.539 INFO:tasks.workunit.client.0.smithi103.stdout:0/841: getdents dd/d16/d4b/d96/d29/d48/d5c/d9b 0 2024-03-22T02:28:50.540 INFO:tasks.workunit.client.1.smithi122.stdout:8/372: dwrite d1/d9/f1f [0,4194304] 0 2024-03-22T02:28:50.540 INFO:tasks.workunit.client.0.smithi103.stdout:0/842: stat dd/d16/d4b/d96/d64/d6a/ca2 0 2024-03-22T02:28:50.541 INFO:tasks.workunit.client.0.smithi103.stdout:0/843: write dd/d16/d4b/ff6 [612604,79041] 0 2024-03-22T02:28:50.543 INFO:tasks.workunit.client.0.smithi103.stdout:0/844: creat dd/d34/d53/d8c/f10e x:0 0 0 2024-03-22T02:28:50.544 INFO:tasks.workunit.client.1.smithi122.stdout:8/373: chown d1/d3f/d45/f68 3527610 1 2024-03-22T02:28:50.544 INFO:tasks.workunit.client.1.smithi122.stdout:8/374: chown d1/d4d 36935400 1 2024-03-22T02:28:50.548 INFO:tasks.workunit.client.0.smithi103.stdout:8/954: dread d4/d2a/f94 [0,4194304] 0 2024-03-22T02:28:50.549 INFO:tasks.workunit.client.1.smithi122.stdout:5/285: dwrite d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:28:50.549 INFO:tasks.workunit.client.0.smithi103.stdout:0/845: getdents dd/d16/d4b/d96/d29/d48/ded 0 2024-03-22T02:28:50.550 INFO:tasks.workunit.client.0.smithi103.stdout:0/846: write dd/d16/d4b/d96/f68 [4156030,125606] 0 2024-03-22T02:28:50.551 INFO:tasks.workunit.client.0.smithi103.stdout:8/955: mkdir d4/d26/d27/d134 0 2024-03-22T02:28:50.552 INFO:tasks.workunit.client.0.smithi103.stdout:0/847: chown dd/d16/d4b/d96/d64/d6a 1357913 1 2024-03-22T02:28:50.553 INFO:tasks.workunit.client.1.smithi122.stdout:5/286: rename d8/d1a/f2f to d8/d1e/d22/d3a/d46/f64 0 2024-03-22T02:28:50.554 INFO:tasks.workunit.client.0.smithi103.stdout:8/956: write d4/d26/d27/d7e/d39/d49/d79/d11e/d35/fa0 [3078742,9246] 0 2024-03-22T02:28:50.555 INFO:tasks.workunit.client.0.smithi103.stdout:0/848: truncate dd/d16/d4b/d96/f5d 918339 0 2024-03-22T02:28:50.556 INFO:tasks.workunit.client.1.smithi122.stdout:5/287: unlink d8/d1e/d22/d3a/d39/f49 0 2024-03-22T02:28:50.558 INFO:tasks.workunit.client.1.smithi122.stdout:5/288: write d8/d1e/f5e [429609,82369] 0 2024-03-22T02:28:50.558 INFO:tasks.workunit.client.0.smithi103.stdout:8/957: creat d4/f135 x:0 0 0 2024-03-22T02:28:50.558 INFO:tasks.workunit.client.0.smithi103.stdout:0/849: dread dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 [0,4194304] 0 2024-03-22T02:28:50.559 INFO:tasks.workunit.client.1.smithi122.stdout:5/289: truncate d8/d1e/f5e 569627 0 2024-03-22T02:28:50.560 INFO:tasks.workunit.client.0.smithi103.stdout:8/958: fsync d4/d19/f9b 0 2024-03-22T02:28:50.560 INFO:tasks.workunit.client.1.smithi122.stdout:5/290: write d8/d1e/f5e [1284833,101627] 0 2024-03-22T02:28:50.561 INFO:tasks.workunit.client.0.smithi103.stdout:0/850: read dd/d16/d4b/d96/d29/f3e [2320958,74011] 0 2024-03-22T02:28:50.561 INFO:tasks.workunit.client.1.smithi122.stdout:5/291: chown d8/d1e/d22/d3a/d46/d4f/c50 16002 1 2024-03-22T02:28:50.563 INFO:tasks.workunit.client.0.smithi103.stdout:8/959: dread d4/d26/d27/d7e/d39/d49/d79/fb9 [0,4194304] 0 2024-03-22T02:28:50.564 INFO:tasks.workunit.client.0.smithi103.stdout:8/960: write f2 [198264,16272] 0 2024-03-22T02:28:50.566 INFO:tasks.workunit.client.1.smithi122.stdout:5/292: link d8/d1a/c56 d8/da/d28/d42/c65 0 2024-03-22T02:28:50.566 INFO:tasks.workunit.client.0.smithi103.stdout:8/961: creat d4/d26/d27/da3/d117/f136 x:0 0 0 2024-03-22T02:28:50.567 INFO:tasks.workunit.client.1.smithi122.stdout:8/375: write d1/d22/f52 [7352006,7369] 0 2024-03-22T02:28:50.569 INFO:tasks.workunit.client.1.smithi122.stdout:5/293: symlink d8/d1e/d22/l66 0 2024-03-22T02:28:50.569 INFO:tasks.workunit.client.0.smithi103.stdout:8/962: mknod d4/d26/d27/d7e/d39/d49/d51/daf/d127/c137 0 2024-03-22T02:28:50.570 INFO:tasks.workunit.client.1.smithi122.stdout:8/376: mkdir d1/d81 0 2024-03-22T02:28:50.572 INFO:tasks.workunit.client.1.smithi122.stdout:5/294: symlink d8/d1e/l67 0 2024-03-22T02:28:50.572 INFO:tasks.workunit.client.1.smithi122.stdout:5/295: chown d8/d1e/f5e 226963 1 2024-03-22T02:28:50.574 INFO:tasks.workunit.client.1.smithi122.stdout:5/296: rename d8/d3b/f45 to d8/da/d28/d35/f68 0 2024-03-22T02:28:50.577 INFO:tasks.workunit.client.1.smithi122.stdout:5/297: rename d8/d3b/f57 to d8/d3b/f69 0 2024-03-22T02:28:50.579 INFO:tasks.workunit.client.1.smithi122.stdout:5/298: mkdir d8/d1e/d22/d3a/d46/d4f/d55/d6a 0 2024-03-22T02:28:50.583 INFO:tasks.workunit.client.1.smithi122.stdout:5/299: dread d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:28:50.584 INFO:tasks.workunit.client.1.smithi122.stdout:5/300: chown d8/l1f 33222788 1 2024-03-22T02:28:50.586 INFO:tasks.workunit.client.1.smithi122.stdout:5/301: creat d8/d1e/d22/d3a/d39/f6b x:0 0 0 2024-03-22T02:28:50.589 INFO:tasks.workunit.client.1.smithi122.stdout:8/377: dread d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:50.590 INFO:tasks.workunit.client.1.smithi122.stdout:5/302: link d8/d1e/l67 d8/d1e/d22/d3a/d38/l6c 0 2024-03-22T02:28:50.590 INFO:tasks.workunit.client.1.smithi122.stdout:8/378: write d1/d8/d2f/f62 [427265,5838] 0 2024-03-22T02:28:50.594 INFO:tasks.workunit.client.1.smithi122.stdout:5/303: rename d8/d1e/d22/d3a/d46/l4d to d8/da/d28/d42/l6d 0 2024-03-22T02:28:50.596 INFO:tasks.workunit.client.1.smithi122.stdout:5/304: write d8/da/d28/d42/d4e/f58 [669222,101288] 0 2024-03-22T02:28:50.597 INFO:tasks.workunit.client.1.smithi122.stdout:5/305: write d8/d1a/f5c [56071,92819] 0 2024-03-22T02:28:50.599 INFO:tasks.workunit.client.1.smithi122.stdout:5/306: symlink d8/d1e/l6e 0 2024-03-22T02:28:50.606 INFO:tasks.workunit.client.1.smithi122.stdout:8/379: dread d1/d40/f4a [0,4194304] 0 2024-03-22T02:28:50.606 INFO:tasks.workunit.client.1.smithi122.stdout:8/380: fsync d1/d3f/d45/f49 0 2024-03-22T02:28:50.608 INFO:tasks.workunit.client.1.smithi122.stdout:8/381: creat d1/d8/d2f/d43/d78/f82 x:0 0 0 2024-03-22T02:28:50.611 INFO:tasks.workunit.client.1.smithi122.stdout:8/382: getdents d1/d40 0 2024-03-22T02:28:50.613 INFO:tasks.workunit.client.1.smithi122.stdout:8/383: mknod d1/d40/d48/c83 0 2024-03-22T02:28:50.620 INFO:tasks.workunit.client.1.smithi122.stdout:8/384: dwrite d1/f3d [0,4194304] 0 2024-03-22T02:28:50.623 INFO:tasks.workunit.client.0.smithi103.stdout:6/594: sync 2024-03-22T02:28:50.623 INFO:tasks.workunit.client.0.smithi103.stdout:1/992: sync 2024-03-22T02:28:50.623 INFO:tasks.workunit.client.0.smithi103.stdout:7/830: sync 2024-03-22T02:28:50.623 INFO:tasks.workunit.client.0.smithi103.stdout:5/841: sync 2024-03-22T02:28:50.624 INFO:tasks.workunit.client.1.smithi122.stdout:8/385: write d1/d22/f7b [559975,106196] 0 2024-03-22T02:28:50.624 INFO:tasks.workunit.client.0.smithi103.stdout:7/831: chown d1/dfa 159 1 2024-03-22T02:28:50.625 INFO:tasks.workunit.client.0.smithi103.stdout:1/993: chown db/d4a/d78/d92/fc5 6 1 2024-03-22T02:28:50.626 INFO:tasks.workunit.client.0.smithi103.stdout:7/832: chown d1/d10/d11/d19/d29/d61/d69/la4 111210805 1 2024-03-22T02:28:50.627 INFO:tasks.workunit.client.0.smithi103.stdout:1/994: chown db/d4a/d78/d92/da8/ddd/d18/d2d/l47 4 1 2024-03-22T02:28:50.637 INFO:tasks.workunit.client.0.smithi103.stdout:7/833: rename d1/d10/d80 to d1/d10/d11/d70/d8a/d9d/dad/d104 0 2024-03-22T02:28:50.639 INFO:tasks.workunit.client.0.smithi103.stdout:5/842: dwrite d0/da2/fb2 [0,4194304] 0 2024-03-22T02:28:50.640 INFO:tasks.workunit.client.0.smithi103.stdout:6/595: dread d0/d14/f56 [0,4194304] 0 2024-03-22T02:28:50.643 INFO:tasks.workunit.client.0.smithi103.stdout:1/995: creat db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dfe/d101/d14b/f157 x:0 0 0 2024-03-22T02:28:50.643 INFO:tasks.workunit.client.1.smithi122.stdout:8/386: dread d1/d8/f33 [4194304,4194304] 0 2024-03-22T02:28:50.644 INFO:tasks.workunit.client.1.smithi122.stdout:8/387: truncate d1/d22/f55 4638918 0 2024-03-22T02:28:50.645 INFO:tasks.workunit.client.0.smithi103.stdout:5/843: dread d0/da2/fb2 [0,4194304] 0 2024-03-22T02:28:50.645 INFO:tasks.workunit.client.0.smithi103.stdout:7/834: mkdir d1/d10/d11/d19/d29/d33/d4b/d5e/d105 0 2024-03-22T02:28:50.645 INFO:tasks.workunit.client.1.smithi122.stdout:8/388: write d1/d9/f77 [664070,106469] 0 2024-03-22T02:28:50.645 INFO:tasks.workunit.client.0.smithi103.stdout:8/963: link d4/d19/d1a/ce6 d4/d19/d1a/c138 0 2024-03-22T02:28:50.647 INFO:tasks.workunit.client.1.smithi122.stdout:8/389: symlink d1/d4d/l84 0 2024-03-22T02:28:50.647 INFO:tasks.workunit.client.1.smithi122.stdout:8/390: dread - d1/d8/f75 zero size 2024-03-22T02:28:50.648 INFO:tasks.workunit.client.0.smithi103.stdout:6/596: truncate d0/d14/d57/d68/f55 371919 0 2024-03-22T02:28:50.649 INFO:tasks.workunit.client.1.smithi122.stdout:8/391: readlink d1/l17 0 2024-03-22T02:28:50.649 INFO:tasks.workunit.client.0.smithi103.stdout:1/996: creat db/d4a/d78/d92/da8/ddd/d18/d98/f158 x:0 0 0 2024-03-22T02:28:50.651 INFO:tasks.workunit.client.0.smithi103.stdout:7/835: creat d1/d10/d11/d19/d86/f106 x:0 0 0 2024-03-22T02:28:50.651 INFO:tasks.workunit.client.1.smithi122.stdout:8/392: link d1/l27 d1/d8/d63/l85 0 2024-03-22T02:28:50.652 INFO:tasks.workunit.client.0.smithi103.stdout:5/844: creat d0/d14/d23/d5c/d77/df2/f129 x:0 0 0 2024-03-22T02:28:50.654 INFO:tasks.workunit.client.0.smithi103.stdout:8/964: truncate d4/d2a/f6c 2243788 0 2024-03-22T02:28:50.656 INFO:tasks.workunit.client.0.smithi103.stdout:8/965: write d4/d26/d27/d7e/dc8/f10b [454427,127544] 0 2024-03-22T02:28:50.657 INFO:tasks.workunit.client.0.smithi103.stdout:1/997: mkdir db/d4a/d78/d92/da8/dc9/d159 0 2024-03-22T02:28:50.660 INFO:tasks.workunit.client.0.smithi103.stdout:6/597: rename d0/d3/d22/c65 to d0/d14/cbf 0 2024-03-22T02:28:50.661 INFO:tasks.workunit.client.0.smithi103.stdout:1/998: mknod db/d4a/d78/d92/da8/ddd/d29/d41/d68/d74/dba/dd6/c15a 0 2024-03-22T02:28:50.663 INFO:tasks.workunit.client.0.smithi103.stdout:1/999: write db/d4a/d78/d92/da8/ddd/d18/dca/fdc [735102,95860] 0 2024-03-22T02:28:50.663 INFO:tasks.workunit.client.0.smithi103.stdout:7/836: rename d1/d10/d11/d19/d29/d61/d69/d5c/l81 to d1/d10/d11/d70/d8a/d9d/l107 0 2024-03-22T02:28:50.666 INFO:tasks.workunit.client.0.smithi103.stdout:8/966: rename d4/fd to d4/d19/f139 0 2024-03-22T02:28:50.668 INFO:tasks.workunit.client.0.smithi103.stdout:6/598: getdents d0/d14/d57 0 2024-03-22T02:28:50.669 INFO:tasks.workunit.client.0.smithi103.stdout:7/837: mkdir d1/d10/d11/d19/d29/d33/d4b/d5e/d105/d108 0 2024-03-22T02:28:50.671 INFO:tasks.workunit.client.0.smithi103.stdout:5/845: dread d0/d14/d23/d5c/d70/f8f [4194304,4194304] 0 2024-03-22T02:28:50.671 INFO:tasks.workunit.client.1.smithi122.stdout:0/288: sync 2024-03-22T02:28:50.672 INFO:tasks.workunit.client.1.smithi122.stdout:1/395: sync 2024-03-22T02:28:50.672 INFO:tasks.workunit.client.1.smithi122.stdout:7/288: sync 2024-03-22T02:28:50.672 INFO:tasks.workunit.client.1.smithi122.stdout:2/323: sync 2024-03-22T02:28:50.673 INFO:tasks.workunit.client.0.smithi103.stdout:7/838: dread d1/d10/d11/d70/d8a/fc9 [0,4194304] 0 2024-03-22T02:28:50.674 INFO:tasks.workunit.client.0.smithi103.stdout:5/846: mknod d0/d14/d23/d2a/d33/db5/dfb/c12a 0 2024-03-22T02:28:50.674 INFO:tasks.workunit.client.1.smithi122.stdout:7/289: chown d0/d4/d1d/c35 44 1 2024-03-22T02:28:50.675 INFO:tasks.workunit.client.1.smithi122.stdout:7/290: dread - d0/f1b zero size 2024-03-22T02:28:50.676 INFO:tasks.workunit.client.0.smithi103.stdout:5/847: unlink d0/d14/d23/d2a/d1f/f120 0 2024-03-22T02:28:50.677 INFO:tasks.workunit.client.1.smithi122.stdout:1/396: symlink dc/d3b/d46/d4d/l89 0 2024-03-22T02:28:50.677 INFO:tasks.workunit.client.1.smithi122.stdout:8/393: dwrite d1/d40/f47 [0,4194304] 0 2024-03-22T02:28:50.678 INFO:tasks.workunit.client.0.smithi103.stdout:5/848: creat d0/d14/d23/d5c/df3/f12b x:0 0 0 2024-03-22T02:28:50.679 INFO:tasks.workunit.client.0.smithi103.stdout:5/849: write d0/d14/d23/d36/d95/f5a [2040546,45211] 0 2024-03-22T02:28:50.681 INFO:tasks.workunit.client.0.smithi103.stdout:5/850: write d0/d14/d23/d36/fb6 [1175304,53993] 0 2024-03-22T02:28:50.684 INFO:tasks.workunit.client.1.smithi122.stdout:2/324: dwrite df/d1f/d2a/d39/f62 [0,4194304] 0 2024-03-22T02:28:50.685 INFO:tasks.workunit.client.0.smithi103.stdout:5/851: mkdir d0/da2/d12c 0 2024-03-22T02:28:50.690 INFO:tasks.workunit.client.1.smithi122.stdout:7/291: dwrite d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:50.691 INFO:tasks.workunit.client.1.smithi122.stdout:8/394: dwrite d1/f3c [0,4194304] 0 2024-03-22T02:28:50.693 INFO:tasks.workunit.client.0.smithi103.stdout:7/839: symlink d1/d10/d11/d19/d86/l109 0 2024-03-22T02:28:50.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/395: fsync d1/d40/f7a 0 2024-03-22T02:28:50.697 INFO:tasks.workunit.client.0.smithi103.stdout:7/840: mkdir d1/d10/d11/d19/d29/d61/d10a 0 2024-03-22T02:28:50.699 INFO:tasks.workunit.client.1.smithi122.stdout:2/325: dwrite df/d1f/f3c [0,4194304] 0 2024-03-22T02:28:50.700 INFO:tasks.workunit.client.0.smithi103.stdout:7/841: rmdir d1 39 2024-03-22T02:28:50.701 INFO:tasks.workunit.client.1.smithi122.stdout:8/396: creat d1/d8/d2f/d43/f86 x:0 0 0 2024-03-22T02:28:50.701 INFO:tasks.workunit.client.1.smithi122.stdout:8/397: fdatasync d1/d8/f5c 0 2024-03-22T02:28:50.702 INFO:tasks.workunit.client.1.smithi122.stdout:2/326: creat df/d1f/d33/d4d/f77 x:0 0 0 2024-03-22T02:28:50.703 INFO:tasks.workunit.client.1.smithi122.stdout:2/327: chown df/c1b 113 1 2024-03-22T02:28:50.707 INFO:tasks.workunit.client.1.smithi122.stdout:1/397: truncate dc/d3b/d46/d47/f59 3948182 0 2024-03-22T02:28:50.712 INFO:tasks.workunit.client.0.smithi103.stdout:8/967: symlink d4/d19/d1a/l13a 0 2024-03-22T02:28:50.712 INFO:tasks.workunit.client.1.smithi122.stdout:2/328: dwrite df/d1f/f5c [0,4194304] 0 2024-03-22T02:28:50.713 INFO:tasks.workunit.client.1.smithi122.stdout:2/329: chown df/l35 31146 1 2024-03-22T02:28:50.713 INFO:tasks.workunit.client.1.smithi122.stdout:2/330: chown df/d1f/d33/d6a 9417 1 2024-03-22T02:28:50.715 INFO:tasks.workunit.client.0.smithi103.stdout:8/968: mknod d4/d26/d27/d7e/dc8/c13b 0 2024-03-22T02:28:50.715 INFO:tasks.workunit.client.1.smithi122.stdout:7/292: truncate d0/f2 3077207 0 2024-03-22T02:28:50.718 INFO:tasks.workunit.client.1.smithi122.stdout:2/331: unlink df/d1f/f21 0 2024-03-22T02:28:50.718 INFO:tasks.workunit.client.0.smithi103.stdout:8/969: unlink d4/d19/f9b 0 2024-03-22T02:28:50.719 INFO:tasks.workunit.client.1.smithi122.stdout:7/293: creat d0/d30/f5b x:0 0 0 2024-03-22T02:28:50.719 INFO:tasks.workunit.client.1.smithi122.stdout:7/294: fdatasync d0/d30/f52 0 2024-03-22T02:28:50.721 INFO:tasks.workunit.client.0.smithi103.stdout:5/852: dwrite d0/d14/d23/d36/d95/d96/daa/daf/fc6 [0,4194304] 0 2024-03-22T02:28:50.721 INFO:tasks.workunit.client.0.smithi103.stdout:5/853: stat d0/d3e/f9e 0 2024-03-22T02:28:50.724 INFO:tasks.workunit.client.0.smithi103.stdout:8/970: symlink d4/d26/d27/d7e/d39/l13c 0 2024-03-22T02:28:50.724 INFO:tasks.workunit.client.0.smithi103.stdout:7/842: write d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/fd1 [1373102,97763] 0 2024-03-22T02:28:50.725 INFO:tasks.workunit.client.0.smithi103.stdout:5/854: mknod d0/d14/d23/d5c/d77/df2/dd9/c12d 0 2024-03-22T02:28:50.737 INFO:tasks.workunit.client.0.smithi103.stdout:5/855: dread d0/d14/d23/d5c/d70/fc4 [0,4194304] 0 2024-03-22T02:28:50.739 INFO:tasks.workunit.client.0.smithi103.stdout:5/856: symlink d0/d14/d23/d36/d95/d123/l12e 0 2024-03-22T02:28:50.745 INFO:tasks.workunit.client.0.smithi103.stdout:7/843: truncate d1/da/fe 148437 0 2024-03-22T02:28:50.746 INFO:tasks.workunit.client.0.smithi103.stdout:8/971: truncate d4/d26/d27/d7e/d39/d49/d79/d11e/d35/fa0 302425 0 2024-03-22T02:28:50.747 INFO:tasks.workunit.client.0.smithi103.stdout:8/972: fdatasync d4/d26/d27/da3/df5/d12e/f118 0 2024-03-22T02:28:50.747 INFO:tasks.workunit.client.0.smithi103.stdout:7/844: mkdir d1/d10b 0 2024-03-22T02:28:50.748 INFO:tasks.workunit.client.0.smithi103.stdout:5/857: write d0/da2/dca/fba [950307,18185] 0 2024-03-22T02:28:50.748 INFO:tasks.workunit.client.0.smithi103.stdout:8/973: fdatasync d4/d26/d27/da3/f126 0 2024-03-22T02:28:50.752 INFO:tasks.workunit.client.0.smithi103.stdout:2/826: sync 2024-03-22T02:28:50.752 INFO:tasks.workunit.client.0.smithi103.stdout:3/939: sync 2024-03-22T02:28:50.752 INFO:tasks.workunit.client.0.smithi103.stdout:9/822: sync 2024-03-22T02:28:50.754 INFO:tasks.workunit.client.0.smithi103.stdout:5/858: dread - d0/d14/d23/f10e zero size 2024-03-22T02:28:50.755 INFO:tasks.workunit.client.0.smithi103.stdout:8/974: stat d4/d19/d1a/l13a 0 2024-03-22T02:28:50.758 INFO:tasks.workunit.client.1.smithi122.stdout:1/398: dread dc/d3b/f68 [0,4194304] 0 2024-03-22T02:28:50.758 INFO:tasks.workunit.client.0.smithi103.stdout:9/823: creat d2/d10/d43/d7e/fed x:0 0 0 2024-03-22T02:28:50.760 INFO:tasks.workunit.client.0.smithi103.stdout:2/827: mknod d11/d25/d63/d45/d47/d83/d78/c113 0 2024-03-22T02:28:50.762 INFO:tasks.workunit.client.1.smithi122.stdout:1/399: link dc/d2b/f2d dc/d3b/d46/d47/f8a 0 2024-03-22T02:28:50.763 INFO:tasks.workunit.client.1.smithi122.stdout:1/400: creat dc/d4c/f8b x:0 0 0 2024-03-22T02:28:50.763 INFO:tasks.workunit.client.0.smithi103.stdout:8/975: creat d4/d19/d1a/dc3/f13d x:0 0 0 2024-03-22T02:28:50.764 INFO:tasks.workunit.client.0.smithi103.stdout:9/824: creat d2/da/d18/d1e/d34/d3d/fee x:0 0 0 2024-03-22T02:28:50.766 INFO:tasks.workunit.client.0.smithi103.stdout:7/845: write d1/d10/f103 [585586,12660] 0 2024-03-22T02:28:50.766 INFO:tasks.workunit.client.1.smithi122.stdout:1/401: rename dc/f40 to dc/d2b/f8c 0 2024-03-22T02:28:50.767 INFO:tasks.workunit.client.0.smithi103.stdout:2/828: symlink d11/d25/d46/d48/l114 0 2024-03-22T02:28:50.767 INFO:tasks.workunit.client.1.smithi122.stdout:1/402: read dc/d3b/d46/f6b [848398,72849] 0 2024-03-22T02:28:50.767 INFO:tasks.workunit.client.0.smithi103.stdout:9/825: truncate d2/d5/d7b/d7d/dd4/dd8/fe2 5163991 0 2024-03-22T02:28:50.769 INFO:tasks.workunit.client.0.smithi103.stdout:2/829: read d11/d25/d63/d45/ddd/f10e [409401,91303] 0 2024-03-22T02:28:50.769 INFO:tasks.workunit.client.0.smithi103.stdout:3/940: mkdir de/d1a/d10f/d9c/d13e 0 2024-03-22T02:28:50.769 INFO:tasks.workunit.client.1.smithi122.stdout:1/403: rename dc/d2b/d6e to dc/d3b/d8d 0 2024-03-22T02:28:50.770 INFO:tasks.workunit.client.0.smithi103.stdout:3/941: stat de/d1a/d10f/c4b 0 2024-03-22T02:28:50.773 INFO:tasks.workunit.client.1.smithi122.stdout:1/404: rename dc/d3b/d8d/l7f to dc/d2b/l8e 0 2024-03-22T02:28:50.774 INFO:tasks.workunit.client.1.smithi122.stdout:1/405: creat dc/d3b/d8d/f8f x:0 0 0 2024-03-22T02:28:50.774 INFO:tasks.workunit.client.0.smithi103.stdout:8/976: dwrite d4/d26/d27/d7e/d39/d49/d79/d11e/d40/fba [0,4194304] 0 2024-03-22T02:28:50.776 INFO:tasks.workunit.client.0.smithi103.stdout:9/826: rename d2/d10/d59/f5e to d2/d5/d7b/d7d/dd4/dd8/fef 0 2024-03-22T02:28:50.776 INFO:tasks.workunit.client.0.smithi103.stdout:5/859: write d0/d14/d23/d2a/d1f/f68 [415991,89630] 0 2024-03-22T02:28:50.777 INFO:tasks.workunit.client.0.smithi103.stdout:2/830: symlink d11/d25/d63/d45/d47/l115 0 2024-03-22T02:28:50.779 INFO:tasks.workunit.client.0.smithi103.stdout:3/942: mkdir de/d12/d5d/d102/d13f 0 2024-03-22T02:28:50.783 INFO:tasks.workunit.client.0.smithi103.stdout:9/827: dwrite d2/da/f23 [0,4194304] 0 2024-03-22T02:28:50.785 INFO:tasks.workunit.client.0.smithi103.stdout:5/860: creat d0/d14/d23/d36/d95/d96/daa/f12f x:0 0 0 2024-03-22T02:28:50.785 INFO:tasks.workunit.client.0.smithi103.stdout:3/943: write de/d12/d5d/d92/fce [829859,90152] 0 2024-03-22T02:28:50.790 INFO:tasks.workunit.client.0.smithi103.stdout:8/977: rename d4/d26/d27/da3/df5/d105 to d4/d26/d13e 0 2024-03-22T02:28:50.792 INFO:tasks.workunit.client.0.smithi103.stdout:7/846: dread d1/da/fe [0,4194304] 0 2024-03-22T02:28:50.795 INFO:tasks.workunit.client.0.smithi103.stdout:5/861: mknod d0/d14/d23/d6f/d83/c130 0 2024-03-22T02:28:50.798 INFO:tasks.workunit.client.0.smithi103.stdout:8/978: dwrite d4/d26/d27/da3/df5/d12e/f88 [0,4194304] 0 2024-03-22T02:28:50.798 INFO:tasks.workunit.client.0.smithi103.stdout:2/831: link d11/d25/d46/d48/lf2 d11/d25/d63/d45/d47/d83/d78/l116 0 2024-03-22T02:28:50.800 INFO:tasks.workunit.client.0.smithi103.stdout:8/979: fdatasync d4/d26/d27/da3/df5/d12e/f129 0 2024-03-22T02:28:50.800 INFO:tasks.workunit.client.0.smithi103.stdout:7/847: symlink d1/d10/d11/d19/d29/d61/d69/d6b/d7a/dc4/l10c 0 2024-03-22T02:28:50.803 INFO:tasks.workunit.client.0.smithi103.stdout:5/862: creat d0/d14/d23/d5c/d77/f131 x:0 0 0 2024-03-22T02:28:50.806 INFO:tasks.workunit.client.0.smithi103.stdout:2/832: mknod d11/d25/d63/d45/d47/dae/c117 0 2024-03-22T02:28:50.806 INFO:tasks.workunit.client.0.smithi103.stdout:4/828: sync 2024-03-22T02:28:50.806 INFO:tasks.workunit.client.0.smithi103.stdout:0/851: sync 2024-03-22T02:28:50.806 INFO:tasks.workunit.client.0.smithi103.stdout:3/944: write de/d12/d19/f71 [2003055,112901] 0 2024-03-22T02:28:50.809 INFO:tasks.workunit.client.0.smithi103.stdout:9/828: dread d2/d5/d4c/d56/f95 [0,4194304] 0 2024-03-22T02:28:50.810 INFO:tasks.workunit.client.0.smithi103.stdout:5/863: write d0/d14/d23/d2a/d33/db9/ff5 [209314,28030] 0 2024-03-22T02:28:50.810 INFO:tasks.workunit.client.0.smithi103.stdout:4/829: write d3/d1e/d25/d65/f96 [1492995,69998] 0 2024-03-22T02:28:50.812 INFO:tasks.workunit.client.0.smithi103.stdout:2/833: write d11/d25/d63/d45/d47/d83/fd7 [610336,633] 0 2024-03-22T02:28:50.812 INFO:tasks.workunit.client.0.smithi103.stdout:8/980: rename d4/d19/d1a/d4a/cbe to d4/d2a/d38/c13f 0 2024-03-22T02:28:50.822 INFO:tasks.workunit.client.0.smithi103.stdout:4/830: chown d3/lf 219145465 1 2024-03-22T02:28:50.823 INFO:tasks.workunit.client.0.smithi103.stdout:2/834: creat d11/d19/da7/dd3/f118 x:0 0 0 2024-03-22T02:28:50.824 INFO:tasks.workunit.client.0.smithi103.stdout:5/864: creat d0/d14/d23/d5c/d77/df2/dd9/f132 x:0 0 0 2024-03-22T02:28:50.826 INFO:tasks.workunit.client.0.smithi103.stdout:2/835: stat d11/d25/d46/db6/lbb 0 2024-03-22T02:28:50.829 INFO:tasks.workunit.client.0.smithi103.stdout:3/945: dwrite de/d12/d30/d70/de5/f13b [0,4194304] 0 2024-03-22T02:28:50.830 INFO:tasks.workunit.client.0.smithi103.stdout:0/852: write dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/fcf [273646,73901] 0 2024-03-22T02:28:50.834 INFO:tasks.workunit.client.0.smithi103.stdout:7/848: dread d1/d10/d11/d19/d29/d61/f24 [0,4194304] 0 2024-03-22T02:28:50.842 INFO:tasks.workunit.client.0.smithi103.stdout:4/831: dread d3/d1e/f2f [0,4194304] 0 2024-03-22T02:28:50.855 INFO:tasks.workunit.client.1.smithi122.stdout:3/253: sync 2024-03-22T02:28:50.855 INFO:tasks.workunit.client.1.smithi122.stdout:4/322: sync 2024-03-22T02:28:50.855 INFO:tasks.workunit.client.1.smithi122.stdout:9/371: sync 2024-03-22T02:28:50.856 INFO:tasks.workunit.client.1.smithi122.stdout:9/372: readlink db/d23/d1f/d21/d3d/l47 0 2024-03-22T02:28:50.858 INFO:tasks.workunit.client.0.smithi103.stdout:8/981: rename d4/d2a/d4f/fd9 to d4/d19/d1a/d4a/f140 0 2024-03-22T02:28:50.858 INFO:tasks.workunit.client.1.smithi122.stdout:3/254: symlink d2/d9/dc/d17/d28/d4c/l50 0 2024-03-22T02:28:50.859 INFO:tasks.workunit.client.1.smithi122.stdout:9/373: symlink db/df/l7c 0 2024-03-22T02:28:50.860 INFO:tasks.workunit.client.1.smithi122.stdout:4/323: truncate d0/d12/d15/d42/d4e/fb 1093511 0 2024-03-22T02:28:50.862 INFO:tasks.workunit.client.0.smithi103.stdout:3/946: mkdir de/d12/d8f/d140 0 2024-03-22T02:28:50.862 INFO:tasks.workunit.client.1.smithi122.stdout:3/255: unlink d2/d9/c1c 0 2024-03-22T02:28:50.862 INFO:tasks.workunit.client.1.smithi122.stdout:3/256: chown d2/d9/dc/l10 762788 1 2024-03-22T02:28:50.863 INFO:tasks.workunit.client.0.smithi103.stdout:0/853: chown dd/d16/c17 1093971 1 2024-03-22T02:28:50.863 INFO:tasks.workunit.client.0.smithi103.stdout:7/849: rmdir d1/d10/d11/d70/d8a/d9d 39 2024-03-22T02:28:50.864 INFO:tasks.workunit.client.1.smithi122.stdout:3/257: mkdir d2/d9/dc/d17/d51 0 2024-03-22T02:28:50.864 INFO:tasks.workunit.client.0.smithi103.stdout:7/850: dread - d1/da/d2f/d36/fe9 zero size 2024-03-22T02:28:50.865 INFO:tasks.workunit.client.0.smithi103.stdout:4/832: fdatasync d3/d1e/d25/d65/d83/dc1/fcc 0 2024-03-22T02:28:50.866 INFO:tasks.workunit.client.0.smithi103.stdout:4/833: chown d3/d1e/d25/c72 3 1 2024-03-22T02:28:50.869 INFO:tasks.workunit.client.0.smithi103.stdout:5/865: symlink d0/d14/d116/l133 0 2024-03-22T02:28:50.871 INFO:tasks.workunit.client.0.smithi103.stdout:0/854: truncate dd/d16/d4b/d96/d29/f85 2434913 0 2024-03-22T02:28:50.875 INFO:tasks.workunit.client.0.smithi103.stdout:5/866: dread d0/d14/d23/d5c/d70/f8f [4194304,4194304] 0 2024-03-22T02:28:50.875 INFO:tasks.workunit.client.0.smithi103.stdout:4/834: write d3/d31/fd8 [4467781,61775] 0 2024-03-22T02:28:50.876 INFO:tasks.workunit.client.0.smithi103.stdout:8/982: dread d4/d2a/d38/f9c [0,4194304] 0 2024-03-22T02:28:50.877 INFO:tasks.workunit.client.0.smithi103.stdout:4/835: write d3/fb [4700841,49234] 0 2024-03-22T02:28:50.878 INFO:tasks.workunit.client.1.smithi122.stdout:4/324: dwrite d0/d25/f1e [0,4194304] 0 2024-03-22T02:28:50.880 INFO:tasks.workunit.client.1.smithi122.stdout:4/325: write d0/d12/d15/d3d/f68 [969748,117276] 0 2024-03-22T02:28:50.881 INFO:tasks.workunit.client.1.smithi122.stdout:3/258: dwrite d2/f15 [0,4194304] 0 2024-03-22T02:28:50.883 INFO:tasks.workunit.client.0.smithi103.stdout:8/983: dread - d4/d26/d27/d7e/d39/d7b/fb8 zero size 2024-03-22T02:28:50.885 INFO:tasks.workunit.client.0.smithi103.stdout:8/984: write d4/f15 [443640,90547] 0 2024-03-22T02:28:50.889 INFO:tasks.workunit.client.1.smithi122.stdout:4/326: dwrite d0/d12/f1c [0,4194304] 0 2024-03-22T02:28:50.889 INFO:tasks.workunit.client.0.smithi103.stdout:4/836: creat d3/f115 x:0 0 0 2024-03-22T02:28:50.891 INFO:tasks.workunit.client.1.smithi122.stdout:3/259: dwrite d2/d9/dc/f2f [0,4194304] 0 2024-03-22T02:28:50.892 INFO:tasks.workunit.client.0.smithi103.stdout:0/855: link dd/d16/d4b/d96/d64/dab/dc6/l3d dd/d16/d4b/d96/l10f 0 2024-03-22T02:28:50.892 INFO:tasks.workunit.client.0.smithi103.stdout:4/837: write d3/d38/fdc [2891249,95421] 0 2024-03-22T02:28:50.893 INFO:tasks.workunit.client.0.smithi103.stdout:8/985: dwrite d4/d2a/d4f/fb7 [0,4194304] 0 2024-03-22T02:28:50.895 INFO:tasks.workunit.client.0.smithi103.stdout:8/986: readlink d4/d26/d27/da3/d12f/lc1 0 2024-03-22T02:28:50.898 INFO:tasks.workunit.client.0.smithi103.stdout:8/987: readlink d4/d26/d27/d7e/d39/d49/d79/d11e/d40/l77 0 2024-03-22T02:28:50.902 INFO:tasks.workunit.client.0.smithi103.stdout:0/856: rename dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d/fe5 to dd/d16/d4b/d96/d29/d48/d94/f110 0 2024-03-22T02:28:50.902 INFO:tasks.workunit.client.1.smithi122.stdout:3/260: rmdir d2/d9/dc/d17/d2b/d3f 39 2024-03-22T02:28:50.903 INFO:tasks.workunit.client.1.smithi122.stdout:3/261: fsync d2/d9/dc/d17/d45/f46 0 2024-03-22T02:28:50.903 INFO:tasks.workunit.client.0.smithi103.stdout:0/857: write dd/d16/d4b/d96/d29/d48/fec [202241,126350] 0 2024-03-22T02:28:50.905 INFO:tasks.workunit.client.1.smithi122.stdout:3/262: dread d2/d9/dc/df/d38/f3b [0,4194304] 0 2024-03-22T02:28:50.906 INFO:tasks.workunit.client.0.smithi103.stdout:4/838: mkdir d3/d1e/d25/d65/d83/dc1/d116 0 2024-03-22T02:28:50.906 INFO:tasks.workunit.client.0.smithi103.stdout:4/839: chown d3/d31/l7b 2 1 2024-03-22T02:28:50.906 INFO:tasks.workunit.client.1.smithi122.stdout:3/263: truncate d2/d9/dc/d17/d45/f46 1374926 0 2024-03-22T02:28:50.909 INFO:tasks.workunit.client.0.smithi103.stdout:4/840: creat d3/d1e/d68/f117 x:0 0 0 2024-03-22T02:28:50.910 INFO:tasks.workunit.client.0.smithi103.stdout:4/841: fdatasync d3/d1e/f36 0 2024-03-22T02:28:50.914 INFO:tasks.workunit.client.0.smithi103.stdout:4/842: rename d3/fb to d3/d1e/d25/d65/f118 0 2024-03-22T02:28:50.916 INFO:tasks.workunit.client.0.smithi103.stdout:4/843: symlink d3/d31/l119 0 2024-03-22T02:28:50.920 INFO:tasks.workunit.client.0.smithi103.stdout:2/836: write d11/d25/d63/f65 [519447,106376] 0 2024-03-22T02:28:50.920 INFO:tasks.workunit.client.0.smithi103.stdout:4/844: getdents d3/d38/d5c/d90/df8 0 2024-03-22T02:28:50.923 INFO:tasks.workunit.client.0.smithi103.stdout:2/837: unlink d11/d25/d46/l2d 0 2024-03-22T02:28:50.923 INFO:tasks.workunit.client.0.smithi103.stdout:2/838: write d11/d25/f35 [4892169,41482] 0 2024-03-22T02:28:50.924 INFO:tasks.workunit.client.0.smithi103.stdout:2/839: chown d11/d25/d46/d48 20798 1 2024-03-22T02:28:50.925 INFO:tasks.workunit.client.0.smithi103.stdout:4/845: creat d3/d30/d45/d107/f11a x:0 0 0 2024-03-22T02:28:50.928 INFO:tasks.workunit.client.1.smithi122.stdout:3/264: truncate d2/d9/dc/df/f48 21056 0 2024-03-22T02:28:50.930 INFO:tasks.workunit.client.0.smithi103.stdout:4/846: creat d3/d1e/d53/d7a/f11b x:0 0 0 2024-03-22T02:28:50.930 INFO:tasks.workunit.client.1.smithi122.stdout:3/265: mknod d2/d9/dc/d17/d28/d4c/c52 0 2024-03-22T02:28:50.931 INFO:tasks.workunit.client.1.smithi122.stdout:4/327: dwrite d0/d12/f21 [0,4194304] 0 2024-03-22T02:28:50.932 INFO:tasks.workunit.client.1.smithi122.stdout:3/266: chown d2/d9/dc/d17/d2b/c3a 146 1 2024-03-22T02:28:50.933 INFO:tasks.workunit.client.0.smithi103.stdout:2/840: rename d11/d25/d63/d45/d47/d83/da3/ce4 to d11/d19/da7/dd3/c119 0 2024-03-22T02:28:50.933 INFO:tasks.workunit.client.0.smithi103.stdout:3/947: truncate de/d12/d4d/fc4 11879325 0 2024-03-22T02:28:50.934 INFO:tasks.workunit.client.1.smithi122.stdout:4/328: mknod d0/d25/c6c 0 2024-03-22T02:28:50.934 INFO:tasks.workunit.client.0.smithi103.stdout:2/841: readlink d11/d19/l3b 0 2024-03-22T02:28:50.935 INFO:tasks.workunit.client.1.smithi122.stdout:3/267: creat d2/d9/dc/df/f53 x:0 0 0 2024-03-22T02:28:50.935 INFO:tasks.workunit.client.0.smithi103.stdout:7/851: dwrite d1/d10/d11/d19/d29/d33/da8/fc6 [0,4194304] 0 2024-03-22T02:28:50.935 INFO:tasks.workunit.client.1.smithi122.stdout:4/329: read - d0/d25/f5e zero size 2024-03-22T02:28:50.938 INFO:tasks.workunit.client.1.smithi122.stdout:4/330: truncate d0/d12/d15/d42/d57/f58 892640 0 2024-03-22T02:28:50.940 INFO:tasks.workunit.client.1.smithi122.stdout:3/268: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:50.941 INFO:tasks.workunit.client.1.smithi122.stdout:4/331: chown d0/d25/fa 87591 1 2024-03-22T02:28:50.941 INFO:tasks.workunit.client.0.smithi103.stdout:3/948: creat de/d12/d30/d70/f141 x:0 0 0 2024-03-22T02:28:50.944 INFO:tasks.workunit.client.0.smithi103.stdout:5/867: dwrite d0/d14/d23/d36/f82 [0,4194304] 0 2024-03-22T02:28:50.944 INFO:tasks.workunit.client.0.smithi103.stdout:4/847: dwrite d3/d38/d5c/faa [0,4194304] 0 2024-03-22T02:28:50.946 INFO:tasks.workunit.client.1.smithi122.stdout:4/332: creat d0/d12/d15/f6d x:0 0 0 2024-03-22T02:28:50.947 INFO:tasks.workunit.client.0.smithi103.stdout:8/988: truncate d4/f15 3054507 0 2024-03-22T02:28:50.948 INFO:tasks.workunit.client.1.smithi122.stdout:3/269: link d2/d9/dc/f36 d2/d9/dc/df/d49/f54 0 2024-03-22T02:28:50.949 INFO:tasks.workunit.client.1.smithi122.stdout:3/270: readlink d2/d9/dc/d17/d28/d4c/l50 0 2024-03-22T02:28:50.950 INFO:tasks.workunit.client.0.smithi103.stdout:2/842: rename d11/f16 to d11/d19/d4e/f11a 0 2024-03-22T02:28:50.950 INFO:tasks.workunit.client.1.smithi122.stdout:3/271: read d2/d9/dc/fd [287871,63073] 0 2024-03-22T02:28:50.950 INFO:tasks.workunit.client.1.smithi122.stdout:4/333: link d0/d12/d15/d2d/c6b d0/d12/d15/d42/c6e 0 2024-03-22T02:28:50.951 INFO:tasks.workunit.client.0.smithi103.stdout:7/852: unlink d1/d10/d11/d19/d29/d33/da8/dc5/fe4 0 2024-03-22T02:28:50.951 INFO:tasks.workunit.client.1.smithi122.stdout:4/334: chown d0/l39 82838445 1 2024-03-22T02:28:50.952 INFO:tasks.workunit.client.0.smithi103.stdout:0/858: truncate dd/d16/d4b/d96/d29/f83 5092738 0 2024-03-22T02:28:50.953 INFO:tasks.workunit.client.0.smithi103.stdout:5/868: symlink d0/da2/l134 0 2024-03-22T02:28:50.954 INFO:tasks.workunit.client.0.smithi103.stdout:2/843: write d11/d25/d46/d40/d90/ffd [424367,6695] 0 2024-03-22T02:28:50.955 INFO:tasks.workunit.client.0.smithi103.stdout:8/989: truncate f1 6404229 0 2024-03-22T02:28:50.956 INFO:tasks.workunit.client.0.smithi103.stdout:4/848: dread d3/d1e/f2f [0,4194304] 0 2024-03-22T02:28:50.958 INFO:tasks.workunit.client.0.smithi103.stdout:3/949: dwrite de/d12/d8a/dda/f111 [0,4194304] 0 2024-03-22T02:28:50.961 INFO:tasks.workunit.client.0.smithi103.stdout:7/853: creat d1/da/d1c/f10d x:0 0 0 2024-03-22T02:28:50.967 INFO:tasks.workunit.client.0.smithi103.stdout:8/990: mknod d4/d26/d27/da3/d117/c141 0 2024-03-22T02:28:50.969 INFO:tasks.workunit.client.0.smithi103.stdout:5/869: dwrite d0/d14/d23/d46/f11e [0,4194304] 0 2024-03-22T02:28:50.971 INFO:tasks.workunit.client.0.smithi103.stdout:3/950: dwrite de/d12/d19/f71 [0,4194304] 0 2024-03-22T02:28:50.974 INFO:tasks.workunit.client.0.smithi103.stdout:2/844: unlink d11/c74 0 2024-03-22T02:28:50.980 INFO:tasks.workunit.client.0.smithi103.stdout:5/870: dwrite d0/d14/d23/d36/d95/ff1 [0,4194304] 0 2024-03-22T02:28:50.982 INFO:tasks.workunit.client.0.smithi103.stdout:8/991: mkdir d4/d2a/d4f/d142 0 2024-03-22T02:28:50.984 INFO:tasks.workunit.client.0.smithi103.stdout:7/854: mkdir d1/d10b/d10e 0 2024-03-22T02:28:50.985 INFO:tasks.workunit.client.0.smithi103.stdout:3/951: mknod de/d12/d30/d70/c142 0 2024-03-22T02:28:50.986 INFO:tasks.workunit.client.0.smithi103.stdout:6/599: sync 2024-03-22T02:28:50.986 INFO:tasks.workunit.client.0.smithi103.stdout:2/845: creat d11/d25/d63/f11b x:0 0 0 2024-03-22T02:28:50.987 INFO:tasks.workunit.client.0.smithi103.stdout:6/600: chown d0/d3/ca7 5644334 1 2024-03-22T02:28:50.988 INFO:tasks.workunit.client.0.smithi103.stdout:5/871: mknod d0/d14/d23/d6f/c135 0 2024-03-22T02:28:50.989 INFO:tasks.workunit.client.0.smithi103.stdout:4/849: rename d3/da2/f108 to d3/f11c 0 2024-03-22T02:28:50.989 INFO:tasks.workunit.client.1.smithi122.stdout:6/370: sync 2024-03-22T02:28:50.990 INFO:tasks.workunit.client.0.smithi103.stdout:7/855: mknod d1/d10/d11/d19/dc3/de8/c10f 0 2024-03-22T02:28:50.991 INFO:tasks.workunit.client.1.smithi122.stdout:3/272: creat d2/f55 x:0 0 0 2024-03-22T02:28:50.992 INFO:tasks.workunit.client.0.smithi103.stdout:7/856: stat d1/d10/d11/d19/d29/d33/da8/dc5 0 2024-03-22T02:28:50.992 INFO:tasks.workunit.client.0.smithi103.stdout:3/952: unlink de/d1a/d25/f42 0 2024-03-22T02:28:50.993 INFO:tasks.workunit.client.1.smithi122.stdout:4/335: write d0/d25/f3b [62077,76722] 0 2024-03-22T02:28:50.993 INFO:tasks.workunit.client.0.smithi103.stdout:2/846: rmdir d11/d25/d63/d45/d47 39 2024-03-22T02:28:50.994 INFO:tasks.workunit.client.1.smithi122.stdout:6/371: unlink d7/de/dd/d20/d2d/d54/f59 0 2024-03-22T02:28:50.995 INFO:tasks.workunit.client.0.smithi103.stdout:7/857: dread d1/d10/d11/d19/dc3/de8/ff1 [0,4194304] 0 2024-03-22T02:28:50.995 INFO:tasks.workunit.client.0.smithi103.stdout:6/601: rmdir d0/d14/d57 39 2024-03-22T02:28:50.999 INFO:tasks.workunit.client.1.smithi122.stdout:4/336: dread d0/d25/f28 [0,4194304] 0 2024-03-22T02:28:51.001 INFO:tasks.workunit.client.1.smithi122.stdout:3/273: getdents d2/d9/dc/d17/d28 0 2024-03-22T02:28:51.001 INFO:tasks.workunit.client.1.smithi122.stdout:4/337: chown d0/d12/d15/d3d/f68 39764 1 2024-03-22T02:28:51.002 INFO:tasks.workunit.client.0.smithi103.stdout:3/953: dwrite de/d1a/d10f/d9c/fca [0,4194304] 0 2024-03-22T02:28:51.005 INFO:tasks.workunit.client.1.smithi122.stdout:6/372: dread d7/de/dd/d39/f5b [0,4194304] 0 2024-03-22T02:28:51.006 INFO:tasks.workunit.client.0.smithi103.stdout:2/847: dread - d11/d25/d63/d45/f94 zero size 2024-03-22T02:28:51.008 INFO:tasks.workunit.client.0.smithi103.stdout:6/602: symlink d0/d88/d8e/dae/lc0 0 2024-03-22T02:28:51.010 INFO:tasks.workunit.client.1.smithi122.stdout:4/338: dwrite d0/d12/f51 [0,4194304] 0 2024-03-22T02:28:51.012 INFO:tasks.workunit.client.1.smithi122.stdout:6/373: symlink d7/de/dd/l78 0 2024-03-22T02:28:51.012 INFO:tasks.workunit.client.0.smithi103.stdout:9/829: sync 2024-03-22T02:28:51.013 INFO:tasks.workunit.client.0.smithi103.stdout:3/954: truncate de/d6b/d80/d73/fba 3091291 0 2024-03-22T02:28:51.014 INFO:tasks.workunit.client.1.smithi122.stdout:4/339: stat d0/d25/f8 0 2024-03-22T02:28:51.015 INFO:tasks.workunit.client.1.smithi122.stdout:4/340: readlink d0/d12/d15/d2d/l2e 0 2024-03-22T02:28:51.016 INFO:tasks.workunit.client.0.smithi103.stdout:2/848: dwrite d11/d19/d5d/fba [0,4194304] 0 2024-03-22T02:28:51.017 INFO:tasks.workunit.client.0.smithi103.stdout:0/859: rename dd/d16/d4b/d96/d29/d48/ded to dd/d16/d4b/d96/d64/dab/dc6/d111 0 2024-03-22T02:28:51.017 INFO:tasks.workunit.client.1.smithi122.stdout:6/374: chown d7/de/dd/d20/d2d/f5f 8079 1 2024-03-22T02:28:51.019 INFO:tasks.workunit.client.0.smithi103.stdout:7/858: dwrite d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:51.020 INFO:tasks.workunit.client.0.smithi103.stdout:7/859: dread - d1/da/d1c/f10d zero size 2024-03-22T02:28:51.022 INFO:tasks.workunit.client.0.smithi103.stdout:0/860: read dd/d34/d53/fa1 [671293,119007] 0 2024-03-22T02:28:51.023 INFO:tasks.workunit.client.0.smithi103.stdout:7/860: read - d1/d10/d11/d19/d29/d61/d1d/da3/db9/ff3 zero size 2024-03-22T02:28:51.024 INFO:tasks.workunit.client.0.smithi103.stdout:5/872: link d0/d14/d23/d36/c52 d0/d14/d23/d5c/d77/d125/c136 0 2024-03-22T02:28:51.025 INFO:tasks.workunit.client.1.smithi122.stdout:4/341: dwrite d0/d12/d15/f55 [0,4194304] 0 2024-03-22T02:28:51.026 INFO:tasks.workunit.client.1.smithi122.stdout:6/375: creat d7/de/f79 x:0 0 0 2024-03-22T02:28:51.027 INFO:tasks.workunit.client.1.smithi122.stdout:3/274: write d2/d9/dc/df/d38/f3b [722644,13731] 0 2024-03-22T02:28:51.028 INFO:tasks.workunit.client.0.smithi103.stdout:9/830: unlink d2/d5/d4c/l55 0 2024-03-22T02:28:51.031 INFO:tasks.workunit.client.1.smithi122.stdout:3/275: rmdir d2/d9/dc/df/d47 39 2024-03-22T02:28:51.032 INFO:tasks.workunit.client.1.smithi122.stdout:3/276: fsync d2/f2c 0 2024-03-22T02:28:51.033 INFO:tasks.workunit.client.0.smithi103.stdout:3/955: rename de/d12/d8a/dda/f108 to de/d12/d8a/db0/d122/f143 0 2024-03-22T02:28:51.036 INFO:tasks.workunit.client.0.smithi103.stdout:6/603: symlink d0/d3/lc1 0 2024-03-22T02:28:51.038 INFO:tasks.workunit.client.1.smithi122.stdout:4/342: dwrite d0/d12/d15/d42/d57/f64 [0,4194304] 0 2024-03-22T02:28:51.039 INFO:tasks.workunit.client.1.smithi122.stdout:4/343: read d0/d12/f31 [1016898,28876] 0 2024-03-22T02:28:51.040 INFO:tasks.workunit.client.0.smithi103.stdout:8/992: dread f2 [0,4194304] 0 2024-03-22T02:28:51.040 INFO:tasks.workunit.client.1.smithi122.stdout:3/277: creat d2/d9/dc/df/d47/f56 x:0 0 0 2024-03-22T02:28:51.043 INFO:tasks.workunit.client.1.smithi122.stdout:6/376: dwrite d7/de/dd/d20/f31 [0,4194304] 0 2024-03-22T02:28:51.043 INFO:tasks.workunit.client.1.smithi122.stdout:3/278: stat d2/d9/dc/d17/d28/d4c/c52 0 2024-03-22T02:28:51.044 INFO:tasks.workunit.client.1.smithi122.stdout:3/279: stat d2/d9/dc/d17/d51 0 2024-03-22T02:28:51.045 INFO:tasks.workunit.client.0.smithi103.stdout:0/861: rename dd/d34/d53/d8c/lb8 to dd/d16/d4b/d96/d35/dc4/df9/l112 0 2024-03-22T02:28:51.046 INFO:tasks.workunit.client.1.smithi122.stdout:6/377: fsync d7/de/dd/d23/f48 0 2024-03-22T02:28:51.047 INFO:tasks.workunit.client.1.smithi122.stdout:6/378: read - d7/de/d27/d52/f57 zero size 2024-03-22T02:28:51.048 INFO:tasks.workunit.client.0.smithi103.stdout:3/956: fdatasync de/d6b/d80/d4f/fa3 0 2024-03-22T02:28:51.048 INFO:tasks.workunit.client.0.smithi103.stdout:6/604: symlink d0/d88/d8e/dae/lc2 0 2024-03-22T02:28:51.049 INFO:tasks.workunit.client.1.smithi122.stdout:3/280: mkdir d2/d9/dc/d57 0 2024-03-22T02:28:51.050 INFO:tasks.workunit.client.0.smithi103.stdout:6/605: chown d0/d3/d22/d58/f8c 210980726 1 2024-03-22T02:28:51.052 INFO:tasks.workunit.client.0.smithi103.stdout:8/993: unlink d4/d26/d27/da3/l122 0 2024-03-22T02:28:51.054 INFO:tasks.workunit.client.0.smithi103.stdout:3/957: rmdir de/d1a 39 2024-03-22T02:28:51.056 INFO:tasks.workunit.client.1.smithi122.stdout:6/379: link d7/de/dd/df/c58 d7/de/c7a 0 2024-03-22T02:28:51.057 INFO:tasks.workunit.client.1.smithi122.stdout:6/380: read - d7/de/dd/d39/f77 zero size 2024-03-22T02:28:51.058 INFO:tasks.workunit.client.0.smithi103.stdout:7/861: creat d1/d10/d11/d70/d8a/d9d/dff/f110 x:0 0 0 2024-03-22T02:28:51.063 INFO:tasks.workunit.client.0.smithi103.stdout:3/958: dread - de/d12/d8a/d93/d103/f129 zero size 2024-03-22T02:28:51.063 INFO:tasks.workunit.client.1.smithi122.stdout:6/381: dwrite d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:28:51.065 INFO:tasks.workunit.client.1.smithi122.stdout:6/382: truncate d7/de/dd/df/f6e 377801 0 2024-03-22T02:28:51.067 INFO:tasks.workunit.client.0.smithi103.stdout:6/606: mknod d0/d3/d22/d58/d6f/dbe/cc3 0 2024-03-22T02:28:51.068 INFO:tasks.workunit.client.0.smithi103.stdout:4/850: dread d3/f24 [0,4194304] 0 2024-03-22T02:28:51.068 INFO:tasks.workunit.client.1.smithi122.stdout:6/383: getdents d7/de/dd/d23 0 2024-03-22T02:28:51.069 INFO:tasks.workunit.client.0.smithi103.stdout:7/862: dwrite d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/ff7 [0,4194304] 0 2024-03-22T02:28:51.069 INFO:tasks.workunit.client.1.smithi122.stdout:6/384: fsync d7/de/dd/d20/d2d/f30 0 2024-03-22T02:28:51.072 INFO:tasks.workunit.client.0.smithi103.stdout:4/851: write d3/d31/fe6 [215731,3918] 0 2024-03-22T02:28:51.072 INFO:tasks.workunit.client.1.smithi122.stdout:3/281: truncate d2/d9/dc/fd 3454006 0 2024-03-22T02:28:51.073 INFO:tasks.workunit.client.0.smithi103.stdout:0/862: rename dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d/fd3 to dd/d16/d4b/d96/d64/d6a/dd5/f113 0 2024-03-22T02:28:51.075 INFO:tasks.workunit.client.0.smithi103.stdout:3/959: write de/d1a/d25/fd1 [2950766,38627] 0 2024-03-22T02:28:51.075 INFO:tasks.workunit.client.1.smithi122.stdout:6/385: getdents d7/de/dd/df 0 2024-03-22T02:28:51.076 INFO:tasks.workunit.client.0.smithi103.stdout:0/863: stat dd/d34/d53/c88 0 2024-03-22T02:28:51.079 INFO:tasks.workunit.client.1.smithi122.stdout:6/386: mknod d7/de/dd/d20/d2d/d54/d76/c7b 0 2024-03-22T02:28:51.081 INFO:tasks.workunit.client.1.smithi122.stdout:3/282: dwrite d2/f7 [0,4194304] 0 2024-03-22T02:28:51.081 INFO:tasks.workunit.client.0.smithi103.stdout:7/863: symlink d1/d10/d11/d19/d29/d61/d69/l111 0 2024-03-22T02:28:51.082 INFO:tasks.workunit.client.0.smithi103.stdout:5/873: dread d0/d14/d23/d2a/d33/db5/dbd/fc5 [0,4194304] 0 2024-03-22T02:28:51.084 INFO:tasks.workunit.client.0.smithi103.stdout:7/864: stat d1/da/l58 0 2024-03-22T02:28:51.085 INFO:tasks.workunit.client.1.smithi122.stdout:6/387: creat d7/de/dd/d20/f7c x:0 0 0 2024-03-22T02:28:51.085 INFO:tasks.workunit.client.0.smithi103.stdout:2/849: dwrite d11/d19/d4e/dd5/f108 [0,4194304] 0 2024-03-22T02:28:51.085 INFO:tasks.workunit.client.1.smithi122.stdout:6/388: dread - d7/f65 zero size 2024-03-22T02:28:51.087 INFO:tasks.workunit.client.0.smithi103.stdout:5/874: chown d0/da2/dca/fd0 216 1 2024-03-22T02:28:51.089 INFO:tasks.workunit.client.0.smithi103.stdout:5/875: fdatasync d0/d14/d23/d36/d95/f5a 0 2024-03-22T02:28:51.089 INFO:tasks.workunit.client.1.smithi122.stdout:6/389: creat d7/de/d27/d52/f7d x:0 0 0 2024-03-22T02:28:51.089 INFO:tasks.workunit.client.0.smithi103.stdout:6/607: rename d0/d14/l82 to d0/d3/d33/lc4 0 2024-03-22T02:28:51.090 INFO:tasks.workunit.client.0.smithi103.stdout:5/876: readlink d0/d14/l43 0 2024-03-22T02:28:51.092 INFO:tasks.workunit.client.0.smithi103.stdout:0/864: truncate dd/d16/d4b/d96/d29/d48/d5c/d103/f99 2047845 0 2024-03-22T02:28:51.092 INFO:tasks.workunit.client.0.smithi103.stdout:8/994: dread d4/d19/ff0 [0,4194304] 0 2024-03-22T02:28:51.095 INFO:tasks.workunit.client.0.smithi103.stdout:3/960: link de/d12/d19/df2/f69 de/d6b/d80/d4f/d84/f144 0 2024-03-22T02:28:51.098 INFO:tasks.workunit.client.1.smithi122.stdout:6/390: dwrite d7/de/dd/d39/f4f [0,4194304] 0 2024-03-22T02:28:51.099 INFO:tasks.workunit.client.0.smithi103.stdout:2/850: creat d11/d25/d63/d45/d10a/f11c x:0 0 0 2024-03-22T02:28:51.100 INFO:tasks.workunit.client.0.smithi103.stdout:2/851: chown d11/d25/d63/d45 83 1 2024-03-22T02:28:51.100 INFO:tasks.workunit.client.0.smithi103.stdout:2/852: chown d11/d19/da7/dd3/df1 0 1 2024-03-22T02:28:51.100 INFO:tasks.workunit.client.1.smithi122.stdout:6/391: fsync d7/de/dd/d39/f5b 0 2024-03-22T02:28:51.100 INFO:tasks.workunit.client.1.smithi122.stdout:6/392: readlink d7/de/l3d 0 2024-03-22T02:28:51.101 INFO:tasks.workunit.client.1.smithi122.stdout:6/393: chown d7/de/dd/c74 3827146 1 2024-03-22T02:28:51.103 INFO:tasks.workunit.client.0.smithi103.stdout:3/961: mknod de/d6b/d80/d4f/c145 0 2024-03-22T02:28:51.104 INFO:tasks.workunit.client.0.smithi103.stdout:4/852: write d3/d69/f46 [76353,60934] 0 2024-03-22T02:28:51.104 INFO:tasks.workunit.client.1.smithi122.stdout:5/307: sync 2024-03-22T02:28:51.104 INFO:tasks.workunit.client.1.smithi122.stdout:0/289: sync 2024-03-22T02:28:51.104 INFO:tasks.workunit.client.0.smithi103.stdout:3/962: readlink de/d12/d8f/dfb/l10c 0 2024-03-22T02:28:51.105 INFO:tasks.workunit.client.0.smithi103.stdout:8/995: symlink d4/d19/d1a/l143 0 2024-03-22T02:28:51.105 INFO:tasks.workunit.client.1.smithi122.stdout:6/394: dread d7/de/f34 [0,4194304] 0 2024-03-22T02:28:51.106 INFO:tasks.workunit.client.1.smithi122.stdout:5/308: chown d8/da/d28/d35/f4c 6132 1 2024-03-22T02:28:51.107 INFO:tasks.workunit.client.1.smithi122.stdout:3/283: write d2/d9/dc/d17/d28/f31 [5166485,111752] 0 2024-03-22T02:28:51.108 INFO:tasks.workunit.client.0.smithi103.stdout:5/877: link d0/d14/f41 d0/d14/d23/d5c/d77/d125/f137 0 2024-03-22T02:28:51.109 INFO:tasks.workunit.client.1.smithi122.stdout:5/309: creat d8/d1e/d22/d3a/d46/f6f x:0 0 0 2024-03-22T02:28:51.111 INFO:tasks.workunit.client.1.smithi122.stdout:6/395: creat d7/de/dd/f7e x:0 0 0 2024-03-22T02:28:51.112 INFO:tasks.workunit.client.1.smithi122.stdout:3/284: truncate d2/d9/dc/d17/d28/f4d 355336 0 2024-03-22T02:28:51.113 INFO:tasks.workunit.client.0.smithi103.stdout:8/996: fdatasync d4/d26/d27/d7e/f52 0 2024-03-22T02:28:51.116 INFO:tasks.workunit.client.0.smithi103.stdout:5/878: mknod d0/d14/d23/d2a/d33/db5/dbd/c138 0 2024-03-22T02:28:51.118 INFO:tasks.workunit.client.0.smithi103.stdout:4/853: mkdir d3/d31/d10f/d11d 0 2024-03-22T02:28:51.119 INFO:tasks.workunit.client.0.smithi103.stdout:5/879: readlink d0/d14/d23/d36/d95/l65 0 2024-03-22T02:28:51.121 INFO:tasks.workunit.client.0.smithi103.stdout:0/865: dread dd/d16/d4b/d96/f21 [0,4194304] 0 2024-03-22T02:28:51.121 INFO:tasks.workunit.client.1.smithi122.stdout:5/310: write d8/d1e/d22/d3a/f32 [99022,123905] 0 2024-03-22T02:28:51.121 INFO:tasks.workunit.client.1.smithi122.stdout:3/285: dread - d2/f55 zero size 2024-03-22T02:28:51.121 INFO:tasks.workunit.client.1.smithi122.stdout:0/290: dwrite d11/d14/d1d/d27/d32/f5f [0,4194304] 0 2024-03-22T02:28:51.122 INFO:tasks.workunit.client.0.smithi103.stdout:6/608: dwrite d0/f6a [0,4194304] 0 2024-03-22T02:28:51.122 INFO:tasks.workunit.client.0.smithi103.stdout:3/963: link de/d12/d19/fb6 de/d12/d5d/da9/db9/d10e/f146 0 2024-03-22T02:28:51.123 INFO:tasks.workunit.client.1.smithi122.stdout:0/291: readlink d11/d14/d23/l43 0 2024-03-22T02:28:51.125 INFO:tasks.workunit.client.0.smithi103.stdout:3/964: fdatasync de/d1a/d10f/d9c/fca 0 2024-03-22T02:28:51.128 INFO:tasks.workunit.client.1.smithi122.stdout:0/292: mknod d11/d16/d37/c64 0 2024-03-22T02:28:51.128 INFO:tasks.workunit.client.1.smithi122.stdout:3/286: dread d2/d9/dc/d17/d45/f46 [0,4194304] 0 2024-03-22T02:28:51.132 INFO:tasks.workunit.client.1.smithi122.stdout:6/396: dwrite d7/de/dd/d39/f4f [4194304,4194304] 0 2024-03-22T02:28:51.133 INFO:tasks.workunit.client.0.smithi103.stdout:8/997: symlink d4/d26/d27/d7e/d39/d49/l144 0 2024-03-22T02:28:51.134 INFO:tasks.workunit.client.1.smithi122.stdout:3/287: read d2/d9/dc/d17/d28/f2d [2328548,121684] 0 2024-03-22T02:28:51.136 INFO:tasks.workunit.client.0.smithi103.stdout:2/853: rename d11/d19/d7e/l8c to d11/d25/d63/d45/d47/d83/da3/l11d 0 2024-03-22T02:28:51.137 INFO:tasks.workunit.client.0.smithi103.stdout:3/965: creat de/d12/d5d/da9/f147 x:0 0 0 2024-03-22T02:28:51.139 INFO:tasks.workunit.client.1.smithi122.stdout:3/288: truncate d2/d9/dc/d17/d45/f46 2138194 0 2024-03-22T02:28:51.140 INFO:tasks.workunit.client.0.smithi103.stdout:0/866: dwrite dd/d16/d4b/f50 [4194304,4194304] 0 2024-03-22T02:28:51.140 INFO:tasks.workunit.client.0.smithi103.stdout:5/880: dwrite d0/d14/d23/d2a/d33/db5/dbd/ded/d103/f122 [0,4194304] 0 2024-03-22T02:28:51.142 INFO:tasks.workunit.client.1.smithi122.stdout:6/397: dwrite d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:51.144 INFO:tasks.workunit.client.1.smithi122.stdout:6/398: chown d7/de/d27/c55 9151253 1 2024-03-22T02:28:51.144 INFO:tasks.workunit.client.1.smithi122.stdout:3/289: symlink d2/d9/dc/d17/d45/l58 0 2024-03-22T02:28:51.146 INFO:tasks.workunit.client.0.smithi103.stdout:6/609: dwrite d0/d3/f1b [0,4194304] 0 2024-03-22T02:28:51.146 INFO:tasks.workunit.client.1.smithi122.stdout:6/399: chown d7/de/dd/d23/f47 14 1 2024-03-22T02:28:51.150 INFO:tasks.workunit.client.1.smithi122.stdout:6/400: dread - d7/de/dd/df/f62 zero size 2024-03-22T02:28:51.152 INFO:tasks.workunit.client.1.smithi122.stdout:6/401: write d7/de/dd/d23/f63 [4869033,86685] 0 2024-03-22T02:28:51.153 INFO:tasks.workunit.client.0.smithi103.stdout:3/966: dwrite de/d12/d19/df2/f53 [0,4194304] 0 2024-03-22T02:28:51.155 INFO:tasks.workunit.client.1.smithi122.stdout:3/290: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:51.156 INFO:tasks.workunit.client.0.smithi103.stdout:5/881: mknod d0/d14/d23/d6f/c139 0 2024-03-22T02:28:51.157 INFO:tasks.workunit.client.0.smithi103.stdout:2/854: mkdir d11/d19/da7/dd3/dd9/d110/d11e 0 2024-03-22T02:28:51.160 INFO:tasks.workunit.client.0.smithi103.stdout:3/967: truncate de/d12/d5d/da9/db9/f139 412170 0 2024-03-22T02:28:51.160 INFO:tasks.workunit.client.0.smithi103.stdout:5/882: symlink d0/d14/d23/d5c/df3/l13a 0 2024-03-22T02:28:51.162 INFO:tasks.workunit.client.0.smithi103.stdout:2/855: creat d11/d25/d63/d45/d47/d83/da3/f11f x:0 0 0 2024-03-22T02:28:51.163 INFO:tasks.workunit.client.1.smithi122.stdout:6/402: truncate d7/de/d27/f36 720844 0 2024-03-22T02:28:51.163 INFO:tasks.workunit.client.1.smithi122.stdout:0/293: dwrite d11/d4f/d5a/f5c [0,4194304] 0 2024-03-22T02:28:51.163 INFO:tasks.workunit.client.0.smithi103.stdout:6/610: creat d0/d3/d22/d58/d6f/d75/db0/db1/fc5 x:0 0 0 2024-03-22T02:28:51.164 INFO:tasks.workunit.client.1.smithi122.stdout:0/294: chown d11/d14/d23/f2d 7 1 2024-03-22T02:28:51.165 INFO:tasks.workunit.client.0.smithi103.stdout:6/611: chown d0/d3/c9b 54877130 1 2024-03-22T02:28:51.166 INFO:tasks.workunit.client.0.smithi103.stdout:8/998: write d4/d26/f81 [396068,120753] 0 2024-03-22T02:28:51.167 INFO:tasks.workunit.client.1.smithi122.stdout:6/403: write d7/de/dd/d20/d2d/f30 [3737658,111458] 0 2024-03-22T02:28:51.167 INFO:tasks.workunit.client.0.smithi103.stdout:5/883: creat d0/d14/d23/d5c/df3/f13b x:0 0 0 2024-03-22T02:28:51.169 INFO:tasks.workunit.client.1.smithi122.stdout:3/291: dread d2/d9/dc/f2f [0,4194304] 0 2024-03-22T02:28:51.169 INFO:tasks.workunit.client.0.smithi103.stdout:2/856: dread d11/d25/d63/d45/f77 [0,4194304] 0 2024-03-22T02:28:51.171 INFO:tasks.workunit.client.0.smithi103.stdout:8/999: truncate d4/d26/d27/d7e/d39/d49/d79/f123 382966 0 2024-03-22T02:28:51.171 INFO:tasks.workunit.client.0.smithi103.stdout:6/612: creat d0/d14/fc6 x:0 0 0 2024-03-22T02:28:51.171 INFO:tasks.workunit.client.1.smithi122.stdout:6/404: mkdir d7/de/dd/d23/d7f 0 2024-03-22T02:28:51.171 INFO:tasks.workunit.client.0.smithi103.stdout:2/857: symlink d11/d19/d7e/l120 0 2024-03-22T02:28:51.172 INFO:tasks.workunit.client.1.smithi122.stdout:3/292: write d2/d9/dc/d17/f42 [204837,108684] 0 2024-03-22T02:28:51.174 INFO:tasks.workunit.client.1.smithi122.stdout:0/295: dwrite d11/d4f/d5a/f61 [0,4194304] 0 2024-03-22T02:28:51.174 INFO:tasks.workunit.client.1.smithi122.stdout:6/405: write d7/de/dd/df/f62 [573681,35059] 0 2024-03-22T02:28:51.176 INFO:tasks.workunit.client.0.smithi103.stdout:2/858: mkdir d11/d19/da7/dd3/df1/d121 0 2024-03-22T02:28:51.178 INFO:tasks.workunit.client.0.smithi103.stdout:5/884: dread d0/d14/d23/d2a/d1f/f78 [0,4194304] 0 2024-03-22T02:28:51.179 INFO:tasks.workunit.client.0.smithi103.stdout:2/859: dread d11/d25/d46/d40/d90/ffd [0,4194304] 0 2024-03-22T02:28:51.186 INFO:tasks.workunit.client.1.smithi122.stdout:3/293: dwrite d2/d9/dc/d17/d28/f2d [0,4194304] 0 2024-03-22T02:28:51.186 INFO:tasks.workunit.client.1.smithi122.stdout:0/296: dwrite d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:51.190 INFO:tasks.workunit.client.0.smithi103.stdout:5/885: dwrite d0/d14/d23/d36/d95/d96/daa/daf/fab [0,4194304] 0 2024-03-22T02:28:51.190 INFO:tasks.workunit.client.0.smithi103.stdout:4/854: dread d3/d69/fcd [4194304,4194304] 0 2024-03-22T02:28:51.191 INFO:tasks.workunit.client.0.smithi103.stdout:2/860: dwrite d11/d25/f7c [0,4194304] 0 2024-03-22T02:28:51.192 INFO:tasks.workunit.client.1.smithi122.stdout:6/406: dwrite d7/de/dd/f42 [0,4194304] 0 2024-03-22T02:28:51.192 INFO:tasks.workunit.client.1.smithi122.stdout:0/297: mknod d11/d14/d23/d31/c65 0 2024-03-22T02:28:51.193 INFO:tasks.workunit.client.0.smithi103.stdout:2/861: chown f4 469216847 1 2024-03-22T02:28:51.195 INFO:tasks.workunit.client.0.smithi103.stdout:4/855: creat d3/d1e/d53/d7a/f11e x:0 0 0 2024-03-22T02:28:51.198 INFO:tasks.workunit.client.1.smithi122.stdout:8/398: sync 2024-03-22T02:28:51.201 INFO:tasks.workunit.client.0.smithi103.stdout:2/862: creat d11/d25/d63/d45/d47/dae/f122 x:0 0 0 2024-03-22T02:28:51.201 INFO:tasks.workunit.client.1.smithi122.stdout:3/294: dwrite d2/d9/dc/f2f [4194304,4194304] 0 2024-03-22T02:28:51.206 INFO:tasks.workunit.client.0.smithi103.stdout:5/886: rename d0/d14/d23/d36/de1/f109 to d0/f13c 0 2024-03-22T02:28:51.206 INFO:tasks.workunit.client.0.smithi103.stdout:4/856: dread d3/d1e/fac [0,4194304] 0 2024-03-22T02:28:51.209 INFO:tasks.workunit.client.1.smithi122.stdout:8/399: symlink d1/d8/d2f/d43/l87 0 2024-03-22T02:28:51.209 INFO:tasks.workunit.client.1.smithi122.stdout:0/298: dwrite d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:51.211 INFO:tasks.workunit.client.0.smithi103.stdout:5/887: mknod d0/d14/c13d 0 2024-03-22T02:28:51.212 INFO:tasks.workunit.client.1.smithi122.stdout:3/295: mkdir d2/d9/dc/df/d3c/d59 0 2024-03-22T02:28:51.212 INFO:tasks.workunit.client.1.smithi122.stdout:6/407: truncate d7/de/dd/df/f4b 330524 0 2024-03-22T02:28:51.214 INFO:tasks.workunit.client.0.smithi103.stdout:2/863: dwrite d11/d25/d46/d48/fde [0,4194304] 0 2024-03-22T02:28:51.214 INFO:tasks.workunit.client.0.smithi103.stdout:6/613: rmdir d0 39 2024-03-22T02:28:51.216 INFO:tasks.workunit.client.1.smithi122.stdout:8/400: mkdir d1/d8/d63/d88 0 2024-03-22T02:28:51.217 INFO:tasks.workunit.client.0.smithi103.stdout:4/857: creat d3/d1e/d53/f11f x:0 0 0 2024-03-22T02:28:51.219 INFO:tasks.workunit.client.0.smithi103.stdout:3/968: dwrite de/d12/d30/f35 [0,4194304] 0 2024-03-22T02:28:51.220 INFO:tasks.workunit.client.0.smithi103.stdout:5/888: read d0/f1e [3155574,11352] 0 2024-03-22T02:28:51.222 INFO:tasks.workunit.client.1.smithi122.stdout:8/401: creat d1/d40/d51/f89 x:0 0 0 2024-03-22T02:28:51.223 INFO:tasks.workunit.client.1.smithi122.stdout:0/299: link f10 d11/d14/d1d/d27/d2f/d4e/f66 0 2024-03-22T02:28:51.224 INFO:tasks.workunit.client.0.smithi103.stdout:5/889: chown d0/d14/d23/d2a/d33/db5/dfb 208642 1 2024-03-22T02:28:51.224 INFO:tasks.workunit.client.0.smithi103.stdout:0/867: dread dd/d16/d4b/d96/d64/f71 [0,4194304] 0 2024-03-22T02:28:51.226 INFO:tasks.workunit.client.1.smithi122.stdout:0/300: chown d11/d14/d1b/f60 269 1 2024-03-22T02:28:51.227 INFO:tasks.workunit.client.0.smithi103.stdout:2/864: dwrite d11/d25/d63/d45/d47/dae/fea [0,4194304] 0 2024-03-22T02:28:51.228 INFO:tasks.workunit.client.0.smithi103.stdout:0/868: stat dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/f95 0 2024-03-22T02:28:51.229 INFO:tasks.workunit.client.1.smithi122.stdout:8/402: chown d1/d8/d2f/l54 569058 1 2024-03-22T02:28:51.229 INFO:tasks.workunit.client.1.smithi122.stdout:6/408: dwrite d7/de/f68 [0,4194304] 0 2024-03-22T02:28:51.230 INFO:tasks.workunit.client.0.smithi103.stdout:2/865: dread - d11/d25/d63/d45/f94 zero size 2024-03-22T02:28:51.230 INFO:tasks.workunit.client.1.smithi122.stdout:0/301: mkdir d11/d14/d23/d24/d33/d67 0 2024-03-22T02:28:51.230 INFO:tasks.workunit.client.0.smithi103.stdout:0/869: chown dd/d34/d53/d8c/fc0 2318 1 2024-03-22T02:28:51.231 INFO:tasks.workunit.client.1.smithi122.stdout:0/302: dread - d11/d14/d1b/f60 zero size 2024-03-22T02:28:51.232 INFO:tasks.workunit.client.0.smithi103.stdout:2/866: fsync d11/d25/d63/d45/d10a/f11c 0 2024-03-22T02:28:51.232 INFO:tasks.workunit.client.1.smithi122.stdout:8/403: write d1/d9/f28 [454191,31993] 0 2024-03-22T02:28:51.233 INFO:tasks.workunit.client.1.smithi122.stdout:6/409: stat d7/de/c7a 0 2024-03-22T02:28:51.233 INFO:tasks.workunit.client.0.smithi103.stdout:5/890: dwrite d0/d14/d23/d2a/d33/db9/ff5 [0,4194304] 0 2024-03-22T02:28:51.236 INFO:tasks.workunit.client.1.smithi122.stdout:0/303: mknod d11/d14/d1b/c68 0 2024-03-22T02:28:51.239 INFO:tasks.workunit.client.1.smithi122.stdout:0/304: symlink d11/d14/d1d/d27/l69 0 2024-03-22T02:28:51.240 INFO:tasks.workunit.client.1.smithi122.stdout:6/410: dread d7/de/dd/d20/f31 [0,4194304] 0 2024-03-22T02:28:51.241 INFO:tasks.workunit.client.1.smithi122.stdout:6/411: readlink d7/de/d27/l2e 0 2024-03-22T02:28:51.242 INFO:tasks.workunit.client.1.smithi122.stdout:6/412: read d7/de/dd/d20/f31 [3595744,58000] 0 2024-03-22T02:28:51.242 INFO:tasks.workunit.client.1.smithi122.stdout:0/305: rename d11/f4a to d11/d14/d1d/f6a 0 2024-03-22T02:28:51.243 INFO:tasks.workunit.client.1.smithi122.stdout:2/332: sync 2024-03-22T02:28:51.243 INFO:tasks.workunit.client.1.smithi122.stdout:9/374: sync 2024-03-22T02:28:51.244 INFO:tasks.workunit.client.1.smithi122.stdout:7/295: sync 2024-03-22T02:28:51.244 INFO:tasks.workunit.client.1.smithi122.stdout:5/311: sync 2024-03-22T02:28:51.246 INFO:tasks.workunit.client.1.smithi122.stdout:5/312: chown d8/d1e/d22/c3c 927836 1 2024-03-22T02:28:51.246 INFO:tasks.workunit.client.1.smithi122.stdout:7/296: write d0/d4/d20/f27 [1366865,82162] 0 2024-03-22T02:28:51.247 INFO:tasks.workunit.client.1.smithi122.stdout:9/375: fdatasync db/d23/f3a 0 2024-03-22T02:28:51.247 INFO:tasks.workunit.client.1.smithi122.stdout:2/333: creat df/d1f/d33/d4d/f78 x:0 0 0 2024-03-22T02:28:51.248 INFO:tasks.workunit.client.1.smithi122.stdout:5/313: write d8/d1e/d22/d3a/d39/f6b [504735,101363] 0 2024-03-22T02:28:51.250 INFO:tasks.workunit.client.1.smithi122.stdout:2/334: mknod df/d1f/d3a/d41/c79 0 2024-03-22T02:28:51.253 INFO:tasks.workunit.client.1.smithi122.stdout:2/335: fsync df/d1f/d3a/d4f/f5e 0 2024-03-22T02:28:51.255 INFO:tasks.workunit.client.1.smithi122.stdout:2/336: chown df/d1f/d3a/f58 7607 1 2024-03-22T02:28:51.255 INFO:tasks.workunit.client.1.smithi122.stdout:9/376: dwrite db/d23/d1f/d57/d66/f7a [0,4194304] 0 2024-03-22T02:28:51.256 INFO:tasks.workunit.client.1.smithi122.stdout:9/377: dread - db/d23/d1f/d21/d29/f5f zero size 2024-03-22T02:28:51.260 INFO:tasks.workunit.client.1.smithi122.stdout:1/406: sync 2024-03-22T02:28:51.260 INFO:tasks.workunit.client.1.smithi122.stdout:4/344: sync 2024-03-22T02:28:51.262 INFO:tasks.workunit.client.0.smithi103.stdout:4/858: mkdir d3/d1e/d25/d65/dc9/d120 0 2024-03-22T02:28:51.263 INFO:tasks.workunit.client.1.smithi122.stdout:2/337: getdents df/d1f/d3d/d52 0 2024-03-22T02:28:51.265 INFO:tasks.workunit.client.1.smithi122.stdout:1/407: write dc/f2f [4053372,109485] 0 2024-03-22T02:28:51.266 INFO:tasks.workunit.client.1.smithi122.stdout:4/345: write d0/d12/d15/d2d/f38 [1094813,31910] 0 2024-03-22T02:28:51.266 INFO:tasks.workunit.client.0.smithi103.stdout:3/969: mknod de/d12/d19/df2/c148 0 2024-03-22T02:28:51.266 INFO:tasks.workunit.client.1.smithi122.stdout:2/338: fdatasync fd 0 2024-03-22T02:28:51.267 INFO:tasks.workunit.client.1.smithi122.stdout:1/408: fsync dc/d55/f5f 0 2024-03-22T02:28:51.268 INFO:tasks.workunit.client.1.smithi122.stdout:4/346: read d0/d12/d15/d42/d57/f64 [1838942,51491] 0 2024-03-22T02:28:51.270 INFO:tasks.workunit.client.1.smithi122.stdout:2/339: chown df/d1f/d33/d34/c40 4929 1 2024-03-22T02:28:51.270 INFO:tasks.workunit.client.0.smithi103.stdout:0/870: fsync fa 0 2024-03-22T02:28:51.271 INFO:tasks.workunit.client.0.smithi103.stdout:0/871: fsync dd/f15 0 2024-03-22T02:28:51.272 INFO:tasks.workunit.client.0.smithi103.stdout:2/867: symlink d11/d19/da7/dd3/l123 0 2024-03-22T02:28:51.273 INFO:tasks.workunit.client.1.smithi122.stdout:2/340: stat df/d1f/c27 0 2024-03-22T02:28:51.273 INFO:tasks.workunit.client.0.smithi103.stdout:0/872: chown dd/d16/d4b/d96/d29/d48/d5c/d9b/ff5 781 1 2024-03-22T02:28:51.274 INFO:tasks.workunit.client.0.smithi103.stdout:2/868: chown d11/d19/da7/dd3/dd9 10550988 1 2024-03-22T02:28:51.274 INFO:tasks.workunit.client.1.smithi122.stdout:2/341: readlink df/l35 0 2024-03-22T02:28:51.274 INFO:tasks.workunit.client.1.smithi122.stdout:5/314: read d8/da/d28/d42/d4e/f58 [599059,122875] 0 2024-03-22T02:28:51.277 INFO:tasks.workunit.client.1.smithi122.stdout:3/296: rmdir d2/d9 39 2024-03-22T02:28:51.277 INFO:tasks.workunit.client.1.smithi122.stdout:5/315: truncate d8/f26 1254618 0 2024-03-22T02:28:51.279 INFO:tasks.workunit.client.0.smithi103.stdout:6/614: rename d0/d3/d22/d58/d6f/dbe to d0/d14/dc7 0 2024-03-22T02:28:51.281 INFO:tasks.workunit.client.0.smithi103.stdout:0/873: truncate dd/d16/d4b/d96/f21 771129 0 2024-03-22T02:28:51.281 INFO:tasks.workunit.client.1.smithi122.stdout:1/409: dwrite dc/f11 [0,4194304] 0 2024-03-22T02:28:51.284 INFO:tasks.workunit.client.1.smithi122.stdout:4/347: read d0/d12/d15/d42/d4e/f16 [123051,118408] 0 2024-03-22T02:28:51.287 INFO:tasks.workunit.client.1.smithi122.stdout:1/410: mkdir dc/d3b/d46/d4d/d90 0 2024-03-22T02:28:51.287 INFO:tasks.workunit.client.1.smithi122.stdout:2/342: dwrite df/f15 [0,4194304] 0 2024-03-22T02:28:51.287 INFO:tasks.workunit.client.1.smithi122.stdout:3/297: dread d2/d9/dc/df/d38/f3b [0,4194304] 0 2024-03-22T02:28:51.289 INFO:tasks.workunit.client.1.smithi122.stdout:1/411: write dc/d3b/f45 [410540,130707] 0 2024-03-22T02:28:51.289 INFO:tasks.workunit.client.1.smithi122.stdout:4/348: mknod d0/d12/d15/d42/d57/c6f 0 2024-03-22T02:28:51.291 INFO:tasks.workunit.client.1.smithi122.stdout:3/298: truncate d2/f8 5392739 0 2024-03-22T02:28:51.291 INFO:tasks.workunit.client.1.smithi122.stdout:1/412: truncate dc/f22 4480657 0 2024-03-22T02:28:51.297 INFO:tasks.workunit.client.1.smithi122.stdout:1/413: unlink dc/d3b/f43 0 2024-03-22T02:28:51.297 INFO:tasks.workunit.client.1.smithi122.stdout:9/378: dread db/d1b/d6c/f4f [0,4194304] 0 2024-03-22T02:28:51.302 INFO:tasks.workunit.client.1.smithi122.stdout:4/349: dwrite d0/d12/d15/d42/d4e/de/f44 [0,4194304] 0 2024-03-22T02:28:51.305 INFO:tasks.workunit.client.1.smithi122.stdout:1/414: fdatasync dc/d3b/d46/f6b 0 2024-03-22T02:28:51.307 INFO:tasks.workunit.client.1.smithi122.stdout:3/299: dwrite d2/d9/dc/df/f53 [0,4194304] 0 2024-03-22T02:28:51.309 INFO:tasks.workunit.client.1.smithi122.stdout:1/415: write dc/f88 [243918,26490] 0 2024-03-22T02:28:51.312 INFO:tasks.workunit.client.1.smithi122.stdout:4/350: write d0/d25/f4d [726051,39693] 0 2024-03-22T02:28:51.314 INFO:tasks.workunit.client.1.smithi122.stdout:8/404: dwrite d1/d8/d2f/f3e [0,4194304] 0 2024-03-22T02:28:51.314 INFO:tasks.workunit.client.1.smithi122.stdout:1/416: fsync dc/d4c/f52 0 2024-03-22T02:28:51.316 INFO:tasks.workunit.client.1.smithi122.stdout:9/379: creat db/df/f7d x:0 0 0 2024-03-22T02:28:51.316 INFO:tasks.workunit.client.0.smithi103.stdout:6/615: dread d0/f2c [0,4194304] 0 2024-03-22T02:28:51.317 INFO:tasks.workunit.client.1.smithi122.stdout:0/306: stat d11/d14/d1d/d27/d2f/f53 0 2024-03-22T02:28:51.319 INFO:tasks.workunit.client.0.smithi103.stdout:5/891: dread d0/d14/f41 [4194304,4194304] 0 2024-03-22T02:28:51.319 INFO:tasks.workunit.client.1.smithi122.stdout:8/405: chown d1/d4d/f58 88 1 2024-03-22T02:28:51.320 INFO:tasks.workunit.client.0.smithi103.stdout:6/616: symlink d0/d14/d9a/lc8 0 2024-03-22T02:28:51.325 INFO:tasks.workunit.client.0.smithi103.stdout:6/617: symlink d0/dbd/lc9 0 2024-03-22T02:28:51.326 INFO:tasks.workunit.client.1.smithi122.stdout:0/307: creat d11/d14/d1d/d27/d32/f6b x:0 0 0 2024-03-22T02:28:51.326 INFO:tasks.workunit.client.1.smithi122.stdout:8/406: chown d1/d8/d63/d88 28566866 1 2024-03-22T02:28:51.327 INFO:tasks.workunit.client.1.smithi122.stdout:3/300: rename d2/d9/dc/d17/l40 to d2/d9/dc/l5a 0 2024-03-22T02:28:51.327 INFO:tasks.workunit.client.0.smithi103.stdout:2/869: write d11/d25/fad [935811,113821] 0 2024-03-22T02:28:51.329 INFO:tasks.workunit.client.1.smithi122.stdout:0/308: write f9 [3570103,50935] 0 2024-03-22T02:28:51.331 INFO:tasks.workunit.client.1.smithi122.stdout:5/316: dread d8/f17 [0,4194304] 0 2024-03-22T02:28:51.332 INFO:tasks.workunit.client.0.smithi103.stdout:5/892: getdents d0/d14/d23/d36/d95/d96/daa 0 2024-03-22T02:28:51.333 INFO:tasks.workunit.client.1.smithi122.stdout:0/309: chown f9 17155 1 2024-03-22T02:28:51.333 INFO:tasks.workunit.client.1.smithi122.stdout:6/413: write d7/de/dd/df/f4b [693419,79928] 0 2024-03-22T02:28:51.334 INFO:tasks.workunit.client.0.smithi103.stdout:3/970: dwrite de/d12/d8a/d93/fac [0,4194304] 0 2024-03-22T02:28:51.334 INFO:tasks.workunit.client.0.smithi103.stdout:2/870: chown d11/d19/d4e/de9/lf9 36506 1 2024-03-22T02:28:51.335 INFO:tasks.workunit.client.1.smithi122.stdout:4/351: rename d0/d12/d15/d2d/l3c to d0/d41/l70 0 2024-03-22T02:28:51.336 INFO:tasks.workunit.client.1.smithi122.stdout:5/317: fdatasync d8/d1e/d22/d3a/f32 0 2024-03-22T02:28:51.336 INFO:tasks.workunit.client.0.smithi103.stdout:4/859: dwrite d3/d38/d5c/d86/fdd [0,4194304] 0 2024-03-22T02:28:51.337 INFO:tasks.workunit.client.1.smithi122.stdout:9/380: dwrite db/d23/d1f/d21/d29/f5f [0,4194304] 0 2024-03-22T02:28:51.337 INFO:tasks.workunit.client.0.smithi103.stdout:2/871: write d11/d19/d5d/f8d [7257627,104580] 0 2024-03-22T02:28:51.340 INFO:tasks.workunit.client.1.smithi122.stdout:2/343: dread df/f10 [0,4194304] 0 2024-03-22T02:28:51.341 INFO:tasks.workunit.client.0.smithi103.stdout:3/971: rename de/d12/d19/df2/f69 to de/d12/d19/f149 0 2024-03-22T02:28:51.342 INFO:tasks.workunit.client.1.smithi122.stdout:4/352: symlink d0/d12/l71 0 2024-03-22T02:28:51.342 INFO:tasks.workunit.client.1.smithi122.stdout:5/318: chown d8/d1e/d22/c4a 504 1 2024-03-22T02:28:51.343 INFO:tasks.workunit.client.0.smithi103.stdout:4/860: dread - d3/d38/d5c/d90/f112 zero size 2024-03-22T02:28:51.343 INFO:tasks.workunit.client.0.smithi103.stdout:5/893: creat d0/d14/d23/d5c/d77/df2/f13e x:0 0 0 2024-03-22T02:28:51.343 INFO:tasks.workunit.client.0.smithi103.stdout:4/861: dread - d3/d1e/d53/f11f zero size 2024-03-22T02:28:51.343 INFO:tasks.workunit.client.1.smithi122.stdout:4/353: chown d0/d12/d15/d42/d57/f64 46734995 1 2024-03-22T02:28:51.344 INFO:tasks.workunit.client.0.smithi103.stdout:2/872: symlink d11/d25/d46/db6/dfb/l124 0 2024-03-22T02:28:51.345 INFO:tasks.workunit.client.1.smithi122.stdout:6/414: truncate d7/de/dd/d20/d2d/d54/d76/f6b 926056 0 2024-03-22T02:28:51.347 INFO:tasks.workunit.client.0.smithi103.stdout:3/972: unlink de/d12/d5d/d102/f11d 0 2024-03-22T02:28:51.349 INFO:tasks.workunit.client.1.smithi122.stdout:5/319: chown d8/d1e/d22/d3a/d46/d4f/f54 1492108 1 2024-03-22T02:28:51.349 INFO:tasks.workunit.client.1.smithi122.stdout:8/407: rename d1/d8/d2f/d43 to d1/d8/d8a 0 2024-03-22T02:28:51.350 INFO:tasks.workunit.client.1.smithi122.stdout:6/415: write d7/de/dd/d39/f77 [130207,76154] 0 2024-03-22T02:28:51.351 INFO:tasks.workunit.client.0.smithi103.stdout:6/618: dread d0/d14/d57/d68/f28 [0,4194304] 0 2024-03-22T02:28:51.352 INFO:tasks.workunit.client.1.smithi122.stdout:0/310: rmdir d11/d47 39 2024-03-22T02:28:51.356 INFO:tasks.workunit.client.0.smithi103.stdout:5/894: rename d0/d14/d23/d36/d95/d96/fcb to d0/f13f 0 2024-03-22T02:28:51.357 INFO:tasks.workunit.client.1.smithi122.stdout:7/297: sync 2024-03-22T02:28:51.357 INFO:tasks.workunit.client.1.smithi122.stdout:9/381: mkdir db/df/d7e 0 2024-03-22T02:28:51.358 INFO:tasks.workunit.client.0.smithi103.stdout:2/873: dwrite d11/d25/d63/d45/d47/d83/da3/fb9 [0,4194304] 0 2024-03-22T02:28:51.360 INFO:tasks.workunit.client.0.smithi103.stdout:3/973: dwrite de/d12/d30/d70/de5/f13b [4194304,4194304] 0 2024-03-22T02:28:51.360 INFO:tasks.workunit.client.1.smithi122.stdout:4/354: dwrite d0/d25/f4d [0,4194304] 0 2024-03-22T02:28:51.362 INFO:tasks.workunit.client.1.smithi122.stdout:5/320: dread d8/d1e/d22/d3a/d39/f6b [0,4194304] 0 2024-03-22T02:28:51.363 INFO:tasks.workunit.client.0.smithi103.stdout:2/874: unlink d11/d25/d46/d48/f97 0 2024-03-22T02:28:51.364 INFO:tasks.workunit.client.0.smithi103.stdout:6/619: link d0/d3/d22/d58/d6f/d75/l77 d0/lca 0 2024-03-22T02:28:51.365 INFO:tasks.workunit.client.1.smithi122.stdout:4/355: write d0/d25/f4d [2224482,27562] 0 2024-03-22T02:28:51.368 INFO:tasks.workunit.client.1.smithi122.stdout:6/416: chown d7/de/dd/f13 59 1 2024-03-22T02:28:51.369 INFO:tasks.workunit.client.0.smithi103.stdout:6/620: write d0/d3/fb8 [101763,91768] 0 2024-03-22T02:28:51.371 INFO:tasks.workunit.client.1.smithi122.stdout:7/298: mkdir d0/d25/d5c 0 2024-03-22T02:28:51.371 INFO:tasks.workunit.client.1.smithi122.stdout:8/408: dwrite d1/d9/f24 [0,4194304] 0 2024-03-22T02:28:51.373 INFO:tasks.workunit.client.0.smithi103.stdout:5/895: dwrite d0/d14/d23/f10e [0,4194304] 0 2024-03-22T02:28:51.375 INFO:tasks.workunit.client.1.smithi122.stdout:5/321: creat d8/d1a/f70 x:0 0 0 2024-03-22T02:28:51.377 INFO:tasks.workunit.client.0.smithi103.stdout:3/974: dwrite de/d12/d4d/f107 [0,4194304] 0 2024-03-22T02:28:51.377 INFO:tasks.workunit.client.0.smithi103.stdout:2/875: mkdir d11/d25/d63/d45/d47/d83/d78/d125 0 2024-03-22T02:28:51.377 INFO:tasks.workunit.client.1.smithi122.stdout:8/409: fdatasync d1/d8/d5e/f65 0 2024-03-22T02:28:51.378 INFO:tasks.workunit.client.1.smithi122.stdout:3/301: rename d2/d9/dc/df/d47/c4e to d2/d9/dc/df/c5b 0 2024-03-22T02:28:51.378 INFO:tasks.workunit.client.1.smithi122.stdout:6/417: creat d7/de/d27/f80 x:0 0 0 2024-03-22T02:28:51.380 INFO:tasks.workunit.client.1.smithi122.stdout:7/299: rmdir d0/d30 39 2024-03-22T02:28:51.381 INFO:tasks.workunit.client.0.smithi103.stdout:5/896: read - d0/d14/d23/d36/d95/d96/daa/feb zero size 2024-03-22T02:28:51.381 INFO:tasks.workunit.client.0.smithi103.stdout:6/621: unlink d0/d3/f9 0 2024-03-22T02:28:51.383 INFO:tasks.workunit.client.1.smithi122.stdout:1/417: dread dc/f80 [0,4194304] 0 2024-03-22T02:28:51.384 INFO:tasks.workunit.client.1.smithi122.stdout:5/322: fsync d8/d1e/d22/d3a/d46/f4b 0 2024-03-22T02:28:51.386 INFO:tasks.workunit.client.1.smithi122.stdout:8/410: creat d1/d22/f8b x:0 0 0 2024-03-22T02:28:51.386 INFO:tasks.workunit.client.0.smithi103.stdout:3/975: mkdir de/d12/d8f/d14a 0 2024-03-22T02:28:51.388 INFO:tasks.workunit.client.0.smithi103.stdout:2/876: creat d11/dc5/f126 x:0 0 0 2024-03-22T02:28:51.388 INFO:tasks.workunit.client.1.smithi122.stdout:3/302: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:51.389 INFO:tasks.workunit.client.1.smithi122.stdout:7/300: unlink d0/d4/d20/d21/c29 0 2024-03-22T02:28:51.392 INFO:tasks.workunit.client.1.smithi122.stdout:4/356: getdents d0/d41 0 2024-03-22T02:28:51.392 INFO:tasks.workunit.client.1.smithi122.stdout:6/418: creat d7/de/dd/d20/f81 x:0 0 0 2024-03-22T02:28:51.393 INFO:tasks.workunit.client.1.smithi122.stdout:7/301: mkdir d0/d4e/d5d 0 2024-03-22T02:28:51.393 INFO:tasks.workunit.client.0.smithi103.stdout:3/976: mkdir de/d12/d8a/db0/d122/d14b 0 2024-03-22T02:28:51.395 INFO:tasks.workunit.client.0.smithi103.stdout:4/862: dwrite d3/fa0 [0,4194304] 0 2024-03-22T02:28:51.395 INFO:tasks.workunit.client.1.smithi122.stdout:4/357: write d0/d12/d15/d3d/f5c [690967,41202] 0 2024-03-22T02:28:51.395 INFO:tasks.workunit.client.1.smithi122.stdout:9/382: dread db/d23/f11 [0,4194304] 0 2024-03-22T02:28:51.396 INFO:tasks.workunit.client.1.smithi122.stdout:3/303: symlink d2/d9/dc/df/d47/l5c 0 2024-03-22T02:28:51.398 INFO:tasks.workunit.client.1.smithi122.stdout:8/411: mknod d1/d8/d63/d88/c8c 0 2024-03-22T02:28:51.399 INFO:tasks.workunit.client.0.smithi103.stdout:3/977: write de/d12/d8a/dda/f111 [4927469,23318] 0 2024-03-22T02:28:51.399 INFO:tasks.workunit.client.0.smithi103.stdout:2/877: dread d11/d25/d63/d45/d47/d83/da3/fb9 [0,4194304] 0 2024-03-22T02:28:51.399 INFO:tasks.workunit.client.1.smithi122.stdout:9/383: write db/d23/d1f/d21/d25/d41/f61 [690732,104816] 0 2024-03-22T02:28:51.399 INFO:tasks.workunit.client.1.smithi122.stdout:1/418: creat dc/d2b/f91 x:0 0 0 2024-03-22T02:28:51.400 INFO:tasks.workunit.client.0.smithi103.stdout:4/863: truncate d3/d1e/d68/f117 873273 0 2024-03-22T02:28:51.401 INFO:tasks.workunit.client.1.smithi122.stdout:9/384: fsync fa 0 2024-03-22T02:28:51.401 INFO:tasks.workunit.client.0.smithi103.stdout:5/897: creat d0/d14/d23/d5c/d77/d125/f140 x:0 0 0 2024-03-22T02:28:51.402 INFO:tasks.workunit.client.1.smithi122.stdout:9/385: chown db/d23/d1f/d21/d29/f3f 13488 1 2024-03-22T02:28:51.404 INFO:tasks.workunit.client.1.smithi122.stdout:7/302: mkdir d0/d4/d20/d21/d39/d5e 0 2024-03-22T02:28:51.405 INFO:tasks.workunit.client.1.smithi122.stdout:6/419: fsync d7/de/dd/d20/f5d 0 2024-03-22T02:28:51.405 INFO:tasks.workunit.client.0.smithi103.stdout:6/622: creat d0/d3/d33/fcb x:0 0 0 2024-03-22T02:28:51.406 INFO:tasks.workunit.client.1.smithi122.stdout:1/419: write dc/f22 [953763,109232] 0 2024-03-22T02:28:51.407 INFO:tasks.workunit.client.0.smithi103.stdout:6/623: rmdir d0/d3/d22 39 2024-03-22T02:28:51.408 INFO:tasks.workunit.client.1.smithi122.stdout:8/412: unlink d1/d22/f7b 0 2024-03-22T02:28:51.409 INFO:tasks.workunit.client.1.smithi122.stdout:4/358: creat d0/d12/d15/d2d/f72 x:0 0 0 2024-03-22T02:28:51.410 INFO:tasks.workunit.client.1.smithi122.stdout:9/386: mknod db/d23/d1e/c7f 0 2024-03-22T02:28:51.411 INFO:tasks.workunit.client.0.smithi103.stdout:4/864: dwrite d3/d1e/d53/f11f [0,4194304] 0 2024-03-22T02:28:51.412 INFO:tasks.workunit.client.0.smithi103.stdout:6/624: symlink d0/d14/d57/d68/lcc 0 2024-03-22T02:28:51.413 INFO:tasks.workunit.client.1.smithi122.stdout:6/420: symlink d7/de/dd/d20/d2d/d54/l82 0 2024-03-22T02:28:51.413 INFO:tasks.workunit.client.0.smithi103.stdout:4/865: symlink d3/d38/l121 0 2024-03-22T02:28:51.414 INFO:tasks.workunit.client.0.smithi103.stdout:4/866: chown d3/d69/f5a 13062 1 2024-03-22T02:28:51.414 INFO:tasks.workunit.client.0.smithi103.stdout:6/625: unlink d0/d14/c1d 0 2024-03-22T02:28:51.415 INFO:tasks.workunit.client.1.smithi122.stdout:6/421: read - d7/de/d27/d52/f7d zero size 2024-03-22T02:28:51.415 INFO:tasks.workunit.client.1.smithi122.stdout:8/413: rmdir d1/d4d 39 2024-03-22T02:28:51.415 INFO:tasks.workunit.client.1.smithi122.stdout:6/422: fsync d7/f65 0 2024-03-22T02:28:51.416 INFO:tasks.workunit.client.0.smithi103.stdout:4/867: symlink d3/d1e/d25/l122 0 2024-03-22T02:28:51.418 INFO:tasks.workunit.client.1.smithi122.stdout:8/414: mkdir d1/d40/d48/d8d 0 2024-03-22T02:28:51.419 INFO:tasks.workunit.client.0.smithi103.stdout:6/626: dread d0/f6a [0,4194304] 0 2024-03-22T02:28:51.421 INFO:tasks.workunit.client.1.smithi122.stdout:9/387: creat db/df/f80 x:0 0 0 2024-03-22T02:28:51.422 INFO:tasks.workunit.client.0.smithi103.stdout:4/868: link d3/d69/fcd d3/d30/f123 0 2024-03-22T02:28:51.424 INFO:tasks.workunit.client.0.smithi103.stdout:4/869: fdatasync d3/d1e/d53/fe2 0 2024-03-22T02:28:51.432 INFO:tasks.workunit.client.1.smithi122.stdout:0/311: dwrite d11/d14/d23/f2d [0,4194304] 0 2024-03-22T02:28:51.433 INFO:tasks.workunit.client.0.smithi103.stdout:6/627: link d0/d14/d9a/lba d0/d88/d8e/dae/lcd 0 2024-03-22T02:28:51.434 INFO:tasks.workunit.client.1.smithi122.stdout:7/303: rename d0/d4/d20/d21/d39 to d0/d4/d22/d5f 0 2024-03-22T02:28:51.437 INFO:tasks.workunit.client.1.smithi122.stdout:8/415: rename d1/d3f/l7e to d1/d40/d51/l8e 0 2024-03-22T02:28:51.439 INFO:tasks.workunit.client.1.smithi122.stdout:3/304: dread d2/d9/dc/f36 [0,4194304] 0 2024-03-22T02:28:51.439 INFO:tasks.workunit.client.1.smithi122.stdout:8/416: stat d1/c1c 0 2024-03-22T02:28:51.440 INFO:tasks.workunit.client.0.smithi103.stdout:4/870: link d3/d31/f32 d3/d1e/d25/d65/d83/dc1/f124 0 2024-03-22T02:28:51.443 INFO:tasks.workunit.client.0.smithi103.stdout:4/871: mkdir d3/d31/d10f/d125 0 2024-03-22T02:28:51.444 INFO:tasks.workunit.client.1.smithi122.stdout:9/388: rename db/d23/d1f/d21 to db/df/d3e/d81 0 2024-03-22T02:28:51.444 INFO:tasks.workunit.client.0.smithi103.stdout:2/878: dread d11/d19/d4e/f11a [0,4194304] 0 2024-03-22T02:28:51.446 INFO:tasks.workunit.client.0.smithi103.stdout:5/898: truncate d0/d14/d23/d6f/d83/f11a 1064124 0 2024-03-22T02:28:51.447 INFO:tasks.workunit.client.1.smithi122.stdout:4/359: dread d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:51.447 INFO:tasks.workunit.client.0.smithi103.stdout:4/872: write d3/d1e/d25/d65/d83/dc1/f101 [821495,33049] 0 2024-03-22T02:28:51.448 INFO:tasks.workunit.client.1.smithi122.stdout:4/360: write d0/d25/f4d [576537,71954] 0 2024-03-22T02:28:51.449 INFO:tasks.workunit.client.0.smithi103.stdout:2/879: chown d11/d19/da7/dd3/laf 58 1 2024-03-22T02:28:51.449 INFO:tasks.workunit.client.1.smithi122.stdout:3/305: creat d2/d9/dc/df/d38/f5d x:0 0 0 2024-03-22T02:28:51.449 INFO:tasks.workunit.client.1.smithi122.stdout:4/361: chown d0/d12/f21 3 1 2024-03-22T02:28:51.449 INFO:tasks.workunit.client.0.smithi103.stdout:2/880: dread - d11/f10c zero size 2024-03-22T02:28:51.450 INFO:tasks.workunit.client.0.smithi103.stdout:4/873: symlink d3/d30/d45/l126 0 2024-03-22T02:28:51.450 INFO:tasks.workunit.client.1.smithi122.stdout:0/312: rename d11/d14/d23/d24/f4d to d11/d14/d1d/d27/d32/f6c 0 2024-03-22T02:28:51.452 INFO:tasks.workunit.client.0.smithi103.stdout:2/881: chown d11/f5a 428698368 1 2024-03-22T02:28:51.453 INFO:tasks.workunit.client.1.smithi122.stdout:0/313: write d11/d14/d23/f2d [4026923,11039] 0 2024-03-22T02:28:51.454 INFO:tasks.workunit.client.1.smithi122.stdout:0/314: chown d11/d14/d1d 125052786 1 2024-03-22T02:28:51.454 INFO:tasks.workunit.client.0.smithi103.stdout:4/874: readlink d3/l9 0 2024-03-22T02:28:51.456 INFO:tasks.workunit.client.1.smithi122.stdout:3/306: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:51.456 INFO:tasks.workunit.client.1.smithi122.stdout:6/423: write d7/de/dd/d20/f3f [680130,56468] 0 2024-03-22T02:28:51.458 INFO:tasks.workunit.client.0.smithi103.stdout:4/875: unlink d3/d1e/f7f 0 2024-03-22T02:28:51.460 INFO:tasks.workunit.client.0.smithi103.stdout:6/628: dread d0/d3/d22/f96 [0,4194304] 0 2024-03-22T02:28:51.461 INFO:tasks.workunit.client.1.smithi122.stdout:7/304: rename d0/d25/d46/d48/l54 to d0/d25/l60 0 2024-03-22T02:28:51.462 INFO:tasks.workunit.client.0.smithi103.stdout:6/629: write d0/d14/d57/d68/f62 [525988,108053] 0 2024-03-22T02:28:51.465 INFO:tasks.workunit.client.1.smithi122.stdout:0/315: creat d11/d14/d23/f6d x:0 0 0 2024-03-22T02:28:51.467 INFO:tasks.workunit.client.0.smithi103.stdout:5/899: dread d0/da2/dca/ff8 [0,4194304] 0 2024-03-22T02:28:51.467 INFO:tasks.workunit.client.1.smithi122.stdout:3/307: dwrite d2/d9/dc/f16 [0,4194304] 0 2024-03-22T02:28:51.468 INFO:tasks.workunit.client.1.smithi122.stdout:4/362: dwrite d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:51.468 INFO:tasks.workunit.client.1.smithi122.stdout:4/363: readlink d0/d25/l65 0 2024-03-22T02:28:51.471 INFO:tasks.workunit.client.0.smithi103.stdout:6/630: dwrite d0/d3/f74 [0,4194304] 0 2024-03-22T02:28:51.474 INFO:tasks.workunit.client.1.smithi122.stdout:8/417: getdents d1 0 2024-03-22T02:28:51.475 INFO:tasks.workunit.client.1.smithi122.stdout:3/308: dwrite d2/f55 [0,4194304] 0 2024-03-22T02:28:51.477 INFO:tasks.workunit.client.1.smithi122.stdout:0/316: write d11/d4f/d5a/f5c [3984233,128194] 0 2024-03-22T02:28:51.478 INFO:tasks.workunit.client.1.smithi122.stdout:4/364: mkdir d0/d12/d73 0 2024-03-22T02:28:51.479 INFO:tasks.workunit.client.0.smithi103.stdout:5/900: dwrite d0/d14/d23/d36/d95/f7d [0,4194304] 0 2024-03-22T02:28:51.479 INFO:tasks.workunit.client.1.smithi122.stdout:9/389: write db/df/f15 [1865999,66458] 0 2024-03-22T02:28:51.485 INFO:tasks.workunit.client.1.smithi122.stdout:8/418: dwrite d1/d9/f77 [0,4194304] 0 2024-03-22T02:28:51.486 INFO:tasks.workunit.client.1.smithi122.stdout:0/317: stat d11/d47/d5d 0 2024-03-22T02:28:51.487 INFO:tasks.workunit.client.1.smithi122.stdout:7/305: rmdir d0 39 2024-03-22T02:28:51.489 INFO:tasks.workunit.client.1.smithi122.stdout:6/424: link d7/de/d27/c49 d7/de/dd/df/c83 0 2024-03-22T02:28:51.491 INFO:tasks.workunit.client.1.smithi122.stdout:9/390: rmdir db/d23/d1f/d57/d66/d73 39 2024-03-22T02:28:51.492 INFO:tasks.workunit.client.1.smithi122.stdout:0/318: dread d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:51.494 INFO:tasks.workunit.client.1.smithi122.stdout:8/419: symlink d1/d3f/d45/l8f 0 2024-03-22T02:28:51.495 INFO:tasks.workunit.client.1.smithi122.stdout:7/306: fdatasync d0/fd 0 2024-03-22T02:28:51.496 INFO:tasks.workunit.client.1.smithi122.stdout:4/365: creat d0/d12/d61/f74 x:0 0 0 2024-03-22T02:28:51.497 INFO:tasks.workunit.client.1.smithi122.stdout:7/307: stat d0/d4/f2d 0 2024-03-22T02:28:51.498 INFO:tasks.workunit.client.1.smithi122.stdout:4/366: read - d0/f6a zero size 2024-03-22T02:28:51.499 INFO:tasks.workunit.client.1.smithi122.stdout:8/420: readlink d1/d40/d51/l8e 0 2024-03-22T02:28:51.499 INFO:tasks.workunit.client.1.smithi122.stdout:7/308: chown d0/d4/d20/l38 6 1 2024-03-22T02:28:51.500 INFO:tasks.workunit.client.0.smithi103.stdout:2/882: dwrite d11/d25/d63/d45/f77 [0,4194304] 0 2024-03-22T02:28:51.501 INFO:tasks.workunit.client.1.smithi122.stdout:0/319: dwrite d11/d14/d23/f2d [0,4194304] 0 2024-03-22T02:28:51.502 INFO:tasks.workunit.client.1.smithi122.stdout:0/320: chown c7 4076189 1 2024-03-22T02:28:51.504 INFO:tasks.workunit.client.1.smithi122.stdout:6/425: mknod d7/de/dd/c84 0 2024-03-22T02:28:51.505 INFO:tasks.workunit.client.1.smithi122.stdout:4/367: mkdir d0/d12/d15/d2d/d75 0 2024-03-22T02:28:51.506 INFO:tasks.workunit.client.1.smithi122.stdout:6/426: read - d7/de/dd/d20/f81 zero size 2024-03-22T02:28:51.507 INFO:tasks.workunit.client.1.smithi122.stdout:7/309: symlink d0/d4e/l61 0 2024-03-22T02:28:51.508 INFO:tasks.workunit.client.0.smithi103.stdout:4/876: dwrite d3/f8 [0,4194304] 0 2024-03-22T02:28:51.508 INFO:tasks.workunit.client.1.smithi122.stdout:9/391: link db/d1b/d6c/l75 db/df/d3e/d81/d25/l82 0 2024-03-22T02:28:51.511 INFO:tasks.workunit.client.1.smithi122.stdout:4/368: symlink d0/d12/d15/d42/l76 0 2024-03-22T02:28:51.513 INFO:tasks.workunit.client.1.smithi122.stdout:8/421: dwrite d1/d9/d50/f59 [0,4194304] 0 2024-03-22T02:28:51.513 INFO:tasks.workunit.client.1.smithi122.stdout:9/392: truncate db/d1b/d28/f6b 4379512 0 2024-03-22T02:28:51.516 INFO:tasks.workunit.client.1.smithi122.stdout:9/393: chown db/df/d3e/d81/d29 36 1 2024-03-22T02:28:51.516 INFO:tasks.workunit.client.1.smithi122.stdout:4/369: chown d0/d12/d15/l26 198251 1 2024-03-22T02:28:51.519 INFO:tasks.workunit.client.0.smithi103.stdout:2/883: dwrite d11/d25/d63/d45/d10a/f11c [0,4194304] 0 2024-03-22T02:28:51.519 INFO:tasks.workunit.client.0.smithi103.stdout:4/877: truncate d3/d1e/fba 644422 0 2024-03-22T02:28:51.520 INFO:tasks.workunit.client.1.smithi122.stdout:7/310: mknod d0/d4/c62 0 2024-03-22T02:28:51.520 INFO:tasks.workunit.client.1.smithi122.stdout:6/427: dread d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:28:51.522 INFO:tasks.workunit.client.0.smithi103.stdout:2/884: mknod d11/d19/da7/dd3/df1/c127 0 2024-03-22T02:28:51.525 INFO:tasks.workunit.client.0.smithi103.stdout:4/878: mkdir d3/d1e/d68/dbc/d127 0 2024-03-22T02:28:51.526 INFO:tasks.workunit.client.1.smithi122.stdout:4/370: mkdir d0/d12/d15/d2d/d77 0 2024-03-22T02:28:51.526 INFO:tasks.workunit.client.1.smithi122.stdout:6/428: chown d7/de/dd/d20/d2d/d54/d76/c7b 17 1 2024-03-22T02:28:51.527 INFO:tasks.workunit.client.1.smithi122.stdout:4/371: readlink d0/d12/d15/d42/l52 0 2024-03-22T02:28:51.532 INFO:tasks.workunit.client.0.smithi103.stdout:2/885: dwrite d11/d25/d63/f11b [0,4194304] 0 2024-03-22T02:28:51.532 INFO:tasks.workunit.client.1.smithi122.stdout:8/422: dwrite d1/d8/d2f/f38 [0,4194304] 0 2024-03-22T02:28:51.532 INFO:tasks.workunit.client.0.smithi103.stdout:4/879: creat d3/d1e/d25/d65/d83/dc1/f128 x:0 0 0 2024-03-22T02:28:51.533 INFO:tasks.workunit.client.0.smithi103.stdout:9/831: sync 2024-03-22T02:28:51.533 INFO:tasks.workunit.client.1.smithi122.stdout:9/394: dwrite db/d23/d1e/f70 [0,4194304] 0 2024-03-22T02:28:51.534 INFO:tasks.workunit.client.0.smithi103.stdout:2/886: stat d11/d25/d46/d48/l4d 0 2024-03-22T02:28:51.539 INFO:tasks.workunit.client.1.smithi122.stdout:9/395: creat db/d23/f83 x:0 0 0 2024-03-22T02:28:51.541 INFO:tasks.workunit.client.1.smithi122.stdout:9/396: write db/df/d3e/d81/d29/f3f [1398244,129883] 0 2024-03-22T02:28:51.541 INFO:tasks.workunit.client.1.smithi122.stdout:6/429: link d7/c41 d7/de/dd/d20/c85 0 2024-03-22T02:28:51.543 INFO:tasks.workunit.client.1.smithi122.stdout:6/430: dread - d7/de/d27/f80 zero size 2024-03-22T02:28:51.546 INFO:tasks.workunit.client.0.smithi103.stdout:2/887: dwrite d11/d25/d63/d45/d47/dae/fe3 [0,4194304] 0 2024-03-22T02:28:51.549 INFO:tasks.workunit.client.0.smithi103.stdout:2/888: symlink d11/d25/d63/d45/d47/d83/da3/l128 0 2024-03-22T02:28:51.550 INFO:tasks.workunit.client.1.smithi122.stdout:9/397: rename db/d1b/d6c/l72 to db/d23/d1f/l84 0 2024-03-22T02:28:51.551 INFO:tasks.workunit.client.0.smithi103.stdout:4/880: dwrite d3/d1e/d25/d65/f6c [0,4194304] 0 2024-03-22T02:28:51.554 INFO:tasks.workunit.client.0.smithi103.stdout:2/889: mknod d11/d25/d63/d45/ddd/c129 0 2024-03-22T02:28:51.556 INFO:tasks.workunit.client.1.smithi122.stdout:9/398: getdents db/df/d3e/d81 0 2024-03-22T02:28:51.556 INFO:tasks.workunit.client.0.smithi103.stdout:2/890: fdatasync f10 0 2024-03-22T02:28:51.558 INFO:tasks.workunit.client.0.smithi103.stdout:5/901: dwrite d0/d14/d23/d5c/d70/fc4 [0,4194304] 0 2024-03-22T02:28:51.561 INFO:tasks.workunit.client.0.smithi103.stdout:2/891: unlink d11/d19/d7e/l120 0 2024-03-22T02:28:51.562 INFO:tasks.workunit.client.0.smithi103.stdout:5/902: symlink d0/d14/d23/d36/d95/d96/de5/l141 0 2024-03-22T02:28:51.563 INFO:tasks.workunit.client.1.smithi122.stdout:9/399: read db/df/f15 [208267,95121] 0 2024-03-22T02:28:51.568 INFO:tasks.workunit.client.1.smithi122.stdout:9/400: rmdir db/df/d7e 0 2024-03-22T02:28:51.568 INFO:tasks.workunit.client.1.smithi122.stdout:9/401: chown db/d1b/f6a 771 1 2024-03-22T02:28:51.571 INFO:tasks.workunit.client.1.smithi122.stdout:9/402: creat db/d23/d1f/d57/d66/f85 x:0 0 0 2024-03-22T02:28:51.572 INFO:tasks.workunit.client.1.smithi122.stdout:9/403: chown db/df/f4a 0 1 2024-03-22T02:28:51.574 INFO:tasks.workunit.client.1.smithi122.stdout:9/404: mkdir db/df/d3e/d81/d3d/d86 0 2024-03-22T02:28:51.575 INFO:tasks.workunit.client.1.smithi122.stdout:9/405: creat db/d23/d1f/d57/f87 x:0 0 0 2024-03-22T02:28:51.576 INFO:tasks.workunit.client.1.smithi122.stdout:9/406: readlink db/df/d3e/d81/d3d/l52 0 2024-03-22T02:28:51.585 INFO:tasks.workunit.client.1.smithi122.stdout:2/344: sync 2024-03-22T02:28:51.587 INFO:tasks.workunit.client.1.smithi122.stdout:2/345: mkdir df/d1f/d33/d4d/d7a 0 2024-03-22T02:28:51.588 INFO:tasks.workunit.client.1.smithi122.stdout:2/346: readlink df/d1f/d3d/l6c 0 2024-03-22T02:28:51.594 INFO:tasks.workunit.client.0.smithi103.stdout:7/865: sync 2024-03-22T02:28:51.594 INFO:tasks.workunit.client.0.smithi103.stdout:3/978: sync 2024-03-22T02:28:51.594 INFO:tasks.workunit.client.0.smithi103.stdout:0/874: sync 2024-03-22T02:28:51.594 INFO:tasks.workunit.client.1.smithi122.stdout:0/321: dread d11/d16/d44/d4c/f4b [0,4194304] 0 2024-03-22T02:28:51.595 INFO:tasks.workunit.client.1.smithi122.stdout:2/347: dwrite df/d1f/d2a/d39/f62 [0,4194304] 0 2024-03-22T02:28:51.595 INFO:tasks.workunit.client.0.smithi103.stdout:6/631: truncate d0/d14/f93 3164366 0 2024-03-22T02:28:51.596 INFO:tasks.workunit.client.1.smithi122.stdout:2/348: chown df/d1f/d3a/d41/l42 2676479 1 2024-03-22T02:28:51.597 INFO:tasks.workunit.client.1.smithi122.stdout:0/322: truncate d11/d14/d1d/d27/d32/f6b 127461 0 2024-03-22T02:28:51.600 INFO:tasks.workunit.client.0.smithi103.stdout:2/892: dread d11/f13 [0,4194304] 0 2024-03-22T02:28:51.601 INFO:tasks.workunit.client.1.smithi122.stdout:2/349: symlink df/d1f/d2a/d51/d67/l7b 0 2024-03-22T02:28:51.601 INFO:tasks.workunit.client.0.smithi103.stdout:2/893: fdatasync f2 0 2024-03-22T02:28:51.602 INFO:tasks.workunit.client.1.smithi122.stdout:2/350: chown df/d1f/d33/l65 3 1 2024-03-22T02:28:51.602 INFO:tasks.workunit.client.1.smithi122.stdout:6/431: rmdir d7/de/d27 39 2024-03-22T02:28:51.603 INFO:tasks.workunit.client.0.smithi103.stdout:6/632: mkdir d0/d14/dc7/dce 0 2024-03-22T02:28:51.604 INFO:tasks.workunit.client.0.smithi103.stdout:0/875: symlink dd/d16/d4b/d96/d35/dc4/l114 0 2024-03-22T02:28:51.605 INFO:tasks.workunit.client.0.smithi103.stdout:9/832: dread d2/d5/f63 [4194304,4194304] 0 2024-03-22T02:28:51.606 INFO:tasks.workunit.client.0.smithi103.stdout:7/866: truncate d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f 1078979 0 2024-03-22T02:28:51.607 INFO:tasks.workunit.client.1.smithi122.stdout:2/351: mkdir df/d1f/d33/d6a/d7c 0 2024-03-22T02:28:51.609 INFO:tasks.workunit.client.0.smithi103.stdout:9/833: chown d2/la7 37424 1 2024-03-22T02:28:51.609 INFO:tasks.workunit.client.1.smithi122.stdout:0/323: getdents d11/d14/d23 0 2024-03-22T02:28:51.609 INFO:tasks.workunit.client.1.smithi122.stdout:2/352: read - df/d1f/d33/f46 zero size 2024-03-22T02:28:51.609 INFO:tasks.workunit.client.0.smithi103.stdout:9/834: dread - d2/d10/d43/d4b/d65/fc0 zero size 2024-03-22T02:28:51.610 INFO:tasks.workunit.client.0.smithi103.stdout:3/979: creat de/d1a/d10f/d9c/f14c x:0 0 0 2024-03-22T02:28:51.610 INFO:tasks.workunit.client.1.smithi122.stdout:2/353: fsync df/f13 0 2024-03-22T02:28:51.611 INFO:tasks.workunit.client.0.smithi103.stdout:3/980: readlink de/d12/d19/df2/l105 0 2024-03-22T02:28:51.611 INFO:tasks.workunit.client.1.smithi122.stdout:2/354: chown df/d1f/l38 5 1 2024-03-22T02:28:51.611 INFO:tasks.workunit.client.1.smithi122.stdout:0/324: mknod d11/d14/d1d/d27/d59/c6e 0 2024-03-22T02:28:51.613 INFO:tasks.workunit.client.0.smithi103.stdout:3/981: truncate de/d12/d19/fcb 142535 0 2024-03-22T02:28:51.615 INFO:tasks.workunit.client.1.smithi122.stdout:2/355: read f4 [341038,123126] 0 2024-03-22T02:28:51.616 INFO:tasks.workunit.client.0.smithi103.stdout:3/982: write de/d12/d8a/dda/f111 [4690899,44935] 0 2024-03-22T02:28:51.617 INFO:tasks.workunit.client.0.smithi103.stdout:0/876: creat dd/d34/f115 x:0 0 0 2024-03-22T02:28:51.618 INFO:tasks.workunit.client.1.smithi122.stdout:2/356: mknod df/d1f/d3a/d41/c7d 0 2024-03-22T02:28:51.618 INFO:tasks.workunit.client.0.smithi103.stdout:9/835: dwrite d2/d5/d4c/d56/fbc [0,4194304] 0 2024-03-22T02:28:51.619 INFO:tasks.workunit.client.0.smithi103.stdout:0/877: readlink dd/d16/d4b/lb3 0 2024-03-22T02:28:51.620 INFO:tasks.workunit.client.0.smithi103.stdout:6/633: link d0/d14/d57/d68/f28 d0/d3/d22/d58/d6f/d75/daa/fcf 0 2024-03-22T02:28:51.622 INFO:tasks.workunit.client.0.smithi103.stdout:2/894: link d11/d25/d63/d45/d47/d83/lb2 d11/d25/d63/d45/ddd/l12a 0 2024-03-22T02:28:51.624 INFO:tasks.workunit.client.1.smithi122.stdout:8/423: truncate d1/d40/f4a 859194 0 2024-03-22T02:28:51.626 INFO:tasks.workunit.client.1.smithi122.stdout:8/424: write d1/f3c [4520031,53382] 0 2024-03-22T02:28:51.627 INFO:tasks.workunit.client.0.smithi103.stdout:7/867: dread d1/d10/d11/d19/d29/d61/f96 [0,4194304] 0 2024-03-22T02:28:51.629 INFO:tasks.workunit.client.0.smithi103.stdout:9/836: rename d2/da/d18/dc4/dce to d2/da/d18/d1e/d34/d3d/d61/da0/df0 0 2024-03-22T02:28:51.630 INFO:tasks.workunit.client.1.smithi122.stdout:4/372: dwrite d0/d25/f28 [0,4194304] 0 2024-03-22T02:28:51.630 INFO:tasks.workunit.client.1.smithi122.stdout:0/325: dread f9 [4194304,4194304] 0 2024-03-22T02:28:51.631 INFO:tasks.workunit.client.0.smithi103.stdout:4/881: dwrite d3/d1e/d25/f2b [4194304,4194304] 0 2024-03-22T02:28:51.631 INFO:tasks.workunit.client.0.smithi103.stdout:0/878: creat dd/d16/d4b/d96/d29/d48/d94/f116 x:0 0 0 2024-03-22T02:28:51.632 INFO:tasks.workunit.client.1.smithi122.stdout:4/373: write d0/d25/f1e [2738048,32418] 0 2024-03-22T02:28:51.633 INFO:tasks.workunit.client.1.smithi122.stdout:8/425: mkdir d1/d60/d73/d90 0 2024-03-22T02:28:51.633 INFO:tasks.workunit.client.1.smithi122.stdout:8/426: stat d1/d22/l35 0 2024-03-22T02:28:51.634 INFO:tasks.workunit.client.0.smithi103.stdout:2/895: mkdir d11/d25/d46/d48/d12b 0 2024-03-22T02:28:51.636 INFO:tasks.workunit.client.0.smithi103.stdout:5/903: truncate d0/d14/d23/d36/d95/d96/fb0 1957984 0 2024-03-22T02:28:51.636 INFO:tasks.workunit.client.1.smithi122.stdout:0/326: symlink d11/d14/d1d/d27/d2f/l6f 0 2024-03-22T02:28:51.638 INFO:tasks.workunit.client.1.smithi122.stdout:6/432: dread d7/f15 [0,4194304] 0 2024-03-22T02:28:51.639 INFO:tasks.workunit.client.1.smithi122.stdout:4/374: creat d0/d41/f78 x:0 0 0 2024-03-22T02:28:51.639 INFO:tasks.workunit.client.0.smithi103.stdout:5/904: stat d0/d14/d23/d2a/d33/c11b 0 2024-03-22T02:28:51.640 INFO:tasks.workunit.client.1.smithi122.stdout:8/427: creat d1/d8/d2f/f91 x:0 0 0 2024-03-22T02:28:51.641 INFO:tasks.workunit.client.0.smithi103.stdout:7/868: rename d1/d10/d11/d70/d8a/d9d/dad to d1/d10/d11/d19/dc3/d112 0 2024-03-22T02:28:51.642 INFO:tasks.workunit.client.0.smithi103.stdout:6/634: creat d0/d3/d33/fd0 x:0 0 0 2024-03-22T02:28:51.643 INFO:tasks.workunit.client.1.smithi122.stdout:9/407: dwrite db/d23/d1f/f2b [0,4194304] 0 2024-03-22T02:28:51.644 INFO:tasks.workunit.client.0.smithi103.stdout:5/905: dread d0/d14/d23/d5c/d77/d125/f137 [8388608,4194304] 0 2024-03-22T02:28:51.645 INFO:tasks.workunit.client.1.smithi122.stdout:6/433: mkdir d7/de/dd/d20/d2d/d54/d76/d86 0 2024-03-22T02:28:51.647 INFO:tasks.workunit.client.1.smithi122.stdout:8/428: symlink d1/d3f/d45/l92 0 2024-03-22T02:28:51.648 INFO:tasks.workunit.client.0.smithi103.stdout:0/879: link dd/d16/d4b/cce dd/d16/d4b/d96/d29/d48/d5c/d103/c117 0 2024-03-22T02:28:51.650 INFO:tasks.workunit.client.1.smithi122.stdout:9/408: chown db/d23/c18 202503 1 2024-03-22T02:28:51.650 INFO:tasks.workunit.client.0.smithi103.stdout:4/882: link d3/d1e/d53/d8a/ld6 d3/d1e/d25/d65/dc9/d111/l129 0 2024-03-22T02:28:51.654 INFO:tasks.workunit.client.0.smithi103.stdout:2/896: rename d11/d25/d63/fe7 to d11/d25/d46/f12c 0 2024-03-22T02:28:51.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/327: link d11/f3d d11/d14/d1d/d27/d2f/f70 0 2024-03-22T02:28:51.655 INFO:tasks.workunit.client.1.smithi122.stdout:2/357: dwrite fd [0,4194304] 0 2024-03-22T02:28:51.658 INFO:tasks.workunit.client.1.smithi122.stdout:4/375: dwrite d0/d12/d15/d3d/f68 [0,4194304] 0 2024-03-22T02:28:51.661 INFO:tasks.workunit.client.0.smithi103.stdout:3/983: write de/d1a/d25/f8b [545938,7997] 0 2024-03-22T02:28:51.661 INFO:tasks.workunit.client.1.smithi122.stdout:0/328: truncate d11/d14/d1b/f3c 101268 0 2024-03-22T02:28:51.664 INFO:tasks.workunit.client.0.smithi103.stdout:4/883: dread d3/d1e/d53/fb7 [0,4194304] 0 2024-03-22T02:28:51.665 INFO:tasks.workunit.client.0.smithi103.stdout:2/897: dwrite d11/d25/d46/fca [0,4194304] 0 2024-03-22T02:28:51.665 INFO:tasks.workunit.client.1.smithi122.stdout:2/358: symlink df/d1f/d33/d4d/l7e 0 2024-03-22T02:28:51.668 INFO:tasks.workunit.client.0.smithi103.stdout:4/884: chown d3/d31/l40 799801 1 2024-03-22T02:28:51.669 INFO:tasks.workunit.client.1.smithi122.stdout:9/409: dwrite db/df/f80 [0,4194304] 0 2024-03-22T02:28:51.670 INFO:tasks.workunit.client.0.smithi103.stdout:9/837: rename d2/da/d18/d1e/d77/d8b/dae/lda to d2/d10/d43/d7e/d96/lf1 0 2024-03-22T02:28:51.671 INFO:tasks.workunit.client.1.smithi122.stdout:0/329: symlink d11/d16/l71 0 2024-03-22T02:28:51.671 INFO:tasks.workunit.client.0.smithi103.stdout:3/984: creat de/d6b/d80/d73/f14d x:0 0 0 2024-03-22T02:28:51.672 INFO:tasks.workunit.client.0.smithi103.stdout:4/885: truncate d3/d1e/d25/d65/dc9/f113 408490 0 2024-03-22T02:28:51.673 INFO:tasks.workunit.client.1.smithi122.stdout:6/434: link d7/de/dd/d20/f5d d7/de/dd/d20/d2d/d54/d76/d86/f87 0 2024-03-22T02:28:51.673 INFO:tasks.workunit.client.1.smithi122.stdout:9/410: dread - db/df/d3e/d81/d29/f32 zero size 2024-03-22T02:28:51.673 INFO:tasks.workunit.client.0.smithi103.stdout:2/898: creat d11/d25/d63/f12d x:0 0 0 2024-03-22T02:28:51.674 INFO:tasks.workunit.client.0.smithi103.stdout:6/635: rename d0/d14/f56 to d0/d88/d8e/fd1 0 2024-03-22T02:28:51.675 INFO:tasks.workunit.client.0.smithi103.stdout:4/886: chown d3/d1e/d25/d65/d83/dc1/fcc 913331 1 2024-03-22T02:28:51.676 INFO:tasks.workunit.client.0.smithi103.stdout:3/985: truncate de/d12/d30/d70/f141 891126 0 2024-03-22T02:28:51.676 INFO:tasks.workunit.client.1.smithi122.stdout:0/330: stat d11/d14/d23/d24/d33/d67 0 2024-03-22T02:28:51.676 INFO:tasks.workunit.client.0.smithi103.stdout:0/880: getdents dd/d16/d4b/d96/d64 0 2024-03-22T02:28:51.680 INFO:tasks.workunit.client.1.smithi122.stdout:2/359: dwrite df/d1f/f3c [0,4194304] 0 2024-03-22T02:28:51.680 INFO:tasks.workunit.client.1.smithi122.stdout:9/411: mknod db/df/d3e/d81/c88 0 2024-03-22T02:28:51.682 INFO:tasks.workunit.client.0.smithi103.stdout:0/881: chown dd/d16/d4b/d96/d29/d48/d94/f116 11 1 2024-03-22T02:28:51.682 INFO:tasks.workunit.client.1.smithi122.stdout:0/331: creat d11/d14/d23/f72 x:0 0 0 2024-03-22T02:28:51.683 INFO:tasks.workunit.client.0.smithi103.stdout:5/906: rename d0/d14/d23/d2a/lef to d0/da2/d12c/l142 0 2024-03-22T02:28:51.685 INFO:tasks.workunit.client.1.smithi122.stdout:4/376: dwrite d0/d25/f27 [0,4194304] 0 2024-03-22T02:28:51.685 INFO:tasks.workunit.client.1.smithi122.stdout:2/360: write df/d1f/f3c [69684,66674] 0 2024-03-22T02:28:51.685 INFO:tasks.workunit.client.0.smithi103.stdout:6/636: mkdir d0/d14/dd2 0 2024-03-22T02:28:51.685 INFO:tasks.workunit.client.0.smithi103.stdout:2/899: dwrite d11/f30 [4194304,4194304] 0 2024-03-22T02:28:51.686 INFO:tasks.workunit.client.1.smithi122.stdout:0/332: fsync d11/d14/d23/f2d 0 2024-03-22T02:28:51.688 INFO:tasks.workunit.client.0.smithi103.stdout:0/882: rmdir dd/d16/d4b/d96/d35/d82 39 2024-03-22T02:28:51.693 INFO:tasks.workunit.client.1.smithi122.stdout:6/435: dwrite d7/de/dd/d20/d2d/d54/f72 [0,4194304] 0 2024-03-22T02:28:51.693 INFO:tasks.workunit.client.1.smithi122.stdout:0/333: mknod d11/d14/d1d/d27/d2f/c73 0 2024-03-22T02:28:51.695 INFO:tasks.workunit.client.1.smithi122.stdout:5/323: sync 2024-03-22T02:28:51.698 INFO:tasks.workunit.client.0.smithi103.stdout:2/900: dwrite d11/f58 [4194304,4194304] 0 2024-03-22T02:28:51.703 INFO:tasks.workunit.client.0.smithi103.stdout:2/901: creat d11/d25/d63/f12e x:0 0 0 2024-03-22T02:28:51.704 INFO:tasks.workunit.client.1.smithi122.stdout:6/436: creat d7/de/dd/d23/f88 x:0 0 0 2024-03-22T02:28:51.704 INFO:tasks.workunit.client.1.smithi122.stdout:2/361: link df/d1f/d23/c45 df/d1f/d33/c7f 0 2024-03-22T02:28:51.705 INFO:tasks.workunit.client.0.smithi103.stdout:7/869: truncate d1/d10/d11/d19/d29/d61/d69/fa5 990305 0 2024-03-22T02:28:51.705 INFO:tasks.workunit.client.0.smithi103.stdout:2/902: chown d11/d19/da7/dd3/df1/lf8 1502 1 2024-03-22T02:28:51.706 INFO:tasks.workunit.client.1.smithi122.stdout:0/334: creat d11/d14/d1d/d27/d32/f74 x:0 0 0 2024-03-22T02:28:51.706 INFO:tasks.workunit.client.1.smithi122.stdout:4/377: dwrite d0/d12/d15/d42/d4e/de/f66 [0,4194304] 0 2024-03-22T02:28:51.706 INFO:tasks.workunit.client.1.smithi122.stdout:6/437: fdatasync d7/de/dd/d39/f6f 0 2024-03-22T02:28:51.707 INFO:tasks.workunit.client.0.smithi103.stdout:2/903: mknod d11/d25/d63/d45/ddd/c12f 0 2024-03-22T02:28:51.708 INFO:tasks.workunit.client.0.smithi103.stdout:2/904: chown d11/d19/da7/dd3/dd9 475082 1 2024-03-22T02:28:51.709 INFO:tasks.workunit.client.1.smithi122.stdout:6/438: chown d7/de/f68 0 1 2024-03-22T02:28:51.710 INFO:tasks.workunit.client.0.smithi103.stdout:2/905: write d11/d25/d63/d45/f77 [3237199,81106] 0 2024-03-22T02:28:51.713 INFO:tasks.workunit.client.1.smithi122.stdout:0/335: getdents d11/d14/d23/d24/d33 0 2024-03-22T02:28:51.715 INFO:tasks.workunit.client.1.smithi122.stdout:0/336: fdatasync d11/d14/d1d/d27/d2f/f70 0 2024-03-22T02:28:51.716 INFO:tasks.workunit.client.1.smithi122.stdout:6/439: dwrite d7/de/dd/d23/f47 [0,4194304] 0 2024-03-22T02:28:51.717 INFO:tasks.workunit.client.1.smithi122.stdout:0/337: fsync d11/d14/d23/f2d 0 2024-03-22T02:28:51.718 INFO:tasks.workunit.client.0.smithi103.stdout:2/906: dwrite d11/d25/d63/f12e [0,4194304] 0 2024-03-22T02:28:51.720 INFO:tasks.workunit.client.0.smithi103.stdout:2/907: chown d11/d25/d63/d45/ddd/c12f 13 1 2024-03-22T02:28:51.725 INFO:tasks.workunit.client.0.smithi103.stdout:5/907: dread d0/d14/d23/fcf [0,4194304] 0 2024-03-22T02:28:51.725 INFO:tasks.workunit.client.1.smithi122.stdout:6/440: creat d7/de/dd/d20/d2d/d54/d76/f89 x:0 0 0 2024-03-22T02:28:51.727 INFO:tasks.workunit.client.1.smithi122.stdout:0/338: creat d11/d14/d23/d31/f75 x:0 0 0 2024-03-22T02:28:51.728 INFO:tasks.workunit.client.0.smithi103.stdout:5/908: stat d0/d14/d23/d36/d95/d96/cc2 0 2024-03-22T02:28:51.728 INFO:tasks.workunit.client.1.smithi122.stdout:6/441: readlink d7/de/d27/l66 0 2024-03-22T02:28:51.730 INFO:tasks.workunit.client.1.smithi122.stdout:6/442: truncate d7/de/dd/d39/f77 1158349 0 2024-03-22T02:28:51.730 INFO:tasks.workunit.client.0.smithi103.stdout:2/908: dwrite d11/d25/d46/f9c [0,4194304] 0 2024-03-22T02:28:51.731 INFO:tasks.workunit.client.1.smithi122.stdout:0/339: dread d11/d14/d1d/f41 [0,4194304] 0 2024-03-22T02:28:51.731 INFO:tasks.workunit.client.1.smithi122.stdout:6/443: stat d7/de/dd/d20/d2d/f5f 0 2024-03-22T02:28:51.733 INFO:tasks.workunit.client.1.smithi122.stdout:0/340: dread - d11/d14/d1d/f6a zero size 2024-03-22T02:28:51.734 INFO:tasks.workunit.client.1.smithi122.stdout:6/444: mkdir d7/de/dd/d20/d2d/d8a 0 2024-03-22T02:28:51.736 INFO:tasks.workunit.client.0.smithi103.stdout:2/909: rename d11/da0/fcf to d11/d19/d4e/dfc/f130 0 2024-03-22T02:28:51.739 INFO:tasks.workunit.client.0.smithi103.stdout:2/910: symlink d11/l131 0 2024-03-22T02:28:51.741 INFO:tasks.workunit.client.1.smithi122.stdout:6/445: dread d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:28:51.741 INFO:tasks.workunit.client.0.smithi103.stdout:6/637: write d0/d3/d22/d58/d6f/d75/f95 [80502,76705] 0 2024-03-22T02:28:51.745 INFO:tasks.workunit.client.1.smithi122.stdout:5/324: dread d8/d3b/f69 [4194304,4194304] 0 2024-03-22T02:28:51.746 INFO:tasks.workunit.client.0.smithi103.stdout:3/986: dwrite de/d12/d19/fbf [0,4194304] 0 2024-03-22T02:28:51.747 INFO:tasks.workunit.client.0.smithi103.stdout:3/987: chown de/d12/d5d 5348084 1 2024-03-22T02:28:51.748 INFO:tasks.workunit.client.0.smithi103.stdout:6/638: creat d0/d3/d22/fd3 x:0 0 0 2024-03-22T02:28:51.749 INFO:tasks.workunit.client.1.smithi122.stdout:2/362: dread df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:51.749 INFO:tasks.workunit.client.0.smithi103.stdout:6/639: chown d0/d14/c2f 0 1 2024-03-22T02:28:51.750 INFO:tasks.workunit.client.1.smithi122.stdout:2/363: chown df/d1f/d33/d6a 295 1 2024-03-22T02:28:51.750 INFO:tasks.workunit.client.0.smithi103.stdout:3/988: mkdir de/d12/d8f/d140/d14e 0 2024-03-22T02:28:51.751 INFO:tasks.workunit.client.1.smithi122.stdout:5/325: rename d8/d1e/d22/d3a/d38 to d8/da/d28/d35/d71 0 2024-03-22T02:28:51.751 INFO:tasks.workunit.client.1.smithi122.stdout:2/364: write df/d1f/d33/d34/f5a [1424133,110186] 0 2024-03-22T02:28:51.752 INFO:tasks.workunit.client.1.smithi122.stdout:0/341: dread d11/d14/d1d/d27/d2f/f3a [0,4194304] 0 2024-03-22T02:28:51.753 INFO:tasks.workunit.client.0.smithi103.stdout:6/640: truncate d0/d88/f8a 498657 0 2024-03-22T02:28:51.753 INFO:tasks.workunit.client.1.smithi122.stdout:0/342: stat d11 0 2024-03-22T02:28:51.754 INFO:tasks.workunit.client.1.smithi122.stdout:5/326: fsync d8/d1e/d22/d3a/d46/d4f/f54 0 2024-03-22T02:28:51.755 INFO:tasks.workunit.client.1.smithi122.stdout:2/365: creat df/d1f/d33/f80 x:0 0 0 2024-03-22T02:28:51.756 INFO:tasks.workunit.client.0.smithi103.stdout:3/989: creat de/d12/d4d/f14f x:0 0 0 2024-03-22T02:28:51.756 INFO:tasks.workunit.client.1.smithi122.stdout:0/343: creat d11/d16/d44/f76 x:0 0 0 2024-03-22T02:28:51.757 INFO:tasks.workunit.client.0.smithi103.stdout:7/870: write d1/dfa/fb0 [618750,92051] 0 2024-03-22T02:28:51.757 INFO:tasks.workunit.client.0.smithi103.stdout:6/641: stat d0/d88/d8e/dae/lcd 0 2024-03-22T02:28:51.759 INFO:tasks.workunit.client.1.smithi122.stdout:5/327: rename d8/f17 to d8/da/d28/d35/d71/f72 0 2024-03-22T02:28:51.760 INFO:tasks.workunit.client.1.smithi122.stdout:0/344: write d11/d16/d44/d4c/f4b [1762129,65156] 0 2024-03-22T02:28:51.761 INFO:tasks.workunit.client.1.smithi122.stdout:5/328: write d8/d3b/f5a [71484,109682] 0 2024-03-22T02:28:51.762 INFO:tasks.workunit.client.1.smithi122.stdout:5/329: fdatasync d8/d1a/f1c 0 2024-03-22T02:28:51.763 INFO:tasks.workunit.client.0.smithi103.stdout:7/871: mkdir d1/d10/d11/d19/d29/d61/d69/d113 0 2024-03-22T02:28:51.764 INFO:tasks.workunit.client.1.smithi122.stdout:2/366: dread df/d1f/f2d [4194304,4194304] 0 2024-03-22T02:28:51.765 INFO:tasks.workunit.client.0.smithi103.stdout:6/642: getdents d0/d3/d22/d58/d6f/d75 0 2024-03-22T02:28:51.767 INFO:tasks.workunit.client.1.smithi122.stdout:2/367: unlink df/d1f/d33/l48 0 2024-03-22T02:28:51.768 INFO:tasks.workunit.client.0.smithi103.stdout:9/838: dread d2/d5/d4c/f5c [0,4194304] 0 2024-03-22T02:28:51.768 INFO:tasks.workunit.client.0.smithi103.stdout:7/872: mknod d1/d10/d11/d19/d29/d61/c114 0 2024-03-22T02:28:51.769 INFO:tasks.workunit.client.1.smithi122.stdout:0/345: getdents d11 0 2024-03-22T02:28:51.770 INFO:tasks.workunit.client.0.smithi103.stdout:6/643: mkdir d0/d3/d22/d58/d6f/d75/daa/dd4 0 2024-03-22T02:28:51.770 INFO:tasks.workunit.client.1.smithi122.stdout:5/330: creat d8/da/f73 x:0 0 0 2024-03-22T02:28:51.772 INFO:tasks.workunit.client.0.smithi103.stdout:7/873: symlink d1/d10/d11/d19/d29/d61/d69/l115 0 2024-03-22T02:28:51.776 INFO:tasks.workunit.client.1.smithi122.stdout:0/346: rename d11/d14/f26 to d11/d14/d23/f77 0 2024-03-22T02:28:51.776 INFO:tasks.workunit.client.1.smithi122.stdout:2/368: dwrite df/d1f/d3a/f58 [0,4194304] 0 2024-03-22T02:28:51.777 INFO:tasks.workunit.client.0.smithi103.stdout:6/644: symlink d0/d14/ld5 0 2024-03-22T02:28:51.778 INFO:tasks.workunit.client.0.smithi103.stdout:5/909: dread d0/d14/d23/f24 [4194304,4194304] 0 2024-03-22T02:28:51.778 INFO:tasks.workunit.client.0.smithi103.stdout:9/839: dwrite d2/da/d18/d1e/d34/d3d/d4d/f8f [0,4194304] 0 2024-03-22T02:28:51.779 INFO:tasks.workunit.client.0.smithi103.stdout:6/645: readlink d0/d3/d33/la6 0 2024-03-22T02:28:51.780 INFO:tasks.workunit.client.0.smithi103.stdout:7/874: truncate d1/d10/d11/d19/d29/d61/f45 70677 0 2024-03-22T02:28:51.781 INFO:tasks.workunit.client.1.smithi122.stdout:5/331: dwrite d8/d1a/f5c [0,4194304] 0 2024-03-22T02:28:51.783 INFO:tasks.workunit.client.1.smithi122.stdout:0/347: symlink d11/d47/d5d/l78 0 2024-03-22T02:28:51.784 INFO:tasks.workunit.client.0.smithi103.stdout:0/883: dread dd/d16/d4b/d96/d64/dab/dc6/d111/fcd [0,4194304] 0 2024-03-22T02:28:51.787 INFO:tasks.workunit.client.0.smithi103.stdout:2/911: dwrite d11/d25/d63/d45/d47/d83/f60 [0,4194304] 0 2024-03-22T02:28:51.788 INFO:tasks.workunit.client.1.smithi122.stdout:9/412: rename db/fc to db/df/d3e/d81/d3d/d86/f89 0 2024-03-22T02:28:51.789 INFO:tasks.workunit.client.0.smithi103.stdout:2/912: chown d11/d19/l4a 8 1 2024-03-22T02:28:51.789 INFO:tasks.workunit.client.0.smithi103.stdout:3/990: write de/d6b/d80/d4f/f64 [269993,84242] 0 2024-03-22T02:28:51.790 INFO:tasks.workunit.client.0.smithi103.stdout:3/991: stat fd 0 2024-03-22T02:28:51.790 INFO:tasks.workunit.client.1.smithi122.stdout:5/332: mkdir d8/d74 0 2024-03-22T02:28:51.792 INFO:tasks.workunit.client.1.smithi122.stdout:0/348: unlink d11/d14/d1d/d27/d2f/c73 0 2024-03-22T02:28:51.793 INFO:tasks.workunit.client.1.smithi122.stdout:6/446: write d7/de/dd/d20/f5d [533688,106676] 0 2024-03-22T02:28:51.794 INFO:tasks.workunit.client.0.smithi103.stdout:5/910: creat d0/d14/d23/d5c/d77/df2/dd9/f143 x:0 0 0 2024-03-22T02:28:51.795 INFO:tasks.workunit.client.1.smithi122.stdout:9/413: rmdir db/df/d3e/d81 39 2024-03-22T02:28:51.797 INFO:tasks.workunit.client.0.smithi103.stdout:6/646: creat d0/d14/dd2/fd6 x:0 0 0 2024-03-22T02:28:51.797 INFO:tasks.workunit.client.0.smithi103.stdout:6/647: read - d0/d3/f92 zero size 2024-03-22T02:28:51.797 INFO:tasks.workunit.client.1.smithi122.stdout:2/369: getdents df/d1f/d3a/d41 0 2024-03-22T02:28:51.798 INFO:tasks.workunit.client.0.smithi103.stdout:7/875: unlink d1/d10/d11/d19/d86/l109 0 2024-03-22T02:28:51.799 INFO:tasks.workunit.client.1.smithi122.stdout:5/333: dwrite d8/d1a/f5c [0,4194304] 0 2024-03-22T02:28:51.799 INFO:tasks.workunit.client.0.smithi103.stdout:5/911: chown d0/d14/d23/d5c/fb8 104 1 2024-03-22T02:28:51.799 INFO:tasks.workunit.client.0.smithi103.stdout:0/884: dwrite dd/d34/f115 [0,4194304] 0 2024-03-22T02:28:51.802 INFO:tasks.workunit.client.1.smithi122.stdout:0/349: creat d11/d14/d23/d24/d33/f79 x:0 0 0 2024-03-22T02:28:51.804 INFO:tasks.workunit.client.0.smithi103.stdout:2/913: chown d11/cfa 519900560 1 2024-03-22T02:28:51.806 INFO:tasks.workunit.client.0.smithi103.stdout:3/992: rmdir de/d12/d5d/d102 39 2024-03-22T02:28:51.807 INFO:tasks.workunit.client.1.smithi122.stdout:9/414: write db/d23/d1f/d57/d66/d73/f76 [2623116,11735] 0 2024-03-22T02:28:51.808 INFO:tasks.workunit.client.1.smithi122.stdout:0/350: symlink d11/d47/d5d/l7a 0 2024-03-22T02:28:51.809 INFO:tasks.workunit.client.1.smithi122.stdout:2/370: mknod df/d1f/d33/d6a/d7c/c81 0 2024-03-22T02:28:51.811 INFO:tasks.workunit.client.1.smithi122.stdout:5/334: creat d8/d1e/d51/f75 x:0 0 0 2024-03-22T02:28:51.812 INFO:tasks.workunit.client.1.smithi122.stdout:2/371: creat df/d1f/d3a/d41/f82 x:0 0 0 2024-03-22T02:28:51.813 INFO:tasks.workunit.client.0.smithi103.stdout:6/648: mkdir d0/d3/d22/d58/d6f/d75/db0/dd7 0 2024-03-22T02:28:51.814 INFO:tasks.workunit.client.0.smithi103.stdout:7/876: write d1/f7e [3882221,9639] 0 2024-03-22T02:28:51.815 INFO:tasks.workunit.client.1.smithi122.stdout:0/351: rename d11/f3b to d11/d14/d1d/d27/d2f/f7b 0 2024-03-22T02:28:51.816 INFO:tasks.workunit.client.0.smithi103.stdout:3/993: dread de/d12/d8a/d93/fac [0,4194304] 0 2024-03-22T02:28:51.816 INFO:tasks.workunit.client.0.smithi103.stdout:6/649: chown d0/d3/d22/f96 2 1 2024-03-22T02:28:51.817 INFO:tasks.workunit.client.0.smithi103.stdout:2/914: dread d11/d25/d63/f12e [0,4194304] 0 2024-03-22T02:28:51.818 INFO:tasks.workunit.client.1.smithi122.stdout:9/415: dwrite db/d23/d1f/d57/d66/f85 [0,4194304] 0 2024-03-22T02:28:51.818 INFO:tasks.workunit.client.1.smithi122.stdout:2/372: mknod df/d1f/d33/c83 0 2024-03-22T02:28:51.818 INFO:tasks.workunit.client.1.smithi122.stdout:5/335: write d8/da/f10 [6793483,103334] 0 2024-03-22T02:28:51.820 INFO:tasks.workunit.client.0.smithi103.stdout:6/650: stat d0/d3/d22/d58/d6f/d75/db0/dd7 0 2024-03-22T02:28:51.820 INFO:tasks.workunit.client.1.smithi122.stdout:2/373: fdatasync df/d1f/d33/d34/f5a 0 2024-03-22T02:28:51.820 INFO:tasks.workunit.client.1.smithi122.stdout:9/416: write db/f13 [4005976,62755] 0 2024-03-22T02:28:51.820 INFO:tasks.workunit.client.1.smithi122.stdout:5/336: chown d8/d1e/d22/c4a 124 1 2024-03-22T02:28:51.822 INFO:tasks.workunit.client.0.smithi103.stdout:5/912: link d0/d14/d23/fbc d0/f144 0 2024-03-22T02:28:51.822 INFO:tasks.workunit.client.1.smithi122.stdout:9/417: write db/df/f4a [15546,17418] 0 2024-03-22T02:28:51.823 INFO:tasks.workunit.client.1.smithi122.stdout:5/337: symlink d8/da/l76 0 2024-03-22T02:28:51.824 INFO:tasks.workunit.client.1.smithi122.stdout:2/374: fdatasync df/d1f/d33/d34/f43 0 2024-03-22T02:28:51.826 INFO:tasks.workunit.client.1.smithi122.stdout:2/375: write df/d1f/d2a/d51/d67/f6e [116512,32612] 0 2024-03-22T02:28:51.826 INFO:tasks.workunit.client.1.smithi122.stdout:5/338: rmdir d8/da/d28 39 2024-03-22T02:28:51.830 INFO:tasks.workunit.client.0.smithi103.stdout:3/994: fdatasync de/d1a/d10f/f68 0 2024-03-22T02:28:51.831 INFO:tasks.workunit.client.1.smithi122.stdout:2/376: rmdir df/d1f/d33/d4d 39 2024-03-22T02:28:51.831 INFO:tasks.workunit.client.0.smithi103.stdout:6/651: dread - d0/d3/f46 zero size 2024-03-22T02:28:51.832 INFO:tasks.workunit.client.1.smithi122.stdout:5/339: write d8/d1e/d22/d3a/d46/f64 [836199,49918] 0 2024-03-22T02:28:51.833 INFO:tasks.workunit.client.1.smithi122.stdout:9/418: link db/df/f7d db/df/d3e/d81/d3d/d79/f8a 0 2024-03-22T02:28:51.834 INFO:tasks.workunit.client.1.smithi122.stdout:9/419: chown db/d1b/d6c/l44 5 1 2024-03-22T02:28:51.834 INFO:tasks.workunit.client.1.smithi122.stdout:5/340: mknod d8/d74/c77 0 2024-03-22T02:28:51.835 INFO:tasks.workunit.client.0.smithi103.stdout:5/913: dread d0/d14/d23/d5c/d77/df2/f4b [4194304,4194304] 0 2024-03-22T02:28:51.842 INFO:tasks.workunit.client.1.smithi122.stdout:6/447: dread f5 [0,4194304] 0 2024-03-22T02:28:51.843 INFO:tasks.workunit.client.1.smithi122.stdout:9/420: dwrite db/df/d3e/d81/f42 [0,4194304] 0 2024-03-22T02:28:51.845 INFO:tasks.workunit.client.0.smithi103.stdout:0/885: dread dd/d16/d4b/d96/d64/d6a/f74 [4194304,4194304] 0 2024-03-22T02:28:51.850 INFO:tasks.workunit.client.0.smithi103.stdout:9/840: write d2/d10/d43/d7e/f91 [804963,343] 0 2024-03-22T02:28:51.852 INFO:tasks.workunit.client.1.smithi122.stdout:5/341: dread d8/d1e/d22/d3a/d46/f64 [0,4194304] 0 2024-03-22T02:28:51.853 INFO:tasks.workunit.client.0.smithi103.stdout:9/841: chown d2/d10/d43/f7a 7 1 2024-03-22T02:28:51.853 INFO:tasks.workunit.client.1.smithi122.stdout:5/342: write d8/d1a/f48 [911641,54676] 0 2024-03-22T02:28:51.855 INFO:tasks.workunit.client.0.smithi103.stdout:0/886: dwrite dd/d16/d4b/fd6 [0,4194304] 0 2024-03-22T02:28:51.857 INFO:tasks.workunit.client.0.smithi103.stdout:9/842: chown d2/d5/d7b/d7d/dd4/dd8/lde 0 1 2024-03-22T02:28:51.860 INFO:tasks.workunit.client.0.smithi103.stdout:2/915: symlink d11/d25/d63/d45/l132 0 2024-03-22T02:28:51.864 INFO:tasks.workunit.client.0.smithi103.stdout:0/887: dwrite dd/d16/d4b/f105 [0,4194304] 0 2024-03-22T02:28:51.865 INFO:tasks.workunit.client.1.smithi122.stdout:6/448: dread d7/de/dd/f3a [0,4194304] 0 2024-03-22T02:28:51.865 INFO:tasks.workunit.client.1.smithi122.stdout:6/449: stat d7/f3b 0 2024-03-22T02:28:51.866 INFO:tasks.workunit.client.0.smithi103.stdout:5/914: symlink d0/da2/l145 0 2024-03-22T02:28:51.867 INFO:tasks.workunit.client.1.smithi122.stdout:6/450: dread - d7/de/dd/d20/d2d/f5f zero size 2024-03-22T02:28:51.867 INFO:tasks.workunit.client.1.smithi122.stdout:6/451: truncate d7/de/dd/d20/f81 993279 0 2024-03-22T02:28:51.867 INFO:tasks.workunit.client.0.smithi103.stdout:0/888: chown dd/d16/d4b/d96/d29/f3e 7516 1 2024-03-22T02:28:51.869 INFO:tasks.workunit.client.0.smithi103.stdout:9/843: unlink d2/da/d18/fe9 0 2024-03-22T02:28:51.872 INFO:tasks.workunit.client.0.smithi103.stdout:3/995: mknod de/d12/d5d/d92/c150 0 2024-03-22T02:28:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:51 smithi103 ceph-mon[29258]: pgmap v19: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 297 MiB/s wr, 551 op/s 2024-03-22T02:28:51.875 INFO:tasks.workunit.client.0.smithi103.stdout:5/915: creat d0/da2/f146 x:0 0 0 2024-03-22T02:28:51.877 INFO:tasks.workunit.client.0.smithi103.stdout:7/877: dwrite d1/d10/d11/d19/d29/d61/d69/f84 [0,4194304] 0 2024-03-22T02:28:51.878 INFO:tasks.workunit.client.1.smithi122.stdout:1/420: sync 2024-03-22T02:28:51.878 INFO:tasks.workunit.client.0.smithi103.stdout:5/916: chown d0/d14/d23/d6f/c135 36445 1 2024-03-22T02:28:51.882 INFO:tasks.workunit.client.0.smithi103.stdout:2/916: dread d11/d25/d63/f96 [0,4194304] 0 2024-03-22T02:28:51.882 INFO:tasks.workunit.client.0.smithi103.stdout:3/996: fsync de/f96 0 2024-03-22T02:28:51.883 INFO:tasks.workunit.client.0.smithi103.stdout:7/878: stat d1/l6 0 2024-03-22T02:28:51.886 INFO:tasks.workunit.client.0.smithi103.stdout:5/917: unlink d0/d14/d23/d5c/d77/fa8 0 2024-03-22T02:28:51.887 INFO:tasks.workunit.client.0.smithi103.stdout:3/997: write de/d12/d30/f35 [1969391,118616] 0 2024-03-22T02:28:51.889 INFO:tasks.workunit.client.0.smithi103.stdout:2/917: read - d11/d25/d46/db6/fec zero size 2024-03-22T02:28:51.890 INFO:tasks.workunit.client.0.smithi103.stdout:0/889: dwrite f4 [0,4194304] 0 2024-03-22T02:28:51.890 INFO:tasks.workunit.client.0.smithi103.stdout:5/918: chown d0/d14/d23/d2a/d33/l3d 0 1 2024-03-22T02:28:51.891 INFO:tasks.workunit.client.0.smithi103.stdout:3/998: chown de/d12/d5d/c112 213310 1 2024-03-22T02:28:51.893 INFO:tasks.workunit.client.0.smithi103.stdout:2/918: write d11/d25/d46/db6/dfb/f10f [85383,129398] 0 2024-03-22T02:28:51.894 INFO:tasks.workunit.client.0.smithi103.stdout:9/844: dread d2/da/f23 [0,4194304] 0 2024-03-22T02:28:51.894 INFO:tasks.workunit.client.0.smithi103.stdout:0/890: write dd/d16/d4b/d96/d64/f71 [31447,8615] 0 2024-03-22T02:28:51.897 INFO:tasks.workunit.client.0.smithi103.stdout:0/891: read - dd/d16/d4b/d96/d29/d48/d5c/d9b/fc1 zero size 2024-03-22T02:28:51.900 INFO:tasks.workunit.client.0.smithi103.stdout:3/999: creat de/d1a/d25/d121/f151 x:0 0 0 2024-03-22T02:28:51.902 INFO:tasks.workunit.client.0.smithi103.stdout:9/845: creat d2/da/d18/d1e/d77/d8b/dae/dcb/ff2 x:0 0 0 2024-03-22T02:28:51.905 INFO:tasks.workunit.client.0.smithi103.stdout:5/919: rename d0/f13c to d0/d14/d23/d36/d95/f147 0 2024-03-22T02:28:51.907 INFO:tasks.workunit.client.0.smithi103.stdout:5/920: readlink d0/d14/d23/d2a/d33/db5/dfb/d10c/l10a 0 2024-03-22T02:28:51.908 INFO:tasks.workunit.client.1.smithi122.stdout:1/421: dread dc/f2c [0,4194304] 0 2024-03-22T02:28:51.908 INFO:tasks.workunit.client.1.smithi122.stdout:1/422: stat dc/f38 0 2024-03-22T02:28:51.910 INFO:tasks.workunit.client.0.smithi103.stdout:0/892: creat dd/d16/d4b/d96/d35/f118 x:0 0 0 2024-03-22T02:28:51.916 INFO:tasks.workunit.client.0.smithi103.stdout:0/893: symlink dd/d16/d4b/d96/d29/d48/d94/l119 0 2024-03-22T02:28:51.916 INFO:tasks.workunit.client.0.smithi103.stdout:5/921: dwrite d0/d14/d23/d36/d95/d96/daa/f12f [0,4194304] 0 2024-03-22T02:28:51.919 INFO:tasks.workunit.client.0.smithi103.stdout:6/652: dwrite d0/d14/d57/d68/f40 [0,4194304] 0 2024-03-22T02:28:51.922 INFO:tasks.workunit.client.0.smithi103.stdout:0/894: truncate dd/fc3 680242 0 2024-03-22T02:28:51.926 INFO:tasks.workunit.client.0.smithi103.stdout:9/846: dread d2/da/f25 [0,4194304] 0 2024-03-22T02:28:51.928 INFO:tasks.workunit.client.0.smithi103.stdout:6/653: mkdir d0/d3/d22/d58/dd8 0 2024-03-22T02:28:51.929 INFO:tasks.workunit.client.0.smithi103.stdout:2/919: write d11/d19/d7e/f80 [545739,129391] 0 2024-03-22T02:28:51.932 INFO:tasks.workunit.client.1.smithi122.stdout:9/421: creat db/df/d3e/f8b x:0 0 0 2024-03-22T02:28:51.933 INFO:tasks.workunit.client.1.smithi122.stdout:5/343: mkdir d8/d1e/d22/d78 0 2024-03-22T02:28:51.934 INFO:tasks.workunit.client.1.smithi122.stdout:6/452: mknod d7/de/dd/d20/d2d/d54/d76/c8b 0 2024-03-22T02:28:51.935 INFO:tasks.workunit.client.1.smithi122.stdout:6/453: fsync d7/de/dd/d20/d2d/d54/d76/f89 0 2024-03-22T02:28:51.936 INFO:tasks.workunit.client.0.smithi103.stdout:9/847: rename d2/d10/d43/d4b to d2/d10/d43/d9b/df3 0 2024-03-22T02:28:51.937 INFO:tasks.workunit.client.1.smithi122.stdout:1/423: unlink l5 0 2024-03-22T02:28:51.940 INFO:tasks.workunit.client.0.smithi103.stdout:2/920: rmdir d11/d19/d4e 39 2024-03-22T02:28:51.941 INFO:tasks.workunit.client.1.smithi122.stdout:9/422: rmdir db/df/d3e/d81/d29 39 2024-03-22T02:28:51.942 INFO:tasks.workunit.client.1.smithi122.stdout:6/454: dwrite f3 [4194304,4194304] 0 2024-03-22T02:28:51.942 INFO:tasks.workunit.client.1.smithi122.stdout:9/423: chown db/d23/f1a 16321 1 2024-03-22T02:28:51.945 INFO:tasks.workunit.client.0.smithi103.stdout:6/654: fsync d0/d3/d22/f3d 0 2024-03-22T02:28:51.947 INFO:tasks.workunit.client.1.smithi122.stdout:6/455: write f5 [1645191,29794] 0 2024-03-22T02:28:51.947 INFO:tasks.workunit.client.0.smithi103.stdout:5/922: dwrite d0/d14/d23/fcf [0,4194304] 0 2024-03-22T02:28:51.949 INFO:tasks.workunit.client.0.smithi103.stdout:2/921: write d11/d19/d4e/dd5/f108 [3611820,19068] 0 2024-03-22T02:28:51.949 INFO:tasks.workunit.client.1.smithi122.stdout:6/456: dread - d7/de/d27/f80 zero size 2024-03-22T02:28:51.952 INFO:tasks.workunit.client.1.smithi122.stdout:2/377: truncate df/d1f/d2a/d39/f62 3028623 0 2024-03-22T02:28:51.954 INFO:tasks.workunit.client.0.smithi103.stdout:9/848: write d2/d5/d7b/d7d/dd4/dd8/fe2 [4332322,60206] 0 2024-03-22T02:28:51.954 INFO:tasks.workunit.client.1.smithi122.stdout:9/424: unlink db/df/d3e/d81/d3d/d86/f89 0 2024-03-22T02:28:51.955 INFO:tasks.workunit.client.0.smithi103.stdout:0/895: getdents dd/d34 0 2024-03-22T02:28:51.956 INFO:tasks.workunit.client.1.smithi122.stdout:9/425: write db/d23/f1a [2415390,52234] 0 2024-03-22T02:28:51.956 INFO:tasks.workunit.client.1.smithi122.stdout:0/352: truncate d11/d14/d1d/d27/d2f/f7b 3563565 0 2024-03-22T02:28:51.958 INFO:tasks.workunit.client.0.smithi103.stdout:2/922: mknod d11/d25/d63/d45/d47/c133 0 2024-03-22T02:28:51.958 INFO:tasks.workunit.client.1.smithi122.stdout:2/378: chown df/d1f/d3d/c55 17683421 1 2024-03-22T02:28:51.958 INFO:tasks.workunit.client.1.smithi122.stdout:2/379: chown df/d1f 546993 1 2024-03-22T02:28:51.959 INFO:tasks.workunit.client.1.smithi122.stdout:5/344: link d8/da/c13 d8/da/c79 0 2024-03-22T02:28:51.960 INFO:tasks.workunit.client.0.smithi103.stdout:0/896: write dd/d16/d4b/d96/d64/dab/dc6/d89/f8b [3999941,32949] 0 2024-03-22T02:28:51.961 INFO:tasks.workunit.client.0.smithi103.stdout:2/923: rmdir d11/d25/d46/db6 39 2024-03-22T02:28:51.962 INFO:tasks.workunit.client.1.smithi122.stdout:6/457: symlink d7/de/dd/d20/l8c 0 2024-03-22T02:28:51.963 INFO:tasks.workunit.client.1.smithi122.stdout:1/424: write dc/d3b/d46/d47/f59 [2058657,90308] 0 2024-03-22T02:28:51.963 INFO:tasks.workunit.client.0.smithi103.stdout:9/849: unlink d2/d5/d7b/d7d/dd4/dd8/lde 0 2024-03-22T02:28:51.963 INFO:tasks.workunit.client.0.smithi103.stdout:2/924: write d11/d25/fad [444859,97721] 0 2024-03-22T02:28:51.964 INFO:tasks.workunit.client.1.smithi122.stdout:9/426: chown db/d1b/d6c/f35 0 1 2024-03-22T02:28:51.965 INFO:tasks.workunit.client.1.smithi122.stdout:2/380: mkdir df/d1f/d3a/d4f/d84 0 2024-03-22T02:28:51.966 INFO:tasks.workunit.client.1.smithi122.stdout:0/353: creat d11/d14/d23/d24/d33/d67/f7c x:0 0 0 2024-03-22T02:28:51.966 INFO:tasks.workunit.client.0.smithi103.stdout:5/923: dwrite d0/d14/d23/d5c/fc9 [0,4194304] 0 2024-03-22T02:28:51.967 INFO:tasks.workunit.client.1.smithi122.stdout:9/427: dread db/df/f4a [0,4194304] 0 2024-03-22T02:28:51.967 INFO:tasks.workunit.client.0.smithi103.stdout:9/850: chown d2/d5/d7b/d7d/dd4/dd8/fe2 66 1 2024-03-22T02:28:51.969 INFO:tasks.workunit.client.0.smithi103.stdout:5/924: write d0/d14/d23/d5c/fc9 [475666,63611] 0 2024-03-22T02:28:51.971 INFO:tasks.workunit.client.1.smithi122.stdout:5/345: mkdir d8/d1e/d22/d3a/d7a 0 2024-03-22T02:28:51.972 INFO:tasks.workunit.client.0.smithi103.stdout:9/851: symlink d2/da/d18/lf4 0 2024-03-22T02:28:51.972 INFO:tasks.workunit.client.0.smithi103.stdout:6/655: write d0/d14/d57/d68/f55 [71762,112335] 0 2024-03-22T02:28:51.973 INFO:tasks.workunit.client.1.smithi122.stdout:2/381: dwrite f4 [4194304,4194304] 0 2024-03-22T02:28:51.974 INFO:tasks.workunit.client.1.smithi122.stdout:2/382: chown df/d1f/d33/f46 98 1 2024-03-22T02:28:51.979 INFO:tasks.workunit.client.1.smithi122.stdout:2/383: dread df/d1f/d2a/d51/d67/f6e [0,4194304] 0 2024-03-22T02:28:51.979 INFO:tasks.workunit.client.0.smithi103.stdout:9/852: rename d2/d10/d43/d7e/f91 to d2/d5/d7b/d7d/ff5 0 2024-03-22T02:28:51.980 INFO:tasks.workunit.client.1.smithi122.stdout:2/384: chown df/d1f/d3a/d41/l42 14833711 1 2024-03-22T02:28:51.980 INFO:tasks.workunit.client.1.smithi122.stdout:2/385: stat df/l35 0 2024-03-22T02:28:51.981 INFO:tasks.workunit.client.0.smithi103.stdout:9/853: stat d2/d10/d43/d7e/da4/dcd/cc2 0 2024-03-22T02:28:51.981 INFO:tasks.workunit.client.1.smithi122.stdout:0/354: symlink d11/d14/d23/l7d 0 2024-03-22T02:28:51.984 INFO:tasks.workunit.client.0.smithi103.stdout:5/925: dwrite d0/da2/dca/fdd [0,4194304] 0 2024-03-22T02:28:51.985 INFO:tasks.workunit.client.0.smithi103.stdout:9/854: chown d2/da/d18/d1e/d34/d3d/d46/f6a 14190675 1 2024-03-22T02:28:51.987 INFO:tasks.workunit.client.0.smithi103.stdout:6/656: dread d0/d14/d57/d68/f28 [0,4194304] 0 2024-03-22T02:28:51.988 INFO:tasks.workunit.client.1.smithi122.stdout:2/386: symlink df/d1f/d2a/d51/l85 0 2024-03-22T02:28:51.990 INFO:tasks.workunit.client.1.smithi122.stdout:1/425: write dc/d3b/d46/d47/f50 [1961750,97599] 0 2024-03-22T02:28:51.991 INFO:tasks.workunit.client.1.smithi122.stdout:5/346: symlink d8/l7b 0 2024-03-22T02:28:51.991 INFO:tasks.workunit.client.0.smithi103.stdout:9/855: fsync d2/d5/d7b/d7d/fa6 0 2024-03-22T02:28:51.991 INFO:tasks.workunit.client.0.smithi103.stdout:5/926: write d0/d14/d23/d2a/d33/db5/dbd/ded/d103/f122 [3091579,55993] 0 2024-03-22T02:28:51.992 INFO:tasks.workunit.client.1.smithi122.stdout:5/347: chown d8/f30 9996 1 2024-03-22T02:28:51.992 INFO:tasks.workunit.client.0.smithi103.stdout:9/856: write d2/da/d18/f6c [4440108,25164] 0 2024-03-22T02:28:51.993 INFO:tasks.workunit.client.1.smithi122.stdout:5/348: readlink d8/d1e/l6e 0 2024-03-22T02:28:51.994 INFO:tasks.workunit.client.1.smithi122.stdout:6/458: truncate d7/de/dd/d20/f3f 279772 0 2024-03-22T02:28:51.996 INFO:tasks.workunit.client.0.smithi103.stdout:2/925: dread d11/d25/d46/f29 [0,4194304] 0 2024-03-22T02:28:51.997 INFO:tasks.workunit.client.1.smithi122.stdout:6/459: write d7/de/f34 [4333986,73848] 0 2024-03-22T02:28:51.997 INFO:tasks.workunit.client.0.smithi103.stdout:2/926: chown d11/d19/d4e/c73 854 1 2024-03-22T02:28:51.997 INFO:tasks.workunit.client.0.smithi103.stdout:5/927: unlink d0/d14/d23/d36/d95/d96/de5/l141 0 2024-03-22T02:28:51.998 INFO:tasks.workunit.client.1.smithi122.stdout:2/387: creat df/d1f/d23/f86 x:0 0 0 2024-03-22T02:28:51.998 INFO:tasks.workunit.client.1.smithi122.stdout:0/355: dwrite f9 [0,4194304] 0 2024-03-22T02:28:51.998 INFO:tasks.workunit.client.0.smithi103.stdout:5/928: readlink d0/d14/l16 0 2024-03-22T02:28:52.001 INFO:tasks.workunit.client.1.smithi122.stdout:1/426: link dc/d3b/d46/d4d/d67/f7b dc/d4c/f92 0 2024-03-22T02:28:52.001 INFO:tasks.workunit.client.0.smithi103.stdout:0/897: dread dd/d16/d4b/f50 [4194304,4194304] 0 2024-03-22T02:28:52.002 INFO:tasks.workunit.client.1.smithi122.stdout:6/460: chown d7/de/dd/d20/d2d/d54/d76/l67 16140407 1 2024-03-22T02:28:52.002 INFO:tasks.workunit.client.1.smithi122.stdout:5/349: unlink d8/da/c13 0 2024-03-22T02:28:52.002 INFO:tasks.workunit.client.0.smithi103.stdout:0/898: write dd/f15 [89062,33365] 0 2024-03-22T02:28:52.003 INFO:tasks.workunit.client.0.smithi103.stdout:2/927: mkdir d11/d25/d63/d45/d47/dae/d106/d134 0 2024-03-22T02:28:52.006 INFO:tasks.workunit.client.0.smithi103.stdout:6/657: write d0/d3/d33/f60 [1116663,44730] 0 2024-03-22T02:28:52.008 INFO:tasks.workunit.client.0.smithi103.stdout:2/928: creat d11/dc5/f135 x:0 0 0 2024-03-22T02:28:52.009 INFO:tasks.workunit.client.1.smithi122.stdout:2/388: dwrite df/d1f/d33/d34/f5f [0,4194304] 0 2024-03-22T02:28:52.009 INFO:tasks.workunit.client.1.smithi122.stdout:6/461: rename d7/de/dd/d20/d2d/d54/d76/l4d to d7/de/d27/l8d 0 2024-03-22T02:28:52.011 INFO:tasks.workunit.client.0.smithi103.stdout:5/929: dwrite d0/d14/d23/d5c/d77/d125/f140 [0,4194304] 0 2024-03-22T02:28:52.012 INFO:tasks.workunit.client.0.smithi103.stdout:2/929: mknod d11/d19/da7/dd3/dd9/c136 0 2024-03-22T02:28:52.014 INFO:tasks.workunit.client.1.smithi122.stdout:5/350: dwrite d8/d3b/f69 [0,4194304] 0 2024-03-22T02:28:52.014 INFO:tasks.workunit.client.1.smithi122.stdout:2/389: unlink df/d1f/d33/d4d/l7e 0 2024-03-22T02:28:52.016 INFO:tasks.workunit.client.1.smithi122.stdout:2/390: write df/d1f/d2a/f2f [1574625,75597] 0 2024-03-22T02:28:52.016 INFO:tasks.workunit.client.0.smithi103.stdout:5/930: rmdir d0/d14/d23/d6f/d83 39 2024-03-22T02:28:52.018 INFO:tasks.workunit.client.1.smithi122.stdout:2/391: write df/d1f/d3d/d52/f5d [194851,126694] 0 2024-03-22T02:28:52.019 INFO:tasks.workunit.client.1.smithi122.stdout:6/462: symlink d7/de/dd/d20/d2d/d54/d76/d86/l8e 0 2024-03-22T02:28:52.019 INFO:tasks.workunit.client.0.smithi103.stdout:6/658: creat d0/d3/d22/d58/fd9 x:0 0 0 2024-03-22T02:28:52.022 INFO:tasks.workunit.client.1.smithi122.stdout:6/463: write d7/de/dd/d20/f5d [330592,86356] 0 2024-03-22T02:28:52.023 INFO:tasks.workunit.client.1.smithi122.stdout:5/351: mknod d8/d1e/d22/d3a/d46/d4f/d55/c7c 0 2024-03-22T02:28:52.023 INFO:tasks.workunit.client.0.smithi103.stdout:5/931: creat d0/d14/d23/d36/d95/d96/de5/f148 x:0 0 0 2024-03-22T02:28:52.024 INFO:tasks.workunit.client.0.smithi103.stdout:2/930: creat d11/d25/d63/d45/d47/d83/d78/d125/f137 x:0 0 0 2024-03-22T02:28:52.027 INFO:tasks.workunit.client.0.smithi103.stdout:6/659: symlink d0/d88/d8e/dae/lda 0 2024-03-22T02:28:52.030 INFO:tasks.workunit.client.0.smithi103.stdout:5/932: unlink d0/d3e/c4a 0 2024-03-22T02:28:52.031 INFO:tasks.workunit.client.1.smithi122.stdout:0/356: dread d11/d14/d1d/d27/d2f/d4e/f66 [0,4194304] 0 2024-03-22T02:28:52.031 INFO:tasks.workunit.client.0.smithi103.stdout:6/660: rename d0/d3/d33/fd0 to d0/d14/d9a/fdb 0 2024-03-22T02:28:52.032 INFO:tasks.workunit.client.1.smithi122.stdout:5/352: write d8/d3b/f62 [422519,77295] 0 2024-03-22T02:28:52.032 INFO:tasks.workunit.client.0.smithi103.stdout:2/931: mknod d11/d25/d46/d48/d12b/c138 0 2024-03-22T02:28:52.032 INFO:tasks.workunit.client.0.smithi103.stdout:6/661: write d0/d3/d22/fab [721750,33573] 0 2024-03-22T02:28:52.033 INFO:tasks.workunit.client.1.smithi122.stdout:5/353: readlink d8/l2a 0 2024-03-22T02:28:52.033 INFO:tasks.workunit.client.0.smithi103.stdout:2/932: dread - d11/dc5/f135 zero size 2024-03-22T02:28:52.035 INFO:tasks.workunit.client.1.smithi122.stdout:9/428: dread db/df/d3e/d81/d25/d41/f54 [0,4194304] 0 2024-03-22T02:28:52.036 INFO:tasks.workunit.client.0.smithi103.stdout:2/933: dread - d11/d25/d46/d40/feb zero size 2024-03-22T02:28:52.036 INFO:tasks.workunit.client.1.smithi122.stdout:2/392: dwrite fd [4194304,4194304] 0 2024-03-22T02:28:52.037 INFO:tasks.workunit.client.0.smithi103.stdout:9/857: dread d2/da/d18/d1e/d34/d3d/d46/f6a [0,4194304] 0 2024-03-22T02:28:52.038 INFO:tasks.workunit.client.0.smithi103.stdout:6/662: mkdir d0/d3/ddc 0 2024-03-22T02:28:52.039 INFO:tasks.workunit.client.1.smithi122.stdout:6/464: mkdir d7/de/dd/d20/d8f 0 2024-03-22T02:28:52.039 INFO:tasks.workunit.client.1.smithi122.stdout:0/357: symlink d11/d14/l7e 0 2024-03-22T02:28:52.040 INFO:tasks.workunit.client.0.smithi103.stdout:2/934: readlink d11/d25/d63/d45/d47/d83/l57 0 2024-03-22T02:28:52.040 INFO:tasks.workunit.client.0.smithi103.stdout:2/935: chown d11/da0 1891670 1 2024-03-22T02:28:52.041 INFO:tasks.workunit.client.1.smithi122.stdout:9/429: creat db/d23/d1e/f8c x:0 0 0 2024-03-22T02:28:52.042 INFO:tasks.workunit.client.1.smithi122.stdout:6/465: chown d7/de/d27/f6a 6866346 1 2024-03-22T02:28:52.044 INFO:tasks.workunit.client.1.smithi122.stdout:5/354: creat d8/d1e/d22/d3a/f7d x:0 0 0 2024-03-22T02:28:52.045 INFO:tasks.workunit.client.0.smithi103.stdout:0/899: dread dd/f5b [0,4194304] 0 2024-03-22T02:28:52.045 INFO:tasks.workunit.client.1.smithi122.stdout:9/430: chown db/df/d3e/d81/d29/f32 5267 1 2024-03-22T02:28:52.046 INFO:tasks.workunit.client.1.smithi122.stdout:0/358: truncate d11/d14/d1d/d27/d32/f6b 170441 0 2024-03-22T02:28:52.046 INFO:tasks.workunit.client.1.smithi122.stdout:6/466: unlink d7/de/dd/d39/f6f 0 2024-03-22T02:28:52.047 INFO:tasks.workunit.client.1.smithi122.stdout:2/393: dwrite df/f15 [0,4194304] 0 2024-03-22T02:28:52.050 INFO:tasks.workunit.client.1.smithi122.stdout:9/431: write db/d23/f11 [2538363,64946] 0 2024-03-22T02:28:52.050 INFO:tasks.workunit.client.0.smithi103.stdout:6/663: dwrite d0/d3/d22/d58/f89 [0,4194304] 0 2024-03-22T02:28:52.051 INFO:tasks.workunit.client.0.smithi103.stdout:0/900: creat dd/d16/d4b/d96/d35/dc4/df9/f11a x:0 0 0 2024-03-22T02:28:52.052 INFO:tasks.workunit.client.1.smithi122.stdout:1/427: read dc/d3b/d46/d47/f59 [896856,73041] 0 2024-03-22T02:28:52.055 INFO:tasks.workunit.client.1.smithi122.stdout:5/355: chown d8/da/d28/d42/l6d 58989 1 2024-03-22T02:28:52.055 INFO:tasks.workunit.client.1.smithi122.stdout:1/428: dread - dc/d3b/d46/d4d/f7a zero size 2024-03-22T02:28:52.056 INFO:tasks.workunit.client.1.smithi122.stdout:2/394: write df/d1f/d2a/d51/d67/f71 [385691,79301] 0 2024-03-22T02:28:52.058 INFO:tasks.workunit.client.1.smithi122.stdout:5/356: chown d8/f19 3 1 2024-03-22T02:28:52.058 INFO:tasks.workunit.client.1.smithi122.stdout:0/359: fsync d11/f3d 0 2024-03-22T02:28:52.060 INFO:tasks.workunit.client.1.smithi122.stdout:1/429: chown dc/d3b/c44 80 1 2024-03-22T02:28:52.061 INFO:tasks.workunit.client.0.smithi103.stdout:2/936: write d11/de8/f104 [660810,40491] 0 2024-03-22T02:28:52.061 INFO:tasks.workunit.client.0.smithi103.stdout:0/901: dread dd/d34/fad [0,4194304] 0 2024-03-22T02:28:52.062 INFO:tasks.workunit.client.1.smithi122.stdout:6/467: dwrite d7/de/f34 [0,4194304] 0 2024-03-22T02:28:52.065 INFO:tasks.workunit.client.0.smithi103.stdout:6/664: dwrite d0/d3/d22/fab [0,4194304] 0 2024-03-22T02:28:52.066 INFO:tasks.workunit.client.1.smithi122.stdout:3/309: sync 2024-03-22T02:28:52.066 INFO:tasks.workunit.client.1.smithi122.stdout:2/395: symlink df/d1f/d2a/d51/d67/l87 0 2024-03-22T02:28:52.066 INFO:tasks.workunit.client.0.smithi103.stdout:2/937: chown d11/dc5/f126 0 1 2024-03-22T02:28:52.066 INFO:tasks.workunit.client.0.smithi103.stdout:9/858: dread d2/d10/d43/f7a [0,4194304] 0 2024-03-22T02:28:52.067 INFO:tasks.workunit.client.0.smithi103.stdout:0/902: readlink dd/d16/d4b/d96/d64/dab/dc6/l41 0 2024-03-22T02:28:52.067 INFO:tasks.workunit.client.0.smithi103.stdout:9/859: stat d2/d5/d7b/d7d/cbd 0 2024-03-22T02:28:52.068 INFO:tasks.workunit.client.1.smithi122.stdout:5/357: write d8/da/d28/f3d [574854,23724] 0 2024-03-22T02:28:52.072 INFO:tasks.workunit.client.1.smithi122.stdout:0/360: dwrite d11/d14/d1d/d27/d32/f74 [0,4194304] 0 2024-03-22T02:28:52.073 INFO:tasks.workunit.client.0.smithi103.stdout:2/938: creat d11/dc5/f139 x:0 0 0 2024-03-22T02:28:52.074 INFO:tasks.workunit.client.0.smithi103.stdout:5/933: dwrite d0/d14/d23/d5c/d70/f87 [0,4194304] 0 2024-03-22T02:28:52.074 INFO:tasks.workunit.client.1.smithi122.stdout:2/396: creat df/d1f/d3d/f88 x:0 0 0 2024-03-22T02:28:52.074 INFO:tasks.workunit.client.0.smithi103.stdout:6/665: creat d0/d3/d33/fdd x:0 0 0 2024-03-22T02:28:52.075 INFO:tasks.workunit.client.1.smithi122.stdout:3/310: truncate d2/d9/dc/df/d38/f3b 745022 0 2024-03-22T02:28:52.081 INFO:tasks.workunit.client.0.smithi103.stdout:2/939: creat d11/dc5/f13a x:0 0 0 2024-03-22T02:28:52.083 INFO:tasks.workunit.client.0.smithi103.stdout:0/903: dwrite dd/d16/d4b/d96/f68 [0,4194304] 0 2024-03-22T02:28:52.083 INFO:tasks.workunit.client.1.smithi122.stdout:0/361: mknod d11/d14/d23/d24/d33/d67/c7f 0 2024-03-22T02:28:52.087 INFO:tasks.workunit.client.1.smithi122.stdout:3/311: rmdir d2 39 2024-03-22T02:28:52.090 INFO:tasks.workunit.client.0.smithi103.stdout:0/904: rmdir dd 39 2024-03-22T02:28:52.092 INFO:tasks.workunit.client.1.smithi122.stdout:6/468: rmdir d7/de/dd/d20/d2d/d8a 0 2024-03-22T02:28:52.092 INFO:tasks.workunit.client.1.smithi122.stdout:0/362: dread d11/d14/d1d/d27/d32/f74 [0,4194304] 0 2024-03-22T02:28:52.093 INFO:tasks.workunit.client.1.smithi122.stdout:1/430: getdents dc/d3b/d8d 0 2024-03-22T02:28:52.095 INFO:tasks.workunit.client.0.smithi103.stdout:2/940: rename d11/d19/d7e/f98 to d11/f13b 0 2024-03-22T02:28:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:51 smithi122 ceph-mon[36403]: pgmap v19: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 297 MiB/s wr, 551 op/s 2024-03-22T02:28:52.100 INFO:tasks.workunit.client.0.smithi103.stdout:0/905: dwrite dd/d34/d53/d8c/fa0 [4194304,4194304] 0 2024-03-22T02:28:52.101 INFO:tasks.workunit.client.1.smithi122.stdout:0/363: dwrite d11/d14/d1d/d27/d32/f5f [0,4194304] 0 2024-03-22T02:28:52.101 INFO:tasks.workunit.client.1.smithi122.stdout:1/431: dwrite dc/d55/f5f [0,4194304] 0 2024-03-22T02:28:52.103 INFO:tasks.workunit.client.1.smithi122.stdout:3/312: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:28:52.103 INFO:tasks.workunit.client.0.smithi103.stdout:2/941: dwrite d11/d25/d63/d45/d47/dae/fe3 [0,4194304] 0 2024-03-22T02:28:52.107 INFO:tasks.workunit.client.0.smithi103.stdout:2/942: symlink d11/d25/d46/db6/l13c 0 2024-03-22T02:28:52.107 INFO:tasks.workunit.client.1.smithi122.stdout:1/432: getdents dc/d3b/d46/d47 0 2024-03-22T02:28:52.112 INFO:tasks.workunit.client.0.smithi103.stdout:2/943: dread d11/d25/d46/db6/dfb/f10f [0,4194304] 0 2024-03-22T02:28:52.114 INFO:tasks.workunit.client.1.smithi122.stdout:9/432: dread db/d23/d1f/d57/d66/d73/f76 [0,4194304] 0 2024-03-22T02:28:52.116 INFO:tasks.workunit.client.1.smithi122.stdout:3/313: dwrite d2/f2c [0,4194304] 0 2024-03-22T02:28:52.117 INFO:tasks.workunit.client.0.smithi103.stdout:9/860: dwrite d2/d5/fb5 [0,4194304] 0 2024-03-22T02:28:52.121 INFO:tasks.workunit.client.0.smithi103.stdout:2/944: rename d11/d25/d46/db6/f102 to d11/d19/d5d/f13d 0 2024-03-22T02:28:52.123 INFO:tasks.workunit.client.1.smithi122.stdout:9/433: unlink db/df/d3e/d81/d25/l82 0 2024-03-22T02:28:52.123 INFO:tasks.workunit.client.1.smithi122.stdout:1/433: dwrite dc/d4c/f62 [0,4194304] 0 2024-03-22T02:28:52.123 INFO:tasks.workunit.client.0.smithi103.stdout:2/945: fsync d11/f5a 0 2024-03-22T02:28:52.125 INFO:tasks.workunit.client.1.smithi122.stdout:5/358: truncate d8/d1a/f5c 341466 0 2024-03-22T02:28:52.128 INFO:tasks.workunit.client.0.smithi103.stdout:5/934: write d0/d14/d23/d36/d95/f147 [814255,20839] 0 2024-03-22T02:28:52.132 INFO:tasks.workunit.client.0.smithi103.stdout:6/666: dwrite d0/f41 [0,4194304] 0 2024-03-22T02:28:52.133 INFO:tasks.workunit.client.1.smithi122.stdout:2/397: dwrite df/d1f/d33/d34/f43 [0,4194304] 0 2024-03-22T02:28:52.136 INFO:tasks.workunit.client.0.smithi103.stdout:2/946: dwrite d11/d25/d46/fc2 [0,4194304] 0 2024-03-22T02:28:52.138 INFO:tasks.workunit.client.0.smithi103.stdout:6/667: dread d0/d88/f8a [0,4194304] 0 2024-03-22T02:28:52.143 INFO:tasks.workunit.client.0.smithi103.stdout:5/935: link d0/d14/d23/d2a/d33/db5/dbd/ded/d103/f122 d0/d14/d23/d6f/d83/f149 0 2024-03-22T02:28:52.143 INFO:tasks.workunit.client.0.smithi103.stdout:6/668: read d0/d14/d57/d68/f6b [2601046,14320] 0 2024-03-22T02:28:52.144 INFO:tasks.workunit.client.1.smithi122.stdout:2/398: mknod df/d1f/d2a/d51/c89 0 2024-03-22T02:28:52.144 INFO:tasks.workunit.client.1.smithi122.stdout:3/314: rename d2/f2c to d2/f5e 0 2024-03-22T02:28:52.145 INFO:tasks.workunit.client.0.smithi103.stdout:6/669: read d0/f64 [158152,51473] 0 2024-03-22T02:28:52.146 INFO:tasks.workunit.client.1.smithi122.stdout:3/315: chown d2/d9/dc/d17/d45/f46 0 1 2024-03-22T02:28:52.147 INFO:tasks.workunit.client.0.smithi103.stdout:5/936: fdatasync d0/d14/d23/d6f/fda 0 2024-03-22T02:28:52.147 INFO:tasks.workunit.client.1.smithi122.stdout:3/316: chown d2/d9/dc/df/d3c 935 1 2024-03-22T02:28:52.147 INFO:tasks.workunit.client.0.smithi103.stdout:2/947: dread d11/d19/d5d/f68 [0,4194304] 0 2024-03-22T02:28:52.149 INFO:tasks.workunit.client.1.smithi122.stdout:2/399: creat df/d1f/d33/d4d/d7a/f8a x:0 0 0 2024-03-22T02:28:52.149 INFO:tasks.workunit.client.0.smithi103.stdout:6/670: unlink d0/d3/d33/l8f 0 2024-03-22T02:28:52.149 INFO:tasks.workunit.client.0.smithi103.stdout:5/937: fsync d0/d14/d23/d36/d95/d96/daa/daf/ff0 0 2024-03-22T02:28:52.150 INFO:tasks.workunit.client.1.smithi122.stdout:3/317: creat d2/d9/dc/d57/f5f x:0 0 0 2024-03-22T02:28:52.151 INFO:tasks.workunit.client.0.smithi103.stdout:5/938: chown d0/d14/d23/d46/c55 3166594 1 2024-03-22T02:28:52.151 INFO:tasks.workunit.client.1.smithi122.stdout:2/400: symlink df/d1f/d23/l8b 0 2024-03-22T02:28:52.151 INFO:tasks.workunit.client.1.smithi122.stdout:3/318: chown d2/d9/dc/d17/d28/d4c 1 1 2024-03-22T02:28:52.152 INFO:tasks.workunit.client.1.smithi122.stdout:2/401: chown df/d1f/d33/c83 1036917 1 2024-03-22T02:28:52.152 INFO:tasks.workunit.client.0.smithi103.stdout:2/948: creat d11/d19/da7/dd3/dd9/d110/d11e/f13e x:0 0 0 2024-03-22T02:28:52.153 INFO:tasks.workunit.client.1.smithi122.stdout:3/319: write d2/d9/dc/df/d38/f3b [1595953,10166] 0 2024-03-22T02:28:52.156 INFO:tasks.workunit.client.0.smithi103.stdout:2/949: link d11/d25/d46/db6/lbb d11/d19/d5d/l13f 0 2024-03-22T02:28:52.157 INFO:tasks.workunit.client.0.smithi103.stdout:2/950: chown d11/d25/d63/d45/d47/d83/c4b 130250 1 2024-03-22T02:28:52.158 INFO:tasks.workunit.client.1.smithi122.stdout:2/402: dwrite df/d1f/d33/d34/f5f [0,4194304] 0 2024-03-22T02:28:52.161 INFO:tasks.workunit.client.1.smithi122.stdout:3/320: dwrite d2/d9/dc/f16 [0,4194304] 0 2024-03-22T02:28:52.161 INFO:tasks.workunit.client.0.smithi103.stdout:2/951: rmdir d11/d25/d63/d45/d47/dae/d106/d134 0 2024-03-22T02:28:52.161 INFO:tasks.workunit.client.1.smithi122.stdout:3/321: chown d2/f7 11 1 2024-03-22T02:28:52.164 INFO:tasks.workunit.client.1.smithi122.stdout:3/322: write d2/d9/dc/d17/f42 [1221605,22750] 0 2024-03-22T02:28:52.166 INFO:tasks.workunit.client.0.smithi103.stdout:5/939: read d0/d14/d23/d36/fb6 [3419792,57117] 0 2024-03-22T02:28:52.167 INFO:tasks.workunit.client.1.smithi122.stdout:2/403: dwrite df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:52.167 INFO:tasks.workunit.client.0.smithi103.stdout:2/952: link d11/d19/da7/dd3/la1 d11/l140 0 2024-03-22T02:28:52.168 INFO:tasks.workunit.client.0.smithi103.stdout:5/940: chown d0/d14/d23/d36/f54 1 1 2024-03-22T02:28:52.169 INFO:tasks.workunit.client.1.smithi122.stdout:3/323: creat d2/d9/dc/df/d38/f60 x:0 0 0 2024-03-22T02:28:52.173 INFO:tasks.workunit.client.0.smithi103.stdout:2/953: mkdir d11/d25/d63/d45/d141 0 2024-03-22T02:28:52.175 INFO:tasks.workunit.client.1.smithi122.stdout:2/404: dwrite df/d1f/f2d [0,4194304] 0 2024-03-22T02:28:52.177 INFO:tasks.workunit.client.0.smithi103.stdout:5/941: dwrite d0/d14/d23/d5c/d77/d125/f140 [0,4194304] 0 2024-03-22T02:28:52.182 INFO:tasks.workunit.client.0.smithi103.stdout:2/954: rename d11/d19/d4e/dd5 to d11/d25/d142 0 2024-03-22T02:28:52.185 INFO:tasks.workunit.client.0.smithi103.stdout:5/942: mknod d0/d14/d23/d5c/d77/d125/c14a 0 2024-03-22T02:28:52.188 INFO:tasks.workunit.client.0.smithi103.stdout:5/943: creat d0/d14/d23/d2a/d33/f14b x:0 0 0 2024-03-22T02:28:52.189 INFO:tasks.workunit.client.0.smithi103.stdout:5/944: readlink d0/d14/l16 0 2024-03-22T02:28:52.189 INFO:tasks.workunit.client.0.smithi103.stdout:5/945: chown d0/d14/d23/d2a/d33/db5/f106 48483 1 2024-03-22T02:28:52.192 INFO:tasks.workunit.client.0.smithi103.stdout:5/946: rename d0/da2/dca/fba to d0/d14/d23/d2a/d33/db5/dbd/f14c 0 2024-03-22T02:28:52.192 INFO:tasks.workunit.client.0.smithi103.stdout:5/947: fsync d0/fd4 0 2024-03-22T02:28:52.194 INFO:tasks.workunit.client.0.smithi103.stdout:5/948: mkdir d0/d14/d23/d5c/d70/d14d 0 2024-03-22T02:28:52.196 INFO:tasks.workunit.client.0.smithi103.stdout:5/949: fdatasync d0/d14/d23/d6f/d83/f149 0 2024-03-22T02:28:52.197 INFO:tasks.workunit.client.0.smithi103.stdout:5/950: chown d0/f9a 409647999 1 2024-03-22T02:28:52.210 INFO:tasks.workunit.client.1.smithi122.stdout:5/359: truncate d8/da/f16 1879756 0 2024-03-22T02:28:52.213 INFO:tasks.workunit.client.1.smithi122.stdout:5/360: creat d8/d1e/f7e x:0 0 0 2024-03-22T02:28:52.219 INFO:tasks.workunit.client.1.smithi122.stdout:1/434: truncate dc/d2b/f8c 532585 0 2024-03-22T02:28:52.220 INFO:tasks.workunit.client.1.smithi122.stdout:5/361: dwrite d8/d1e/d22/d3a/d46/d4f/f54 [0,4194304] 0 2024-03-22T02:28:52.223 INFO:tasks.workunit.client.1.smithi122.stdout:5/362: creat d8/da/d28/d42/f7f x:0 0 0 2024-03-22T02:28:52.224 INFO:tasks.workunit.client.1.smithi122.stdout:5/363: fdatasync d8/da/d28/f3d 0 2024-03-22T02:28:52.225 INFO:tasks.workunit.client.1.smithi122.stdout:1/435: getdents dc/d3b/d46/d4d/d90 0 2024-03-22T02:28:52.227 INFO:tasks.workunit.client.1.smithi122.stdout:5/364: symlink d8/d3b/l80 0 2024-03-22T02:28:52.229 INFO:tasks.workunit.client.0.smithi103.stdout:5/951: write d0/da2/fb3 [899888,1546] 0 2024-03-22T02:28:52.232 INFO:tasks.workunit.client.0.smithi103.stdout:6/671: dread d0/d14/f93 [0,4194304] 0 2024-03-22T02:28:52.232 INFO:tasks.workunit.client.0.smithi103.stdout:5/952: unlink d0/d14/c110 0 2024-03-22T02:28:52.233 INFO:tasks.workunit.client.0.smithi103.stdout:6/672: rename d0 to d0/d3/d22/d58/d6f/d75/db0/db1/dde 22 2024-03-22T02:28:52.233 INFO:tasks.workunit.client.1.smithi122.stdout:1/436: dwrite dc/d3b/d46/f82 [0,4194304] 0 2024-03-22T02:28:52.235 INFO:tasks.workunit.client.0.smithi103.stdout:5/953: creat d0/d14/d23/d2a/df7/f14e x:0 0 0 2024-03-22T02:28:52.235 INFO:tasks.workunit.client.1.smithi122.stdout:9/434: dread db/d1b/d28/f6b [0,4194304] 0 2024-03-22T02:28:52.235 INFO:tasks.workunit.client.1.smithi122.stdout:1/437: dread - dc/d3b/d8d/f8f zero size 2024-03-22T02:28:52.236 INFO:tasks.workunit.client.0.smithi103.stdout:6/673: chown d0/lca 7 1 2024-03-22T02:28:52.238 INFO:tasks.workunit.client.0.smithi103.stdout:5/954: mkdir d0/d14/d14f 0 2024-03-22T02:28:52.239 INFO:tasks.workunit.client.0.smithi103.stdout:6/674: fdatasync d0/d3/d22/d58/d6f/d75/f7b 0 2024-03-22T02:28:52.239 INFO:tasks.workunit.client.0.smithi103.stdout:5/955: stat d0/d14/d23/d46/c58 0 2024-03-22T02:28:52.239 INFO:tasks.workunit.client.1.smithi122.stdout:8/429: sync 2024-03-22T02:28:52.239 INFO:tasks.workunit.client.1.smithi122.stdout:7/311: sync 2024-03-22T02:28:52.239 INFO:tasks.workunit.client.1.smithi122.stdout:1/438: rename dc/d55/f5f to dc/d3b/d46/d4d/d90/f93 0 2024-03-22T02:28:52.241 INFO:tasks.workunit.client.1.smithi122.stdout:8/430: write d1/d40/d51/f6b [1523784,38871] 0 2024-03-22T02:28:52.242 INFO:tasks.workunit.client.0.smithi103.stdout:5/956: mkdir d0/d14/d23/d2a/d7b/d150 0 2024-03-22T02:28:52.245 INFO:tasks.workunit.client.0.smithi103.stdout:5/957: chown d0/d14/d23/d5c/d77/df2/f3f 213440 1 2024-03-22T02:28:52.245 INFO:tasks.workunit.client.1.smithi122.stdout:1/439: dread dc/f10 [0,4194304] 0 2024-03-22T02:28:52.246 INFO:tasks.workunit.client.0.smithi103.stdout:6/675: dwrite d0/d3/f92 [0,4194304] 0 2024-03-22T02:28:52.247 INFO:tasks.workunit.client.1.smithi122.stdout:1/440: dread - dc/d55/f75 zero size 2024-03-22T02:28:52.248 INFO:tasks.workunit.client.0.smithi103.stdout:5/958: mkdir d0/d14/d23/d6f/d83/d151 0 2024-03-22T02:28:52.248 INFO:tasks.workunit.client.1.smithi122.stdout:8/431: dwrite d1/d8/d2f/f3e [0,4194304] 0 2024-03-22T02:28:52.248 INFO:tasks.workunit.client.1.smithi122.stdout:1/441: write dc/f4a [279333,60199] 0 2024-03-22T02:28:52.251 INFO:tasks.workunit.client.1.smithi122.stdout:5/365: truncate d8/f30 667173 0 2024-03-22T02:28:52.251 INFO:tasks.workunit.client.1.smithi122.stdout:5/366: write d8/f33 [337237,22014] 0 2024-03-22T02:28:52.252 INFO:tasks.workunit.client.0.smithi103.stdout:6/676: unlink d0/d3/d22/c31 0 2024-03-22T02:28:52.253 INFO:tasks.workunit.client.1.smithi122.stdout:7/312: mknod d0/d4/d20/d57/c63 0 2024-03-22T02:28:52.254 INFO:tasks.workunit.client.0.smithi103.stdout:5/959: unlink d0/d14/d23/d2a/d33/l37 0 2024-03-22T02:28:52.255 INFO:tasks.workunit.client.1.smithi122.stdout:9/435: getdents db/df/d3e/d59 0 2024-03-22T02:28:52.258 INFO:tasks.workunit.client.0.smithi103.stdout:5/960: creat d0/da2/f152 x:0 0 0 2024-03-22T02:28:52.259 INFO:tasks.workunit.client.0.smithi103.stdout:6/677: dwrite d0/d3/d22/fab [0,4194304] 0 2024-03-22T02:28:52.259 INFO:tasks.workunit.client.1.smithi122.stdout:8/432: rename d1/d4d/l84 to d1/d8/d8a/l93 0 2024-03-22T02:28:52.261 INFO:tasks.workunit.client.1.smithi122.stdout:1/442: mkdir dc/d3b/d8d/d94 0 2024-03-22T02:28:52.261 INFO:tasks.workunit.client.0.smithi103.stdout:5/961: stat d0/da2/cb4 0 2024-03-22T02:28:52.264 INFO:tasks.workunit.client.0.smithi103.stdout:4/887: sync 2024-03-22T02:28:52.265 INFO:tasks.workunit.client.0.smithi103.stdout:5/962: creat d0/d14/d23/d6f/d83/d126/f153 x:0 0 0 2024-03-22T02:28:52.267 INFO:tasks.workunit.client.1.smithi122.stdout:8/433: creat d1/d60/d73/f94 x:0 0 0 2024-03-22T02:28:52.267 INFO:tasks.workunit.client.0.smithi103.stdout:4/888: creat d3/d1e/d25/d65/dc9/f12a x:0 0 0 2024-03-22T02:28:52.268 INFO:tasks.workunit.client.1.smithi122.stdout:9/436: dread db/df/d3e/d81/d29/f3f [0,4194304] 0 2024-03-22T02:28:52.269 INFO:tasks.workunit.client.0.smithi103.stdout:4/889: mkdir d3/d69/d12b 0 2024-03-22T02:28:52.269 INFO:tasks.workunit.client.1.smithi122.stdout:9/437: chown db/df/c14 10 1 2024-03-22T02:28:52.270 INFO:tasks.workunit.client.0.smithi103.stdout:4/890: rmdir d3/d38/d5c/d86 39 2024-03-22T02:28:52.272 INFO:tasks.workunit.client.0.smithi103.stdout:4/891: chown d3/d38/l121 9175045 1 2024-03-22T02:28:52.272 INFO:tasks.workunit.client.0.smithi103.stdout:5/963: dwrite d0/d14/d23/d46/de7/f114 [0,4194304] 0 2024-03-22T02:28:52.273 INFO:tasks.workunit.client.0.smithi103.stdout:4/892: readlink d3/d1e/d53/l5f 0 2024-03-22T02:28:52.274 INFO:tasks.workunit.client.1.smithi122.stdout:7/313: link d0/d4/d20/f27 d0/d4/d22/f64 0 2024-03-22T02:28:52.275 INFO:tasks.workunit.client.0.smithi103.stdout:5/964: truncate d0/d14/d23/d5c/d77/df2/f13e 361295 0 2024-03-22T02:28:52.275 INFO:tasks.workunit.client.1.smithi122.stdout:8/434: mknod d1/d8/d2f/c95 0 2024-03-22T02:28:52.276 INFO:tasks.workunit.client.1.smithi122.stdout:9/438: mknod db/df/d3e/d81/d29/c8d 0 2024-03-22T02:28:52.278 INFO:tasks.workunit.client.0.smithi103.stdout:4/893: getdents d3/d1e/d53 0 2024-03-22T02:28:52.279 INFO:tasks.workunit.client.1.smithi122.stdout:5/367: getdents d8/d74 0 2024-03-22T02:28:52.283 INFO:tasks.workunit.client.0.smithi103.stdout:5/965: dwrite d0/d14/d23/d2a/d33/db9/ff5 [0,4194304] 0 2024-03-22T02:28:52.283 INFO:tasks.workunit.client.0.smithi103.stdout:4/894: truncate d3/d1e/f28 1021566 0 2024-03-22T02:28:52.283 INFO:tasks.workunit.client.1.smithi122.stdout:8/435: rename d1/d3f/d45/f67 to d1/d40/d48/f96 0 2024-03-22T02:28:52.285 INFO:tasks.workunit.client.1.smithi122.stdout:9/439: dwrite db/df/d3e/d81/d25/f55 [4194304,4194304] 0 2024-03-22T02:28:52.285 INFO:tasks.workunit.client.1.smithi122.stdout:8/436: dread - d1/d22/f64 zero size 2024-03-22T02:28:52.287 INFO:tasks.workunit.client.1.smithi122.stdout:9/440: read db/d23/d1f/d57/d66/d73/f76 [2062155,79712] 0 2024-03-22T02:28:52.288 INFO:tasks.workunit.client.1.smithi122.stdout:8/437: mkdir d1/d40/d71/d97 0 2024-03-22T02:28:52.289 INFO:tasks.workunit.client.1.smithi122.stdout:8/438: write d1/d40/d51/f89 [407865,65435] 0 2024-03-22T02:28:52.290 INFO:tasks.workunit.client.1.smithi122.stdout:9/441: mknod db/df/d3e/d81/d25/c8e 0 2024-03-22T02:28:52.290 INFO:tasks.workunit.client.1.smithi122.stdout:8/439: chown d1/f5d 6841 1 2024-03-22T02:28:52.291 INFO:tasks.workunit.client.1.smithi122.stdout:9/442: chown db/d23/d1f/d57/d66/f7a 1017753171 1 2024-03-22T02:28:52.293 INFO:tasks.workunit.client.1.smithi122.stdout:8/440: creat d1/d8/d2f/f98 x:0 0 0 2024-03-22T02:28:52.294 INFO:tasks.workunit.client.0.smithi103.stdout:4/895: dwrite d3/d38/fda [0,4194304] 0 2024-03-22T02:28:52.296 INFO:tasks.workunit.client.1.smithi122.stdout:5/368: stat d8/da/f16 0 2024-03-22T02:28:52.297 INFO:tasks.workunit.client.1.smithi122.stdout:8/441: rename d1/d8/l14 to d1/d40/d51/l99 0 2024-03-22T02:28:52.301 INFO:tasks.workunit.client.1.smithi122.stdout:9/443: getdents db/d23 0 2024-03-22T02:28:52.302 INFO:tasks.workunit.client.0.smithi103.stdout:5/966: dwrite d0/d14/d23/d36/d95/d96/fb0 [0,4194304] 0 2024-03-22T02:28:52.303 INFO:tasks.workunit.client.0.smithi103.stdout:4/896: rename d3/d1e/d25/d65/d83/dc1/fcc to d3/deb/f12c 0 2024-03-22T02:28:52.304 INFO:tasks.workunit.client.1.smithi122.stdout:9/444: write db/df/f15 [1650264,92421] 0 2024-03-22T02:28:52.304 INFO:tasks.workunit.client.0.smithi103.stdout:6/678: dread d0/f6a [0,4194304] 0 2024-03-22T02:28:52.306 INFO:tasks.workunit.client.0.smithi103.stdout:4/897: mknod d3/da2/c12d 0 2024-03-22T02:28:52.306 INFO:tasks.workunit.client.0.smithi103.stdout:5/967: rename d0/d14/d23/d2a/d33/db5 to d0/d14/d23/d2a/d33/db5/dbd/ded/d154 22 2024-03-22T02:28:52.307 INFO:tasks.workunit.client.1.smithi122.stdout:8/442: dwrite d1/d40/d48/f6d [0,4194304] 0 2024-03-22T02:28:52.309 INFO:tasks.workunit.client.1.smithi122.stdout:8/443: chown d1/d9/d50/f59 76406696 1 2024-03-22T02:28:52.310 INFO:tasks.workunit.client.0.smithi103.stdout:4/898: creat d3/d1e/d53/d7a/f12e x:0 0 0 2024-03-22T02:28:52.312 INFO:tasks.workunit.client.1.smithi122.stdout:1/443: truncate dc/f10 281984 0 2024-03-22T02:28:52.312 INFO:tasks.workunit.client.0.smithi103.stdout:4/899: chown d3/d1e/d25/d65/d83/da1/dbd/fe4 1724952801 1 2024-03-22T02:28:52.314 INFO:tasks.workunit.client.1.smithi122.stdout:8/444: creat d1/d8/d63/d88/f9a x:0 0 0 2024-03-22T02:28:52.315 INFO:tasks.workunit.client.1.smithi122.stdout:8/445: chown d1/d8/d63 1 1 2024-03-22T02:28:52.318 INFO:tasks.workunit.client.1.smithi122.stdout:8/446: creat d1/d60/d73/d90/f9b x:0 0 0 2024-03-22T02:28:52.319 INFO:tasks.workunit.client.1.smithi122.stdout:7/314: dwrite d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:52.320 INFO:tasks.workunit.client.1.smithi122.stdout:7/315: readlink d0/d4/d20/l38 0 2024-03-22T02:28:52.321 INFO:tasks.workunit.client.1.smithi122.stdout:8/447: rmdir d1/d4d 39 2024-03-22T02:28:52.324 INFO:tasks.workunit.client.1.smithi122.stdout:7/316: mknod d0/d4/c65 0 2024-03-22T02:28:52.324 INFO:tasks.workunit.client.1.smithi122.stdout:7/317: chown d0/l2e 84 1 2024-03-22T02:28:52.325 INFO:tasks.workunit.client.1.smithi122.stdout:8/448: dread - d1/d8/d5e/f72 zero size 2024-03-22T02:28:52.329 INFO:tasks.workunit.client.1.smithi122.stdout:8/449: write d1/d3f/d45/f49 [2587698,92614] 0 2024-03-22T02:28:52.333 INFO:tasks.workunit.client.1.smithi122.stdout:7/318: dwrite d0/fd [0,4194304] 0 2024-03-22T02:28:52.334 INFO:tasks.workunit.client.1.smithi122.stdout:8/450: mknod d1/d22/d6a/c9c 0 2024-03-22T02:28:52.335 INFO:tasks.workunit.client.0.smithi103.stdout:4/900: getdents d3/d1e/d25/d65/d83/dc1 0 2024-03-22T02:28:52.336 INFO:tasks.workunit.client.1.smithi122.stdout:7/319: symlink d0/d25/d46/d48/l66 0 2024-03-22T02:28:52.338 INFO:tasks.workunit.client.1.smithi122.stdout:7/320: chown d0/d25/d46 2743 1 2024-03-22T02:28:52.338 INFO:tasks.workunit.client.0.smithi103.stdout:4/901: fdatasync d3/d1e/d25/f3d 0 2024-03-22T02:28:52.339 INFO:tasks.workunit.client.1.smithi122.stdout:8/451: creat d1/d22/f9d x:0 0 0 2024-03-22T02:28:52.339 INFO:tasks.workunit.client.0.smithi103.stdout:6/679: write d0/d3/d22/d58/d6f/f8b [785309,5896] 0 2024-03-22T02:28:52.339 INFO:tasks.workunit.client.1.smithi122.stdout:9/445: write db/df/d3e/d81/d29/f38 [4570389,76538] 0 2024-03-22T02:28:52.340 INFO:tasks.workunit.client.0.smithi103.stdout:4/902: dread - d3/da2/da6/f10d zero size 2024-03-22T02:28:52.340 INFO:tasks.workunit.client.1.smithi122.stdout:7/321: fsync d0/d4/d20/d21/f2b 0 2024-03-22T02:28:52.341 INFO:tasks.workunit.client.0.smithi103.stdout:6/680: fsync d0/d3/d33/fad 0 2024-03-22T02:28:52.342 INFO:tasks.workunit.client.1.smithi122.stdout:7/322: fdatasync d0/d4/d22/f32 0 2024-03-22T02:28:52.343 INFO:tasks.workunit.client.0.smithi103.stdout:4/903: fsync d3/d1e/d53/f11f 0 2024-03-22T02:28:52.344 INFO:tasks.workunit.client.0.smithi103.stdout:5/968: dwrite d0/d3e/f107 [0,4194304] 0 2024-03-22T02:28:52.346 INFO:tasks.workunit.client.1.smithi122.stdout:9/446: dwrite db/d23/f1a [0,4194304] 0 2024-03-22T02:28:52.348 INFO:tasks.workunit.client.0.smithi103.stdout:5/969: creat d0/d14/d23/d36/d95/d96/f155 x:0 0 0 2024-03-22T02:28:52.350 INFO:tasks.workunit.client.0.smithi103.stdout:4/904: creat d3/d1e/d78/df9/f12f x:0 0 0 2024-03-22T02:28:52.350 INFO:tasks.workunit.client.0.smithi103.stdout:5/970: fsync d0/d14/d23/d36/d95/d96/fb0 0 2024-03-22T02:28:52.351 INFO:tasks.workunit.client.1.smithi122.stdout:7/323: getdents d0/d4/d22/d5f 0 2024-03-22T02:28:52.351 INFO:tasks.workunit.client.0.smithi103.stdout:4/905: chown d3/d31 0 1 2024-03-22T02:28:52.353 INFO:tasks.workunit.client.0.smithi103.stdout:5/971: write d0/d14/d23/d36/d95/d96/daa/daf/ff0 [4432093,937] 0 2024-03-22T02:28:52.354 INFO:tasks.workunit.client.0.smithi103.stdout:4/906: read d3/d1e/d25/d65/f118 [360039,45437] 0 2024-03-22T02:28:52.355 INFO:tasks.workunit.client.1.smithi122.stdout:9/447: read db/df/d3e/d81/d25/f55 [1252427,122212] 0 2024-03-22T02:28:52.355 INFO:tasks.workunit.client.1.smithi122.stdout:7/324: write d0/d4/d20/d21/f2b [947108,9381] 0 2024-03-22T02:28:52.356 INFO:tasks.workunit.client.0.smithi103.stdout:6/681: write d0/d14/f4c [1012064,5671] 0 2024-03-22T02:28:52.356 INFO:tasks.workunit.client.0.smithi103.stdout:5/972: chown d0/d14/d23/d2a/d1f 2 1 2024-03-22T02:28:52.357 INFO:tasks.workunit.client.0.smithi103.stdout:4/907: fsync d3/f11 0 2024-03-22T02:28:52.357 INFO:tasks.workunit.client.1.smithi122.stdout:7/325: write d0/d4/d22/f55 [759169,36699] 0 2024-03-22T02:28:52.358 INFO:tasks.workunit.client.1.smithi122.stdout:9/448: rmdir db/d23/d1f/d57 39 2024-03-22T02:28:52.361 INFO:tasks.workunit.client.0.smithi103.stdout:5/973: readlink d0/d14/d23/d5c/d77/df2/l56 0 2024-03-22T02:28:52.361 INFO:tasks.workunit.client.1.smithi122.stdout:9/449: write db/d23/d1f/d57/d66/f85 [3454896,112775] 0 2024-03-22T02:28:52.363 INFO:tasks.workunit.client.1.smithi122.stdout:9/450: chown db/d1b/d28/l45 1419 1 2024-03-22T02:28:52.365 INFO:tasks.workunit.client.0.smithi103.stdout:6/682: dwrite d0/d3/d33/f7c [0,4194304] 0 2024-03-22T02:28:52.365 INFO:tasks.workunit.client.1.smithi122.stdout:9/451: unlink db/l4d 0 2024-03-22T02:28:52.366 INFO:tasks.workunit.client.0.smithi103.stdout:7/879: sync 2024-03-22T02:28:52.366 INFO:tasks.workunit.client.0.smithi103.stdout:6/683: chown d0/d3/d22/d58/d6f/f94 7 1 2024-03-22T02:28:52.370 INFO:tasks.workunit.client.0.smithi103.stdout:6/684: read d0/d88/f9f [346517,52781] 0 2024-03-22T02:28:52.371 INFO:tasks.workunit.client.0.smithi103.stdout:7/880: mknod d1/d10/d11/d19/d29/d61/d69/d6b/d7a/db5/c116 0 2024-03-22T02:28:52.372 INFO:tasks.workunit.client.0.smithi103.stdout:7/881: fdatasync d1/d10/d11/dd9/fec 0 2024-03-22T02:28:52.372 INFO:tasks.workunit.client.0.smithi103.stdout:6/685: mknod d0/dbd/cdf 0 2024-03-22T02:28:52.375 INFO:tasks.workunit.client.0.smithi103.stdout:6/686: fsync d0/d3/f49 0 2024-03-22T02:28:52.379 INFO:tasks.workunit.client.0.smithi103.stdout:6/687: read d0/d14/f6c [273803,62316] 0 2024-03-22T02:28:52.380 INFO:tasks.workunit.client.0.smithi103.stdout:7/882: dwrite d1/dfa/fd5 [0,4194304] 0 2024-03-22T02:28:52.386 INFO:tasks.workunit.client.0.smithi103.stdout:7/883: truncate d1/da/d1c/f8b 3868154 0 2024-03-22T02:28:52.387 INFO:tasks.workunit.client.0.smithi103.stdout:4/908: dread d3/d38/d5c/d86/fdd [0,4194304] 0 2024-03-22T02:28:52.388 INFO:tasks.workunit.client.0.smithi103.stdout:5/974: truncate d0/fe 1881643 0 2024-03-22T02:28:52.390 INFO:tasks.workunit.client.0.smithi103.stdout:7/884: creat d1/d10/d11/d19/d29/d61/d69/d5c/f117 x:0 0 0 2024-03-22T02:28:52.392 INFO:tasks.workunit.client.1.smithi122.stdout:9/452: dwrite db/df/f7d [0,4194304] 0 2024-03-22T02:28:52.392 INFO:tasks.workunit.client.0.smithi103.stdout:4/909: truncate d3/d1e/d25/d65/d83/fe1 743449 0 2024-03-22T02:28:52.393 INFO:tasks.workunit.client.1.smithi122.stdout:9/453: write db/d1b/f6a [741960,74017] 0 2024-03-22T02:28:52.397 INFO:tasks.workunit.client.1.smithi122.stdout:7/326: dread d0/d4/d20/f27 [0,4194304] 0 2024-03-22T02:28:52.399 INFO:tasks.workunit.client.1.smithi122.stdout:9/454: dread db/d23/d1f/d57/d66/d73/f76 [4194304,4194304] 0 2024-03-22T02:28:52.399 INFO:tasks.workunit.client.0.smithi103.stdout:4/910: link d3/d1e/d53/d7a/f10a d3/da2/da6/f130 0 2024-03-22T02:28:52.399 INFO:tasks.workunit.client.0.smithi103.stdout:6/688: dwrite d0/d3/f49 [0,4194304] 0 2024-03-22T02:28:52.401 INFO:tasks.workunit.client.1.smithi122.stdout:9/455: write db/df/d3e/f8b [906319,63683] 0 2024-03-22T02:28:52.406 INFO:tasks.workunit.client.0.smithi103.stdout:6/689: write d0/f1e [9095746,55500] 0 2024-03-22T02:28:52.407 INFO:tasks.workunit.client.1.smithi122.stdout:7/327: dwrite d0/d30/f53 [0,4194304] 0 2024-03-22T02:28:52.409 INFO:tasks.workunit.client.1.smithi122.stdout:9/456: link db/d1b/d28/c40 db/d1b/c8f 0 2024-03-22T02:28:52.410 INFO:tasks.workunit.client.0.smithi103.stdout:4/911: read d3/d38/fdc [1468609,89525] 0 2024-03-22T02:28:52.410 INFO:tasks.workunit.client.0.smithi103.stdout:5/975: dread d0/d14/d23/d6f/d83/f11a [0,4194304] 0 2024-03-22T02:28:52.411 INFO:tasks.workunit.client.1.smithi122.stdout:7/328: symlink d0/d4/d22/d5f/d5e/l67 0 2024-03-22T02:28:52.411 INFO:tasks.workunit.client.1.smithi122.stdout:9/457: write db/df/f80 [4754489,119554] 0 2024-03-22T02:28:52.411 INFO:tasks.workunit.client.0.smithi103.stdout:7/885: dwrite d1/d10/d11/d19/d29/d61/d69/d6b/fac [0,4194304] 0 2024-03-22T02:28:52.412 INFO:tasks.workunit.client.1.smithi122.stdout:9/458: readlink db/d23/l4b 0 2024-03-22T02:28:52.414 INFO:tasks.workunit.client.0.smithi103.stdout:0/906: sync 2024-03-22T02:28:52.414 INFO:tasks.workunit.client.0.smithi103.stdout:9/861: sync 2024-03-22T02:28:52.414 INFO:tasks.workunit.client.0.smithi103.stdout:2/955: sync 2024-03-22T02:28:52.415 INFO:tasks.workunit.client.1.smithi122.stdout:7/329: mknod d0/d4/d22/d5f/c68 0 2024-03-22T02:28:52.415 INFO:tasks.workunit.client.0.smithi103.stdout:9/862: chown d2/da/d18/d1e/d34/d3d/c44 46242577 1 2024-03-22T02:28:52.415 INFO:tasks.workunit.client.0.smithi103.stdout:2/956: chown d11/d25/d63/d45/d47 79101 1 2024-03-22T02:28:52.417 INFO:tasks.workunit.client.0.smithi103.stdout:5/976: unlink d0/d14/d23/d46/f61 0 2024-03-22T02:28:52.418 INFO:tasks.workunit.client.0.smithi103.stdout:7/886: chown d1/d10/cb6 13913 1 2024-03-22T02:28:52.419 INFO:tasks.workunit.client.0.smithi103.stdout:5/977: write d0/d14/d23/d5c/d77/d125/f140 [1052776,30808] 0 2024-03-22T02:28:52.419 INFO:tasks.workunit.client.0.smithi103.stdout:4/912: rename d3/d1e/d25/d65/d83/da1/dbd/fea to d3/d1e/d25/d65/d83/dc1/f131 0 2024-03-22T02:28:52.420 INFO:tasks.workunit.client.1.smithi122.stdout:7/330: read d0/d4/fb [4109216,129545] 0 2024-03-22T02:28:52.421 INFO:tasks.workunit.client.0.smithi103.stdout:0/907: fdatasync fa 0 2024-03-22T02:28:52.422 INFO:tasks.workunit.client.1.smithi122.stdout:9/459: dwrite db/d1b/d6c/d58/f67 [0,4194304] 0 2024-03-22T02:28:52.422 INFO:tasks.workunit.client.0.smithi103.stdout:9/863: unlink d2/da/d18/d1e/d34/d3d/c44 0 2024-03-22T02:28:52.422 INFO:tasks.workunit.client.0.smithi103.stdout:0/908: chown dd/d34/fac 1898 1 2024-03-22T02:28:52.423 INFO:tasks.workunit.client.1.smithi122.stdout:9/460: fdatasync db/df/d3e/d81/d29/f5f 0 2024-03-22T02:28:52.425 INFO:tasks.workunit.client.0.smithi103.stdout:7/887: symlink d1/d10/d11/d19/dc3/de8/l118 0 2024-03-22T02:28:52.425 INFO:tasks.workunit.client.0.smithi103.stdout:0/909: dread - dd/d16/d4b/d96/d64/d6a/dd5/ffd zero size 2024-03-22T02:28:52.426 INFO:tasks.workunit.client.1.smithi122.stdout:9/461: rename db/df/d3e/c6e to db/d23/c90 0 2024-03-22T02:28:52.429 INFO:tasks.workunit.client.0.smithi103.stdout:2/957: rename d11/d19/d5d/f68 to d11/d25/d63/d45/d47/d83/d78/d125/f143 0 2024-03-22T02:28:52.432 INFO:tasks.workunit.client.0.smithi103.stdout:0/910: dread dd/f5b [0,4194304] 0 2024-03-22T02:28:52.432 INFO:tasks.workunit.client.0.smithi103.stdout:4/913: fdatasync d3/d1e/fba 0 2024-03-22T02:28:52.433 INFO:tasks.workunit.client.1.smithi122.stdout:9/462: dwrite db/df/f15 [0,4194304] 0 2024-03-22T02:28:52.433 INFO:tasks.workunit.client.0.smithi103.stdout:9/864: creat d2/d10/d43/d7e/da4/dcd/ff6 x:0 0 0 2024-03-22T02:28:52.435 INFO:tasks.workunit.client.0.smithi103.stdout:9/865: fdatasync d2/d10/d43/d7e/fed 0 2024-03-22T02:28:52.436 INFO:tasks.workunit.client.0.smithi103.stdout:0/911: readlink dd/d16/d4b/d96/d35/l47 0 2024-03-22T02:28:52.437 INFO:tasks.workunit.client.0.smithi103.stdout:2/958: truncate f2 4712764 0 2024-03-22T02:28:52.439 INFO:tasks.workunit.client.0.smithi103.stdout:9/866: chown d2/d5/l90 30955775 1 2024-03-22T02:28:52.440 INFO:tasks.workunit.client.1.smithi122.stdout:9/463: dwrite db/df/f15 [0,4194304] 0 2024-03-22T02:28:52.442 INFO:tasks.workunit.client.0.smithi103.stdout:7/888: dread d1/da/f74 [0,4194304] 0 2024-03-22T02:28:52.443 INFO:tasks.workunit.client.0.smithi103.stdout:6/690: dread d0/d3/d22/f50 [0,4194304] 0 2024-03-22T02:28:52.444 INFO:tasks.workunit.client.1.smithi122.stdout:4/378: sync 2024-03-22T02:28:52.446 INFO:tasks.workunit.client.0.smithi103.stdout:9/867: mkdir d2/d10/d43/d9b/df3/df7 0 2024-03-22T02:28:52.448 INFO:tasks.workunit.client.1.smithi122.stdout:9/464: dread db/df/d3e/d81/d3d/d79/f8a [0,4194304] 0 2024-03-22T02:28:52.448 INFO:tasks.workunit.client.1.smithi122.stdout:7/331: dwrite d0/f2 [0,4194304] 0 2024-03-22T02:28:52.449 INFO:tasks.workunit.client.1.smithi122.stdout:4/379: getdents d0/d12/d61 0 2024-03-22T02:28:52.452 INFO:tasks.workunit.client.0.smithi103.stdout:7/889: truncate d1/d10/d11/d19/dc3/de8/ff1 163194 0 2024-03-22T02:28:52.454 INFO:tasks.workunit.client.1.smithi122.stdout:4/380: creat d0/d12/f79 x:0 0 0 2024-03-22T02:28:52.456 INFO:tasks.workunit.client.0.smithi103.stdout:7/890: stat d1/d10/d11/l12 0 2024-03-22T02:28:52.460 INFO:tasks.workunit.client.1.smithi122.stdout:9/465: dwrite db/d23/d1f/d57/d66/f85 [0,4194304] 0 2024-03-22T02:28:52.460 INFO:tasks.workunit.client.0.smithi103.stdout:7/891: mknod d1/d10b/c119 0 2024-03-22T02:28:52.462 INFO:tasks.workunit.client.1.smithi122.stdout:4/381: rmdir d0/d12/d15/d42/d57 39 2024-03-22T02:28:52.463 INFO:tasks.workunit.client.0.smithi103.stdout:7/892: mkdir d1/da/d1c/d11a 0 2024-03-22T02:28:52.464 INFO:tasks.workunit.client.1.smithi122.stdout:9/466: creat db/d23/d1f/d57/d66/d73/f91 x:0 0 0 2024-03-22T02:28:52.466 INFO:tasks.workunit.client.1.smithi122.stdout:4/382: mkdir d0/d7a 0 2024-03-22T02:28:52.467 INFO:tasks.workunit.client.1.smithi122.stdout:9/467: write db/d23/d1f/d57/d66/d73/f76 [4462747,50662] 0 2024-03-22T02:28:52.468 INFO:tasks.workunit.client.0.smithi103.stdout:6/691: write d0/d3/d22/d58/f80 [777472,27724] 0 2024-03-22T02:28:52.468 INFO:tasks.workunit.client.1.smithi122.stdout:9/468: dread - db/df/d3e/d81/d25/f53 zero size 2024-03-22T02:28:52.470 INFO:tasks.workunit.client.1.smithi122.stdout:4/383: creat d0/d12/d15/d42/f7b x:0 0 0 2024-03-22T02:28:52.471 INFO:tasks.workunit.client.0.smithi103.stdout:5/978: dwrite d0/d14/d23/d2a/df7/f111 [0,4194304] 0 2024-03-22T02:28:52.472 INFO:tasks.workunit.client.1.smithi122.stdout:7/332: truncate d0/fd 744818 0 2024-03-22T02:28:52.476 INFO:tasks.workunit.client.0.smithi103.stdout:6/692: unlink d0/d3/d33/f36 0 2024-03-22T02:28:52.477 INFO:tasks.workunit.client.1.smithi122.stdout:9/469: rename db/d1b/c2e to db/d23/d1f/d57/d66/c92 0 2024-03-22T02:28:52.477 INFO:tasks.workunit.client.0.smithi103.stdout:2/959: dwrite f4 [0,4194304] 0 2024-03-22T02:28:52.480 INFO:tasks.workunit.client.1.smithi122.stdout:4/384: creat d0/d25/f7c x:0 0 0 2024-03-22T02:28:52.481 INFO:tasks.workunit.client.0.smithi103.stdout:6/693: stat d0/d3/d22/d58/d6f/d75/la5 0 2024-03-22T02:28:52.481 INFO:tasks.workunit.client.0.smithi103.stdout:7/893: creat d1/d10/d11/d70/d8a/d9d/dff/f11b x:0 0 0 2024-03-22T02:28:52.481 INFO:tasks.workunit.client.0.smithi103.stdout:0/912: dwrite dd/d16/d4b/d96/d64/dab/dc6/fd7 [0,4194304] 0 2024-03-22T02:28:52.482 INFO:tasks.workunit.client.1.smithi122.stdout:9/470: creat db/df/d3e/d81/d3d/d79/f93 x:0 0 0 2024-03-22T02:28:52.483 INFO:tasks.workunit.client.1.smithi122.stdout:4/385: truncate d0/d12/d15/d42/d4e/f4a 724996 0 2024-03-22T02:28:52.484 INFO:tasks.workunit.client.1.smithi122.stdout:9/471: mknod db/df/d3e/d81/d29/c94 0 2024-03-22T02:28:52.485 INFO:tasks.workunit.client.0.smithi103.stdout:9/868: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:52.486 INFO:tasks.workunit.client.1.smithi122.stdout:7/333: dread d0/fd [0,4194304] 0 2024-03-22T02:28:52.486 INFO:tasks.workunit.client.0.smithi103.stdout:2/960: truncate d11/d19/d4e/f7d 1178919 0 2024-03-22T02:28:52.489 INFO:tasks.workunit.client.1.smithi122.stdout:7/334: rmdir d0/d4e 39 2024-03-22T02:28:52.492 INFO:tasks.workunit.client.1.smithi122.stdout:9/472: link db/df/d3e/d81/c5e db/d23/d1f/d57/d66/d73/c95 0 2024-03-22T02:28:52.494 INFO:tasks.workunit.client.0.smithi103.stdout:6/694: mkdir d0/d3/d22/d58/de0 0 2024-03-22T02:28:52.494 INFO:tasks.workunit.client.0.smithi103.stdout:9/869: write d2/d5/d7b/d7d/dd4/dd8/fe2 [27988,96796] 0 2024-03-22T02:28:52.495 INFO:tasks.workunit.client.0.smithi103.stdout:0/913: dwrite dd/d16/d4b/d96/d29/d48/d94/f116 [0,4194304] 0 2024-03-22T02:28:52.497 INFO:tasks.workunit.client.1.smithi122.stdout:7/335: dwrite d0/d4/fb [0,4194304] 0 2024-03-22T02:28:52.498 INFO:tasks.workunit.client.1.smithi122.stdout:7/336: fdatasync d0/f1b 0 2024-03-22T02:28:52.500 INFO:tasks.workunit.client.0.smithi103.stdout:7/894: creat d1/d10/d11/d19/d86/d92/f11c x:0 0 0 2024-03-22T02:28:52.502 INFO:tasks.workunit.client.0.smithi103.stdout:9/870: rmdir d2/da/d18/d1e/d77/d8b 39 2024-03-22T02:28:52.503 INFO:tasks.workunit.client.1.smithi122.stdout:9/473: dwrite db/d23/d1e/f8c [0,4194304] 0 2024-03-22T02:28:52.503 INFO:tasks.workunit.client.0.smithi103.stdout:9/871: chown d2/d5/d4c/d56/ddc 11098 1 2024-03-22T02:28:52.504 INFO:tasks.workunit.client.1.smithi122.stdout:9/474: write db/d23/d1f/d57/d66/f7a [4013451,114564] 0 2024-03-22T02:28:52.505 INFO:tasks.workunit.client.1.smithi122.stdout:7/337: rename d0/d4/d1d/c4f to d0/c69 0 2024-03-22T02:28:52.506 INFO:tasks.workunit.client.0.smithi103.stdout:2/961: truncate d11/d25/d46/d40/feb 923087 0 2024-03-22T02:28:52.509 INFO:tasks.workunit.client.0.smithi103.stdout:2/962: fdatasync d11/d25/d63/d45/d47/d83/da3/f11f 0 2024-03-22T02:28:52.513 INFO:tasks.workunit.client.0.smithi103.stdout:7/895: creat d1/d10/d11/d19/d86/f11d x:0 0 0 2024-03-22T02:28:52.514 INFO:tasks.workunit.client.0.smithi103.stdout:2/963: write d11/d19/d5d/f13d [11230,40044] 0 2024-03-22T02:28:52.514 INFO:tasks.workunit.client.1.smithi122.stdout:9/475: dwrite db/df/d3e/d81/d29/f38 [0,4194304] 0 2024-03-22T02:28:52.515 INFO:tasks.workunit.client.0.smithi103.stdout:9/872: mkdir d2/da/d18/d1e/d34/d3d/d61/df8 0 2024-03-22T02:28:52.515 INFO:tasks.workunit.client.0.smithi103.stdout:2/964: fdatasync d11/dc5/f13a 0 2024-03-22T02:28:52.515 INFO:tasks.workunit.client.0.smithi103.stdout:7/896: stat d1/d10/d11/d19/d29/d61/d69/c25 0 2024-03-22T02:28:52.516 INFO:tasks.workunit.client.0.smithi103.stdout:2/965: fsync f10 0 2024-03-22T02:28:52.517 INFO:tasks.workunit.client.1.smithi122.stdout:7/338: dread d0/d4/d22/f55 [0,4194304] 0 2024-03-22T02:28:52.518 INFO:tasks.workunit.client.0.smithi103.stdout:7/897: chown d1/d10/d11/d19/d29/d61/d69/l115 2399 1 2024-03-22T02:28:52.518 INFO:tasks.workunit.client.1.smithi122.stdout:7/339: dread - d0/d30/f52 zero size 2024-03-22T02:28:52.520 INFO:tasks.workunit.client.1.smithi122.stdout:7/340: fsync d0/d4/f12 0 2024-03-22T02:28:52.520 INFO:tasks.workunit.client.0.smithi103.stdout:7/898: readlink d1/d10/d11/d19/d29/d61/d69/l115 0 2024-03-22T02:28:52.523 INFO:tasks.workunit.client.1.smithi122.stdout:9/476: dwrite fa [0,4194304] 0 2024-03-22T02:28:52.527 INFO:tasks.workunit.client.1.smithi122.stdout:9/477: read db/d1b/f6a [644782,42895] 0 2024-03-22T02:28:52.527 INFO:tasks.workunit.client.0.smithi103.stdout:2/966: dread d11/d25/d63/d45/d47/d83/f76 [0,4194304] 0 2024-03-22T02:28:52.528 INFO:tasks.workunit.client.1.smithi122.stdout:7/341: dread d0/f2 [0,4194304] 0 2024-03-22T02:28:52.531 INFO:tasks.workunit.client.0.smithi103.stdout:6/695: write d0/d3/f6d [898216,117543] 0 2024-03-22T02:28:52.531 INFO:tasks.workunit.client.1.smithi122.stdout:9/478: mknod db/df/c96 0 2024-03-22T02:28:52.532 INFO:tasks.workunit.client.1.smithi122.stdout:9/479: chown db/df/d3e/d81/d3d/d79/f93 60 1 2024-03-22T02:28:52.533 INFO:tasks.workunit.client.1.smithi122.stdout:7/342: creat d0/d4/d20/d57/f6a x:0 0 0 2024-03-22T02:28:52.534 INFO:tasks.workunit.client.1.smithi122.stdout:9/480: chown db/d1b/d28/l45 8 1 2024-03-22T02:28:52.534 INFO:tasks.workunit.client.1.smithi122.stdout:9/481: readlink db/d23/l12 0 2024-03-22T02:28:52.535 INFO:tasks.workunit.client.1.smithi122.stdout:9/482: chown db/d23/l3b 57802 1 2024-03-22T02:28:52.539 INFO:tasks.workunit.client.1.smithi122.stdout:9/483: dread db/d23/d1f/d57/d66/f85 [0,4194304] 0 2024-03-22T02:28:52.540 INFO:tasks.workunit.client.0.smithi103.stdout:2/967: dwrite d11/d25/d63/d45/d47/d83/d78/d125/f137 [0,4194304] 0 2024-03-22T02:28:52.541 INFO:tasks.workunit.client.0.smithi103.stdout:0/914: dwrite dd/d16/d4b/d96/d29/d48/d5c/d9b/fda [0,4194304] 0 2024-03-22T02:28:52.542 INFO:tasks.workunit.client.1.smithi122.stdout:9/484: mknod db/d1b/d6c/c97 0 2024-03-22T02:28:52.545 INFO:tasks.workunit.client.1.smithi122.stdout:9/485: rename db/df/d3e/d81/d29/f3f to db/d23/d1e/f98 0 2024-03-22T02:28:52.546 INFO:tasks.workunit.client.0.smithi103.stdout:7/899: dwrite d1/da/d1c/f8b [0,4194304] 0 2024-03-22T02:28:52.547 INFO:tasks.workunit.client.0.smithi103.stdout:0/915: rmdir dd/d16/d4b/d96/d64/d6a/dd5 39 2024-03-22T02:28:52.548 INFO:tasks.workunit.client.0.smithi103.stdout:0/916: chown dd/d16/d4b/d96/d64 0 1 2024-03-22T02:28:52.549 INFO:tasks.workunit.client.1.smithi122.stdout:7/343: write d0/fd [583719,43786] 0 2024-03-22T02:28:52.549 INFO:tasks.workunit.client.0.smithi103.stdout:7/900: write d1/d10/d11/d19/d29/d61/d1d/f101 [811477,4047] 0 2024-03-22T02:28:52.550 INFO:tasks.workunit.client.1.smithi122.stdout:9/486: dread fa [0,4194304] 0 2024-03-22T02:28:52.554 INFO:tasks.workunit.client.1.smithi122.stdout:7/344: rename d0/d4/d22/l49 to d0/d4/d22/d5f/l6b 0 2024-03-22T02:28:52.556 INFO:tasks.workunit.client.1.smithi122.stdout:9/487: symlink db/d1b/d6c/l99 0 2024-03-22T02:28:52.556 INFO:tasks.workunit.client.0.smithi103.stdout:0/917: truncate dd/d16/d4b/d96/d29/d48/f73 3135118 0 2024-03-22T02:28:52.557 INFO:tasks.workunit.client.1.smithi122.stdout:7/345: symlink d0/d25/l6c 0 2024-03-22T02:28:52.557 INFO:tasks.workunit.client.1.smithi122.stdout:7/346: dread - d0/d25/f56 zero size 2024-03-22T02:28:52.558 INFO:tasks.workunit.client.1.smithi122.stdout:9/488: mkdir db/d23/d9a 0 2024-03-22T02:28:52.558 INFO:tasks.workunit.client.0.smithi103.stdout:0/918: write dd/d16/d4b/ff6 [1622948,73022] 0 2024-03-22T02:28:52.560 INFO:tasks.workunit.client.1.smithi122.stdout:7/347: creat d0/d25/d5c/f6d x:0 0 0 2024-03-22T02:28:52.560 INFO:tasks.workunit.client.1.smithi122.stdout:9/489: symlink db/df/d3e/d81/d25/l9b 0 2024-03-22T02:28:52.561 INFO:tasks.workunit.client.0.smithi103.stdout:7/901: rename d1/d10/d11/d19/d29/d61/d69/ffc to d1/d10/d11/d70/d8a/f11e 0 2024-03-22T02:28:52.562 INFO:tasks.workunit.client.0.smithi103.stdout:0/919: dread dd/d16/d4b/d96/d29/d48/d5c/d9b/fa7 [0,4194304] 0 2024-03-22T02:28:52.562 INFO:tasks.workunit.client.0.smithi103.stdout:2/968: write d11/d25/d63/d45/d47/d83/d78/fa4 [92014,103195] 0 2024-03-22T02:28:52.565 INFO:tasks.workunit.client.1.smithi122.stdout:7/348: link d0/d4/d20/d57/f6a d0/d4e/d5d/f6e 0 2024-03-22T02:28:52.565 INFO:tasks.workunit.client.0.smithi103.stdout:9/873: dread d2/d10/d43/f66 [0,4194304] 0 2024-03-22T02:28:52.566 INFO:tasks.workunit.client.0.smithi103.stdout:0/920: read dd/d16/d4b/d96/d29/f5a [186207,119425] 0 2024-03-22T02:28:52.566 INFO:tasks.workunit.client.1.smithi122.stdout:9/490: rename db/df/d3e/d81/c88 to db/d1b/d6c/c9c 0 2024-03-22T02:28:52.567 INFO:tasks.workunit.client.1.smithi122.stdout:7/349: write d0/d4/d22/f64 [4796909,77695] 0 2024-03-22T02:28:52.568 INFO:tasks.workunit.client.1.smithi122.stdout:9/491: mkdir db/d1b/d6c/d58/d9d 0 2024-03-22T02:28:52.569 INFO:tasks.workunit.client.0.smithi103.stdout:0/921: mkdir dd/d34/d53/d11b 0 2024-03-22T02:28:52.569 INFO:tasks.workunit.client.1.smithi122.stdout:9/492: fdatasync db/df/d3e/d81/d29/f32 0 2024-03-22T02:28:52.570 INFO:tasks.workunit.client.0.smithi103.stdout:9/874: creat d2/da/d18/d1e/d77/d8b/dae/dcb/ff9 x:0 0 0 2024-03-22T02:28:52.570 INFO:tasks.workunit.client.0.smithi103.stdout:2/969: link d11/d19/da7/dd3/dd9/lf0 d11/d19/da7/l144 0 2024-03-22T02:28:52.570 INFO:tasks.workunit.client.0.smithi103.stdout:9/875: readlink d2/d10/l16 0 2024-03-22T02:28:52.571 INFO:tasks.workunit.client.1.smithi122.stdout:9/493: mknod db/d23/c9e 0 2024-03-22T02:28:52.571 INFO:tasks.workunit.client.1.smithi122.stdout:7/350: link d0/d4/d22/l36 d0/d4/d20/d57/l6f 0 2024-03-22T02:28:52.573 INFO:tasks.workunit.client.1.smithi122.stdout:9/494: truncate db/d23/d1f/d57/f64 242689 0 2024-03-22T02:28:52.573 INFO:tasks.workunit.client.1.smithi122.stdout:9/495: chown db/d23/d1f/d57/c60 73546 1 2024-03-22T02:28:52.574 INFO:tasks.workunit.client.0.smithi103.stdout:0/922: rename dd/d34/d53/d8c/le3 to dd/d16/d4b/d96/d29/d48/l11c 0 2024-03-22T02:28:52.575 INFO:tasks.workunit.client.1.smithi122.stdout:9/496: unlink db/d23/d1e/f34 0 2024-03-22T02:28:52.578 INFO:tasks.workunit.client.0.smithi103.stdout:2/970: getdents d11/d25/d63/d45/d47/dae/d106 0 2024-03-22T02:28:52.578 INFO:tasks.workunit.client.1.smithi122.stdout:9/497: unlink db/d23/d1e/c26 0 2024-03-22T02:28:52.578 INFO:tasks.workunit.client.0.smithi103.stdout:9/876: dwrite d2/d5/fb [0,4194304] 0 2024-03-22T02:28:52.579 INFO:tasks.workunit.client.1.smithi122.stdout:7/351: fsync d0/d4/d20/f27 0 2024-03-22T02:28:52.581 INFO:tasks.workunit.client.1.smithi122.stdout:7/352: symlink d0/d30/l70 0 2024-03-22T02:28:52.584 INFO:tasks.workunit.client.1.smithi122.stdout:7/353: rmdir d0/d4e 39 2024-03-22T02:28:52.585 INFO:tasks.workunit.client.0.smithi103.stdout:9/877: read d2/d5/d7b/d7d/dd4/dd8/fe2 [2428841,60058] 0 2024-03-22T02:28:52.587 INFO:tasks.workunit.client.1.smithi122.stdout:9/498: dwrite db/d23/d1f/d57/d66/d73/f76 [4194304,4194304] 0 2024-03-22T02:28:52.588 INFO:tasks.workunit.client.0.smithi103.stdout:0/923: dwrite dd/d34/f115 [0,4194304] 0 2024-03-22T02:28:52.588 INFO:tasks.workunit.client.1.smithi122.stdout:7/354: creat d0/d25/d46/f71 x:0 0 0 2024-03-22T02:28:52.589 INFO:tasks.workunit.client.0.smithi103.stdout:9/878: dread - d2/da/d18/d1e/d34/d3d/fee zero size 2024-03-22T02:28:52.590 INFO:tasks.workunit.client.1.smithi122.stdout:7/355: write d0/d4/d22/f64 [835682,72000] 0 2024-03-22T02:28:52.590 INFO:tasks.workunit.client.0.smithi103.stdout:0/924: chown fa 2322 1 2024-03-22T02:28:52.591 INFO:tasks.workunit.client.1.smithi122.stdout:7/356: chown d0/d4/d20/d21/d28/c33 38723909 1 2024-03-22T02:28:52.592 INFO:tasks.workunit.client.0.smithi103.stdout:2/971: dwrite d11/d19/d5d/fba [0,4194304] 0 2024-03-22T02:28:52.593 INFO:tasks.workunit.client.1.smithi122.stdout:9/499: creat db/df/d3e/d81/d29/f9f x:0 0 0 2024-03-22T02:28:52.596 INFO:tasks.workunit.client.0.smithi103.stdout:2/972: write d11/dc5/fe6 [111407,102223] 0 2024-03-22T02:28:52.597 INFO:tasks.workunit.client.1.smithi122.stdout:9/500: creat db/d23/d1f/fa0 x:0 0 0 2024-03-22T02:28:52.597 INFO:tasks.workunit.client.1.smithi122.stdout:7/357: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:52.599 INFO:tasks.workunit.client.0.smithi103.stdout:7/902: dread d1/d10/d11/d19/d29/d61/d69/f62 [0,4194304] 0 2024-03-22T02:28:52.601 INFO:tasks.workunit.client.0.smithi103.stdout:2/973: chown d11/d25/d46/db6/lbb 12859 1 2024-03-22T02:28:52.602 INFO:tasks.workunit.client.0.smithi103.stdout:9/879: rename d2/da/d18/d1e/d34/d3d/d61/da0/fb8 to d2/da/d18/d1e/d34/d3d/d61/df8/ffa 0 2024-03-22T02:28:52.604 INFO:tasks.workunit.client.1.smithi122.stdout:7/358: rename d0/d25/c26 to d0/d4/d22/c72 0 2024-03-22T02:28:52.604 INFO:tasks.workunit.client.0.smithi103.stdout:9/880: chown d2/da/d18/d1e/d34/d3d/d61/db7/fd5 128188 1 2024-03-22T02:28:52.606 INFO:tasks.workunit.client.0.smithi103.stdout:7/903: rmdir d1/d10/d11/d19/d29/d33/d4b/d5e/d105 39 2024-03-22T02:28:52.607 INFO:tasks.workunit.client.1.smithi122.stdout:7/359: truncate d0/f16 346690 0 2024-03-22T02:28:52.608 INFO:tasks.workunit.client.1.smithi122.stdout:9/501: dwrite db/df/d3e/d81/d3d/d79/f8a [0,4194304] 0 2024-03-22T02:28:52.609 INFO:tasks.workunit.client.0.smithi103.stdout:0/925: link dd/d16/d4b/d96/d29/d48/d5c/fa8 dd/d16/f11d 0 2024-03-22T02:28:52.610 INFO:tasks.workunit.client.0.smithi103.stdout:9/881: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:52.610 INFO:tasks.workunit.client.1.smithi122.stdout:7/360: truncate d0/d25/d5c/f6d 296259 0 2024-03-22T02:28:52.610 INFO:tasks.workunit.client.1.smithi122.stdout:9/502: read - db/df/d3e/d81/d25/f53 zero size 2024-03-22T02:28:52.610 INFO:tasks.workunit.client.1.smithi122.stdout:7/361: write d0/fd [213817,13063] 0 2024-03-22T02:28:52.611 INFO:tasks.workunit.client.0.smithi103.stdout:2/974: mknod d11/d25/d46/c145 0 2024-03-22T02:28:52.612 INFO:tasks.workunit.client.0.smithi103.stdout:7/904: mkdir d1/d10/d11/d19/d11f 0 2024-03-22T02:28:52.612 INFO:tasks.workunit.client.0.smithi103.stdout:0/926: rmdir dd/d16/d4b/d96/d64/dab/dc6/d111 39 2024-03-22T02:28:52.614 INFO:tasks.workunit.client.1.smithi122.stdout:9/503: creat db/df/d3e/d81/d3d/fa1 x:0 0 0 2024-03-22T02:28:52.616 INFO:tasks.workunit.client.1.smithi122.stdout:9/504: read - db/df/d3e/d81/d29/f32 zero size 2024-03-22T02:28:52.617 INFO:tasks.workunit.client.1.smithi122.stdout:7/362: dwrite d0/d4/d22/f32 [0,4194304] 0 2024-03-22T02:28:52.617 INFO:tasks.workunit.client.1.smithi122.stdout:9/505: readlink db/df/d3e/d81/d3d/l71 0 2024-03-22T02:28:52.622 INFO:tasks.workunit.client.1.smithi122.stdout:9/506: link db/d23/d1e/c7f db/df/d3e/d81/d25/ca2 0 2024-03-22T02:28:52.625 INFO:tasks.workunit.client.1.smithi122.stdout:9/507: rename db/d1b/d6c/f35 to db/d23/d9a/fa3 0 2024-03-22T02:28:52.632 INFO:tasks.workunit.client.0.smithi103.stdout:0/927: mkdir dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/d11e 0 2024-03-22T02:28:52.635 INFO:tasks.workunit.client.0.smithi103.stdout:0/928: fsync dd/d16/d4b/d96/d35/f46 0 2024-03-22T02:28:52.637 INFO:tasks.workunit.client.0.smithi103.stdout:0/929: symlink dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4/d11e/l11f 0 2024-03-22T02:28:52.652 INFO:tasks.workunit.client.0.smithi103.stdout:9/882: dread d2/d10/d43/d9b/df3/f82 [0,4194304] 0 2024-03-22T02:28:52.652 INFO:tasks.workunit.client.1.smithi122.stdout:6/469: sync 2024-03-22T02:28:52.654 INFO:tasks.workunit.client.1.smithi122.stdout:6/470: chown d7/de/dd/d20/d2d/f5f 2 1 2024-03-22T02:28:52.656 INFO:tasks.workunit.client.0.smithi103.stdout:9/883: rename d2/d5/caa to d2/da/d18/d1e/d34/d3d/d61/cfb 0 2024-03-22T02:28:52.656 INFO:tasks.workunit.client.1.smithi122.stdout:6/471: mknod d7/de/c90 0 2024-03-22T02:28:52.656 INFO:tasks.workunit.client.0.smithi103.stdout:9/884: chown d2/d10/d43/d7e/d96/d9d/lb4 111482 1 2024-03-22T02:28:52.658 INFO:tasks.workunit.client.1.smithi122.stdout:6/472: unlink d7/de/c90 0 2024-03-22T02:28:52.659 INFO:tasks.workunit.client.0.smithi103.stdout:9/885: truncate d2/da/d18/d1e/d34/d3d/d61/f87 876877 0 2024-03-22T02:28:52.663 INFO:tasks.workunit.client.1.smithi122.stdout:6/473: getdents d7/de/dd/d20/d2d/d54/d76 0 2024-03-22T02:28:52.667 INFO:tasks.workunit.client.1.smithi122.stdout:6/474: rename d7/de/dd/d20/l35 to d7/de/dd/d20/d2d/d54/d76/l91 0 2024-03-22T02:28:52.669 INFO:tasks.workunit.client.0.smithi103.stdout:9/886: write d2/d10/d43/f3e [582614,123499] 0 2024-03-22T02:28:52.672 INFO:tasks.workunit.client.0.smithi103.stdout:2/975: dwrite d11/f5a [0,4194304] 0 2024-03-22T02:28:52.672 INFO:tasks.workunit.client.1.smithi122.stdout:6/475: getdents d7/de/d27 0 2024-03-22T02:28:52.675 INFO:tasks.workunit.client.0.smithi103.stdout:0/930: dwrite dd/d16/d4b/d96/d29/d48/d5c/d9b/fa9 [0,4194304] 0 2024-03-22T02:28:52.675 INFO:tasks.workunit.client.0.smithi103.stdout:9/887: mknod d2/d10/d43/d7e/dc1/cfc 0 2024-03-22T02:28:52.676 INFO:tasks.workunit.client.0.smithi103.stdout:2/976: write d11/d25/d46/fc2 [2101863,120736] 0 2024-03-22T02:28:52.679 INFO:tasks.workunit.client.0.smithi103.stdout:2/977: chown d11/d19/d4e/dfc 2034 1 2024-03-22T02:28:52.679 INFO:tasks.workunit.client.0.smithi103.stdout:7/905: dread d1/d10/d11/d19/d29/d61/d1d/f41 [0,4194304] 0 2024-03-22T02:28:52.682 INFO:tasks.workunit.client.0.smithi103.stdout:9/888: symlink d2/d5/d4c/d56/lfd 0 2024-03-22T02:28:52.682 INFO:tasks.workunit.client.1.smithi122.stdout:6/476: dwrite d7/de/dd/d20/f45 [0,4194304] 0 2024-03-22T02:28:52.683 INFO:tasks.workunit.client.0.smithi103.stdout:0/931: chown dd/d16/d4b/d96/d64/dab/lb6 211 1 2024-03-22T02:28:52.684 INFO:tasks.workunit.client.0.smithi103.stdout:2/978: mkdir d11/d19/d5d/d146 0 2024-03-22T02:28:52.685 INFO:tasks.workunit.client.0.smithi103.stdout:9/889: chown d2/da/d18/d1e/d34/d3d/d61 0 1 2024-03-22T02:28:52.687 INFO:tasks.workunit.client.0.smithi103.stdout:7/906: dwrite d1/da/fc7 [4194304,4194304] 0 2024-03-22T02:28:52.687 INFO:tasks.workunit.client.0.smithi103.stdout:9/890: read d2/da/fd [931698,44116] 0 2024-03-22T02:28:52.689 INFO:tasks.workunit.client.0.smithi103.stdout:2/979: chown d11/d25/d63/d45/d47/d83/da3/fd0 49 1 2024-03-22T02:28:52.691 INFO:tasks.workunit.client.1.smithi122.stdout:6/477: dread d7/de/d27/f6a [0,4194304] 0 2024-03-22T02:28:52.693 INFO:tasks.workunit.client.0.smithi103.stdout:2/980: write f4 [4560586,48101] 0 2024-03-22T02:28:52.693 INFO:tasks.workunit.client.0.smithi103.stdout:9/891: dread - d2/da/d18/d1e/d77/d8b/fb6 zero size 2024-03-22T02:28:52.696 INFO:tasks.workunit.client.0.smithi103.stdout:9/892: truncate d2/d10/d43/d7e/da4/fe3 684897 0 2024-03-22T02:28:52.698 INFO:tasks.workunit.client.0.smithi103.stdout:7/907: dread d1/d10/d11/d19/d29/d61/d69/d6b/d7a/f7f [0,4194304] 0 2024-03-22T02:28:52.701 INFO:tasks.workunit.client.0.smithi103.stdout:2/981: dwrite d11/d25/f111 [0,4194304] 0 2024-03-22T02:28:52.703 INFO:tasks.workunit.client.0.smithi103.stdout:0/932: write dd/d34/d53/d8c/fc0 [1766880,5765] 0 2024-03-22T02:28:52.706 INFO:tasks.workunit.client.0.smithi103.stdout:9/893: rename d2/d5/d4c/c86 to d2/da/d18/d1e/d77/cfe 0 2024-03-22T02:28:52.708 INFO:tasks.workunit.client.1.smithi122.stdout:7/363: link d0/f16 d0/d4e/f73 0 2024-03-22T02:28:52.708 INFO:tasks.workunit.client.0.smithi103.stdout:0/933: mkdir dd/d16/d120 0 2024-03-22T02:28:52.711 INFO:tasks.workunit.client.0.smithi103.stdout:7/908: rename d1/d10/d11/d19/d29/d61/fc8 to d1/d10/d11/d70/f120 0 2024-03-22T02:28:52.713 INFO:tasks.workunit.client.0.smithi103.stdout:2/982: getdents d11/d25/d63/d45/d47/dae 0 2024-03-22T02:28:52.714 INFO:tasks.workunit.client.1.smithi122.stdout:7/364: link d0/d4e/f73 d0/d25/d5c/f74 0 2024-03-22T02:28:52.715 INFO:tasks.workunit.client.1.smithi122.stdout:7/365: fdatasync d0/d4/f2d 0 2024-03-22T02:28:52.715 INFO:tasks.workunit.client.0.smithi103.stdout:2/983: chown d11/d19/da7/dd3/f118 159636938 1 2024-03-22T02:28:52.715 INFO:tasks.workunit.client.0.smithi103.stdout:0/934: mknod dd/d16/d4b/d96/d64/dab/dc6/c121 0 2024-03-22T02:28:52.716 INFO:tasks.workunit.client.0.smithi103.stdout:9/894: dwrite d2/da/d18/d1e/f88 [0,4194304] 0 2024-03-22T02:28:52.718 INFO:tasks.workunit.client.1.smithi122.stdout:7/366: getdents d0/d4/d22 0 2024-03-22T02:28:52.720 INFO:tasks.workunit.client.1.smithi122.stdout:7/367: rmdir d0/d4/d20/d21 39 2024-03-22T02:28:52.722 INFO:tasks.workunit.client.0.smithi103.stdout:0/935: symlink dd/d16/d4b/d96/d35/dc4/df9/l122 0 2024-03-22T02:28:52.723 INFO:tasks.workunit.client.0.smithi103.stdout:2/984: mknod d11/d19/da7/dd3/dd9/d110/c147 0 2024-03-22T02:28:52.723 INFO:tasks.workunit.client.1.smithi122.stdout:7/368: creat d0/d4/d20/d21/f75 x:0 0 0 2024-03-22T02:28:52.723 INFO:tasks.workunit.client.0.smithi103.stdout:7/909: getdents d1/d10/d11/d19/dc3 0 2024-03-22T02:28:52.725 INFO:tasks.workunit.client.1.smithi122.stdout:7/369: write d0/d4/f9 [3779790,95510] 0 2024-03-22T02:28:52.726 INFO:tasks.workunit.client.0.smithi103.stdout:9/895: rename d2/d10/d43/d7e/fed to d2/d10/d43/d7e/d96/fff 0 2024-03-22T02:28:52.727 INFO:tasks.workunit.client.1.smithi122.stdout:7/370: rmdir d0/d25/d46/d48 39 2024-03-22T02:28:52.729 INFO:tasks.workunit.client.0.smithi103.stdout:2/985: symlink d11/d25/d63/d45/d47/d83/d78/d125/l148 0 2024-03-22T02:28:52.730 INFO:tasks.workunit.client.0.smithi103.stdout:9/896: dread d2/da/d18/d1e/f88 [0,4194304] 0 2024-03-22T02:28:52.730 INFO:tasks.workunit.client.1.smithi122.stdout:6/478: dread d7/de/f6c [0,4194304] 0 2024-03-22T02:28:52.731 INFO:tasks.workunit.client.1.smithi122.stdout:6/479: fdatasync d7/de/dd/df/f62 0 2024-03-22T02:28:52.732 INFO:tasks.workunit.client.0.smithi103.stdout:2/986: symlink d11/d25/d63/d45/ddd/l149 0 2024-03-22T02:28:52.734 INFO:tasks.workunit.client.0.smithi103.stdout:7/910: truncate d1/d10/d11/d19/d29/d61/d69/fa5 1144242 0 2024-03-22T02:28:52.734 INFO:tasks.workunit.client.0.smithi103.stdout:4/914: sync 2024-03-22T02:28:52.736 INFO:tasks.workunit.client.1.smithi122.stdout:7/371: truncate d0/d4/d22/f32 1874066 0 2024-03-22T02:28:52.737 INFO:tasks.workunit.client.0.smithi103.stdout:2/987: rmdir d11/d25/d46/d40 39 2024-03-22T02:28:52.737 INFO:tasks.workunit.client.0.smithi103.stdout:9/897: dwrite d2/da/d18/f6c [0,4194304] 0 2024-03-22T02:28:52.737 INFO:tasks.workunit.client.1.smithi122.stdout:6/480: truncate d7/de/dd/f13 3818112 0 2024-03-22T02:28:52.738 INFO:tasks.workunit.client.0.smithi103.stdout:2/988: chown d11/d19/da7/dd3/df1 411 1 2024-03-22T02:28:52.739 INFO:tasks.workunit.client.1.smithi122.stdout:7/372: symlink d0/d4/d20/l76 0 2024-03-22T02:28:52.739 INFO:tasks.workunit.client.0.smithi103.stdout:7/911: mknod d1/d10/d11/d70/d8a/d9d/c121 0 2024-03-22T02:28:52.740 INFO:tasks.workunit.client.1.smithi122.stdout:7/373: write d0/d4/f9 [391210,21525] 0 2024-03-22T02:28:52.741 INFO:tasks.workunit.client.0.smithi103.stdout:0/936: write dd/d16/d4b/d96/d35/dc4/fd4 [497481,29360] 0 2024-03-22T02:28:52.743 INFO:tasks.workunit.client.0.smithi103.stdout:4/915: dwrite d3/d1e/d25/f2b [4194304,4194304] 0 2024-03-22T02:28:52.743 INFO:tasks.workunit.client.1.smithi122.stdout:7/374: creat d0/d4/d22/d5f/d5e/f77 x:0 0 0 2024-03-22T02:28:52.745 INFO:tasks.workunit.client.0.smithi103.stdout:2/989: creat d11/d19/da7/dd3/dd9/f14a x:0 0 0 2024-03-22T02:28:52.745 INFO:tasks.workunit.client.0.smithi103.stdout:7/912: read d1/d10/d11/f28 [3446308,23205] 0 2024-03-22T02:28:52.747 INFO:tasks.workunit.client.0.smithi103.stdout:9/898: mkdir d2/d10/d43/d9b/df3/df7/d100 0 2024-03-22T02:28:52.749 INFO:tasks.workunit.client.0.smithi103.stdout:7/913: write d1/da/d2f/d36/fe9 [923640,105166] 0 2024-03-22T02:28:52.751 INFO:tasks.workunit.client.0.smithi103.stdout:0/937: truncate dd/d16/d4b/d96/d29/d48/d5c/fe8 728945 0 2024-03-22T02:28:52.753 INFO:tasks.workunit.client.0.smithi103.stdout:2/990: creat d11/de8/f14b x:0 0 0 2024-03-22T02:28:52.753 INFO:tasks.workunit.client.0.smithi103.stdout:7/914: chown d1/da/d1c/l47 0 1 2024-03-22T02:28:52.755 INFO:tasks.workunit.client.1.smithi122.stdout:6/481: dread d7/de/dd/d20/d2d/f30 [0,4194304] 0 2024-03-22T02:28:52.755 INFO:tasks.workunit.client.0.smithi103.stdout:9/899: read d2/da/d18/d1e/d34/f97 [388054,60308] 0 2024-03-22T02:28:52.757 INFO:tasks.workunit.client.1.smithi122.stdout:6/482: read d7/de/dd/d39/f5b [191679,79543] 0 2024-03-22T02:28:52.758 INFO:tasks.workunit.client.0.smithi103.stdout:2/991: creat d11/d25/d46/f14c x:0 0 0 2024-03-22T02:28:52.758 INFO:tasks.workunit.client.1.smithi122.stdout:6/483: readlink d7/de/dd/df/l4c 0 2024-03-22T02:28:52.759 INFO:tasks.workunit.client.0.smithi103.stdout:7/915: rename d1/da/d1c/c20 to d1/d10/d11/d19/d29/d61/d69/d5c/c122 0 2024-03-22T02:28:52.760 INFO:tasks.workunit.client.0.smithi103.stdout:7/916: fsync d1/d10/d11/d19/d29/fbb 0 2024-03-22T02:28:52.761 INFO:tasks.workunit.client.0.smithi103.stdout:2/992: creat d11/d19/da7/dd3/f14d x:0 0 0 2024-03-22T02:28:52.761 INFO:tasks.workunit.client.0.smithi103.stdout:6/696: sync 2024-03-22T02:28:52.761 INFO:tasks.workunit.client.0.smithi103.stdout:5/979: sync 2024-03-22T02:28:52.762 INFO:tasks.workunit.client.1.smithi122.stdout:6/484: link d7/de/dd/d20/f31 d7/de/f92 0 2024-03-22T02:28:52.764 INFO:tasks.workunit.client.0.smithi103.stdout:2/993: chown d11/l1c 39949612 1 2024-03-22T02:28:52.764 INFO:tasks.workunit.client.0.smithi103.stdout:6/697: unlink d0/fa2 0 2024-03-22T02:28:52.765 INFO:tasks.workunit.client.0.smithi103.stdout:2/994: read - d11/dc5/f139 zero size 2024-03-22T02:28:52.767 INFO:tasks.workunit.client.0.smithi103.stdout:7/917: symlink d1/d6f/l123 0 2024-03-22T02:28:52.767 INFO:tasks.workunit.client.0.smithi103.stdout:5/980: symlink d0/d14/d23/d5c/d77/df2/l156 0 2024-03-22T02:28:52.768 INFO:tasks.workunit.client.0.smithi103.stdout:9/900: dwrite d2/d10/d59/f89 [0,4194304] 0 2024-03-22T02:28:52.770 INFO:tasks.workunit.client.0.smithi103.stdout:6/698: creat d0/d14/dd2/fe1 x:0 0 0 2024-03-22T02:28:52.770 INFO:tasks.workunit.client.0.smithi103.stdout:5/981: chown d0/d14/d23/d36/d95/d96/de5 791568 1 2024-03-22T02:28:52.771 INFO:tasks.workunit.client.0.smithi103.stdout:7/918: symlink d1/d10/d11/d70/l124 0 2024-03-22T02:28:52.772 INFO:tasks.workunit.client.0.smithi103.stdout:4/916: write d3/d1e/d25/d65/f82 [5108271,25404] 0 2024-03-22T02:28:52.774 INFO:tasks.workunit.client.0.smithi103.stdout:7/919: readlink d1/da/d2f/d36/l48 0 2024-03-22T02:28:52.775 INFO:tasks.workunit.client.1.smithi122.stdout:6/485: dwrite d7/de/dd/d20/f21 [0,4194304] 0 2024-03-22T02:28:52.777 INFO:tasks.workunit.client.0.smithi103.stdout:9/901: rename d2/d10/d43/d7e/da4/dcd/l67 to d2/d10/d43/l101 0 2024-03-22T02:28:52.779 INFO:tasks.workunit.client.0.smithi103.stdout:6/699: mknod d0/d3/d22/d58/d6f/d75/db0/ce2 0 2024-03-22T02:28:52.781 INFO:tasks.workunit.client.1.smithi122.stdout:7/375: dread d0/d4/f5 [0,4194304] 0 2024-03-22T02:28:52.782 INFO:tasks.workunit.client.1.smithi122.stdout:6/486: rmdir d7/de/dd 39 2024-03-22T02:28:52.782 INFO:tasks.workunit.client.1.smithi122.stdout:7/376: chown d0/d4/d20/d21 0 1 2024-03-22T02:28:52.784 INFO:tasks.workunit.client.0.smithi103.stdout:0/938: dread dd/d16/d4b/fd6 [0,4194304] 0 2024-03-22T02:28:52.785 INFO:tasks.workunit.client.0.smithi103.stdout:6/700: symlink d0/d88/d8e/le3 0 2024-03-22T02:28:52.787 INFO:tasks.workunit.client.0.smithi103.stdout:6/701: truncate d0/d3/d22/f50 4565077 0 2024-03-22T02:28:52.789 INFO:tasks.workunit.client.0.smithi103.stdout:5/982: truncate d0/d14/d23/d36/d95/d96/fb0 4071190 0 2024-03-22T02:28:52.789 INFO:tasks.workunit.client.1.smithi122.stdout:7/377: link d0/d25/l60 d0/d4/d22/d5f/l78 0 2024-03-22T02:28:52.790 INFO:tasks.workunit.client.0.smithi103.stdout:7/920: mkdir d1/d10/d11/d19/d29/d33/d4b/d5e/d125 0 2024-03-22T02:28:52.792 INFO:tasks.workunit.client.0.smithi103.stdout:4/917: rename d3/d1e/f2f to d3/da2/da6/f132 0 2024-03-22T02:28:52.793 INFO:tasks.workunit.client.0.smithi103.stdout:2/995: truncate d11/d19/d4e/f11a 333196 0 2024-03-22T02:28:52.794 INFO:tasks.workunit.client.0.smithi103.stdout:9/902: creat d2/d5/d4c/d56/ddc/f102 x:0 0 0 2024-03-22T02:28:52.794 INFO:tasks.workunit.client.1.smithi122.stdout:7/378: dread d0/d4/fb [0,4194304] 0 2024-03-22T02:28:52.795 INFO:tasks.workunit.client.1.smithi122.stdout:6/487: dwrite d7/de/d27/f69 [0,4194304] 0 2024-03-22T02:28:52.798 INFO:tasks.workunit.client.0.smithi103.stdout:5/983: write d0/d14/d23/d2a/d33/db5/dbd/f14c [369418,90439] 0 2024-03-22T02:28:52.799 INFO:tasks.workunit.client.1.smithi122.stdout:6/488: creat d7/de/dd/d23/f93 x:0 0 0 2024-03-22T02:28:52.800 INFO:tasks.workunit.client.1.smithi122.stdout:6/489: stat d7/de/dd/d20/d2d/d54/f72 0 2024-03-22T02:28:52.800 INFO:tasks.workunit.client.0.smithi103.stdout:4/918: write d3/da2/da6/f132 [4902494,27793] 0 2024-03-22T02:28:52.802 INFO:tasks.workunit.client.0.smithi103.stdout:0/939: creat dd/d16/d4b/d96/d35/d82/f123 x:0 0 0 2024-03-22T02:28:52.802 INFO:tasks.workunit.client.1.smithi122.stdout:6/490: chown d7/de/d27/c49 1 1 2024-03-22T02:28:52.803 INFO:tasks.workunit.client.0.smithi103.stdout:2/996: dwrite d11/d25/d46/fc2 [0,4194304] 0 2024-03-22T02:28:52.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/903: mkdir d2/d10/d43/d7e/dc1/d103 0 2024-03-22T02:28:52.803 INFO:tasks.workunit.client.1.smithi122.stdout:6/491: chown d7/de/dd/l26 30849075 1 2024-03-22T02:28:52.804 INFO:tasks.workunit.client.0.smithi103.stdout:6/702: symlink d0/d14/d57/d68/le4 0 2024-03-22T02:28:52.805 INFO:tasks.workunit.client.0.smithi103.stdout:7/921: unlink d1/lc 0 2024-03-22T02:28:52.807 INFO:tasks.workunit.client.1.smithi122.stdout:6/492: link d7/de/dd/d23/f48 d7/de/dd/df/f94 0 2024-03-22T02:28:52.809 INFO:tasks.workunit.client.0.smithi103.stdout:6/703: creat d0/d3/d22/d58/d6f/fe5 x:0 0 0 2024-03-22T02:28:52.809 INFO:tasks.workunit.client.0.smithi103.stdout:9/904: rmdir d2/d10/d43/d9b/df3/d65 39 2024-03-22T02:28:52.813 INFO:tasks.workunit.client.1.smithi122.stdout:6/493: rename d7/de/d27/l66 to d7/de/dd/d39/l95 0 2024-03-22T02:28:52.814 INFO:tasks.workunit.client.1.smithi122.stdout:6/494: unlink d7/de/l25 0 2024-03-22T02:28:52.815 INFO:tasks.workunit.client.1.smithi122.stdout:6/495: unlink d7/de/dd/d20/d2d/d54/d76/c7b 0 2024-03-22T02:28:52.816 INFO:tasks.workunit.client.0.smithi103.stdout:0/940: dwrite dd/d16/d4b/d96/d29/d48/fec [0,4194304] 0 2024-03-22T02:28:52.818 INFO:tasks.workunit.client.1.smithi122.stdout:6/496: mkdir d7/de/dd/d20/d2d/d54/d76/d86/d96 0 2024-03-22T02:28:52.819 INFO:tasks.workunit.client.0.smithi103.stdout:9/905: rename d2/d5/fb5 to d2/d10/d59/da5/f104 0 2024-03-22T02:28:52.820 INFO:tasks.workunit.client.1.smithi122.stdout:6/497: symlink d7/de/dd/d20/d2d/d54/l97 0 2024-03-22T02:28:52.822 INFO:tasks.workunit.client.1.smithi122.stdout:6/498: mknod d7/de/dd/d20/d2d/d54/d76/d86/d96/c98 0 2024-03-22T02:28:52.826 INFO:tasks.workunit.client.1.smithi122.stdout:6/499: dread f3 [4194304,4194304] 0 2024-03-22T02:28:52.829 INFO:tasks.workunit.client.0.smithi103.stdout:9/906: dread d2/d10/d43/f3e [0,4194304] 0 2024-03-22T02:28:52.837 INFO:tasks.workunit.client.0.smithi103.stdout:7/922: creat d1/d10/d11/d19/d29/d33/d4b/f126 x:0 0 0 2024-03-22T02:28:52.838 INFO:tasks.workunit.client.0.smithi103.stdout:6/704: truncate d0/f1e 16153 0 2024-03-22T02:28:52.840 INFO:tasks.workunit.client.0.smithi103.stdout:5/984: dread d0/d14/f41 [8388608,4194304] 0 2024-03-22T02:28:52.840 INFO:tasks.workunit.client.1.smithi122.stdout:6/500: dwrite f6 [0,4194304] 0 2024-03-22T02:28:52.843 INFO:tasks.workunit.client.1.smithi122.stdout:6/501: creat d7/de/dd/d23/f99 x:0 0 0 2024-03-22T02:28:52.844 INFO:tasks.workunit.client.0.smithi103.stdout:2/997: dwrite d11/d25/d46/d48/f6e [4194304,4194304] 0 2024-03-22T02:28:52.845 INFO:tasks.workunit.client.0.smithi103.stdout:0/941: dread dd/d16/d4b/d96/d29/f3e [0,4194304] 0 2024-03-22T02:28:52.848 INFO:tasks.workunit.client.0.smithi103.stdout:4/919: dwrite d3/d1e/d25/d65/d83/dc1/f131 [0,4194304] 0 2024-03-22T02:28:52.850 INFO:tasks.workunit.client.1.smithi122.stdout:6/502: truncate d7/de/dd/d20/d2d/d54/d76/d86/f87 323548 0 2024-03-22T02:28:52.855 INFO:tasks.workunit.client.1.smithi122.stdout:6/503: rename d7/de/dd/f7e to d7/de/dd/d20/d2d/f9a 0 2024-03-22T02:28:52.858 INFO:tasks.workunit.client.1.smithi122.stdout:0/364: sync 2024-03-22T02:28:52.860 INFO:tasks.workunit.client.1.smithi122.stdout:6/504: read d7/de/dd/d23/f47 [2251222,53408] 0 2024-03-22T02:28:52.863 INFO:tasks.workunit.client.1.smithi122.stdout:6/505: mkdir d7/de/dd/d9b 0 2024-03-22T02:28:52.864 INFO:tasks.workunit.client.1.smithi122.stdout:0/365: read d11/d14/d23/d24/f25 [3459312,63166] 0 2024-03-22T02:28:52.869 INFO:tasks.workunit.client.1.smithi122.stdout:0/366: link d11/d14/d1d/d27/d2f/d4e/f66 d11/d14/d1d/f80 0 2024-03-22T02:28:52.870 INFO:tasks.workunit.client.0.smithi103.stdout:9/907: creat d2/d5/d4c/f105 x:0 0 0 2024-03-22T02:28:52.870 INFO:tasks.workunit.client.0.smithi103.stdout:6/705: fdatasync d0/d14/d57/f99 0 2024-03-22T02:28:52.871 INFO:tasks.workunit.client.0.smithi103.stdout:9/908: dread - d2/da/f6b zero size 2024-03-22T02:28:52.871 INFO:tasks.workunit.client.0.smithi103.stdout:6/706: dread - d0/d14/fc6 zero size 2024-03-22T02:28:52.872 INFO:tasks.workunit.client.1.smithi122.stdout:6/506: dwrite d7/de/dd/d20/d2d/d54/d76/f89 [0,4194304] 0 2024-03-22T02:28:52.872 INFO:tasks.workunit.client.0.smithi103.stdout:6/707: chown d0/d14/c2f 29 1 2024-03-22T02:28:52.873 INFO:tasks.workunit.client.0.smithi103.stdout:5/985: creat d0/d14/d23/d2a/d33/db5/dfb/f157 x:0 0 0 2024-03-22T02:28:52.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:52 smithi103 ceph-mon[29258]: pgmap v20: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 64 MiB/s rd, 216 MiB/s wr, 390 op/s 2024-03-22T02:28:52.875 INFO:tasks.workunit.client.0.smithi103.stdout:0/942: mkdir dd/d16/d4b/d96/d35/d82/d124 0 2024-03-22T02:28:52.876 INFO:tasks.workunit.client.0.smithi103.stdout:4/920: symlink d3/d38/d5c/d90/df8/l133 0 2024-03-22T02:28:52.877 INFO:tasks.workunit.client.1.smithi122.stdout:0/367: rmdir d11/d16/d44 39 2024-03-22T02:28:52.878 INFO:tasks.workunit.client.0.smithi103.stdout:4/921: chown d3/l9 567 1 2024-03-22T02:28:52.879 INFO:tasks.workunit.client.1.smithi122.stdout:0/368: creat d11/d14/d1b/f81 x:0 0 0 2024-03-22T02:28:52.880 INFO:tasks.workunit.client.1.smithi122.stdout:6/507: getdents d7/de/dd/d20/d2d/d54/d76 0 2024-03-22T02:28:52.882 INFO:tasks.workunit.client.0.smithi103.stdout:0/943: truncate dd/d16/d4b/d96/d35/dc4/df9/f11a 543259 0 2024-03-22T02:28:52.882 INFO:tasks.workunit.client.1.smithi122.stdout:0/369: read d11/d16/d44/d4c/f3e [757261,123339] 0 2024-03-22T02:28:52.882 INFO:tasks.workunit.client.0.smithi103.stdout:5/986: dwrite d0/d14/d23/d5c/d77/f131 [0,4194304] 0 2024-03-22T02:28:52.883 INFO:tasks.workunit.client.1.smithi122.stdout:6/508: symlink d7/l9c 0 2024-03-22T02:28:52.883 INFO:tasks.workunit.client.0.smithi103.stdout:6/708: truncate d0/d14/d57/d68/f2d 2693596 0 2024-03-22T02:28:52.883 INFO:tasks.workunit.client.0.smithi103.stdout:0/944: chown dd/d16/d4b/d96 357 1 2024-03-22T02:28:52.885 INFO:tasks.workunit.client.0.smithi103.stdout:7/923: rename d1/dfa/l7c to d1/d10/d11/d19/d29/d61/d69/d6b/l127 0 2024-03-22T02:28:52.885 INFO:tasks.workunit.client.1.smithi122.stdout:6/509: creat d7/de/dd/d20/d2d/d54/f9d x:0 0 0 2024-03-22T02:28:52.886 INFO:tasks.workunit.client.1.smithi122.stdout:6/510: dread - d7/de/dd/d39/f73 zero size 2024-03-22T02:28:52.892 INFO:tasks.workunit.client.1.smithi122.stdout:6/511: read d7/de/dd/d20/f3f [132498,107719] 0 2024-03-22T02:28:52.892 INFO:tasks.workunit.client.1.smithi122.stdout:0/370: fsync f10 0 2024-03-22T02:28:52.899 INFO:tasks.workunit.client.1.smithi122.stdout:6/512: creat d7/de/dd/d9b/f9e x:0 0 0 2024-03-22T02:28:52.900 INFO:tasks.workunit.client.1.smithi122.stdout:0/371: creat d11/d4f/f82 x:0 0 0 2024-03-22T02:28:52.902 INFO:tasks.workunit.client.1.smithi122.stdout:6/513: fdatasync d7/de/dd/d23/f48 0 2024-03-22T02:28:52.904 INFO:tasks.workunit.client.1.smithi122.stdout:6/514: rmdir d7/de/dd/d23 39 2024-03-22T02:28:52.905 INFO:tasks.workunit.client.0.smithi103.stdout:2/998: dread f7 [0,4194304] 0 2024-03-22T02:28:52.906 INFO:tasks.workunit.client.1.smithi122.stdout:0/372: rename d11/d14/d23/d31 to d11/d83 0 2024-03-22T02:28:52.908 INFO:tasks.workunit.client.1.smithi122.stdout:0/373: write d11/d4f/d5a/f5c [1953434,43543] 0 2024-03-22T02:28:52.910 INFO:tasks.workunit.client.1.smithi122.stdout:0/374: write d11/d14/d23/f72 [782273,38497] 0 2024-03-22T02:28:52.910 INFO:tasks.workunit.client.0.smithi103.stdout:7/924: rename d1/d10/d11/d19/d29/d61 to d1/d10/d11/d19/d86/d92/dbf/d128 0 2024-03-22T02:28:52.912 INFO:tasks.workunit.client.0.smithi103.stdout:0/945: write dd/d16/d4b/d96/d29/d48/d5c/f80 [442782,106466] 0 2024-03-22T02:28:52.913 INFO:tasks.workunit.client.0.smithi103.stdout:6/709: write d0/d3/d22/d58/d6f/d75/daa/fcf [112034,90833] 0 2024-03-22T02:28:52.914 INFO:tasks.workunit.client.0.smithi103.stdout:9/909: dwrite d2/da/d18/d1e/d77/fec [0,4194304] 0 2024-03-22T02:28:52.914 INFO:tasks.workunit.client.0.smithi103.stdout:5/987: write d0/fe [1542766,80028] 0 2024-03-22T02:28:52.919 INFO:tasks.workunit.client.0.smithi103.stdout:7/925: write d1/d10/d11/d19/d86/d92/f11c [578076,60894] 0 2024-03-22T02:28:52.921 INFO:tasks.workunit.client.0.smithi103.stdout:6/710: creat d0/d88/d8e/fe6 x:0 0 0 2024-03-22T02:28:52.925 INFO:tasks.workunit.client.0.smithi103.stdout:6/711: write d0/d3/fb8 [489839,111233] 0 2024-03-22T02:28:52.926 INFO:tasks.workunit.client.0.smithi103.stdout:4/922: dwrite d3/d30/f123 [4194304,4194304] 0 2024-03-22T02:28:52.927 INFO:tasks.workunit.client.0.smithi103.stdout:7/926: dread - d1/d10/d11/d19/d29/fe2 zero size 2024-03-22T02:28:52.928 INFO:tasks.workunit.client.0.smithi103.stdout:6/712: fdatasync d0/d3/f74 0 2024-03-22T02:28:52.932 INFO:tasks.workunit.client.0.smithi103.stdout:4/923: mkdir d3/d1e/d25/d65/d83/dc1/d134 0 2024-03-22T02:28:52.932 INFO:tasks.workunit.client.0.smithi103.stdout:2/999: dread f10 [0,4194304] 0 2024-03-22T02:28:52.933 INFO:tasks.workunit.client.0.smithi103.stdout:0/946: dwrite dd/d16/d4b/d96/d35/dc4/fd4 [0,4194304] 0 2024-03-22T02:28:52.934 INFO:tasks.workunit.client.0.smithi103.stdout:6/713: stat d0/d3/d22/d58/d6f/d75/db0/dd7 0 2024-03-22T02:28:52.935 INFO:tasks.workunit.client.0.smithi103.stdout:4/924: chown d3/d31/f47 15 1 2024-03-22T02:28:52.939 INFO:tasks.workunit.client.0.smithi103.stdout:5/988: truncate d0/d14/d23/f10e 3227599 0 2024-03-22T02:28:52.940 INFO:tasks.workunit.client.0.smithi103.stdout:6/714: fsync d0/d3/d33/f5d 0 2024-03-22T02:28:52.941 INFO:tasks.workunit.client.0.smithi103.stdout:7/927: dread d1/d10/d11/d19/d86/d92/dbf/d128/d1d/f41 [4194304,4194304] 0 2024-03-22T02:28:52.943 INFO:tasks.workunit.client.0.smithi103.stdout:9/910: truncate d2/da/d18/d1e/d34/d3d/d61/db7/fd5 177060 0 2024-03-22T02:28:52.944 INFO:tasks.workunit.client.0.smithi103.stdout:5/989: dread - d0/d14/d23/d5c/d77/fa4 zero size 2024-03-22T02:28:52.947 INFO:tasks.workunit.client.0.smithi103.stdout:9/911: symlink d2/da/l106 0 2024-03-22T02:28:52.949 INFO:tasks.workunit.client.0.smithi103.stdout:6/715: mkdir d0/d3/d22/d58/dd8/de7 0 2024-03-22T02:28:52.950 INFO:tasks.workunit.client.0.smithi103.stdout:7/928: rename d1/da/d1c/f8b to d1/d10/d11/d19/dc3/f129 0 2024-03-22T02:28:52.953 INFO:tasks.workunit.client.0.smithi103.stdout:9/912: creat d2/da/f107 x:0 0 0 2024-03-22T02:28:52.956 INFO:tasks.workunit.client.0.smithi103.stdout:4/925: write d3/d1e/d25/d65/f99 [1433120,39364] 0 2024-03-22T02:28:52.959 INFO:tasks.workunit.client.0.smithi103.stdout:6/716: truncate d0/d3/d33/f5d 3863754 0 2024-03-22T02:28:52.961 INFO:tasks.workunit.client.0.smithi103.stdout:9/913: creat d2/da/d18/d1e/d77/d8b/dae/dcb/f108 x:0 0 0 2024-03-22T02:28:52.961 INFO:tasks.workunit.client.0.smithi103.stdout:5/990: link d0/d14/ccd d0/d14/d23/d2a/d33/db9/c158 0 2024-03-22T02:28:52.963 INFO:tasks.workunit.client.0.smithi103.stdout:0/947: truncate dd/d16/d4b/d96/d29/d48/d5c/d9b/fa9 556705 0 2024-03-22T02:28:52.963 INFO:tasks.workunit.client.0.smithi103.stdout:9/914: write d2/da/d18/f6c [3907986,41976] 0 2024-03-22T02:28:52.964 INFO:tasks.workunit.client.0.smithi103.stdout:4/926: symlink d3/d1e/d25/d65/d83/da1/dbd/l135 0 2024-03-22T02:28:52.965 INFO:tasks.workunit.client.0.smithi103.stdout:7/929: dwrite d1/d10/d11/d19/d86/d92/dbf/d128/d1d/f101 [0,4194304] 0 2024-03-22T02:28:52.965 INFO:tasks.workunit.client.0.smithi103.stdout:0/948: chown dd/d16/d4b/ff6 6625353 1 2024-03-22T02:28:52.967 INFO:tasks.workunit.client.0.smithi103.stdout:9/915: write d2/da/d18/d1e/d34/d3d/fee [247789,118708] 0 2024-03-22T02:28:52.968 INFO:tasks.workunit.client.0.smithi103.stdout:0/949: write dd/d34/d53/df3/f93 [3020145,52309] 0 2024-03-22T02:28:52.969 INFO:tasks.workunit.client.0.smithi103.stdout:9/916: read - d2/d5/d4c/d56/ddc/f102 zero size 2024-03-22T02:28:52.969 INFO:tasks.workunit.client.0.smithi103.stdout:6/717: symlink d0/d3/d22/d58/d6f/d75/daa/dd4/le8 0 2024-03-22T02:28:52.971 INFO:tasks.workunit.client.0.smithi103.stdout:9/917: fsync d2/da/d18/d1e/d34/d3d/d4d/f58 0 2024-03-22T02:28:52.972 INFO:tasks.workunit.client.0.smithi103.stdout:4/927: truncate d3/d1e/d53/d7a/f11b 595916 0 2024-03-22T02:28:52.977 INFO:tasks.workunit.client.0.smithi103.stdout:7/930: mkdir d1/da/d2f/d12a 0 2024-03-22T02:28:52.978 INFO:tasks.workunit.client.0.smithi103.stdout:7/931: stat d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/f85 0 2024-03-22T02:28:52.981 INFO:tasks.workunit.client.0.smithi103.stdout:4/928: rename d3/d1e/d53/d8a/f9a to d3/d30/d45/f136 0 2024-03-22T02:28:52.987 INFO:tasks.workunit.client.0.smithi103.stdout:0/950: creat dd/d16/d4b/d96/d35/f125 x:0 0 0 2024-03-22T02:28:52.989 INFO:tasks.workunit.client.0.smithi103.stdout:5/991: getdents d0/d14/d23/d5c/df3 0 2024-03-22T02:28:52.991 INFO:tasks.workunit.client.0.smithi103.stdout:4/929: rmdir d3/d1e/d68 39 2024-03-22T02:28:52.994 INFO:tasks.workunit.client.0.smithi103.stdout:9/918: write d2/da/f6b [628203,73875] 0 2024-03-22T02:28:52.997 INFO:tasks.workunit.client.0.smithi103.stdout:0/951: rmdir dd/d34/d53/d11b 0 2024-03-22T02:28:52.997 INFO:tasks.workunit.client.0.smithi103.stdout:6/718: dwrite d0/d3/f46 [0,4194304] 0 2024-03-22T02:28:52.999 INFO:tasks.workunit.client.0.smithi103.stdout:5/992: link d0/d14/d23/d2a/d33/db9/ff5 d0/d14/d23/d36/d95/d96/daa/f159 0 2024-03-22T02:28:53.003 INFO:tasks.workunit.client.0.smithi103.stdout:9/919: mkdir d2/da/d109 0 2024-03-22T02:28:53.005 INFO:tasks.workunit.client.0.smithi103.stdout:9/920: chown d2/da/d18/d1e/d34/l5f 3 1 2024-03-22T02:28:53.005 INFO:tasks.workunit.client.0.smithi103.stdout:0/952: creat dd/d16/d4b/d96/d35/dc4/df9/f126 x:0 0 0 2024-03-22T02:28:53.006 INFO:tasks.workunit.client.0.smithi103.stdout:6/719: rmdir d0/d3/d22/d58/d6f/d75/daa 39 2024-03-22T02:28:53.007 INFO:tasks.workunit.client.0.smithi103.stdout:5/993: fsync d0/d14/d23/d5c/d77/df2/f4b 0 2024-03-22T02:28:53.007 INFO:tasks.workunit.client.0.smithi103.stdout:9/921: chown d2/d10/d43/d7e/d96/d9d/f7c 3091253 1 2024-03-22T02:28:53.008 INFO:tasks.workunit.client.0.smithi103.stdout:4/930: write d3/d31/f32 [362454,119025] 0 2024-03-22T02:28:53.011 INFO:tasks.workunit.client.0.smithi103.stdout:7/932: dwrite d1/d10/d11/f5a [4194304,4194304] 0 2024-03-22T02:28:53.012 INFO:tasks.workunit.client.0.smithi103.stdout:7/933: chown d1/d10/d11/d70/d8a 29477 1 2024-03-22T02:28:53.012 INFO:tasks.workunit.client.0.smithi103.stdout:0/953: truncate dd/d16/d4b/d96/f106 1040920 0 2024-03-22T02:28:53.013 INFO:tasks.workunit.client.0.smithi103.stdout:9/922: mkdir d2/d5/d4c/d56/d10a 0 2024-03-22T02:28:53.014 INFO:tasks.workunit.client.0.smithi103.stdout:5/994: creat d0/d14/d23/d6f/d83/f15a x:0 0 0 2024-03-22T02:28:53.017 INFO:tasks.workunit.client.0.smithi103.stdout:4/931: mkdir d3/d38/d5c/d90/df8/d137 0 2024-03-22T02:28:53.020 INFO:tasks.workunit.client.0.smithi103.stdout:7/934: write d1/d10/d11/d70/d8a/f11e [872365,1692] 0 2024-03-22T02:28:53.021 INFO:tasks.workunit.client.0.smithi103.stdout:5/995: dread - d0/d14/d23/d5c/d77/f10f zero size 2024-03-22T02:28:53.021 INFO:tasks.workunit.client.0.smithi103.stdout:4/932: mknod d3/da2/da6/c138 0 2024-03-22T02:28:53.022 INFO:tasks.workunit.client.0.smithi103.stdout:9/923: chown d2/da/d18/d1e/d34/d3d/d46/c70 347 1 2024-03-22T02:28:53.022 INFO:tasks.workunit.client.0.smithi103.stdout:6/720: dwrite d0/d14/d9a/fdb [0,4194304] 0 2024-03-22T02:28:53.025 INFO:tasks.workunit.client.1.smithi122.stdout:3/324: sync 2024-03-22T02:28:53.025 INFO:tasks.workunit.client.0.smithi103.stdout:6/721: chown d0/f2c 15406 1 2024-03-22T02:28:53.028 INFO:tasks.workunit.client.0.smithi103.stdout:5/996: mkdir d0/da2/d12c/d15b 0 2024-03-22T02:28:53.029 INFO:tasks.workunit.client.1.smithi122.stdout:3/325: rename d2/d9/dc/d17/d28/d4c/l50 to d2/d9/dc/df/d3c/d59/l61 0 2024-03-22T02:28:53.030 INFO:tasks.workunit.client.0.smithi103.stdout:0/954: dwrite dd/d16/d4b/d96/d35/dc4/df9/f11a [0,4194304] 0 2024-03-22T02:28:53.030 INFO:tasks.workunit.client.0.smithi103.stdout:9/924: write d2/d10/d43/d7e/d96/fff [976353,94973] 0 2024-03-22T02:28:53.036 INFO:tasks.workunit.client.0.smithi103.stdout:5/997: creat d0/d14/d23/d36/f15c x:0 0 0 2024-03-22T02:28:53.037 INFO:tasks.workunit.client.0.smithi103.stdout:5/998: fdatasync d0/d14/f41 0 2024-03-22T02:28:53.038 INFO:tasks.workunit.client.0.smithi103.stdout:9/925: fdatasync d2/d10/f8a 0 2024-03-22T02:28:53.040 INFO:tasks.workunit.client.1.smithi122.stdout:3/326: getdents d2/d9/dc/df/d3c/d59 0 2024-03-22T02:28:53.040 INFO:tasks.workunit.client.0.smithi103.stdout:7/935: creat d1/d10/d11/d19/d29/d33/d4b/d5e/d125/f12b x:0 0 0 2024-03-22T02:28:53.042 INFO:tasks.workunit.client.1.smithi122.stdout:3/327: dread d2/d9/dc/d17/d28/f4d [0,4194304] 0 2024-03-22T02:28:53.044 INFO:tasks.workunit.client.1.smithi122.stdout:3/328: getdents d2/d9/dc/df/d47 0 2024-03-22T02:28:53.045 INFO:tasks.workunit.client.1.smithi122.stdout:3/329: readlink d2/d9/dc/df/l1f 0 2024-03-22T02:28:53.046 INFO:tasks.workunit.client.0.smithi103.stdout:6/722: creat d0/d14/fe9 x:0 0 0 2024-03-22T02:28:53.046 INFO:tasks.workunit.client.0.smithi103.stdout:4/933: write d3/d1e/f6d [350474,51802] 0 2024-03-22T02:28:53.047 INFO:tasks.workunit.client.0.smithi103.stdout:7/936: unlink d1/d10/d11/d19/d29/d33/d4b/d5e/l50 0 2024-03-22T02:28:53.048 INFO:tasks.workunit.client.0.smithi103.stdout:0/955: write dd/d16/f11d [1313741,14905] 0 2024-03-22T02:28:53.048 INFO:tasks.workunit.client.1.smithi122.stdout:3/330: rename d2/d9/dc/df/d38/f60 to d2/f62 0 2024-03-22T02:28:53.051 INFO:tasks.workunit.client.0.smithi103.stdout:6/723: creat d0/dbd/fea x:0 0 0 2024-03-22T02:28:53.051 INFO:tasks.workunit.client.0.smithi103.stdout:7/937: creat d1/d10/d11/d19/dc3/de8/f12c x:0 0 0 2024-03-22T02:28:53.055 INFO:tasks.workunit.client.0.smithi103.stdout:6/724: truncate d0/d14/d57/d68/fa8 5332366 0 2024-03-22T02:28:53.055 INFO:tasks.workunit.client.0.smithi103.stdout:5/999: dread d0/d14/d23/d5c/d70/f8f [0,4194304] 0 2024-03-22T02:28:53.055 INFO:tasks.workunit.client.0.smithi103.stdout:0/956: mknod dd/d16/d4b/d96/d64/dab/c127 0 2024-03-22T02:28:53.058 INFO:tasks.workunit.client.0.smithi103.stdout:6/725: write d0/d3/f6d [1556674,55693] 0 2024-03-22T02:28:53.060 INFO:tasks.workunit.client.0.smithi103.stdout:0/957: write dd/d16/d4b/d96/f106 [501438,84986] 0 2024-03-22T02:28:53.062 INFO:tasks.workunit.client.1.smithi122.stdout:3/331: dread d2/f7 [0,4194304] 0 2024-03-22T02:28:53.063 INFO:tasks.workunit.client.0.smithi103.stdout:4/934: dread d3/d38/f85 [0,4194304] 0 2024-03-22T02:28:53.063 INFO:tasks.workunit.client.0.smithi103.stdout:6/726: mkdir d0/d3/d22/d58/db4/deb 0 2024-03-22T02:28:53.064 INFO:tasks.workunit.client.0.smithi103.stdout:9/926: dread d2/da/d18/d1e/d34/d3d/d46/fad [0,4194304] 0 2024-03-22T02:28:53.064 INFO:tasks.workunit.client.0.smithi103.stdout:7/938: link d1/d10/d11/dd9/ffb d1/d10/d11/d19/d29/d33/d4b/f12d 0 2024-03-22T02:28:53.067 INFO:tasks.workunit.client.0.smithi103.stdout:4/935: write d3/d30/d45/d107/f11a [407265,46754] 0 2024-03-22T02:28:53.067 INFO:tasks.workunit.client.0.smithi103.stdout:6/727: read d0/d3/d33/f4a [790312,17645] 0 2024-03-22T02:28:53.068 INFO:tasks.workunit.client.0.smithi103.stdout:9/927: creat d2/d10/d43/d7e/f10b x:0 0 0 2024-03-22T02:28:53.068 INFO:tasks.workunit.client.1.smithi122.stdout:3/332: dread d2/d9/dc/d17/f42 [0,4194304] 0 2024-03-22T02:28:53.069 INFO:tasks.workunit.client.1.smithi122.stdout:3/333: stat d2/d9/dc/d17/d2b/c3a 0 2024-03-22T02:28:53.070 INFO:tasks.workunit.client.0.smithi103.stdout:7/939: creat d1/f12e x:0 0 0 2024-03-22T02:28:53.070 INFO:tasks.workunit.client.0.smithi103.stdout:4/936: chown d3/d1e/d53/d8a 111 1 2024-03-22T02:28:53.070 INFO:tasks.workunit.client.1.smithi122.stdout:3/334: read d2/f8 [408304,104987] 0 2024-03-22T02:28:53.072 INFO:tasks.workunit.client.0.smithi103.stdout:6/728: mknod d0/d3/d22/d58/d6f/d75/db0/dd7/cec 0 2024-03-22T02:28:53.075 INFO:tasks.workunit.client.0.smithi103.stdout:4/937: mknod d3/d38/d5c/d90/c139 0 2024-03-22T02:28:53.075 INFO:tasks.workunit.client.1.smithi122.stdout:3/335: write d2/f15 [5223325,56509] 0 2024-03-22T02:28:53.076 INFO:tasks.workunit.client.0.smithi103.stdout:7/940: dread d1/dfa/fb0 [0,4194304] 0 2024-03-22T02:28:53.076 INFO:tasks.workunit.client.1.smithi122.stdout:3/336: read - d2/f62 zero size 2024-03-22T02:28:53.076 INFO:tasks.workunit.client.0.smithi103.stdout:6/729: creat d0/fed x:0 0 0 2024-03-22T02:28:53.077 INFO:tasks.workunit.client.1.smithi122.stdout:3/337: chown d2/d9/dc/d17/d28/d4c/c52 62 1 2024-03-22T02:28:53.078 INFO:tasks.workunit.client.1.smithi122.stdout:3/338: rmdir d2/d9 39 2024-03-22T02:28:53.078 INFO:tasks.workunit.client.0.smithi103.stdout:9/928: link d2/d10/d43/d7e/d96/lf1 d2/d10/d43/d9b/l10c 0 2024-03-22T02:28:53.078 INFO:tasks.workunit.client.0.smithi103.stdout:7/941: write d1/da/d2f/d36/fe9 [1481914,22297] 0 2024-03-22T02:28:53.080 INFO:tasks.workunit.client.1.smithi122.stdout:3/339: creat d2/d9/dc/d17/f63 x:0 0 0 2024-03-22T02:28:53.081 INFO:tasks.workunit.client.1.smithi122.stdout:3/340: stat d2/d9/dc/d17/d28/f4d 0 2024-03-22T02:28:53.081 INFO:tasks.workunit.client.0.smithi103.stdout:7/942: mkdir d1/da/d1c/d12f 0 2024-03-22T02:28:53.082 INFO:tasks.workunit.client.0.smithi103.stdout:7/943: chown d1/d10/d11/d70/d8a 10 1 2024-03-22T02:28:53.083 INFO:tasks.workunit.client.1.smithi122.stdout:3/341: symlink d2/d9/dc/d17/d2b/l64 0 2024-03-22T02:28:53.083 INFO:tasks.workunit.client.0.smithi103.stdout:4/938: dread d3/d1e/d53/d8a/f9c [0,4194304] 0 2024-03-22T02:28:53.084 INFO:tasks.workunit.client.0.smithi103.stdout:9/929: link d2/da/f25 d2/da/d109/f10d 0 2024-03-22T02:28:53.085 INFO:tasks.workunit.client.0.smithi103.stdout:6/730: link d0/d14/d57/d68/c3a d0/d3/d22/d58/db4/cee 0 2024-03-22T02:28:53.087 INFO:tasks.workunit.client.0.smithi103.stdout:7/944: mkdir d1/d10/d11/d19/d29/d33/d130 0 2024-03-22T02:28:53.088 INFO:tasks.workunit.client.0.smithi103.stdout:4/939: creat d3/d69/d12b/f13a x:0 0 0 2024-03-22T02:28:53.090 INFO:tasks.workunit.client.0.smithi103.stdout:6/731: dread d0/d3/d22/f50 [0,4194304] 0 2024-03-22T02:28:53.091 INFO:tasks.workunit.client.0.smithi103.stdout:7/945: symlink d1/da/d2f/d12a/l131 0 2024-03-22T02:28:53.093 INFO:tasks.workunit.client.0.smithi103.stdout:7/946: rmdir d1/d10/d11/d70 39 2024-03-22T02:28:53.096 INFO:tasks.workunit.client.0.smithi103.stdout:6/732: mknod d0/d3/d22/d58/dd8/cef 0 2024-03-22T02:28:53.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:52 smithi122 ceph-mon[36403]: pgmap v20: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 64 MiB/s rd, 216 MiB/s wr, 390 op/s 2024-03-22T02:28:53.100 INFO:tasks.workunit.client.0.smithi103.stdout:6/733: creat d0/d3/d22/d58/de0/ff0 x:0 0 0 2024-03-22T02:28:53.104 INFO:tasks.workunit.client.0.smithi103.stdout:6/734: mknod d0/d3/d22/d58/d6f/d75/daa/cf1 0 2024-03-22T02:28:53.109 INFO:tasks.workunit.client.0.smithi103.stdout:6/735: link d0/d3/fb8 d0/d3/d22/d58/dd8/ff2 0 2024-03-22T02:28:53.111 INFO:tasks.workunit.client.0.smithi103.stdout:9/930: write d2/d5/d4c/d56/fbc [4274943,82204] 0 2024-03-22T02:28:53.112 INFO:tasks.workunit.client.0.smithi103.stdout:6/736: symlink d0/d3/d22/d58/d6f/d75/daa/dd4/lf3 0 2024-03-22T02:28:53.113 INFO:tasks.workunit.client.0.smithi103.stdout:6/737: chown d0/d3/d33/l4d 99779 1 2024-03-22T02:28:53.115 INFO:tasks.workunit.client.0.smithi103.stdout:9/931: rename d2/da/d18/d1e/cd3 to d2/d5/d7b/d7d/dd4/c10e 0 2024-03-22T02:28:53.116 INFO:tasks.workunit.client.0.smithi103.stdout:9/932: stat d2/la7 0 2024-03-22T02:28:53.121 INFO:tasks.workunit.client.0.smithi103.stdout:4/940: dwrite d3/d1e/d53/d8a/fe9 [0,4194304] 0 2024-03-22T02:28:53.123 INFO:tasks.workunit.client.0.smithi103.stdout:7/947: dwrite d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/f85 [0,4194304] 0 2024-03-22T02:28:53.124 INFO:tasks.workunit.client.0.smithi103.stdout:7/948: fdatasync d1/d10/d11/d19/d29/d33/f5f 0 2024-03-22T02:28:53.125 INFO:tasks.workunit.client.0.smithi103.stdout:4/941: mknod d3/d38/d5c/d90/df8/c13b 0 2024-03-22T02:28:53.127 INFO:tasks.workunit.client.0.smithi103.stdout:6/738: write d0/d3/d22/f37 [3233756,124735] 0 2024-03-22T02:28:53.129 INFO:tasks.workunit.client.0.smithi103.stdout:4/942: mkdir d3/d13c 0 2024-03-22T02:28:53.132 INFO:tasks.workunit.client.0.smithi103.stdout:7/949: truncate d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/fef 811375 0 2024-03-22T02:28:53.138 INFO:tasks.workunit.client.0.smithi103.stdout:6/739: dwrite d0/d3/d22/f37 [0,4194304] 0 2024-03-22T02:28:53.139 INFO:tasks.workunit.client.0.smithi103.stdout:7/950: creat d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/f132 x:0 0 0 2024-03-22T02:28:53.143 INFO:tasks.workunit.client.0.smithi103.stdout:7/951: truncate d1/d10/d11/d19/d86/f106 356163 0 2024-03-22T02:28:53.145 INFO:tasks.workunit.client.0.smithi103.stdout:6/740: symlink d0/d3/d22/d58/db4/deb/lf4 0 2024-03-22T02:28:53.146 INFO:tasks.workunit.client.0.smithi103.stdout:7/952: rmdir d1/d10/d11/d70/d8a/d9d/dff 39 2024-03-22T02:28:53.148 INFO:tasks.workunit.client.0.smithi103.stdout:6/741: creat d0/d14/d57/ff5 x:0 0 0 2024-03-22T02:28:53.149 INFO:tasks.workunit.client.0.smithi103.stdout:7/953: stat d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/fef 0 2024-03-22T02:28:53.149 INFO:tasks.workunit.client.0.smithi103.stdout:6/742: stat d0/d3/d22/d58/ca1 0 2024-03-22T02:28:53.153 INFO:tasks.workunit.client.0.smithi103.stdout:7/954: dread - d1/d10/d11/d19/d29/fbc zero size 2024-03-22T02:28:53.154 INFO:tasks.workunit.client.0.smithi103.stdout:6/743: mkdir d0/d3/d22/d58/d6f/d75/db0/df6 0 2024-03-22T02:28:53.154 INFO:tasks.workunit.client.0.smithi103.stdout:4/943: dwrite d3/d1e/d25/d65/d83/da1/fad [0,4194304] 0 2024-03-22T02:28:53.155 INFO:tasks.workunit.client.0.smithi103.stdout:4/944: chown d3/d69/d12b 2981109 1 2024-03-22T02:28:53.156 INFO:tasks.workunit.client.0.smithi103.stdout:7/955: fdatasync d1/dfa/f6d 0 2024-03-22T02:28:53.162 INFO:tasks.workunit.client.1.smithi122.stdout:2/405: sync 2024-03-22T02:28:53.167 INFO:tasks.workunit.client.1.smithi122.stdout:2/406: dwrite f6 [0,4194304] 0 2024-03-22T02:28:53.171 INFO:tasks.workunit.client.1.smithi122.stdout:2/407: unlink df/d1f/d3d/f88 0 2024-03-22T02:28:53.172 INFO:tasks.workunit.client.1.smithi122.stdout:2/408: write df/d1f/d2a/f2f [1555125,22092] 0 2024-03-22T02:28:53.174 INFO:tasks.workunit.client.1.smithi122.stdout:2/409: creat df/d1f/d33/d6a/f8c x:0 0 0 2024-03-22T02:28:53.180 INFO:tasks.workunit.client.1.smithi122.stdout:2/410: dwrite df/f36 [0,4194304] 0 2024-03-22T02:28:53.180 INFO:tasks.workunit.client.1.smithi122.stdout:2/411: stat df/d1f/d3a/f58 0 2024-03-22T02:28:53.182 INFO:tasks.workunit.client.1.smithi122.stdout:2/412: unlink df/l1d 0 2024-03-22T02:28:53.183 INFO:tasks.workunit.client.1.smithi122.stdout:2/413: write df/d1f/d33/f80 [243297,123504] 0 2024-03-22T02:28:53.185 INFO:tasks.workunit.client.1.smithi122.stdout:2/414: creat df/d1f/d33/d6a/f8d x:0 0 0 2024-03-22T02:28:53.186 INFO:tasks.workunit.client.1.smithi122.stdout:2/415: write df/d1f/f3e [3786763,57327] 0 2024-03-22T02:28:53.188 INFO:tasks.workunit.client.1.smithi122.stdout:2/416: write df/d1f/d33/d6a/f8d [244229,67385] 0 2024-03-22T02:28:53.191 INFO:tasks.workunit.client.1.smithi122.stdout:2/417: dread df/d1f/d33/f80 [0,4194304] 0 2024-03-22T02:28:53.191 INFO:tasks.workunit.client.1.smithi122.stdout:2/418: chown df/d1f/d33/d34/l63 1005 1 2024-03-22T02:28:53.192 INFO:tasks.workunit.client.1.smithi122.stdout:2/419: chown df/d1f/d2a/d39 3712423 1 2024-03-22T02:28:53.192 INFO:tasks.workunit.client.1.smithi122.stdout:2/420: stat df/d1f/d2a/l61 0 2024-03-22T02:28:53.194 INFO:tasks.workunit.client.1.smithi122.stdout:2/421: mkdir df/d1f/d33/d6a/d7c/d8e 0 2024-03-22T02:28:53.195 INFO:tasks.workunit.client.1.smithi122.stdout:2/422: fdatasync df/d1f/d33/d34/f43 0 2024-03-22T02:28:53.197 INFO:tasks.workunit.client.1.smithi122.stdout:2/423: chown df/d1f/d2a/d39/f62 34392145 1 2024-03-22T02:28:53.197 INFO:tasks.workunit.client.1.smithi122.stdout:2/424: chown df/d1f/d3d/c55 10824617 1 2024-03-22T02:28:53.199 INFO:tasks.workunit.client.1.smithi122.stdout:2/425: symlink df/d1f/d2a/d72/l8f 0 2024-03-22T02:28:53.200 INFO:tasks.workunit.client.1.smithi122.stdout:2/426: creat df/d1f/d33/d6a/f90 x:0 0 0 2024-03-22T02:28:53.202 INFO:tasks.workunit.client.1.smithi122.stdout:2/427: mknod df/c91 0 2024-03-22T02:28:53.203 INFO:tasks.workunit.client.1.smithi122.stdout:7/379: sync 2024-03-22T02:28:53.205 INFO:tasks.workunit.client.1.smithi122.stdout:2/428: link f6 df/d1f/d3d/f92 0 2024-03-22T02:28:53.206 INFO:tasks.workunit.client.0.smithi103.stdout:6/744: mkdir d0/d14/dd2/df7 0 2024-03-22T02:28:53.206 INFO:tasks.workunit.client.0.smithi103.stdout:4/945: creat d3/d38/d5c/d86/f13d x:0 0 0 2024-03-22T02:28:53.209 INFO:tasks.workunit.client.1.smithi122.stdout:7/380: rename d0/c69 to d0/d25/d5c/c79 0 2024-03-22T02:28:53.209 INFO:tasks.workunit.client.0.smithi103.stdout:4/946: chown d3/d1e/d78/df9/ld2 1 1 2024-03-22T02:28:53.210 INFO:tasks.workunit.client.1.smithi122.stdout:7/381: read - d0/d4/d22/d5f/d5e/f77 zero size 2024-03-22T02:28:53.210 INFO:tasks.workunit.client.1.smithi122.stdout:7/382: dread - d0/d30/f5b zero size 2024-03-22T02:28:53.211 INFO:tasks.workunit.client.1.smithi122.stdout:2/429: write df/f11 [2781301,80904] 0 2024-03-22T02:28:53.213 INFO:tasks.workunit.client.1.smithi122.stdout:2/430: write df/d1f/d33/d34/f5a [783255,118267] 0 2024-03-22T02:28:53.215 INFO:tasks.workunit.client.0.smithi103.stdout:4/947: rename d3/deb/cf4 to d3/d1e/d25/d65/c13e 0 2024-03-22T02:28:53.216 INFO:tasks.workunit.client.0.smithi103.stdout:4/948: fsync d3/d31/f32 0 2024-03-22T02:28:53.217 INFO:tasks.workunit.client.1.smithi122.stdout:7/383: dwrite d0/d4/d22/f64 [0,4194304] 0 2024-03-22T02:28:53.217 INFO:tasks.workunit.client.1.smithi122.stdout:2/431: link f4 df/d1f/d33/d6a/f93 0 2024-03-22T02:28:53.218 INFO:tasks.workunit.client.0.smithi103.stdout:6/745: dwrite d0/d3/d22/d58/fd9 [0,4194304] 0 2024-03-22T02:28:53.219 INFO:tasks.workunit.client.1.smithi122.stdout:7/384: chown d0/d4/d20/d57/l6f 12938437 1 2024-03-22T02:28:53.220 INFO:tasks.workunit.client.0.smithi103.stdout:7/956: dwrite d1/d10/d11/f18 [0,4194304] 0 2024-03-22T02:28:53.221 INFO:tasks.workunit.client.1.smithi122.stdout:7/385: truncate d0/d25/f56 545547 0 2024-03-22T02:28:53.221 INFO:tasks.workunit.client.1.smithi122.stdout:2/432: mkdir df/d1f/d33/d4d/d94 0 2024-03-22T02:28:53.225 INFO:tasks.workunit.client.1.smithi122.stdout:7/386: dread d0/d4/d20/d21/f2b [0,4194304] 0 2024-03-22T02:28:53.225 INFO:tasks.workunit.client.1.smithi122.stdout:2/433: rename df/d1f/d3d/f60 to df/d1f/d33/d4d/d94/f95 0 2024-03-22T02:28:53.225 INFO:tasks.workunit.client.1.smithi122.stdout:2/434: stat df/f10 0 2024-03-22T02:28:53.231 INFO:tasks.workunit.client.1.smithi122.stdout:2/435: link df/f36 df/d1f/d2a/d39/f96 0 2024-03-22T02:28:53.233 INFO:tasks.workunit.client.1.smithi122.stdout:7/387: link d0/d4/d22/d5f/l4b d0/d25/d46/d48/l7a 0 2024-03-22T02:28:53.235 INFO:tasks.workunit.client.1.smithi122.stdout:2/436: rmdir df/d1f/d33/d4d/d7a 39 2024-03-22T02:28:53.237 INFO:tasks.workunit.client.1.smithi122.stdout:7/388: mknod d0/c7b 0 2024-03-22T02:28:53.242 INFO:tasks.workunit.client.1.smithi122.stdout:2/437: dwrite df/d1f/f5c [0,4194304] 0 2024-03-22T02:28:53.263 INFO:tasks.workunit.client.1.smithi122.stdout:2/438: dwrite df/f13 [0,4194304] 0 2024-03-22T02:28:53.264 INFO:tasks.workunit.client.1.smithi122.stdout:2/439: chown df/d1f/d2a/d51 1378 1 2024-03-22T02:28:53.266 INFO:tasks.workunit.client.1.smithi122.stdout:2/440: fdatasync df/d1f/d33/d4d/d7a/f8a 0 2024-03-22T02:28:53.271 INFO:tasks.workunit.client.1.smithi122.stdout:2/441: symlink df/d1f/d3a/d4f/d84/l97 0 2024-03-22T02:28:53.272 INFO:tasks.workunit.client.1.smithi122.stdout:2/442: write df/f10 [2634319,42303] 0 2024-03-22T02:28:53.274 INFO:tasks.workunit.client.1.smithi122.stdout:2/443: symlink df/d1f/d2a/d39/l98 0 2024-03-22T02:28:53.281 INFO:tasks.workunit.client.1.smithi122.stdout:2/444: write df/f2b [466283,56807] 0 2024-03-22T02:28:53.283 INFO:tasks.workunit.client.1.smithi122.stdout:2/445: read df/d1f/d33/d6a/f8d [209295,56175] 0 2024-03-22T02:28:53.288 INFO:tasks.workunit.client.1.smithi122.stdout:2/446: dwrite df/f15 [0,4194304] 0 2024-03-22T02:28:53.293 INFO:tasks.workunit.client.1.smithi122.stdout:2/447: dread df/d1f/d33/d34/f5a [0,4194304] 0 2024-03-22T02:28:53.295 INFO:tasks.workunit.client.1.smithi122.stdout:2/448: mknod df/d1f/d33/d4d/d7a/c99 0 2024-03-22T02:28:53.300 INFO:tasks.workunit.client.1.smithi122.stdout:2/449: dwrite fd [4194304,4194304] 0 2024-03-22T02:28:53.300 INFO:tasks.workunit.client.1.smithi122.stdout:2/450: readlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:53.304 INFO:tasks.workunit.client.1.smithi122.stdout:2/451: link df/c91 df/d1f/d33/d4d/d7a/c9a 0 2024-03-22T02:28:53.308 INFO:tasks.workunit.client.1.smithi122.stdout:2/452: dread df/d1f/d2a/d51/d67/f71 [0,4194304] 0 2024-03-22T02:28:53.318 INFO:tasks.workunit.client.0.smithi103.stdout:4/949: creat d3/d1e/d78/df9/f13f x:0 0 0 2024-03-22T02:28:53.320 INFO:tasks.workunit.client.0.smithi103.stdout:6/746: symlink d0/d3/lf8 0 2024-03-22T02:28:53.320 INFO:tasks.workunit.client.0.smithi103.stdout:4/950: write d3/d1e/d53/d7a/f12e [926889,111586] 0 2024-03-22T02:28:53.320 INFO:tasks.workunit.client.0.smithi103.stdout:7/957: mkdir d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d133 0 2024-03-22T02:28:53.322 INFO:tasks.workunit.client.0.smithi103.stdout:6/747: write d0/d3/d22/fab [3801979,105591] 0 2024-03-22T02:28:53.326 INFO:tasks.workunit.client.1.smithi122.stdout:2/453: dread df/d1f/d3a/d41/f47 [0,4194304] 0 2024-03-22T02:28:53.332 INFO:tasks.workunit.client.1.smithi122.stdout:2/454: dwrite df/d1f/d33/d6a/f8c [0,4194304] 0 2024-03-22T02:28:53.336 INFO:tasks.workunit.client.1.smithi122.stdout:2/455: getdents df/d1f/d2a/d51 0 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:4/386: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:5/369: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:8/452: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:1/444: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:0/375: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:6/515: sync 2024-03-22T02:28:53.338 INFO:tasks.workunit.client.1.smithi122.stdout:3/342: sync 2024-03-22T02:28:53.339 INFO:tasks.workunit.client.1.smithi122.stdout:2/456: mkdir df/d1f/d9b 0 2024-03-22T02:28:53.340 INFO:tasks.workunit.client.1.smithi122.stdout:9/508: sync 2024-03-22T02:28:53.340 INFO:tasks.workunit.client.0.smithi103.stdout:4/951: getdents d3/d1e/d25/d65/dc9 0 2024-03-22T02:28:53.343 INFO:tasks.workunit.client.1.smithi122.stdout:4/387: fsync d0/d25/f3a 0 2024-03-22T02:28:53.343 INFO:tasks.workunit.client.1.smithi122.stdout:3/343: fdatasync d2/d9/dc/d17/d28/f2d 0 2024-03-22T02:28:53.344 INFO:tasks.workunit.client.0.smithi103.stdout:4/952: creat d3/d1e/d53/d8a/f140 x:0 0 0 2024-03-22T02:28:53.344 INFO:tasks.workunit.client.0.smithi103.stdout:6/748: dwrite d0/f6a [0,4194304] 0 2024-03-22T02:28:53.345 INFO:tasks.workunit.client.1.smithi122.stdout:8/453: creat d1/d40/f9e x:0 0 0 2024-03-22T02:28:53.346 INFO:tasks.workunit.client.0.smithi103.stdout:6/749: write d0/d3/d22/fd3 [874736,94912] 0 2024-03-22T02:28:53.347 INFO:tasks.workunit.client.0.smithi103.stdout:6/750: write d0/d3/d22/d58/d6f/f94 [302945,53625] 0 2024-03-22T02:28:53.350 INFO:tasks.workunit.client.0.smithi103.stdout:4/953: rename d3/d1e/f28 to d3/d1e/d25/d65/d83/da1/f141 0 2024-03-22T02:28:53.353 INFO:tasks.workunit.client.1.smithi122.stdout:1/445: rename dc/d2b/c60 to dc/d3b/d46/d4d/d90/c95 0 2024-03-22T02:28:53.355 INFO:tasks.workunit.client.0.smithi103.stdout:4/954: creat d3/d1e/d68/dbc/f142 x:0 0 0 2024-03-22T02:28:53.355 INFO:tasks.workunit.client.1.smithi122.stdout:4/388: creat d0/d12/d15/d2d/f7d x:0 0 0 2024-03-22T02:28:53.355 INFO:tasks.workunit.client.1.smithi122.stdout:5/370: dread d8/d1e/d22/d3a/d46/d4f/f54 [0,4194304] 0 2024-03-22T02:28:53.355 INFO:tasks.workunit.client.1.smithi122.stdout:8/454: write d1/d8/d2f/f98 [183006,126401] 0 2024-03-22T02:28:53.355 INFO:tasks.workunit.client.1.smithi122.stdout:9/509: dwrite db/d23/f11 [0,4194304] 0 2024-03-22T02:28:53.356 INFO:tasks.workunit.client.0.smithi103.stdout:6/751: dwrite d0/d3/d22/fab [4194304,4194304] 0 2024-03-22T02:28:53.357 INFO:tasks.workunit.client.1.smithi122.stdout:2/457: write f6 [1265963,72272] 0 2024-03-22T02:28:53.359 INFO:tasks.workunit.client.1.smithi122.stdout:2/458: readlink df/d1f/d33/l75 0 2024-03-22T02:28:53.361 INFO:tasks.workunit.client.1.smithi122.stdout:2/459: dread - df/d1f/d3a/d4f/f5e zero size 2024-03-22T02:28:53.362 INFO:tasks.workunit.client.0.smithi103.stdout:4/955: unlink d3/d31/l5b 0 2024-03-22T02:28:53.362 INFO:tasks.workunit.client.0.smithi103.stdout:6/752: write d0/d3/d33/fdd [586889,2217] 0 2024-03-22T02:28:53.362 INFO:tasks.workunit.client.1.smithi122.stdout:7/389: sync 2024-03-22T02:28:53.362 INFO:tasks.workunit.client.1.smithi122.stdout:2/460: readlink df/d1f/d2a/d39/l5b 0 2024-03-22T02:28:53.365 INFO:tasks.workunit.client.1.smithi122.stdout:5/371: read d8/f30 [603492,85666] 0 2024-03-22T02:28:53.365 INFO:tasks.workunit.client.1.smithi122.stdout:4/389: mknod d0/d12/d15/d3d/c7e 0 2024-03-22T02:28:53.365 INFO:tasks.workunit.client.0.smithi103.stdout:6/753: symlink d0/d14/dd2/lf9 0 2024-03-22T02:28:53.366 INFO:tasks.workunit.client.0.smithi103.stdout:6/754: write d0/d3/d22/d58/f89 [3139772,60773] 0 2024-03-22T02:28:53.369 INFO:tasks.workunit.client.1.smithi122.stdout:9/510: creat db/d23/d1f/d57/d66/d73/fa4 x:0 0 0 2024-03-22T02:28:53.370 INFO:tasks.workunit.client.1.smithi122.stdout:5/372: dread - d8/d3b/f52 zero size 2024-03-22T02:28:53.370 INFO:tasks.workunit.client.1.smithi122.stdout:0/376: getdents d11/d14/d1d 0 2024-03-22T02:28:53.372 INFO:tasks.workunit.client.1.smithi122.stdout:8/455: creat d1/d40/d71/d97/f9f x:0 0 0 2024-03-22T02:28:53.372 INFO:tasks.workunit.client.1.smithi122.stdout:4/390: truncate d0/d25/f28 4640427 0 2024-03-22T02:28:53.373 INFO:tasks.workunit.client.1.smithi122.stdout:2/461: write df/d1f/d33/d6a/f8d [884276,114185] 0 2024-03-22T02:28:53.373 INFO:tasks.workunit.client.0.smithi103.stdout:6/755: rmdir d0/d3/d22/d58/d6f/d75/db0/df6 0 2024-03-22T02:28:53.374 INFO:tasks.workunit.client.0.smithi103.stdout:6/756: fsync d0/d3/f1b 0 2024-03-22T02:28:53.374 INFO:tasks.workunit.client.1.smithi122.stdout:9/511: mknod db/d23/d9a/ca5 0 2024-03-22T02:28:53.376 INFO:tasks.workunit.client.1.smithi122.stdout:8/456: dread - d1/d8/d8a/d78/f82 zero size 2024-03-22T02:28:53.377 INFO:tasks.workunit.client.1.smithi122.stdout:7/390: dwrite d0/d4/d20/f27 [4194304,4194304] 0 2024-03-22T02:28:53.378 INFO:tasks.workunit.client.1.smithi122.stdout:5/373: rename d8/l29 to d8/d74/l81 0 2024-03-22T02:28:53.380 INFO:tasks.workunit.client.1.smithi122.stdout:9/512: chown db/d1b/d28/f6b 1615234442 1 2024-03-22T02:28:53.380 INFO:tasks.workunit.client.1.smithi122.stdout:4/391: read d0/d12/d15/d42/d4e/fb [522556,85891] 0 2024-03-22T02:28:53.380 INFO:tasks.workunit.client.0.smithi103.stdout:6/757: dread d0/d14/d57/d68/f28 [0,4194304] 0 2024-03-22T02:28:53.381 INFO:tasks.workunit.client.1.smithi122.stdout:8/457: dread - d1/d60/d73/f94 zero size 2024-03-22T02:28:53.381 INFO:tasks.workunit.client.1.smithi122.stdout:7/391: rmdir d0/d4 39 2024-03-22T02:28:53.383 INFO:tasks.workunit.client.1.smithi122.stdout:5/374: truncate d8/d1a/f34 4280269 0 2024-03-22T02:28:53.383 INFO:tasks.workunit.client.1.smithi122.stdout:8/458: readlink d1/d60/d73/l7c 0 2024-03-22T02:28:53.385 INFO:tasks.workunit.client.0.smithi103.stdout:6/758: mkdir d0/d14/dc7/dfa 0 2024-03-22T02:28:53.386 INFO:tasks.workunit.client.1.smithi122.stdout:3/344: truncate d2/d9/dc/f2f 6157238 0 2024-03-22T02:28:53.386 INFO:tasks.workunit.client.1.smithi122.stdout:4/392: stat d0/d12/d15/d42/d4e/f34 0 2024-03-22T02:28:53.387 INFO:tasks.workunit.client.1.smithi122.stdout:8/459: chown d1/d8/d5e/f72 1010272 1 2024-03-22T02:28:53.389 INFO:tasks.workunit.client.1.smithi122.stdout:3/345: truncate d2/f55 5090083 0 2024-03-22T02:28:53.390 INFO:tasks.workunit.client.1.smithi122.stdout:9/513: link f7 db/d1b/d6c/d58/d9d/fa6 0 2024-03-22T02:28:53.393 INFO:tasks.workunit.client.0.smithi103.stdout:6/759: truncate d0/d3/d22/f96 1575663 0 2024-03-22T02:28:53.394 INFO:tasks.workunit.client.1.smithi122.stdout:6/516: dread d7/de/dd/d23/f47 [0,4194304] 0 2024-03-22T02:28:53.394 INFO:tasks.workunit.client.1.smithi122.stdout:2/462: getdents df/d1f/d3a/d4f 0 2024-03-22T02:28:53.394 INFO:tasks.workunit.client.1.smithi122.stdout:5/375: rename d8/da/d28/d42/l6d to d8/d1e/d22/d3a/d46/d4f/d55/l82 0 2024-03-22T02:28:53.394 INFO:tasks.workunit.client.0.smithi103.stdout:6/760: creat d0/d88/d8e/ffb x:0 0 0 2024-03-22T02:28:53.395 INFO:tasks.workunit.client.1.smithi122.stdout:9/514: fsync db/df/f15 0 2024-03-22T02:28:53.395 INFO:tasks.workunit.client.1.smithi122.stdout:2/463: chown df 1918 1 2024-03-22T02:28:53.396 INFO:tasks.workunit.client.1.smithi122.stdout:9/515: chown db/df/d3e/d81/d25/d41/c48 0 1 2024-03-22T02:28:53.398 INFO:tasks.workunit.client.0.smithi103.stdout:4/956: dread d3/f77 [0,4194304] 0 2024-03-22T02:28:53.398 INFO:tasks.workunit.client.1.smithi122.stdout:1/446: dwrite dc/d2b/f36 [0,4194304] 0 2024-03-22T02:28:53.399 INFO:tasks.workunit.client.1.smithi122.stdout:4/393: mknod d0/d12/d15/d42/c7f 0 2024-03-22T02:28:53.400 INFO:tasks.workunit.client.0.smithi103.stdout:4/957: unlink d3/d38/d5c/faa 0 2024-03-22T02:28:53.402 INFO:tasks.workunit.client.1.smithi122.stdout:8/460: rename d1/d9/l34 to d1/d60/d73/la0 0 2024-03-22T02:28:53.403 INFO:tasks.workunit.client.1.smithi122.stdout:7/392: getdents d0/d25/d46 0 2024-03-22T02:28:53.408 INFO:tasks.workunit.client.1.smithi122.stdout:0/377: dread d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:53.410 INFO:tasks.workunit.client.0.smithi103.stdout:6/761: dwrite d0/d14/f6c [0,4194304] 0 2024-03-22T02:28:53.412 INFO:tasks.workunit.client.0.smithi103.stdout:4/958: dwrite d3/d1e/d53/f56 [0,4194304] 0 2024-03-22T02:28:53.412 INFO:tasks.workunit.client.1.smithi122.stdout:1/447: truncate dc/d3b/d46/d4d/d67/f7b 930596 0 2024-03-22T02:28:53.413 INFO:tasks.workunit.client.0.smithi103.stdout:6/762: write d0/d3/d22/d58/d6f/f8b [1225565,43121] 0 2024-03-22T02:28:53.415 INFO:tasks.workunit.client.1.smithi122.stdout:3/346: dwrite d2/f62 [0,4194304] 0 2024-03-22T02:28:53.417 INFO:tasks.workunit.client.0.smithi103.stdout:6/763: write d0/d3/d22/d58/de0/ff0 [966096,35862] 0 2024-03-22T02:28:53.417 INFO:tasks.workunit.client.1.smithi122.stdout:5/376: dread d8/da/f16 [0,4194304] 0 2024-03-22T02:28:53.417 INFO:tasks.workunit.client.1.smithi122.stdout:4/394: symlink d0/d12/d15/d2d/l80 0 2024-03-22T02:28:53.420 INFO:tasks.workunit.client.0.smithi103.stdout:4/959: mkdir d3/d31/d10f/d125/d143 0 2024-03-22T02:28:53.420 INFO:tasks.workunit.client.1.smithi122.stdout:2/464: creat df/d1f/f9c x:0 0 0 2024-03-22T02:28:53.421 INFO:tasks.workunit.client.0.smithi103.stdout:6/764: creat d0/d3/d22/d58/de0/ffc x:0 0 0 2024-03-22T02:28:53.422 INFO:tasks.workunit.client.1.smithi122.stdout:6/517: dwrite d7/de/dd/d20/d2d/d54/d76/f6b [0,4194304] 0 2024-03-22T02:28:53.425 INFO:tasks.workunit.client.0.smithi103.stdout:6/765: mknod d0/d14/d57/d68/cfd 0 2024-03-22T02:28:53.426 INFO:tasks.workunit.client.1.smithi122.stdout:4/395: stat d0/d12/d15/d42/d4e/de 0 2024-03-22T02:28:53.426 INFO:tasks.workunit.client.1.smithi122.stdout:2/465: dread - df/d1f/d23/f86 zero size 2024-03-22T02:28:53.427 INFO:tasks.workunit.client.0.smithi103.stdout:4/960: dwrite d3/d1e/f6d [0,4194304] 0 2024-03-22T02:28:53.434 INFO:tasks.workunit.client.1.smithi122.stdout:5/377: mkdir d8/d1a/d83 0 2024-03-22T02:28:53.434 INFO:tasks.workunit.client.0.smithi103.stdout:6/766: read d0/f3e [715081,81624] 0 2024-03-22T02:28:53.434 INFO:tasks.workunit.client.0.smithi103.stdout:6/767: chown d0/d3/d22/d58/d6f/d75/db0 9327127 1 2024-03-22T02:28:53.436 INFO:tasks.workunit.client.1.smithi122.stdout:7/393: symlink d0/d4/l7c 0 2024-03-22T02:28:53.438 INFO:tasks.workunit.client.1.smithi122.stdout:0/378: dwrite d11/d4f/f82 [0,4194304] 0 2024-03-22T02:28:53.439 INFO:tasks.workunit.client.1.smithi122.stdout:8/461: getdents d1 0 2024-03-22T02:28:53.439 INFO:tasks.workunit.client.1.smithi122.stdout:9/516: dwrite db/d23/f3a [0,4194304] 0 2024-03-22T02:28:53.439 INFO:tasks.workunit.client.1.smithi122.stdout:4/396: rename d0/d12/c2a to d0/d12/d15/d42/d4e/de/d40/c81 0 2024-03-22T02:28:53.440 INFO:tasks.workunit.client.1.smithi122.stdout:7/394: fsync d0/d25/f56 0 2024-03-22T02:28:53.441 INFO:tasks.workunit.client.1.smithi122.stdout:2/466: mknod df/d1f/d33/d34/c9d 0 2024-03-22T02:28:53.443 INFO:tasks.workunit.client.1.smithi122.stdout:1/448: getdents dc/d3b/d8d 0 2024-03-22T02:28:53.451 INFO:tasks.workunit.client.1.smithi122.stdout:0/379: mkdir d11/d16/d37/d84 0 2024-03-22T02:28:53.452 INFO:tasks.workunit.client.1.smithi122.stdout:9/517: unlink db/df/d3e/d81/d29/f2c 0 2024-03-22T02:28:53.454 INFO:tasks.workunit.client.1.smithi122.stdout:4/397: creat d0/d12/f82 x:0 0 0 2024-03-22T02:28:53.455 INFO:tasks.workunit.client.1.smithi122.stdout:8/462: getdents d1/d40/d71/d97 0 2024-03-22T02:28:53.456 INFO:tasks.workunit.client.1.smithi122.stdout:7/395: getdents d0/d4/d20 0 2024-03-22T02:28:53.457 INFO:tasks.workunit.client.1.smithi122.stdout:6/518: dwrite d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:28:53.459 INFO:tasks.workunit.client.1.smithi122.stdout:1/449: dwrite dc/f88 [0,4194304] 0 2024-03-22T02:28:53.461 INFO:tasks.workunit.client.1.smithi122.stdout:8/463: rmdir d1/d40/d51 39 2024-03-22T02:28:53.462 INFO:tasks.workunit.client.1.smithi122.stdout:9/518: write db/df/d3e/d81/d25/d41/f61 [151291,57556] 0 2024-03-22T02:28:53.463 INFO:tasks.workunit.client.1.smithi122.stdout:1/450: chown dc/d4c/f92 404612 1 2024-03-22T02:28:53.465 INFO:tasks.workunit.client.1.smithi122.stdout:6/519: mknod d7/c9f 0 2024-03-22T02:28:53.466 INFO:tasks.workunit.client.1.smithi122.stdout:7/396: mkdir d0/d4/d22/d5f/d7d 0 2024-03-22T02:28:53.467 INFO:tasks.workunit.client.1.smithi122.stdout:4/398: rename d0/d12/d15/f6d to d0/d12/d73/f83 0 2024-03-22T02:28:53.469 INFO:tasks.workunit.client.1.smithi122.stdout:9/519: fdatasync db/df/d3e/d81/d25/d41/f54 0 2024-03-22T02:28:53.471 INFO:tasks.workunit.client.0.smithi103.stdout:4/961: truncate d3/d38/fdf 2141678 0 2024-03-22T02:28:53.474 INFO:tasks.workunit.client.1.smithi122.stdout:7/397: mknod d0/d4/d20/d21/d28/c7e 0 2024-03-22T02:28:53.477 INFO:tasks.workunit.client.0.smithi103.stdout:6/768: dwrite d0/d14/f44 [0,4194304] 0 2024-03-22T02:28:53.479 INFO:tasks.workunit.client.1.smithi122.stdout:6/520: creat d7/de/dd/d20/d2d/d54/d76/d86/fa0 x:0 0 0 2024-03-22T02:28:53.480 INFO:tasks.workunit.client.1.smithi122.stdout:1/451: dwrite dc/d4c/f63 [0,4194304] 0 2024-03-22T02:28:53.482 INFO:tasks.workunit.client.0.smithi103.stdout:6/769: creat d0/d3/d22/d58/ffe x:0 0 0 2024-03-22T02:28:53.483 INFO:tasks.workunit.client.1.smithi122.stdout:9/520: unlink db/df/d3e/d81/d29/l50 0 2024-03-22T02:28:53.483 INFO:tasks.workunit.client.0.smithi103.stdout:6/770: fdatasync d0/d14/dd2/fe1 0 2024-03-22T02:28:53.484 INFO:tasks.workunit.client.1.smithi122.stdout:4/399: getdents d0/d7a 0 2024-03-22T02:28:53.487 INFO:tasks.workunit.client.0.smithi103.stdout:6/771: chown d0/d3/d22/c85 1233079 1 2024-03-22T02:28:53.489 INFO:tasks.workunit.client.0.smithi103.stdout:6/772: chown d0/d14/d9a/fdb 468705 1 2024-03-22T02:28:53.491 INFO:tasks.workunit.client.0.smithi103.stdout:4/962: dwrite d3/f24 [0,4194304] 0 2024-03-22T02:28:53.493 INFO:tasks.workunit.client.0.smithi103.stdout:4/963: truncate d3/d1e/d53/d7a/f11b 1386359 0 2024-03-22T02:28:53.496 INFO:tasks.workunit.client.0.smithi103.stdout:0/958: sync 2024-03-22T02:28:53.497 INFO:tasks.workunit.client.0.smithi103.stdout:7/958: sync 2024-03-22T02:28:53.497 INFO:tasks.workunit.client.0.smithi103.stdout:9/933: sync 2024-03-22T02:28:53.506 INFO:tasks.workunit.client.0.smithi103.stdout:7/959: fdatasync d1/da/d2f/fb1 0 2024-03-22T02:28:53.506 INFO:tasks.workunit.client.0.smithi103.stdout:6/773: getdents d0/d3/d22/d58/db4/deb 0 2024-03-22T02:28:53.507 INFO:tasks.workunit.client.1.smithi122.stdout:8/464: link d1/d8/d8a/l87 d1/d40/d71/d97/la1 0 2024-03-22T02:28:53.507 INFO:tasks.workunit.client.1.smithi122.stdout:1/452: unlink dc/d3b/d46/d4d/l89 0 2024-03-22T02:28:53.507 INFO:tasks.workunit.client.0.smithi103.stdout:6/774: chown d0/d14/d9a/lc8 3 1 2024-03-22T02:28:53.508 INFO:tasks.workunit.client.1.smithi122.stdout:6/521: unlink d7/de/dd/d20/d2d/d54/d76/c8b 0 2024-03-22T02:28:53.508 INFO:tasks.workunit.client.0.smithi103.stdout:0/959: mkdir dd/d16/d4b/d96/d64/d6a/dd5/d128 0 2024-03-22T02:28:53.509 INFO:tasks.workunit.client.0.smithi103.stdout:7/960: fsync d1/d10/d11/d70/d8a/d9d/dff/f11b 0 2024-03-22T02:28:53.510 INFO:tasks.workunit.client.1.smithi122.stdout:4/400: dwrite d0/d12/d15/d42/d4e/de/f66 [0,4194304] 0 2024-03-22T02:28:53.511 INFO:tasks.workunit.client.0.smithi103.stdout:0/960: write dd/d16/d4b/d96/d29/d48/d5c/fa8 [2145879,58294] 0 2024-03-22T02:28:53.512 INFO:tasks.workunit.client.1.smithi122.stdout:6/522: chown d7/de/dd/d20/d2d/f30 2254 1 2024-03-22T02:28:53.512 INFO:tasks.workunit.client.1.smithi122.stdout:1/453: readlink dc/d2b/l8e 0 2024-03-22T02:28:53.513 INFO:tasks.workunit.client.0.smithi103.stdout:7/961: chown d1/d10/d11/d19/d86/d92/dbf/d128/d69/l2a 369026623 1 2024-03-22T02:28:53.513 INFO:tasks.workunit.client.0.smithi103.stdout:6/775: creat d0/d14/d57/d68/fff x:0 0 0 2024-03-22T02:28:53.513 INFO:tasks.workunit.client.1.smithi122.stdout:6/523: chown d7/de/l3d 21993 1 2024-03-22T02:28:53.514 INFO:tasks.workunit.client.1.smithi122.stdout:7/398: link d0/d4/d20/d21/l3e d0/d25/d5c/l7f 0 2024-03-22T02:28:53.515 INFO:tasks.workunit.client.0.smithi103.stdout:7/962: write d1/d10/d11/d19/d86/d92/dbf/ff5 [317956,103525] 0 2024-03-22T02:28:53.515 INFO:tasks.workunit.client.1.smithi122.stdout:6/524: chown d7/fa 224 1 2024-03-22T02:28:53.515 INFO:tasks.workunit.client.1.smithi122.stdout:9/521: rename db/d1b/d28/c40 to db/df/d3e/d59/ca7 0 2024-03-22T02:28:53.518 INFO:tasks.workunit.client.1.smithi122.stdout:6/525: dread - d7/de/dd/d23/f93 zero size 2024-03-22T02:28:53.518 INFO:tasks.workunit.client.1.smithi122.stdout:1/454: creat dc/d3b/d8d/d72/f96 x:0 0 0 2024-03-22T02:28:53.518 INFO:tasks.workunit.client.0.smithi103.stdout:7/963: dread - d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/ffe zero size 2024-03-22T02:28:53.519 INFO:tasks.workunit.client.1.smithi122.stdout:4/401: chown d0/d25/l36 1205655 1 2024-03-22T02:28:53.520 INFO:tasks.workunit.client.1.smithi122.stdout:7/399: mkdir d0/d4/d80 0 2024-03-22T02:28:53.520 INFO:tasks.workunit.client.0.smithi103.stdout:0/961: getdents dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/df4 0 2024-03-22T02:28:53.521 INFO:tasks.workunit.client.1.smithi122.stdout:4/402: stat d0/d12/d15/c35 0 2024-03-22T02:28:53.522 INFO:tasks.workunit.client.1.smithi122.stdout:1/455: readlink dc/l1f 0 2024-03-22T02:28:53.522 INFO:tasks.workunit.client.0.smithi103.stdout:7/964: creat d1/d10b/d10e/f134 x:0 0 0 2024-03-22T02:28:53.523 INFO:tasks.workunit.client.0.smithi103.stdout:0/962: rmdir dd/d16/d4b/d96/d64/dab/dc6/d89 39 2024-03-22T02:28:53.525 INFO:tasks.workunit.client.1.smithi122.stdout:6/526: symlink d7/de/dd/d20/d2d/la1 0 2024-03-22T02:28:53.525 INFO:tasks.workunit.client.1.smithi122.stdout:4/403: write d0/d25/f3a [606741,3637] 0 2024-03-22T02:28:53.525 INFO:tasks.workunit.client.1.smithi122.stdout:1/456: mknod dc/d3b/d46/c97 0 2024-03-22T02:28:53.526 INFO:tasks.workunit.client.0.smithi103.stdout:7/965: rmdir d1/d10/d11/d19/d29/d33/d4b/d5e 39 2024-03-22T02:28:53.527 INFO:tasks.workunit.client.0.smithi103.stdout:6/776: dread d0/f2c [0,4194304] 0 2024-03-22T02:28:53.527 INFO:tasks.workunit.client.1.smithi122.stdout:4/404: dread - d0/d12/f79 zero size 2024-03-22T02:28:53.528 INFO:tasks.workunit.client.0.smithi103.stdout:4/964: write d3/d1e/d53/d7a/da3/ff5 [652173,112833] 0 2024-03-22T02:28:53.528 INFO:tasks.workunit.client.0.smithi103.stdout:7/966: stat d1/f7e 0 2024-03-22T02:28:53.529 INFO:tasks.workunit.client.1.smithi122.stdout:1/457: unlink dc/f28 0 2024-03-22T02:28:53.530 INFO:tasks.workunit.client.0.smithi103.stdout:9/934: write d2/d5/d7b/f84 [2915676,57156] 0 2024-03-22T02:28:53.531 INFO:tasks.workunit.client.1.smithi122.stdout:9/522: dwrite db/f13 [0,4194304] 0 2024-03-22T02:28:53.533 INFO:tasks.workunit.client.0.smithi103.stdout:0/963: creat dd/d16/d4b/d96/f129 x:0 0 0 2024-03-22T02:28:53.533 INFO:tasks.workunit.client.0.smithi103.stdout:7/967: truncate d1/d10/d11/d19/dc3/fd8 320842 0 2024-03-22T02:28:53.534 INFO:tasks.workunit.client.1.smithi122.stdout:0/380: dread d11/d14/d1d/d27/d2f/f7b [0,4194304] 0 2024-03-22T02:28:53.535 INFO:tasks.workunit.client.0.smithi103.stdout:6/777: write d0/d14/d57/d68/f6b [4456699,105165] 0 2024-03-22T02:28:53.535 INFO:tasks.workunit.client.0.smithi103.stdout:0/964: readlink dd/d16/d4b/d96/d29/d48/d94/l9e 0 2024-03-22T02:28:53.537 INFO:tasks.workunit.client.0.smithi103.stdout:7/968: write d1/da/d2f/fa0 [662541,122752] 0 2024-03-22T02:28:53.539 INFO:tasks.workunit.client.1.smithi122.stdout:7/400: link d0/d4/d22/f32 d0/d4/d22/d5f/f81 0 2024-03-22T02:28:53.542 INFO:tasks.workunit.client.0.smithi103.stdout:6/778: unlink d0/d14/fc6 0 2024-03-22T02:28:53.543 INFO:tasks.workunit.client.0.smithi103.stdout:9/935: link d2/d10/l16 d2/d5/d7b/d7d/dd4/dd8/l10f 0 2024-03-22T02:28:53.543 INFO:tasks.workunit.client.1.smithi122.stdout:7/401: read d0/d4/d22/f64 [2871692,113689] 0 2024-03-22T02:28:53.545 INFO:tasks.workunit.client.0.smithi103.stdout:9/936: truncate d2/d10/d43/f3e 1467467 0 2024-03-22T02:28:53.545 INFO:tasks.workunit.client.1.smithi122.stdout:4/405: dwrite d0/d12/d15/d42/d4e/f4a [0,4194304] 0 2024-03-22T02:28:53.545 INFO:tasks.workunit.client.1.smithi122.stdout:6/527: rename f3 to d7/de/dd/d39/fa2 0 2024-03-22T02:28:53.545 INFO:tasks.workunit.client.0.smithi103.stdout:0/965: creat dd/d16/d4b/d96/d29/d48/f12a x:0 0 0 2024-03-22T02:28:53.548 INFO:tasks.workunit.client.0.smithi103.stdout:9/937: readlink d2/d10/d43/d7e/d96/lf1 0 2024-03-22T02:28:53.549 INFO:tasks.workunit.client.0.smithi103.stdout:6/779: symlink d0/d3/d33/l100 0 2024-03-22T02:28:53.550 INFO:tasks.workunit.client.0.smithi103.stdout:0/966: chown dd/d16/d4b/d96/d29/f85 1955385 1 2024-03-22T02:28:53.550 INFO:tasks.workunit.client.1.smithi122.stdout:9/523: dwrite db/d23/f83 [0,4194304] 0 2024-03-22T02:28:53.550 INFO:tasks.workunit.client.1.smithi122.stdout:0/381: mkdir d11/d14/d23/d24/d33/d67/d85 0 2024-03-22T02:28:53.551 INFO:tasks.workunit.client.0.smithi103.stdout:0/967: fdatasync dd/d16/f11d 0 2024-03-22T02:28:53.552 INFO:tasks.workunit.client.1.smithi122.stdout:7/402: symlink d0/d25/d5c/l82 0 2024-03-22T02:28:53.553 INFO:tasks.workunit.client.1.smithi122.stdout:8/465: dread d1/d40/f47 [0,4194304] 0 2024-03-22T02:28:53.554 INFO:tasks.workunit.client.1.smithi122.stdout:8/466: read - d1/d4d/f58 zero size 2024-03-22T02:28:53.555 INFO:tasks.workunit.client.0.smithi103.stdout:4/965: dwrite d3/d30/d45/f136 [0,4194304] 0 2024-03-22T02:28:53.557 INFO:tasks.workunit.client.1.smithi122.stdout:0/382: write d11/d14/d1d/d27/d32/f5f [651537,102181] 0 2024-03-22T02:28:53.559 INFO:tasks.workunit.client.1.smithi122.stdout:0/383: truncate d11/d14/d1b/f2c 737725 0 2024-03-22T02:28:53.561 INFO:tasks.workunit.client.0.smithi103.stdout:9/938: dwrite d2/d10/d43/d7e/fc6 [0,4194304] 0 2024-03-22T02:28:53.564 INFO:tasks.workunit.client.1.smithi122.stdout:4/406: link d0/d12/f69 d0/d25/f84 0 2024-03-22T02:28:53.566 INFO:tasks.workunit.client.0.smithi103.stdout:0/968: creat dd/d16/d4b/f12b x:0 0 0 2024-03-22T02:28:53.567 INFO:tasks.workunit.client.1.smithi122.stdout:8/467: creat d1/d22/d6a/fa2 x:0 0 0 2024-03-22T02:28:53.568 INFO:tasks.workunit.client.1.smithi122.stdout:1/458: getdents dc/d3b/d46 0 2024-03-22T02:28:53.569 INFO:tasks.workunit.client.1.smithi122.stdout:7/403: link d0/d25/l5a d0/d4/d22/d5f/l83 0 2024-03-22T02:28:53.570 INFO:tasks.workunit.client.1.smithi122.stdout:7/404: write d0/f1b [840281,16196] 0 2024-03-22T02:28:53.571 INFO:tasks.workunit.client.0.smithi103.stdout:0/969: rmdir dd/d16/d4b/d96/d64/dab 39 2024-03-22T02:28:53.572 INFO:tasks.workunit.client.1.smithi122.stdout:7/405: write d0/f1b [1631658,53648] 0 2024-03-22T02:28:53.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/468: dread d1/d40/f47 [0,4194304] 0 2024-03-22T02:28:53.573 INFO:tasks.workunit.client.1.smithi122.stdout:6/528: rename d7/de/dd/d20/d2d/d54/f72 to d7/de/dd/d39/fa3 0 2024-03-22T02:28:53.575 INFO:tasks.workunit.client.1.smithi122.stdout:4/407: creat d0/d12/d15/d42/d4e/de/f85 x:0 0 0 2024-03-22T02:28:53.575 INFO:tasks.workunit.client.0.smithi103.stdout:0/970: dread - dd/d16/d4b/d96/d64/dab/dc6/d89/fcc zero size 2024-03-22T02:28:53.576 INFO:tasks.workunit.client.1.smithi122.stdout:0/384: stat d11/d47/l49 0 2024-03-22T02:28:53.577 INFO:tasks.workunit.client.0.smithi103.stdout:0/971: write dd/d16/d4b/d96/d64/f71 [1741793,1736] 0 2024-03-22T02:28:53.577 INFO:tasks.workunit.client.1.smithi122.stdout:0/385: truncate d11/d14/d23/f6d 416925 0 2024-03-22T02:28:53.581 INFO:tasks.workunit.client.1.smithi122.stdout:8/469: mkdir d1/d60/d73/da3 0 2024-03-22T02:28:53.582 INFO:tasks.workunit.client.1.smithi122.stdout:6/529: truncate d7/fa 85885 0 2024-03-22T02:28:53.582 INFO:tasks.workunit.client.0.smithi103.stdout:0/972: dread dd/d16/d4b/d96/d29/d48/d5c/fa8 [0,4194304] 0 2024-03-22T02:28:53.583 INFO:tasks.workunit.client.0.smithi103.stdout:0/973: readlink dd/d16/d4b/d96/l87 0 2024-03-22T02:28:53.584 INFO:tasks.workunit.client.1.smithi122.stdout:1/459: creat dc/d3b/d8d/d94/f98 x:0 0 0 2024-03-22T02:28:53.584 INFO:tasks.workunit.client.1.smithi122.stdout:6/530: fdatasync d7/de/d27/f64 0 2024-03-22T02:28:53.585 INFO:tasks.workunit.client.1.smithi122.stdout:9/524: fdatasync db/d23/f83 0 2024-03-22T02:28:53.586 INFO:tasks.workunit.client.1.smithi122.stdout:0/386: creat d11/d14/d23/d24/f86 x:0 0 0 2024-03-22T02:28:53.586 INFO:tasks.workunit.client.0.smithi103.stdout:0/974: rename dd/d16/d4b/d96/f39 to dd/d16/d10a/f12c 0 2024-03-22T02:28:53.587 INFO:tasks.workunit.client.1.smithi122.stdout:8/470: dread d1/d3f/f41 [0,4194304] 0 2024-03-22T02:28:53.587 INFO:tasks.workunit.client.0.smithi103.stdout:0/975: chown dd/d16/d4b/d96/d29/d48/d94/f116 9519314 1 2024-03-22T02:28:53.588 INFO:tasks.workunit.client.0.smithi103.stdout:9/939: write d2/da/f24 [1467180,53257] 0 2024-03-22T02:28:53.589 INFO:tasks.workunit.client.0.smithi103.stdout:9/940: chown d2/f3 91882 1 2024-03-22T02:28:53.591 INFO:tasks.workunit.client.0.smithi103.stdout:0/976: mknod dd/d16/d4b/d96/d29/d48/d5c/d9b/ddd/c12d 0 2024-03-22T02:28:53.592 INFO:tasks.workunit.client.1.smithi122.stdout:6/531: mknod d7/de/dd/d20/d2d/d54/d76/d86/d96/ca4 0 2024-03-22T02:28:53.592 INFO:tasks.workunit.client.1.smithi122.stdout:8/471: write d1/d22/f8b [595363,14783] 0 2024-03-22T02:28:53.593 INFO:tasks.workunit.client.0.smithi103.stdout:4/966: dwrite d3/f9b [0,4194304] 0 2024-03-22T02:28:53.593 INFO:tasks.workunit.client.0.smithi103.stdout:9/941: creat d2/da/d18/d1e/d34/d3d/f110 x:0 0 0 2024-03-22T02:28:53.595 INFO:tasks.workunit.client.0.smithi103.stdout:4/967: write d3/d38/d5c/d90/df8/f10c [158965,91542] 0 2024-03-22T02:28:53.596 INFO:tasks.workunit.client.1.smithi122.stdout:4/408: dwrite d0/d25/f1d [0,4194304] 0 2024-03-22T02:28:53.596 INFO:tasks.workunit.client.0.smithi103.stdout:4/968: truncate d3/f115 1022694 0 2024-03-22T02:28:53.598 INFO:tasks.workunit.client.0.smithi103.stdout:0/977: creat dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/f12e x:0 0 0 2024-03-22T02:28:53.598 INFO:tasks.workunit.client.1.smithi122.stdout:9/525: dread db/d23/d1e/f98 [0,4194304] 0 2024-03-22T02:28:53.600 INFO:tasks.workunit.client.1.smithi122.stdout:8/472: mkdir d1/d60/da4 0 2024-03-22T02:28:53.602 INFO:tasks.workunit.client.1.smithi122.stdout:9/526: unlink f7 0 2024-03-22T02:28:53.603 INFO:tasks.workunit.client.0.smithi103.stdout:9/942: unlink d2/d10/d43/d9b/df3/d65/fb2 0 2024-03-22T02:28:53.605 INFO:tasks.workunit.client.0.smithi103.stdout:9/943: fsync d2/d5/d4c/f85 0 2024-03-22T02:28:53.605 INFO:tasks.workunit.client.0.smithi103.stdout:0/978: mknod dd/d16/c12f 0 2024-03-22T02:28:53.605 INFO:tasks.workunit.client.0.smithi103.stdout:0/979: stat dd 0 2024-03-22T02:28:53.608 INFO:tasks.workunit.client.0.smithi103.stdout:0/980: truncate dd/d16/d4b/d96/d29/d48/f12a 262480 0 2024-03-22T02:28:53.609 INFO:tasks.workunit.client.0.smithi103.stdout:4/969: getdents d3/d1e/d25/d65/d83/da1/dbd 0 2024-03-22T02:28:53.609 INFO:tasks.workunit.client.1.smithi122.stdout:1/460: dread dc/f2a [0,4194304] 0 2024-03-22T02:28:53.613 INFO:tasks.workunit.client.1.smithi122.stdout:8/473: dwrite d1/d4d/f58 [0,4194304] 0 2024-03-22T02:28:53.613 INFO:tasks.workunit.client.0.smithi103.stdout:9/944: getdents d2 0 2024-03-22T02:28:53.613 INFO:tasks.workunit.client.0.smithi103.stdout:0/981: dread dd/d16/d4b/d96/d29/d48/d5c/fa8 [0,4194304] 0 2024-03-22T02:28:53.614 INFO:tasks.workunit.client.1.smithi122.stdout:6/532: dwrite d7/de/dd/d20/d2d/f9a [0,4194304] 0 2024-03-22T02:28:53.615 INFO:tasks.workunit.client.0.smithi103.stdout:9/945: stat d2/d10/d43/d9b/df3/d65/caf 0 2024-03-22T02:28:53.617 INFO:tasks.workunit.client.0.smithi103.stdout:9/946: fsync d2/f9 0 2024-03-22T02:28:53.623 INFO:tasks.workunit.client.1.smithi122.stdout:0/387: dread d11/d16/d44/d4c/f4b [0,4194304] 0 2024-03-22T02:28:53.623 INFO:tasks.workunit.client.1.smithi122.stdout:8/474: link d1/d22/f9d d1/d81/fa5 0 2024-03-22T02:28:53.624 INFO:tasks.workunit.client.1.smithi122.stdout:1/461: dwrite dc/d55/f75 [0,4194304] 0 2024-03-22T02:28:53.626 INFO:tasks.workunit.client.0.smithi103.stdout:9/947: dwrite d2/f9 [8388608,4194304] 0 2024-03-22T02:28:53.627 INFO:tasks.workunit.client.1.smithi122.stdout:8/475: chown d1/d8/d2f/f91 90893 1 2024-03-22T02:28:53.629 INFO:tasks.workunit.client.1.smithi122.stdout:1/462: mkdir dc/d55/d99 0 2024-03-22T02:28:53.630 INFO:tasks.workunit.client.1.smithi122.stdout:8/476: stat d1/d40/d48/c83 0 2024-03-22T02:28:53.630 INFO:tasks.workunit.client.1.smithi122.stdout:6/533: dwrite d7/de/f68 [0,4194304] 0 2024-03-22T02:28:53.631 INFO:tasks.workunit.client.1.smithi122.stdout:8/477: fdatasync d1/d40/f9e 0 2024-03-22T02:28:53.636 INFO:tasks.workunit.client.1.smithi122.stdout:9/527: write db/d1b/d6c/d58/d9d/fa6 [970037,20063] 0 2024-03-22T02:28:53.636 INFO:tasks.workunit.client.0.smithi103.stdout:9/948: dwrite d2/da/d18/d1e/d34/d3d/d4d/f8f [4194304,4194304] 0 2024-03-22T02:28:53.639 INFO:tasks.workunit.client.1.smithi122.stdout:9/528: fsync db/d1b/d6c/f4f 0 2024-03-22T02:28:53.640 INFO:tasks.workunit.client.1.smithi122.stdout:8/478: write d1/d40/d51/f6b [762722,7249] 0 2024-03-22T02:28:53.640 INFO:tasks.workunit.client.1.smithi122.stdout:1/463: mknod dc/d2b/c9a 0 2024-03-22T02:28:53.640 INFO:tasks.workunit.client.0.smithi103.stdout:9/949: fdatasync d2/d10/d43/f7a 0 2024-03-22T02:28:53.641 INFO:tasks.workunit.client.1.smithi122.stdout:1/464: chown dc/c15 195 1 2024-03-22T02:28:53.644 INFO:tasks.workunit.client.1.smithi122.stdout:0/388: rmdir d11/d83/d54 0 2024-03-22T02:28:53.646 INFO:tasks.workunit.client.0.smithi103.stdout:9/950: symlink d2/d5/d7b/d7d/l111 0 2024-03-22T02:28:53.650 INFO:tasks.workunit.client.0.smithi103.stdout:4/970: dwrite d3/d1e/d53/d8a/f9c [0,4194304] 0 2024-03-22T02:28:53.651 INFO:tasks.workunit.client.1.smithi122.stdout:8/479: mkdir d1/d9/d50/da6 0 2024-03-22T02:28:53.653 INFO:tasks.workunit.client.0.smithi103.stdout:0/982: dwrite dd/d16/d4b/d96/d64/d6a/dd5/f113 [0,4194304] 0 2024-03-22T02:28:53.658 INFO:tasks.workunit.client.1.smithi122.stdout:8/480: mknod d1/d8/d5e/ca7 0 2024-03-22T02:28:53.661 INFO:tasks.workunit.client.1.smithi122.stdout:8/481: write d1/d3f/d45/f68 [70097,108980] 0 2024-03-22T02:28:53.661 INFO:tasks.workunit.client.0.smithi103.stdout:4/971: dwrite d3/d1e/f36 [0,4194304] 0 2024-03-22T02:28:53.662 INFO:tasks.workunit.client.1.smithi122.stdout:1/465: getdents dc/d3b/d46/d4d/d71 0 2024-03-22T02:28:53.664 INFO:tasks.workunit.client.0.smithi103.stdout:4/972: fdatasync d3/fa0 0 2024-03-22T02:28:53.669 INFO:tasks.workunit.client.1.smithi122.stdout:1/466: symlink dc/d4c/l9b 0 2024-03-22T02:28:53.669 INFO:tasks.workunit.client.1.smithi122.stdout:1/467: chown dc/d2b/c9a 391232262 1 2024-03-22T02:28:53.669 INFO:tasks.workunit.client.1.smithi122.stdout:8/482: read d1/d22/f32 [3437019,74535] 0 2024-03-22T02:28:53.678 INFO:tasks.workunit.client.1.smithi122.stdout:8/483: rename d1/d60/d73/l7c to d1/d8/d8a/d78/la8 0 2024-03-22T02:28:53.679 INFO:tasks.workunit.client.1.smithi122.stdout:8/484: readlink d1/d9/d50/l5f 0 2024-03-22T02:28:53.680 INFO:tasks.workunit.client.1.smithi122.stdout:8/485: write d1/d9/f26 [887734,73285] 0 2024-03-22T02:28:53.684 INFO:tasks.workunit.client.1.smithi122.stdout:1/468: truncate dc/f39 947290 0 2024-03-22T02:28:53.687 INFO:tasks.workunit.client.1.smithi122.stdout:6/534: truncate d7/de/dd/d20/d2d/d54/d76/f89 4121156 0 2024-03-22T02:28:53.689 INFO:tasks.workunit.client.1.smithi122.stdout:8/486: symlink d1/d60/d73/la9 0 2024-03-22T02:28:53.691 INFO:tasks.workunit.client.1.smithi122.stdout:9/529: truncate db/d23/d1e/f98 471865 0 2024-03-22T02:28:53.692 INFO:tasks.workunit.client.1.smithi122.stdout:9/530: fdatasync db/df/f7d 0 2024-03-22T02:28:53.692 INFO:tasks.workunit.client.1.smithi122.stdout:8/487: fsync d1/d4d/f4e 0 2024-03-22T02:28:53.693 INFO:tasks.workunit.client.1.smithi122.stdout:1/469: mkdir dc/d3b/d8d/d72/d9c 0 2024-03-22T02:28:53.697 INFO:tasks.workunit.client.1.smithi122.stdout:0/389: dwrite d11/d14/d1d/f41 [0,4194304] 0 2024-03-22T02:28:53.702 INFO:tasks.workunit.client.1.smithi122.stdout:8/488: write d1/d22/f32 [238633,125455] 0 2024-03-22T02:28:53.702 INFO:tasks.workunit.client.1.smithi122.stdout:8/489: readlink d1/d3f/d45/l8f 0 2024-03-22T02:28:53.704 INFO:tasks.workunit.client.1.smithi122.stdout:9/531: dwrite db/d23/f83 [0,4194304] 0 2024-03-22T02:28:53.705 INFO:tasks.workunit.client.1.smithi122.stdout:6/535: creat d7/de/dd/d20/fa5 x:0 0 0 2024-03-22T02:28:53.708 INFO:tasks.workunit.client.1.smithi122.stdout:8/490: mkdir d1/d4d/daa 0 2024-03-22T02:28:53.711 INFO:tasks.workunit.client.1.smithi122.stdout:9/532: mkdir db/d1b/da8 0 2024-03-22T02:28:53.712 INFO:tasks.workunit.client.1.smithi122.stdout:9/533: chown db/df/l1d 412006788 1 2024-03-22T02:28:53.712 INFO:tasks.workunit.client.1.smithi122.stdout:9/534: chown db/d23/f83 6 1 2024-03-22T02:28:53.712 INFO:tasks.workunit.client.1.smithi122.stdout:0/390: mkdir d11/d16/d37/d84/d87 0 2024-03-22T02:28:53.714 INFO:tasks.workunit.client.1.smithi122.stdout:6/536: mknod d7/de/dd/d20/d2d/d54/d76/d86/d96/ca6 0 2024-03-22T02:28:53.717 INFO:tasks.workunit.client.1.smithi122.stdout:3/347: sync 2024-03-22T02:28:53.719 INFO:tasks.workunit.client.1.smithi122.stdout:8/491: mkdir d1/d8/d63/d88/dab 0 2024-03-22T02:28:53.722 INFO:tasks.workunit.client.1.smithi122.stdout:9/535: creat db/d23/fa9 x:0 0 0 2024-03-22T02:28:53.722 INFO:tasks.workunit.client.1.smithi122.stdout:9/536: stat db/ce 0 2024-03-22T02:28:53.725 INFO:tasks.workunit.client.1.smithi122.stdout:6/537: rename d7/de/dd/d20/d2d/d54/d76/d86/fa0 to d7/de/dd/d20/d8f/fa7 0 2024-03-22T02:28:53.727 INFO:tasks.workunit.client.1.smithi122.stdout:0/391: getdents d11/d16/d37/d84/d87 0 2024-03-22T02:28:53.728 INFO:tasks.workunit.client.0.smithi103.stdout:6/780: sync 2024-03-22T02:28:53.728 INFO:tasks.workunit.client.0.smithi103.stdout:7/969: sync 2024-03-22T02:28:53.730 INFO:tasks.workunit.client.1.smithi122.stdout:1/470: dread dc/f1a [0,4194304] 0 2024-03-22T02:28:53.730 INFO:tasks.workunit.client.0.smithi103.stdout:0/983: write dd/d16/f84 [2457831,3827] 0 2024-03-22T02:28:53.730 INFO:tasks.workunit.client.0.smithi103.stdout:4/973: rmdir d3/d69 39 2024-03-22T02:28:53.730 INFO:tasks.workunit.client.0.smithi103.stdout:7/970: stat d1/d10/d11/d70/d8a/d9d/l107 0 2024-03-22T02:28:53.731 INFO:tasks.workunit.client.1.smithi122.stdout:9/537: rename db/l31 to db/df/d3e/d81/d29/laa 0 2024-03-22T02:28:53.733 INFO:tasks.workunit.client.1.smithi122.stdout:8/492: getdents d1/d9/d50/da6 0 2024-03-22T02:28:53.734 INFO:tasks.workunit.client.1.smithi122.stdout:3/348: dwrite d2/f5e [0,4194304] 0 2024-03-22T02:28:53.735 INFO:tasks.workunit.client.0.smithi103.stdout:4/974: symlink d3/deb/l144 0 2024-03-22T02:28:53.736 INFO:tasks.workunit.client.1.smithi122.stdout:9/538: read db/d23/f11 [412289,59457] 0 2024-03-22T02:28:53.737 INFO:tasks.workunit.client.1.smithi122.stdout:0/392: symlink d11/d47/d5d/l88 0 2024-03-22T02:28:53.737 INFO:tasks.workunit.client.0.smithi103.stdout:6/781: dread d0/d14/d57/d68/f55 [0,4194304] 0 2024-03-22T02:28:53.738 INFO:tasks.workunit.client.1.smithi122.stdout:9/539: write db/df/d3e/d81/d25/d41/f61 [967495,87699] 0 2024-03-22T02:28:53.738 INFO:tasks.workunit.client.0.smithi103.stdout:7/971: dwrite d1/da/d2f/d36/f91 [0,4194304] 0 2024-03-22T02:28:53.739 INFO:tasks.workunit.client.0.smithi103.stdout:4/975: truncate d3/d1e/d53/fe2 837326 0 2024-03-22T02:28:53.740 INFO:tasks.workunit.client.1.smithi122.stdout:8/493: symlink d1/d22/d6a/lac 0 2024-03-22T02:28:53.742 INFO:tasks.workunit.client.1.smithi122.stdout:8/494: fsync d1/d40/d71/d97/f9f 0 2024-03-22T02:28:53.742 INFO:tasks.workunit.client.1.smithi122.stdout:8/495: chown d1/l2a 82077 1 2024-03-22T02:28:53.745 INFO:tasks.workunit.client.1.smithi122.stdout:1/471: fdatasync dc/f10 0 2024-03-22T02:28:53.745 INFO:tasks.workunit.client.0.smithi103.stdout:7/972: readlink d1/l77 0 2024-03-22T02:28:53.745 INFO:tasks.workunit.client.1.smithi122.stdout:8/496: write d1/f53 [925046,18637] 0 2024-03-22T02:28:53.748 INFO:tasks.workunit.client.0.smithi103.stdout:6/782: rename d0/d3/f46 to d0/d88/d8e/dae/f101 0 2024-03-22T02:28:53.750 INFO:tasks.workunit.client.0.smithi103.stdout:0/984: creat dd/d16/d4b/d96/f130 x:0 0 0 2024-03-22T02:28:53.751 INFO:tasks.workunit.client.0.smithi103.stdout:4/976: creat d3/da2/da6/d103/f145 x:0 0 0 2024-03-22T02:28:53.752 INFO:tasks.workunit.client.1.smithi122.stdout:3/349: mknod d2/d9/dc/d17/d28/d4c/c65 0 2024-03-22T02:28:53.753 INFO:tasks.workunit.client.1.smithi122.stdout:3/350: dread - d2/d9/dc/df/d49/f4f zero size 2024-03-22T02:28:53.755 INFO:tasks.workunit.client.1.smithi122.stdout:3/351: dread d2/d9/dc/d17/f42 [0,4194304] 0 2024-03-22T02:28:53.756 INFO:tasks.workunit.client.1.smithi122.stdout:6/538: write d7/de/d27/f36 [483593,76848] 0 2024-03-22T02:28:53.757 INFO:tasks.workunit.client.0.smithi103.stdout:6/783: stat d0/c34 0 2024-03-22T02:28:53.758 INFO:tasks.workunit.client.0.smithi103.stdout:6/784: fdatasync d0/f41 0 2024-03-22T02:28:53.758 INFO:tasks.workunit.client.1.smithi122.stdout:1/472: unlink dc/d3b/f45 0 2024-03-22T02:28:53.759 INFO:tasks.workunit.client.1.smithi122.stdout:1/473: dread - dc/d3b/d46/d4d/f7d zero size 2024-03-22T02:28:53.760 INFO:tasks.workunit.client.1.smithi122.stdout:8/497: creat d1/d8/d8a/fad x:0 0 0 2024-03-22T02:28:53.761 INFO:tasks.workunit.client.0.smithi103.stdout:4/977: readlink d3/d100/l10b 0 2024-03-22T02:28:53.762 INFO:tasks.workunit.client.1.smithi122.stdout:9/540: creat db/d1b/da8/fab x:0 0 0 2024-03-22T02:28:53.762 INFO:tasks.workunit.client.0.smithi103.stdout:4/978: chown d3/d31/fd8 200241 1 2024-03-22T02:28:53.762 INFO:tasks.workunit.client.0.smithi103.stdout:7/973: mknod d1/d10/d11/d19/d11f/c135 0 2024-03-22T02:28:53.763 INFO:tasks.workunit.client.1.smithi122.stdout:3/352: read d2/d9/dc/d17/d28/f4d [63381,118374] 0 2024-03-22T02:28:53.763 INFO:tasks.workunit.client.1.smithi122.stdout:8/498: read d1/d40/f7a [491346,78965] 0 2024-03-22T02:28:53.764 INFO:tasks.workunit.client.0.smithi103.stdout:7/974: dread - d1/d10/d11/d19/d86/d92/dbf/d128/d69/d5c/f117 zero size 2024-03-22T02:28:53.764 INFO:tasks.workunit.client.1.smithi122.stdout:3/353: dread - d2/d9/dc/df/d47/f56 zero size 2024-03-22T02:28:53.765 INFO:tasks.workunit.client.0.smithi103.stdout:6/785: rmdir d0/d3/d22/d58/dd8 39 2024-03-22T02:28:53.768 INFO:tasks.workunit.client.1.smithi122.stdout:9/541: unlink db/d23/d1e/f8c 0 2024-03-22T02:28:53.769 INFO:tasks.workunit.client.0.smithi103.stdout:6/786: mknod d0/d3/d22/d58/d6f/d75/db0/db1/c102 0 2024-03-22T02:28:53.770 INFO:tasks.workunit.client.1.smithi122.stdout:1/474: mknod dc/d3b/d8d/d72/c9d 0 2024-03-22T02:28:53.770 INFO:tasks.workunit.client.0.smithi103.stdout:4/979: mkdir d3/d1e/d25/d65/dc9/d120/d146 0 2024-03-22T02:28:53.770 INFO:tasks.workunit.client.1.smithi122.stdout:0/393: dwrite d11/d14/f56 [0,4194304] 0 2024-03-22T02:28:53.771 INFO:tasks.workunit.client.1.smithi122.stdout:1/475: readlink dc/d2b/l8e 0 2024-03-22T02:28:53.772 INFO:tasks.workunit.client.1.smithi122.stdout:8/499: mkdir d1/d8/d63/d88/dae 0 2024-03-22T02:28:53.773 INFO:tasks.workunit.client.0.smithi103.stdout:6/787: stat d0/d14/d57/d68/l98 0 2024-03-22T02:28:53.773 INFO:tasks.workunit.client.1.smithi122.stdout:0/394: dread - d11/d14/d23/d24/d33/f79 zero size 2024-03-22T02:28:53.773 INFO:tasks.workunit.client.0.smithi103.stdout:6/788: chown d0/f6a 6 1 2024-03-22T02:28:53.774 INFO:tasks.workunit.client.1.smithi122.stdout:3/354: fsync d2/d9/dc/df/d49/f54 0 2024-03-22T02:28:53.777 INFO:tasks.workunit.client.0.smithi103.stdout:6/789: mkdir d0/dbd/d103 0 2024-03-22T02:28:53.778 INFO:tasks.workunit.client.1.smithi122.stdout:6/539: creat d7/de/fa8 x:0 0 0 2024-03-22T02:28:53.781 INFO:tasks.workunit.client.0.smithi103.stdout:6/790: creat d0/d3/d22/d58/d6f/d75/db0/dd7/f104 x:0 0 0 2024-03-22T02:28:53.783 INFO:tasks.workunit.client.1.smithi122.stdout:3/355: chown d2/d9/dc/d17/d2b/d3f/l2e 2 1 2024-03-22T02:28:53.784 INFO:tasks.workunit.client.0.smithi103.stdout:6/791: creat d0/d3/d22/d58/de0/f105 x:0 0 0 2024-03-22T02:28:53.786 INFO:tasks.workunit.client.1.smithi122.stdout:0/395: dwrite d11/d14/d1d/d27/d2f/f58 [0,4194304] 0 2024-03-22T02:28:53.786 INFO:tasks.workunit.client.0.smithi103.stdout:6/792: chown d0/d14/d57/l83 3020005 1 2024-03-22T02:28:53.789 INFO:tasks.workunit.client.0.smithi103.stdout:9/951: truncate d2/d5/d7b/d7d/dd4/dd8/fe5 3768046 0 2024-03-22T02:28:53.793 INFO:tasks.workunit.client.0.smithi103.stdout:6/793: dwrite d0/d3/d22/d58/d6f/f94 [0,4194304] 0 2024-03-22T02:28:53.795 INFO:tasks.workunit.client.0.smithi103.stdout:6/794: fsync d0/d14/d57/ff5 0 2024-03-22T02:28:53.796 INFO:tasks.workunit.client.1.smithi122.stdout:9/542: rename db/df/d3e/d81/d25/l9b to db/d23/d1f/lac 0 2024-03-22T02:28:53.796 INFO:tasks.workunit.client.1.smithi122.stdout:8/500: truncate d1/d9/d50/f59 3205331 0 2024-03-22T02:28:53.800 INFO:tasks.workunit.client.1.smithi122.stdout:3/356: rename d2/d9/dc/df/d49/c4a to d2/d9/dc/df/d47/c66 0 2024-03-22T02:28:53.802 INFO:tasks.workunit.client.1.smithi122.stdout:1/476: dwrite dc/fd [8388608,4194304] 0 2024-03-22T02:28:53.803 INFO:tasks.workunit.client.0.smithi103.stdout:6/795: symlink d0/d3/d22/d58/d6f/d75/l106 0 2024-03-22T02:28:53.803 INFO:tasks.workunit.client.1.smithi122.stdout:9/543: mkdir db/d23/d1e/dad 0 2024-03-22T02:28:53.806 INFO:tasks.workunit.client.1.smithi122.stdout:3/357: read d2/d9/dc/d17/d28/f4d [39614,87429] 0 2024-03-22T02:28:53.807 INFO:tasks.workunit.client.1.smithi122.stdout:1/477: rmdir dc/d3b/d8d 39 2024-03-22T02:28:53.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/396: dwrite d11/d16/f55 [0,4194304] 0 2024-03-22T02:28:53.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/358: write d2/d9/dc/df/d38/f3b [50520,107522] 0 2024-03-22T02:28:53.816 INFO:tasks.workunit.client.1.smithi122.stdout:9/544: rename db/d23/d1f/d57/d66/f85 to db/df/d3e/d81/d25/fae 0 2024-03-22T02:28:53.820 INFO:tasks.workunit.client.1.smithi122.stdout:9/545: write db/df/d3e/d81/d29/f9f [872584,76526] 0 2024-03-22T02:28:53.820 INFO:tasks.workunit.client.1.smithi122.stdout:3/359: stat d2/d9/dc/d17/c4b 0 2024-03-22T02:28:53.821 INFO:tasks.workunit.client.1.smithi122.stdout:8/501: getdents d1 0 2024-03-22T02:28:53.824 INFO:tasks.workunit.client.1.smithi122.stdout:8/502: creat d1/d40/d48/faf x:0 0 0 2024-03-22T02:28:53.826 INFO:tasks.workunit.client.0.smithi103.stdout:4/980: rmdir d3/d1e/d53/d7a 39 2024-03-22T02:28:53.826 INFO:tasks.workunit.client.1.smithi122.stdout:9/546: symlink db/d23/d1f/laf 0 2024-03-22T02:28:53.827 INFO:tasks.workunit.client.0.smithi103.stdout:4/981: write d3/d31/f32 [3356277,7300] 0 2024-03-22T02:28:53.828 INFO:tasks.workunit.client.1.smithi122.stdout:9/547: write db/df/f7d [5107060,50284] 0 2024-03-22T02:28:53.831 INFO:tasks.workunit.client.1.smithi122.stdout:3/360: getdents d2/d9/dc/df 0 2024-03-22T02:28:53.831 INFO:tasks.workunit.client.1.smithi122.stdout:8/503: dread d1/d8/f5c [0,4194304] 0 2024-03-22T02:28:53.833 INFO:tasks.workunit.client.1.smithi122.stdout:8/504: stat d1/l17 0 2024-03-22T02:28:53.834 INFO:tasks.workunit.client.1.smithi122.stdout:8/505: dread - d1/d8/d8a/f86 zero size 2024-03-22T02:28:53.835 INFO:tasks.workunit.client.1.smithi122.stdout:0/397: dread d11/d14/d1d/f80 [0,4194304] 0 2024-03-22T02:28:53.836 INFO:tasks.workunit.client.1.smithi122.stdout:3/361: creat d2/d9/dc/d17/d2b/f67 x:0 0 0 2024-03-22T02:28:53.837 INFO:tasks.workunit.client.1.smithi122.stdout:3/362: stat d2/d9/dc/d17/f42 0 2024-03-22T02:28:53.837 INFO:tasks.workunit.client.1.smithi122.stdout:8/506: symlink d1/d9/d50/lb0 0 2024-03-22T02:28:53.840 INFO:tasks.workunit.client.1.smithi122.stdout:8/507: mkdir d1/d9/d50/db1 0 2024-03-22T02:28:53.842 INFO:tasks.workunit.client.1.smithi122.stdout:3/363: rename d2/d9/dc/l5a to d2/d9/dc/d17/d2b/d3f/l68 0 2024-03-22T02:28:53.844 INFO:tasks.workunit.client.1.smithi122.stdout:8/508: rename d1/d40/d51/f89 to d1/d40/d51/fb2 0 2024-03-22T02:28:53.846 INFO:tasks.workunit.client.1.smithi122.stdout:1/478: dread dc/f38 [0,4194304] 0 2024-03-22T02:28:53.846 INFO:tasks.workunit.client.1.smithi122.stdout:3/364: creat d2/d9/dc/d17/d28/f69 x:0 0 0 2024-03-22T02:28:53.847 INFO:tasks.workunit.client.0.smithi103.stdout:0/985: write dd/f5b [4147147,24640] 0 2024-03-22T02:28:53.847 INFO:tasks.workunit.client.1.smithi122.stdout:3/365: dread - d2/d9/dc/d17/f63 zero size 2024-03-22T02:28:53.848 INFO:tasks.workunit.client.1.smithi122.stdout:1/479: creat dc/d3b/d46/f9e x:0 0 0 2024-03-22T02:28:53.849 INFO:tasks.workunit.client.1.smithi122.stdout:1/480: chown dc 385782 1 2024-03-22T02:28:53.849 INFO:tasks.workunit.client.1.smithi122.stdout:8/509: fsync d1/d40/d48/faf 0 2024-03-22T02:28:53.849 INFO:tasks.workunit.client.1.smithi122.stdout:1/481: dread - dc/d4c/f8b zero size 2024-03-22T02:28:53.851 INFO:tasks.workunit.client.1.smithi122.stdout:8/510: rmdir d1/d3f/d45 39 2024-03-22T02:28:53.852 INFO:tasks.workunit.client.1.smithi122.stdout:1/482: write dc/d4c/f8b [499437,117303] 0 2024-03-22T02:28:53.852 INFO:tasks.workunit.client.1.smithi122.stdout:9/548: dread db/df/d3e/d81/d29/f5f [0,4194304] 0 2024-03-22T02:28:53.854 INFO:tasks.workunit.client.1.smithi122.stdout:8/511: write d1/d60/d73/d90/f9b [781041,115369] 0 2024-03-22T02:28:53.855 INFO:tasks.workunit.client.1.smithi122.stdout:3/366: dread d2/d9/dc/d17/d45/f46 [0,4194304] 0 2024-03-22T02:28:53.858 INFO:tasks.workunit.client.0.smithi103.stdout:7/975: write d1/f1e [1485585,8847] 0 2024-03-22T02:28:53.859 INFO:tasks.workunit.client.1.smithi122.stdout:3/367: mkdir d2/d9/dc/d17/d2b/d6a 0 2024-03-22T02:28:53.861 INFO:tasks.workunit.client.1.smithi122.stdout:8/512: dread d1/d9/f13 [0,4194304] 0 2024-03-22T02:28:53.864 INFO:tasks.workunit.client.1.smithi122.stdout:9/549: truncate db/df/d3e/d81/d29/f38 3308373 0 2024-03-22T02:28:53.866 INFO:tasks.workunit.client.1.smithi122.stdout:3/368: dread d2/f39 [0,4194304] 0 2024-03-22T02:28:53.871 INFO:tasks.workunit.client.1.smithi122.stdout:8/513: truncate d1/d40/f47 1936835 0 2024-03-22T02:28:53.874 INFO:tasks.workunit.client.1.smithi122.stdout:9/550: dwrite db/df/d3e/f8b [0,4194304] 0 2024-03-22T02:28:53.874 INFO:tasks.workunit.client.1.smithi122.stdout:8/514: write d1/d8/f75 [215272,3518] 0 2024-03-22T02:28:53.877 INFO:tasks.workunit.client.1.smithi122.stdout:9/551: readlink db/d23/d1f/laf 0 2024-03-22T02:28:53.878 INFO:tasks.workunit.client.1.smithi122.stdout:1/483: dread dc/f2f [4194304,4194304] 0 2024-03-22T02:28:53.880 INFO:tasks.workunit.client.0.smithi103.stdout:9/952: truncate d2/da/d18/d1e/d34/d3d/d61/f9a 3840090 0 2024-03-22T02:28:53.881 INFO:tasks.workunit.client.1.smithi122.stdout:3/369: dwrite d2/d9/dc/df/f53 [4194304,4194304] 0 2024-03-22T02:28:53.883 INFO:tasks.workunit.client.1.smithi122.stdout:8/515: mknod d1/d8/d8a/d78/cb3 0 2024-03-22T02:28:53.883 INFO:tasks.workunit.client.1.smithi122.stdout:3/370: truncate d2/d9/dc/df/d47/f56 5628 0 2024-03-22T02:28:53.887 INFO:tasks.workunit.client.1.smithi122.stdout:1/484: creat dc/d2b/d86/f9f x:0 0 0 2024-03-22T02:28:53.888 INFO:tasks.workunit.client.0.smithi103.stdout:6/796: truncate d0/d14/d9a/fdb 568611 0 2024-03-22T02:28:53.888 INFO:tasks.workunit.client.1.smithi122.stdout:8/516: truncate d1/d8/f33 6760828 0 2024-03-22T02:28:53.889 INFO:tasks.workunit.client.0.smithi103.stdout:6/797: readlink d0/d88/d8e/dae/lc0 0 2024-03-22T02:28:53.890 INFO:tasks.workunit.client.1.smithi122.stdout:8/517: chown d1/l2a 406713 1 2024-03-22T02:28:53.890 INFO:tasks.workunit.client.1.smithi122.stdout:8/518: chown d1/d9/d50/lb0 64 1 2024-03-22T02:28:53.890 INFO:tasks.workunit.client.0.smithi103.stdout:6/798: dread - d0/d14/d57/d68/fff zero size 2024-03-22T02:28:53.891 INFO:tasks.workunit.client.1.smithi122.stdout:3/371: mkdir d2/d9/dc/d17/d2b/d6b 0 2024-03-22T02:28:53.894 INFO:tasks.workunit.client.1.smithi122.stdout:9/552: rename db/d1b/d6c/l99 to db/d23/d1f/d57/d66/lb0 0 2024-03-22T02:28:53.896 INFO:tasks.workunit.client.1.smithi122.stdout:8/519: mkdir d1/d40/d71/db4 0 2024-03-22T02:28:53.898 INFO:tasks.workunit.client.1.smithi122.stdout:3/372: dwrite d2/f7 [4194304,4194304] 0 2024-03-22T02:28:53.898 INFO:tasks.workunit.client.0.smithi103.stdout:4/982: truncate d3/d1e/d68/f117 793146 0 2024-03-22T02:28:53.899 INFO:tasks.workunit.client.1.smithi122.stdout:9/553: readlink db/d1b/d6c/l3c 0 2024-03-22T02:28:53.899 INFO:tasks.workunit.client.0.smithi103.stdout:4/983: truncate d3/d31/fe6 267813 0 2024-03-22T02:28:53.900 INFO:tasks.workunit.client.1.smithi122.stdout:8/520: creat d1/d40/d48/fb5 x:0 0 0 2024-03-22T02:28:53.901 INFO:tasks.workunit.client.1.smithi122.stdout:8/521: chown d1/d9/d50/da6 2861 1 2024-03-22T02:28:53.904 INFO:tasks.workunit.client.1.smithi122.stdout:3/373: creat d2/d9/dc/d17/d28/f6c x:0 0 0 2024-03-22T02:28:53.906 INFO:tasks.workunit.client.1.smithi122.stdout:3/374: stat d2/d9/dc/df/d47/l5c 0 2024-03-22T02:28:53.909 INFO:tasks.workunit.client.1.smithi122.stdout:1/485: fsync dc/d4c/f92 0 2024-03-22T02:28:53.909 INFO:tasks.workunit.client.1.smithi122.stdout:8/522: dwrite d1/d40/d71/d97/f9f [0,4194304] 0 2024-03-22T02:28:53.910 INFO:tasks.workunit.client.1.smithi122.stdout:1/486: chown dc/f49 37560 1 2024-03-22T02:28:53.910 INFO:tasks.workunit.client.0.smithi103.stdout:9/953: fdatasync d2/d5/f63 0 2024-03-22T02:28:53.911 INFO:tasks.workunit.client.1.smithi122.stdout:1/487: write dc/d3b/d46/f82 [3558198,93103] 0 2024-03-22T02:28:53.912 INFO:tasks.workunit.client.0.smithi103.stdout:9/954: write d2/da/d18/d1e/d77/d8b/dae/dcb/ff9 [638531,44746] 0 2024-03-22T02:28:53.914 INFO:tasks.workunit.client.1.smithi122.stdout:8/523: symlink d1/d60/d73/da3/lb6 0 2024-03-22T02:28:53.915 INFO:tasks.workunit.client.1.smithi122.stdout:1/488: read dc/d3b/d46/d47/f8a [1815390,39021] 0 2024-03-22T02:28:53.918 INFO:tasks.workunit.client.1.smithi122.stdout:9/554: truncate db/d1b/d6c/d58/f67 866192 0 2024-03-22T02:28:53.920 INFO:tasks.workunit.client.0.smithi103.stdout:4/984: rename d3/d1e/d25/d65/l8d to d3/d69/l147 0 2024-03-22T02:28:53.921 INFO:tasks.workunit.client.1.smithi122.stdout:8/524: symlink d1/d8/d63/d88/dab/lb7 0 2024-03-22T02:28:53.921 INFO:tasks.workunit.client.1.smithi122.stdout:8/525: readlink d1/d8/l29 0 2024-03-22T02:28:53.924 INFO:tasks.workunit.client.0.smithi103.stdout:9/955: getdents d2/d5/d7b/d7d/dd4 0 2024-03-22T02:28:53.924 INFO:tasks.workunit.client.1.smithi122.stdout:1/489: mknod dc/d3b/d8d/ca0 0 2024-03-22T02:28:53.924 INFO:tasks.workunit.client.1.smithi122.stdout:8/526: dread d1/d40/f7a [0,4194304] 0 2024-03-22T02:28:53.925 INFO:tasks.workunit.client.0.smithi103.stdout:9/956: chown d2/d10/d43/d7e/da4/dcd 8 1 2024-03-22T02:28:53.926 INFO:tasks.workunit.client.0.smithi103.stdout:4/985: getdents d3/d38/d5c 0 2024-03-22T02:28:53.926 INFO:tasks.workunit.client.1.smithi122.stdout:8/527: read - d1/d60/d73/f94 zero size 2024-03-22T02:28:53.927 INFO:tasks.workunit.client.1.smithi122.stdout:1/490: symlink dc/d3b/d46/d47/la1 0 2024-03-22T02:28:53.928 INFO:tasks.workunit.client.1.smithi122.stdout:9/555: getdents db/d1b 0 2024-03-22T02:28:53.929 INFO:tasks.workunit.client.0.smithi103.stdout:4/986: rename d3/d69/l74 to d3/da2/da6/d103/l148 0 2024-03-22T02:28:53.932 INFO:tasks.workunit.client.1.smithi122.stdout:9/556: link db/d23/d1f/d57/d66/d73/f76 db/fb1 0 2024-03-22T02:28:53.933 INFO:tasks.workunit.client.0.smithi103.stdout:9/957: getdents d2/da/d18/d1e/d34/d3d/d61/da0/df0 0 2024-03-22T02:28:53.934 INFO:tasks.workunit.client.1.smithi122.stdout:9/557: fsync db/d1b/f6a 0 2024-03-22T02:28:53.936 INFO:tasks.workunit.client.0.smithi103.stdout:9/958: truncate d2/da/d18/fb0 2173421 0 2024-03-22T02:28:53.937 INFO:tasks.workunit.client.1.smithi122.stdout:8/528: write d1/d9/d50/f59 [1774812,128070] 0 2024-03-22T02:28:53.938 INFO:tasks.workunit.client.1.smithi122.stdout:8/529: write d1/d9/f26 [4079306,8069] 0 2024-03-22T02:28:53.939 INFO:tasks.workunit.client.0.smithi103.stdout:4/987: dread d3/f24 [4194304,4194304] 0 2024-03-22T02:28:53.940 INFO:tasks.workunit.client.1.smithi122.stdout:8/530: truncate d1/d40/d48/faf 875093 0 2024-03-22T02:28:53.941 INFO:tasks.workunit.client.0.smithi103.stdout:4/988: write d3/d1e/d25/d65/dc9/f113 [1012254,91109] 0 2024-03-22T02:28:53.942 INFO:tasks.workunit.client.1.smithi122.stdout:8/531: mkdir d1/d40/d71/d97/db8 0 2024-03-22T02:28:53.945 INFO:tasks.workunit.client.0.smithi103.stdout:4/989: creat d3/d38/d5c/d90/df8/d137/f149 x:0 0 0 2024-03-22T02:28:53.946 INFO:tasks.workunit.client.1.smithi122.stdout:8/532: dread d1/d8/d2f/f98 [0,4194304] 0 2024-03-22T02:28:53.956 INFO:tasks.workunit.client.1.smithi122.stdout:9/558: dread db/d23/f1a [0,4194304] 0 2024-03-22T02:28:53.956 INFO:tasks.workunit.client.0.smithi103.stdout:6/799: write d0/d88/f9f [35835,43676] 0 2024-03-22T02:28:53.956 INFO:tasks.workunit.client.0.smithi103.stdout:0/986: truncate dd/d16/d4b/d96/d29/d48/d94/f110 3358599 0 2024-03-22T02:28:53.957 INFO:tasks.workunit.client.1.smithi122.stdout:9/559: write db/df/f7d [291841,62526] 0 2024-03-22T02:28:53.958 INFO:tasks.workunit.client.0.smithi103.stdout:7/976: write d1/d10/d11/d19/dc3/de8/ff1 [626653,123610] 0 2024-03-22T02:28:53.959 INFO:tasks.workunit.client.1.smithi122.stdout:9/560: chown db/df/d3e/d59/ca7 83532606 1 2024-03-22T02:28:53.960 INFO:tasks.workunit.client.0.smithi103.stdout:7/977: chown d1/da/d1c/l72 3 1 2024-03-22T02:28:53.960 INFO:tasks.workunit.client.0.smithi103.stdout:9/959: dread d2/da/fd [0,4194304] 0 2024-03-22T02:28:53.961 INFO:tasks.workunit.client.1.smithi122.stdout:9/561: truncate db/d23/d1f/d57/d66/d73/fa4 516785 0 2024-03-22T02:28:53.961 INFO:tasks.workunit.client.1.smithi122.stdout:9/562: stat db/df/c96 0 2024-03-22T02:28:53.962 INFO:tasks.workunit.client.1.smithi122.stdout:9/563: truncate db/d1b/f6a 1460107 0 2024-03-22T02:28:53.962 INFO:tasks.workunit.client.0.smithi103.stdout:6/800: creat d0/d14/dc7/f107 x:0 0 0 2024-03-22T02:28:53.962 INFO:tasks.workunit.client.0.smithi103.stdout:7/978: read d1/da/fb [1386737,9272] 0 2024-03-22T02:28:53.963 INFO:tasks.workunit.client.1.smithi122.stdout:9/564: dread - db/df/d3e/d81/d3d/d79/f93 zero size 2024-03-22T02:28:53.963 INFO:tasks.workunit.client.0.smithi103.stdout:0/987: dread dd/d16/d4b/d96/d29/d48/d5c/f80 [0,4194304] 0 2024-03-22T02:28:53.964 INFO:tasks.workunit.client.0.smithi103.stdout:6/801: truncate d0/d3/fbc 702727 0 2024-03-22T02:28:53.966 INFO:tasks.workunit.client.0.smithi103.stdout:0/988: readlink dd/d16/d4b/d96/d35/d82/ldb 0 2024-03-22T02:28:53.966 INFO:tasks.workunit.client.0.smithi103.stdout:6/802: fdatasync d0/d3/d22/d58/d6f/d75/db0/db1/fc5 0 2024-03-22T02:28:53.968 INFO:tasks.workunit.client.0.smithi103.stdout:0/989: chown dd/d16/d4b/d96/d29/d48/fec 585 1 2024-03-22T02:28:53.969 INFO:tasks.workunit.client.0.smithi103.stdout:4/990: dread d3/d69/fb0 [0,4194304] 0 2024-03-22T02:28:53.970 INFO:tasks.workunit.client.0.smithi103.stdout:9/960: symlink d2/d10/d43/d9b/df3/df7/d100/l112 0 2024-03-22T02:28:53.971 INFO:tasks.workunit.client.0.smithi103.stdout:6/803: mknod d0/d88/d8e/c108 0 2024-03-22T02:28:53.972 INFO:tasks.workunit.client.0.smithi103.stdout:6/804: chown d0/c26 101086 1 2024-03-22T02:28:53.973 INFO:tasks.workunit.client.0.smithi103.stdout:7/979: mknod d1/d10/c136 0 2024-03-22T02:28:53.976 INFO:tasks.workunit.client.0.smithi103.stdout:0/990: fdatasync dd/d16/d4b/d96/d29/f85 0 2024-03-22T02:28:53.978 INFO:tasks.workunit.client.0.smithi103.stdout:7/980: dread - d1/d10/d11/d70/d8a/d9d/dff/f110 zero size 2024-03-22T02:28:53.980 INFO:tasks.workunit.client.0.smithi103.stdout:7/981: stat d1/d10/c136 0 2024-03-22T02:28:53.981 INFO:tasks.workunit.client.0.smithi103.stdout:9/961: dwrite d2/da/d18/d1e/d34/d3d/d61/db7/fc3 [0,4194304] 0 2024-03-22T02:28:53.983 INFO:tasks.workunit.client.0.smithi103.stdout:4/991: dread d3/d1e/fc3 [0,4194304] 0 2024-03-22T02:28:53.983 INFO:tasks.workunit.client.0.smithi103.stdout:6/805: dwrite d0/f41 [0,4194304] 0 2024-03-22T02:28:53.990 INFO:tasks.workunit.client.0.smithi103.stdout:6/806: write d0/d88/d8e/fd1 [2171099,116464] 0 2024-03-22T02:28:53.990 INFO:tasks.workunit.client.0.smithi103.stdout:0/991: write dd/d16/d4b/d96/d64/dab/dc6/d111/fcd [4561000,113237] 0 2024-03-22T02:28:53.992 INFO:tasks.workunit.client.0.smithi103.stdout:4/992: symlink d3/d69/d12b/l14a 0 2024-03-22T02:28:53.993 INFO:tasks.workunit.client.0.smithi103.stdout:4/993: readlink d3/d31/l7b 0 2024-03-22T02:28:53.994 INFO:tasks.workunit.client.0.smithi103.stdout:9/962: rmdir d2/d5/d7b/d7d/dd4/dd8 39 2024-03-22T02:28:53.998 INFO:tasks.workunit.client.1.smithi122.stdout:2/467: sync 2024-03-22T02:28:53.998 INFO:tasks.workunit.client.1.smithi122.stdout:7/406: sync 2024-03-22T02:28:53.998 INFO:tasks.workunit.client.1.smithi122.stdout:5/378: sync 2024-03-22T02:28:53.999 INFO:tasks.workunit.client.0.smithi103.stdout:7/982: rename d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/fac to d1/d10/d11/d19/dc3/d112/d104/f137 0 2024-03-22T02:28:54.000 INFO:tasks.workunit.client.1.smithi122.stdout:8/533: dread d1/d40/d48/f6d [0,4194304] 0 2024-03-22T02:28:54.001 INFO:tasks.workunit.client.0.smithi103.stdout:6/807: write d0/d14/d57/d68/fa8 [71404,28772] 0 2024-03-22T02:28:54.001 INFO:tasks.workunit.client.0.smithi103.stdout:4/994: symlink d3/d30/d45/d107/l14b 0 2024-03-22T02:28:54.003 INFO:tasks.workunit.client.1.smithi122.stdout:8/534: write d1/d8/d8a/d78/f82 [34563,52930] 0 2024-03-22T02:28:54.003 INFO:tasks.workunit.client.0.smithi103.stdout:9/963: creat d2/d10/d43/d7e/da4/f113 x:0 0 0 2024-03-22T02:28:54.003 INFO:tasks.workunit.client.1.smithi122.stdout:9/565: dwrite db/d1b/d6c/d58/f67 [0,4194304] 0 2024-03-22T02:28:54.004 INFO:tasks.workunit.client.0.smithi103.stdout:9/964: write d2/d5/f63 [736212,44680] 0 2024-03-22T02:28:54.007 INFO:tasks.workunit.client.1.smithi122.stdout:2/468: unlink df/c91 0 2024-03-22T02:28:54.007 INFO:tasks.workunit.client.0.smithi103.stdout:0/992: rename dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/d9d to dd/d16/d4b/d96/d29/d48/d5c/d103/d131 0 2024-03-22T02:28:54.011 INFO:tasks.workunit.client.0.smithi103.stdout:4/995: creat d3/d38/d5c/d86/f14c x:0 0 0 2024-03-22T02:28:54.011 INFO:tasks.workunit.client.1.smithi122.stdout:8/535: mkdir d1/d8/d63/d88/dab/db9 0 2024-03-22T02:28:54.012 INFO:tasks.workunit.client.1.smithi122.stdout:2/469: mkdir df/d1f/d33/d4d/d94/d9e 0 2024-03-22T02:28:54.013 INFO:tasks.workunit.client.0.smithi103.stdout:9/965: creat d2/da/d18/d1e/d77/d8b/dae/f114 x:0 0 0 2024-03-22T02:28:54.014 INFO:tasks.workunit.client.1.smithi122.stdout:7/407: creat d0/f84 x:0 0 0 2024-03-22T02:28:54.015 INFO:tasks.workunit.client.1.smithi122.stdout:8/536: chown d1/d40/d71/d97/la1 4 1 2024-03-22T02:28:54.015 INFO:tasks.workunit.client.0.smithi103.stdout:7/983: mknod d1/d10/d11/d19/d29/d33/d4b/d5e/d125/c138 0 2024-03-22T02:28:54.016 INFO:tasks.workunit.client.0.smithi103.stdout:7/984: fdatasync d1/da/d1c/f10d 0 2024-03-22T02:28:54.016 INFO:tasks.workunit.client.1.smithi122.stdout:2/470: rename df/c1b to df/d1f/d2a/c9f 0 2024-03-22T02:28:54.018 INFO:tasks.workunit.client.1.smithi122.stdout:8/537: mkdir d1/d8/dba 0 2024-03-22T02:28:54.020 INFO:tasks.workunit.client.1.smithi122.stdout:7/408: link d0/f2 d0/d25/d46/f85 0 2024-03-22T02:28:54.020 INFO:tasks.workunit.client.1.smithi122.stdout:5/379: dread d8/d3b/f5a [0,4194304] 0 2024-03-22T02:28:54.021 INFO:tasks.workunit.client.1.smithi122.stdout:7/409: write d0/f84 [524747,68040] 0 2024-03-22T02:28:54.022 INFO:tasks.workunit.client.1.smithi122.stdout:2/471: mkdir df/d1f/d33/d6a/d7c/d8e/da0 0 2024-03-22T02:28:54.022 INFO:tasks.workunit.client.1.smithi122.stdout:5/380: dread - d8/d1e/d22/d3a/d46/d4f/f63 zero size 2024-03-22T02:28:54.023 INFO:tasks.workunit.client.0.smithi103.stdout:9/966: unlink d2/d10/d43/d9b/df3/cc9 0 2024-03-22T02:28:54.023 INFO:tasks.workunit.client.1.smithi122.stdout:8/538: creat d1/d40/d71/db4/fbb x:0 0 0 2024-03-22T02:28:54.024 INFO:tasks.workunit.client.1.smithi122.stdout:5/381: truncate d8/f20 141171 0 2024-03-22T02:28:54.025 INFO:tasks.workunit.client.0.smithi103.stdout:0/993: rename dd/d16/c1f to dd/c132 0 2024-03-22T02:28:54.025 INFO:tasks.workunit.client.1.smithi122.stdout:2/472: write df/d1f/d33/d4d/f78 [654082,71726] 0 2024-03-22T02:28:54.025 INFO:tasks.workunit.client.1.smithi122.stdout:7/410: unlink d0/d4/f5 0 2024-03-22T02:28:54.027 INFO:tasks.workunit.client.0.smithi103.stdout:0/994: read dd/d16/d4b/d96/d64/dab/dc6/f58 [2661536,28712] 0 2024-03-22T02:28:54.028 INFO:tasks.workunit.client.1.smithi122.stdout:8/539: unlink d1/d8/d2f/f3e 0 2024-03-22T02:28:54.032 INFO:tasks.workunit.client.1.smithi122.stdout:5/382: mknod d8/d1e/d22/d3a/d46/d4f/d55/c84 0 2024-03-22T02:28:54.032 INFO:tasks.workunit.client.1.smithi122.stdout:7/411: dwrite d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:54.033 INFO:tasks.workunit.client.0.smithi103.stdout:0/995: link dd/d16/d4b/d96/d29/d48/d5c/d103/d9a/c9f dd/d34/d53/c133 0 2024-03-22T02:28:54.034 INFO:tasks.workunit.client.1.smithi122.stdout:5/383: read d8/da/f16 [557301,107407] 0 2024-03-22T02:28:54.035 INFO:tasks.workunit.client.1.smithi122.stdout:7/412: readlink d0/d4/d20/d21/l31 0 2024-03-22T02:28:54.035 INFO:tasks.workunit.client.0.smithi103.stdout:4/996: write d3/d1e/d25/d65/d83/fcf [988691,91209] 0 2024-03-22T02:28:54.036 INFO:tasks.workunit.client.1.smithi122.stdout:9/566: dwrite db/d23/d1f/d57/d66/d73/f76 [4194304,4194304] 0 2024-03-22T02:28:54.039 INFO:tasks.workunit.client.1.smithi122.stdout:5/384: rename d8/da/l76 to d8/d1e/d22/d78/l85 0 2024-03-22T02:28:54.040 INFO:tasks.workunit.client.1.smithi122.stdout:7/413: symlink d0/d25/l86 0 2024-03-22T02:28:54.043 INFO:tasks.workunit.client.0.smithi103.stdout:0/996: dread - dd/d16/d4b/d96/d29/d48/d94/ffc zero size 2024-03-22T02:28:54.044 INFO:tasks.workunit.client.1.smithi122.stdout:5/385: truncate d8/da/d28/d35/f37 4743482 0 2024-03-22T02:28:54.045 INFO:tasks.workunit.client.1.smithi122.stdout:5/386: fdatasync d8/d3b/f52 0 2024-03-22T02:28:54.045 INFO:tasks.workunit.client.0.smithi103.stdout:6/808: dwrite d0/f69 [0,4194304] 0 2024-03-22T02:28:54.046 INFO:tasks.workunit.client.1.smithi122.stdout:7/414: dread d0/fd [0,4194304] 0 2024-03-22T02:28:54.046 INFO:tasks.workunit.client.1.smithi122.stdout:9/567: creat db/d23/d1e/dad/fb2 x:0 0 0 2024-03-22T02:28:54.049 INFO:tasks.workunit.client.0.smithi103.stdout:7/985: dwrite d1/d10/d11/d19/d86/d92/dbf/d128/f46 [0,4194304] 0 2024-03-22T02:28:54.050 INFO:tasks.workunit.client.1.smithi122.stdout:5/387: creat d8/d74/f86 x:0 0 0 2024-03-22T02:28:54.051 INFO:tasks.workunit.client.0.smithi103.stdout:9/967: dread d2/d10/d43/d7e/f9c [0,4194304] 0 2024-03-22T02:28:54.051 INFO:tasks.workunit.client.1.smithi122.stdout:5/388: stat d8/d74/c77 0 2024-03-22T02:28:54.056 INFO:tasks.workunit.client.0.smithi103.stdout:6/809: read d0/d3/f92 [1202437,10894] 0 2024-03-22T02:28:54.056 INFO:tasks.workunit.client.1.smithi122.stdout:5/389: mknod d8/da/d28/d42/c87 0 2024-03-22T02:28:54.057 INFO:tasks.workunit.client.0.smithi103.stdout:6/810: readlink d0/d14/ld5 0 2024-03-22T02:28:54.058 INFO:tasks.workunit.client.0.smithi103.stdout:4/997: truncate d3/d69/fcd 6854768 0 2024-03-22T02:28:54.058 INFO:tasks.workunit.client.1.smithi122.stdout:9/568: dwrite db/df/d3e/d81/d25/d41/f54 [4194304,4194304] 0 2024-03-22T02:28:54.060 INFO:tasks.workunit.client.0.smithi103.stdout:7/986: rename d1/d10/d11/d19/dc3/de8/l118 to d1/da/d1c/l139 0 2024-03-22T02:28:54.060 INFO:tasks.workunit.client.1.smithi122.stdout:7/415: getdents d0/d4/d22 0 2024-03-22T02:28:54.062 INFO:tasks.workunit.client.0.smithi103.stdout:9/968: chown d2/d10/d43/d9b/df3/df7/d100/l112 1473536 1 2024-03-22T02:28:54.062 INFO:tasks.workunit.client.0.smithi103.stdout:0/997: rename dd/d16/d4b/d96/d29/d48/d5c/la5 to dd/d16/d4b/d96/d29/d48/d5c/l134 0 2024-03-22T02:28:54.064 INFO:tasks.workunit.client.1.smithi122.stdout:2/473: truncate df/f15 1306922 0 2024-03-22T02:28:54.064 INFO:tasks.workunit.client.0.smithi103.stdout:6/811: mkdir d0/d88/d8e/dae/d109 0 2024-03-22T02:28:54.064 INFO:tasks.workunit.client.0.smithi103.stdout:7/987: dread - d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/ffe zero size 2024-03-22T02:28:54.064 INFO:tasks.workunit.client.0.smithi103.stdout:9/969: write d2/d10/d43/d7e/f10b [716953,27009] 0 2024-03-22T02:28:54.065 INFO:tasks.workunit.client.1.smithi122.stdout:2/474: chown df/d1f/d2a/d39/f53 11 1 2024-03-22T02:28:54.068 INFO:tasks.workunit.client.1.smithi122.stdout:9/569: unlink db/f13 0 2024-03-22T02:28:54.069 INFO:tasks.workunit.client.0.smithi103.stdout:6/812: mknod d0/d88/c10a 0 2024-03-22T02:28:54.069 INFO:tasks.workunit.client.1.smithi122.stdout:9/570: chown db/df/c96 169248 1 2024-03-22T02:28:54.070 INFO:tasks.workunit.client.1.smithi122.stdout:5/390: symlink d8/da/l88 0 2024-03-22T02:28:54.070 INFO:tasks.workunit.client.0.smithi103.stdout:7/988: mkdir d1/d6f/d13a 0 2024-03-22T02:28:54.071 INFO:tasks.workunit.client.0.smithi103.stdout:6/813: read - d0/d3/d22/d58/d6f/fe5 zero size 2024-03-22T02:28:54.071 INFO:tasks.workunit.client.0.smithi103.stdout:7/989: truncate d1/d10/f103 1589979 0 2024-03-22T02:28:54.071 INFO:tasks.workunit.client.1.smithi122.stdout:5/391: stat d8/d1e/d22/d3a/d46/d4f/c50 0 2024-03-22T02:28:54.071 INFO:tasks.workunit.client.1.smithi122.stdout:2/475: creat df/d1f/d33/d6a/d7c/d8e/fa1 x:0 0 0 2024-03-22T02:28:54.072 INFO:tasks.workunit.client.0.smithi103.stdout:4/998: rename d3/d69/d12b to d3/d1e/d25/d14d 0 2024-03-22T02:28:54.073 INFO:tasks.workunit.client.0.smithi103.stdout:9/970: creat d2/da/d18/d1e/d34/d3d/d46/f115 x:0 0 0 2024-03-22T02:28:54.073 INFO:tasks.workunit.client.1.smithi122.stdout:9/571: stat f8 0 2024-03-22T02:28:54.073 INFO:tasks.workunit.client.1.smithi122.stdout:2/476: chown df/d1f/d2a/d51/c89 6949 1 2024-03-22T02:28:54.074 INFO:tasks.workunit.client.0.smithi103.stdout:6/814: creat d0/d88/d8e/dae/f10b x:0 0 0 2024-03-22T02:28:54.074 INFO:tasks.workunit.client.0.smithi103.stdout:9/971: chown d2/d10/d43/d7e/d96/d9d/lb4 47 1 2024-03-22T02:28:54.076 INFO:tasks.workunit.client.1.smithi122.stdout:9/572: creat db/fb3 x:0 0 0 2024-03-22T02:28:54.076 INFO:tasks.workunit.client.0.smithi103.stdout:6/815: fsync d0/d3/d22/d58/d6f/d75/db0/dd7/f104 0 2024-03-22T02:28:54.077 INFO:tasks.workunit.client.0.smithi103.stdout:0/998: rename dd/d16/d120 to dd/d16/d4b/d96/d29/d48/d94/d135 0 2024-03-22T02:28:54.078 INFO:tasks.workunit.client.1.smithi122.stdout:9/573: write db/df/d3e/d81/f42 [2354082,87348] 0 2024-03-22T02:28:54.078 INFO:tasks.workunit.client.1.smithi122.stdout:2/477: truncate df/f37 3137801 0 2024-03-22T02:28:54.078 INFO:tasks.workunit.client.0.smithi103.stdout:9/972: write d2/d10/d59/da5/f104 [2479812,2739] 0 2024-03-22T02:28:54.079 INFO:tasks.workunit.client.1.smithi122.stdout:5/392: link d8/l7b d8/da/d28/d42/l89 0 2024-03-22T02:28:54.080 INFO:tasks.workunit.client.1.smithi122.stdout:2/478: mkdir df/d1f/d3a/da2 0 2024-03-22T02:28:54.081 INFO:tasks.workunit.client.1.smithi122.stdout:9/574: symlink db/df/lb4 0 2024-03-22T02:28:54.081 INFO:tasks.workunit.client.0.smithi103.stdout:7/990: symlink d1/d10/d11/d19/d29/d33/d4b/d5e/d105/d108/l13b 0 2024-03-22T02:28:54.083 INFO:tasks.workunit.client.0.smithi103.stdout:9/973: mknod d2/da/d18/d1e/c116 0 2024-03-22T02:28:54.084 INFO:tasks.workunit.client.1.smithi122.stdout:2/479: unlink df/d1f/d3a/d41/l57 0 2024-03-22T02:28:54.085 INFO:tasks.workunit.client.1.smithi122.stdout:2/480: fsync fd 0 2024-03-22T02:28:54.086 INFO:tasks.workunit.client.0.smithi103.stdout:4/999: mkdir d3/d1e/d53/d8a/def/d14e 0 2024-03-22T02:28:54.088 INFO:tasks.workunit.client.0.smithi103.stdout:6/816: creat d0/d3/d22/f10c x:0 0 0 2024-03-22T02:28:54.089 INFO:tasks.workunit.client.0.smithi103.stdout:7/991: symlink d1/da/l13c 0 2024-03-22T02:28:54.091 INFO:tasks.workunit.client.1.smithi122.stdout:9/575: dwrite db/df/d3e/d81/d25/d41/f61 [0,4194304] 0 2024-03-22T02:28:54.095 INFO:tasks.workunit.client.0.smithi103.stdout:6/817: dwrite d0/d88/f9f [0,4194304] 0 2024-03-22T02:28:54.097 INFO:tasks.workunit.client.0.smithi103.stdout:7/992: link d1/d10/d11/dd9/fec d1/d10/d11/d19/d29/d33/d130/f13d 0 2024-03-22T02:28:54.099 INFO:tasks.workunit.client.1.smithi122.stdout:9/576: rename db/df/d3e/d81/d3d/d79 to db/df/d3e/d59/db5 0 2024-03-22T02:28:54.101 INFO:tasks.workunit.client.0.smithi103.stdout:6/818: rename d0/f64 to d0/dbd/f10d 0 2024-03-22T02:28:54.101 INFO:tasks.workunit.client.1.smithi122.stdout:9/577: write db/d23/d1f/d57/f64 [696284,128490] 0 2024-03-22T02:28:54.103 INFO:tasks.workunit.client.1.smithi122.stdout:7/416: dread d0/d4/f2d [0,4194304] 0 2024-03-22T02:28:54.103 INFO:tasks.workunit.client.0.smithi103.stdout:7/993: dread d1/da/d2f/d36/f91 [0,4194304] 0 2024-03-22T02:28:54.104 INFO:tasks.workunit.client.1.smithi122.stdout:7/417: chown d0/d4/d22/d5f 1497197877 1 2024-03-22T02:28:54.105 INFO:tasks.workunit.client.1.smithi122.stdout:9/578: rmdir db/d1b 39 2024-03-22T02:28:54.106 INFO:tasks.workunit.client.0.smithi103.stdout:6/819: rename d0/d3/d33/cbb to d0/d3/d22/d58/db4/deb/c10e 0 2024-03-22T02:28:54.112 INFO:tasks.workunit.client.0.smithi103.stdout:7/994: rename d1/d10/d11/d19/d86/d92/dbf/d128/d1d/fcf to d1/d10/d11/d19/d29/f13e 0 2024-03-22T02:28:54.112 INFO:tasks.workunit.client.0.smithi103.stdout:6/820: dread d0/f2c [0,4194304] 0 2024-03-22T02:28:54.113 INFO:tasks.workunit.client.1.smithi122.stdout:9/579: creat db/df/d3e/d81/fb6 x:0 0 0 2024-03-22T02:28:54.114 INFO:tasks.workunit.client.0.smithi103.stdout:7/995: write d1/d10/d11/d19/d29/d33/f5f [2101448,95464] 0 2024-03-22T02:28:54.114 INFO:tasks.workunit.client.1.smithi122.stdout:7/418: getdents d0/d4/d22/d5f/d5e 0 2024-03-22T02:28:54.115 INFO:tasks.workunit.client.0.smithi103.stdout:7/996: chown d1/d10/d11/d19/d29/d33/d4b/f126 231 1 2024-03-22T02:28:54.116 INFO:tasks.workunit.client.1.smithi122.stdout:7/419: chown d0/d4/d20/d57/l6f 24 1 2024-03-22T02:28:54.116 INFO:tasks.workunit.client.0.smithi103.stdout:6/821: link d0/d14/c66 d0/d88/d8e/c10f 0 2024-03-22T02:28:54.117 INFO:tasks.workunit.client.1.smithi122.stdout:9/580: creat db/d1b/da8/fb7 x:0 0 0 2024-03-22T02:28:54.118 INFO:tasks.workunit.client.0.smithi103.stdout:6/822: write d0/d14/f6c [3729016,118773] 0 2024-03-22T02:28:54.119 INFO:tasks.workunit.client.1.smithi122.stdout:5/393: dwrite d8/da/d28/f2b [0,4194304] 0 2024-03-22T02:28:54.121 INFO:tasks.workunit.client.0.smithi103.stdout:6/823: rmdir d0/d88 39 2024-03-22T02:28:54.123 INFO:tasks.workunit.client.1.smithi122.stdout:9/581: creat db/d23/d1f/d57/fb8 x:0 0 0 2024-03-22T02:28:54.126 INFO:tasks.workunit.client.0.smithi103.stdout:7/997: dwrite d1/d10/d11/d19/d86/d92/dbf/d128/d69/d6b/d7a/db5/f132 [0,4194304] 0 2024-03-22T02:28:54.126 INFO:tasks.workunit.client.1.smithi122.stdout:5/394: truncate d8/da/d28/f2c 1250678 0 2024-03-22T02:28:54.127 INFO:tasks.workunit.client.1.smithi122.stdout:2/481: dwrite df/d1f/d33/f54 [0,4194304] 0 2024-03-22T02:28:54.129 INFO:tasks.workunit.client.1.smithi122.stdout:7/420: rename d0/d25/d5c/f74 to d0/f87 0 2024-03-22T02:28:54.129 INFO:tasks.workunit.client.1.smithi122.stdout:2/482: write df/d1f/d2a/f2f [3188118,106789] 0 2024-03-22T02:28:54.132 INFO:tasks.workunit.client.1.smithi122.stdout:2/483: readlink df/d1f/d2a/l61 0 2024-03-22T02:28:54.133 INFO:tasks.workunit.client.1.smithi122.stdout:7/421: dread d0/d25/f56 [0,4194304] 0 2024-03-22T02:28:54.133 INFO:tasks.workunit.client.0.smithi103.stdout:6/824: rename d0/d88/d8e/dae/cb3 to d0/d3/d22/d58/d6f/d75/db0/db1/c110 0 2024-03-22T02:28:54.136 INFO:tasks.workunit.client.0.smithi103.stdout:7/998: mkdir d1/d10/d11/d19/d29/d33/da8/dc5/d13f 0 2024-03-22T02:28:54.138 INFO:tasks.workunit.client.1.smithi122.stdout:2/484: rmdir df 39 2024-03-22T02:28:54.139 INFO:tasks.workunit.client.0.smithi103.stdout:6/825: link d0/d14/d57/d68/f40 d0/d88/d8e/dae/f111 0 2024-03-22T02:28:54.139 INFO:tasks.workunit.client.0.smithi103.stdout:7/999: creat d1/da/d2f/d36/f140 x:0 0 0 2024-03-22T02:28:54.141 INFO:tasks.workunit.client.1.smithi122.stdout:7/422: dwrite d0/d4/d20/d21/f75 [0,4194304] 0 2024-03-22T02:28:54.141 INFO:tasks.workunit.client.0.smithi103.stdout:6/826: link d0/d14/dc7/f107 d0/dbd/d103/f112 0 2024-03-22T02:28:54.144 INFO:tasks.workunit.client.1.smithi122.stdout:2/485: creat df/d1f/d9b/fa3 x:0 0 0 2024-03-22T02:28:54.147 INFO:tasks.workunit.client.1.smithi122.stdout:2/486: creat df/d1f/d3d/fa4 x:0 0 0 2024-03-22T02:28:54.148 INFO:tasks.workunit.client.1.smithi122.stdout:2/487: stat df/d1f/d33/d34/c9d 0 2024-03-22T02:28:54.149 INFO:tasks.workunit.client.0.smithi103.stdout:6/827: dwrite d0/d88/d8e/dae/f10b [0,4194304] 0 2024-03-22T02:28:54.150 INFO:tasks.workunit.client.1.smithi122.stdout:7/423: dread d0/d4e/f73 [0,4194304] 0 2024-03-22T02:28:54.150 INFO:tasks.workunit.client.1.smithi122.stdout:2/488: mknod df/d1f/d3a/d41/ca5 0 2024-03-22T02:28:54.153 INFO:tasks.workunit.client.1.smithi122.stdout:7/424: mknod d0/d4/d80/c88 0 2024-03-22T02:28:54.155 INFO:tasks.workunit.client.0.smithi103.stdout:6/828: dread d0/d3/d22/fd3 [0,4194304] 0 2024-03-22T02:28:54.156 INFO:tasks.workunit.client.1.smithi122.stdout:4/409: sync 2024-03-22T02:28:54.157 INFO:tasks.workunit.client.1.smithi122.stdout:4/410: fsync d0/d25/f8 0 2024-03-22T02:28:54.158 INFO:tasks.workunit.client.1.smithi122.stdout:2/489: truncate df/d1f/d33/d6a/f93 2595865 0 2024-03-22T02:28:54.161 INFO:tasks.workunit.client.1.smithi122.stdout:7/425: dwrite d0/f1b [0,4194304] 0 2024-03-22T02:28:54.162 INFO:tasks.workunit.client.1.smithi122.stdout:4/411: mknod d0/d12/d15/d3d/c86 0 2024-03-22T02:28:54.163 INFO:tasks.workunit.client.1.smithi122.stdout:7/426: fsync d0/d30/f53 0 2024-03-22T02:28:54.163 INFO:tasks.workunit.client.1.smithi122.stdout:4/412: fsync d0/d25/f8 0 2024-03-22T02:28:54.165 INFO:tasks.workunit.client.1.smithi122.stdout:2/490: symlink df/d1f/d3d/la6 0 2024-03-22T02:28:54.170 INFO:tasks.workunit.client.0.smithi103.stdout:6/829: rename d0/d14/d57 to d0/d3/d22/d58/d6f/d75/db0/db1/d113 0 2024-03-22T02:28:54.170 INFO:tasks.workunit.client.1.smithi122.stdout:9/582: dread db/df/f80 [0,4194304] 0 2024-03-22T02:28:54.170 INFO:tasks.workunit.client.1.smithi122.stdout:4/413: creat d0/d12/d61/f87 x:0 0 0 2024-03-22T02:28:54.171 INFO:tasks.workunit.client.1.smithi122.stdout:5/395: dread d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:28:54.171 INFO:tasks.workunit.client.1.smithi122.stdout:5/396: stat d8/f26 0 2024-03-22T02:28:54.172 INFO:tasks.workunit.client.1.smithi122.stdout:4/414: write d0/d12/f21 [2480373,90925] 0 2024-03-22T02:28:54.173 INFO:tasks.workunit.client.1.smithi122.stdout:5/397: write d8/da/d28/d35/f5d [271471,102470] 0 2024-03-22T02:28:54.175 INFO:tasks.workunit.client.1.smithi122.stdout:4/415: write d0/d12/d15/d2d/f38 [308579,121943] 0 2024-03-22T02:28:54.176 INFO:tasks.workunit.client.0.smithi103.stdout:6/830: dwrite d0/d3/d33/fad [0,4194304] 0 2024-03-22T02:28:54.179 INFO:tasks.workunit.client.0.smithi103.stdout:6/831: creat d0/d14/dd2/f114 x:0 0 0 2024-03-22T02:28:54.180 INFO:tasks.workunit.client.0.smithi103.stdout:6/832: fsync d0/d3/f6d 0 2024-03-22T02:28:54.181 INFO:tasks.workunit.client.1.smithi122.stdout:5/398: symlink d8/d1e/d22/d3a/d46/l8a 0 2024-03-22T02:28:54.184 INFO:tasks.workunit.client.1.smithi122.stdout:9/583: link db/l46 db/d23/d1f/d57/d66/d73/lb9 0 2024-03-22T02:28:54.185 INFO:tasks.workunit.client.0.smithi103.stdout:6/833: symlink d0/d14/dc7/l115 0 2024-03-22T02:28:54.186 INFO:tasks.workunit.client.1.smithi122.stdout:4/416: symlink d0/d12/d15/d42/d57/l88 0 2024-03-22T02:28:54.186 INFO:tasks.workunit.client.1.smithi122.stdout:2/491: write df/d1f/d3a/d4f/f68 [290483,102155] 0 2024-03-22T02:28:54.186 INFO:tasks.workunit.client.1.smithi122.stdout:7/427: write d0/f87 [459876,105513] 0 2024-03-22T02:28:54.188 INFO:tasks.workunit.client.1.smithi122.stdout:5/399: rename d8/d1e/d22/d3a/d39/f6b to d8/d1e/d22/d3a/d39/f8b 0 2024-03-22T02:28:54.189 INFO:tasks.workunit.client.1.smithi122.stdout:9/584: mkdir db/d1b/da8/dba 0 2024-03-22T02:28:54.191 INFO:tasks.workunit.client.1.smithi122.stdout:2/492: creat df/d1f/d9b/fa7 x:0 0 0 2024-03-22T02:28:54.192 INFO:tasks.workunit.client.1.smithi122.stdout:5/400: dread d8/f30 [0,4194304] 0 2024-03-22T02:28:54.194 INFO:tasks.workunit.client.1.smithi122.stdout:7/428: rename d0/d4/c3c to d0/d4/d20/d21/d28/c89 0 2024-03-22T02:28:54.195 INFO:tasks.workunit.client.1.smithi122.stdout:5/401: chown d8/c9 1 1 2024-03-22T02:28:54.198 INFO:tasks.workunit.client.1.smithi122.stdout:7/429: symlink d0/d25/d46/l8a 0 2024-03-22T02:28:54.205 INFO:tasks.workunit.client.1.smithi122.stdout:4/417: dread d0/d25/f28 [0,4194304] 0 2024-03-22T02:28:54.207 INFO:tasks.workunit.client.1.smithi122.stdout:4/418: creat d0/d12/d15/d42/f89 x:0 0 0 2024-03-22T02:28:54.209 INFO:tasks.workunit.client.1.smithi122.stdout:4/419: creat d0/d12/d15/d2d/d75/f8a x:0 0 0 2024-03-22T02:28:54.214 INFO:tasks.workunit.client.1.smithi122.stdout:4/420: dwrite d0/d12/d61/f74 [0,4194304] 0 2024-03-22T02:28:54.286 INFO:tasks.workunit.client.0.smithi103.stdout:6/834: dread d0/d14/f4c [0,4194304] 0 2024-03-22T02:28:54.289 INFO:tasks.workunit.client.0.smithi103.stdout:6/835: symlink d0/d14/dc7/l116 0 2024-03-22T02:28:54.291 INFO:tasks.workunit.client.1.smithi122.stdout:2/493: write df/f13 [2061665,69162] 0 2024-03-22T02:28:54.292 INFO:tasks.workunit.client.1.smithi122.stdout:5/402: truncate d8/d3b/f5a 120151 0 2024-03-22T02:28:54.293 INFO:tasks.workunit.client.0.smithi103.stdout:6/836: creat d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f117 x:0 0 0 2024-03-22T02:28:54.293 INFO:tasks.workunit.client.1.smithi122.stdout:2/494: mknod df/ca8 0 2024-03-22T02:28:54.295 INFO:tasks.workunit.client.1.smithi122.stdout:5/403: mkdir d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c 0 2024-03-22T02:28:54.296 INFO:tasks.workunit.client.1.smithi122.stdout:2/495: rmdir df/d1f/d3a/d41 39 2024-03-22T02:28:54.297 INFO:tasks.workunit.client.0.smithi103.stdout:6/837: dread d0/d3/d22/d58/fd9 [0,4194304] 0 2024-03-22T02:28:54.299 INFO:tasks.workunit.client.1.smithi122.stdout:5/404: creat d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d x:0 0 0 2024-03-22T02:28:54.300 INFO:tasks.workunit.client.1.smithi122.stdout:4/421: dwrite d0/d12/d15/d42/d4e/f23 [0,4194304] 0 2024-03-22T02:28:54.300 INFO:tasks.workunit.client.0.smithi103.stdout:6/838: stat d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/l4e 0 2024-03-22T02:28:54.301 INFO:tasks.workunit.client.1.smithi122.stdout:2/496: creat df/d1f/d33/d4d/d94/d9e/fa9 x:0 0 0 2024-03-22T02:28:54.301 INFO:tasks.workunit.client.1.smithi122.stdout:4/422: write d0/d25/f27 [1240785,55695] 0 2024-03-22T02:28:54.303 INFO:tasks.workunit.client.0.smithi103.stdout:6/839: creat d0/d3/d22/d58/d6f/d75/db0/f118 x:0 0 0 2024-03-22T02:28:54.305 INFO:tasks.workunit.client.0.smithi103.stdout:6/840: dread - d0/d3/d22/d58/d6f/d75/db0/db1/fc5 zero size 2024-03-22T02:28:54.305 INFO:tasks.workunit.client.1.smithi122.stdout:4/423: read d0/d12/d15/f55 [459129,39776] 0 2024-03-22T02:28:54.305 INFO:tasks.workunit.client.0.smithi103.stdout:6/841: chown d0/dbd 882994 1 2024-03-22T02:28:54.306 INFO:tasks.workunit.client.1.smithi122.stdout:2/497: symlink df/d1f/d33/d6a/laa 0 2024-03-22T02:28:54.307 INFO:tasks.workunit.client.0.smithi103.stdout:6/842: mkdir d0/d14/d9a/d119 0 2024-03-22T02:28:54.309 INFO:tasks.workunit.client.1.smithi122.stdout:2/498: mknod df/d1f/d2a/d51/cab 0 2024-03-22T02:28:54.310 INFO:tasks.workunit.client.0.smithi103.stdout:6/843: mkdir d0/d14/d9a/d11a 0 2024-03-22T02:28:54.310 INFO:tasks.workunit.client.1.smithi122.stdout:4/424: link d0/d41/f78 d0/d12/d15/d42/d4e/de/f8b 0 2024-03-22T02:28:54.311 INFO:tasks.workunit.client.1.smithi122.stdout:4/425: chown d0/d25/f1d 412999170 1 2024-03-22T02:28:54.322 INFO:tasks.workunit.client.1.smithi122.stdout:4/426: rmdir d0/d12/d15/d42/d4e/de 39 2024-03-22T02:28:54.322 INFO:tasks.workunit.client.0.smithi103.stdout:6/844: dread d0/f3e [0,4194304] 0 2024-03-22T02:28:54.323 INFO:tasks.workunit.client.0.smithi103.stdout:6/845: write d0/d3/d22/f15 [2154061,53063] 0 2024-03-22T02:28:54.324 INFO:tasks.workunit.client.1.smithi122.stdout:5/405: dwrite d8/da/d28/d35/f4c [0,4194304] 0 2024-03-22T02:28:54.329 INFO:tasks.workunit.client.1.smithi122.stdout:4/427: chown d0/d12/d15/d42/d4e/de/l45 10213 1 2024-03-22T02:28:54.329 INFO:tasks.workunit.client.0.smithi103.stdout:6/846: write d0/f5b [993199,73247] 0 2024-03-22T02:28:54.331 INFO:tasks.workunit.client.0.smithi103.stdout:6/847: chown d0/d3/d22/d58/d6f/d75/lb6 865 1 2024-03-22T02:28:54.332 INFO:tasks.workunit.client.1.smithi122.stdout:2/499: dread df/f37 [0,4194304] 0 2024-03-22T02:28:54.335 INFO:tasks.workunit.client.1.smithi122.stdout:5/406: getdents d8/d1e/d22/d3a/d46/d4f/d55/d6a 0 2024-03-22T02:28:54.338 INFO:tasks.workunit.client.1.smithi122.stdout:5/407: unlink d8/d1e/d22/c3c 0 2024-03-22T02:28:54.342 INFO:tasks.workunit.client.0.smithi103.stdout:6/848: write d0/d88/f8a [459963,25462] 0 2024-03-22T02:28:54.345 INFO:tasks.workunit.client.1.smithi122.stdout:5/408: dwrite d8/d1e/d22/d3a/d46/f64 [0,4194304] 0 2024-03-22T02:28:54.347 INFO:tasks.workunit.client.0.smithi103.stdout:6/849: dread d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/fa8 [0,4194304] 0 2024-03-22T02:28:54.349 INFO:tasks.workunit.client.1.smithi122.stdout:6/540: sync 2024-03-22T02:28:54.349 INFO:tasks.workunit.client.1.smithi122.stdout:5/409: mkdir d8/da/d28/d42/d8e 0 2024-03-22T02:28:54.351 INFO:tasks.workunit.client.1.smithi122.stdout:5/410: rmdir d8/d3b 39 2024-03-22T02:28:54.353 INFO:tasks.workunit.client.1.smithi122.stdout:4/428: dread d0/f9 [4194304,4194304] 0 2024-03-22T02:28:54.354 INFO:tasks.workunit.client.1.smithi122.stdout:6/541: rename d7/c2a to d7/de/dd/d20/d2d/d54/d76/ca9 0 2024-03-22T02:28:54.354 INFO:tasks.workunit.client.1.smithi122.stdout:5/411: dread d8/f30 [0,4194304] 0 2024-03-22T02:28:54.355 INFO:tasks.workunit.client.1.smithi122.stdout:6/542: stat d7/de/d27/c49 0 2024-03-22T02:28:54.355 INFO:tasks.workunit.client.1.smithi122.stdout:2/500: dread df/d1f/d2a/f2f [0,4194304] 0 2024-03-22T02:28:54.356 INFO:tasks.workunit.client.0.smithi103.stdout:6/850: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f117 [0,4194304] 0 2024-03-22T02:28:54.358 INFO:tasks.workunit.client.1.smithi122.stdout:6/543: write d7/de/dd/d20/d2d/d54/d76/f6b [3804092,111106] 0 2024-03-22T02:28:54.360 INFO:tasks.workunit.client.1.smithi122.stdout:4/429: truncate d0/d12/d15/d42/d57/f64 1619529 0 2024-03-22T02:28:54.364 INFO:tasks.workunit.client.1.smithi122.stdout:6/544: symlink d7/de/d27/d52/laa 0 2024-03-22T02:28:54.365 INFO:tasks.workunit.client.1.smithi122.stdout:5/412: link d8/d74/c77 d8/da/d28/d35/d71/c8f 0 2024-03-22T02:28:54.367 INFO:tasks.workunit.client.1.smithi122.stdout:2/501: dwrite df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:54.368 INFO:tasks.workunit.client.1.smithi122.stdout:6/545: rmdir d7/de/dd/df 39 2024-03-22T02:28:54.374 INFO:tasks.workunit.client.1.smithi122.stdout:4/430: rename d0/d12/f51 to d0/d12/d15/f8c 0 2024-03-22T02:28:54.376 INFO:tasks.workunit.client.0.smithi103.stdout:6/851: dread d0/d3/d22/f3c [0,4194304] 0 2024-03-22T02:28:54.383 INFO:tasks.workunit.client.1.smithi122.stdout:6/546: dwrite d7/de/dd/d39/fa2 [0,4194304] 0 2024-03-22T02:28:54.392 INFO:tasks.workunit.client.0.smithi103.stdout:6/852: dwrite d0/d3/f42 [0,4194304] 0 2024-03-22T02:28:54.393 INFO:tasks.workunit.client.0.smithi103.stdout:6/853: dread - d0/d3/d22/d58/de0/ffc zero size 2024-03-22T02:28:54.394 INFO:tasks.workunit.client.0.smithi103.stdout:6/854: dread - d0/d14/dc7/f107 zero size 2024-03-22T02:28:54.396 INFO:tasks.workunit.client.1.smithi122.stdout:5/413: dread d8/d1a/f34 [0,4194304] 0 2024-03-22T02:28:54.405 INFO:tasks.workunit.client.0.smithi103.stdout:6/855: write d0/d14/f63 [106453,75846] 0 2024-03-22T02:28:54.406 INFO:tasks.workunit.client.0.smithi103.stdout:6/856: fsync d0/d14/dd2/f114 0 2024-03-22T02:28:54.407 INFO:tasks.workunit.client.0.smithi103.stdout:6/857: write d0/d3/d22/d58/d6f/f8b [1955253,77610] 0 2024-03-22T02:28:54.439 INFO:tasks.workunit.client.1.smithi122.stdout:2/502: symlink df/d1f/d33/d4d/lac 0 2024-03-22T02:28:54.440 INFO:tasks.workunit.client.1.smithi122.stdout:2/503: chown df/d1f/d3d/d52/l73 882498434 1 2024-03-22T02:28:54.443 INFO:tasks.workunit.client.1.smithi122.stdout:5/414: stat d8/d1e/d22/d78/l85 0 2024-03-22T02:28:54.447 INFO:tasks.workunit.client.1.smithi122.stdout:4/431: mknod d0/d12/d15/d42/d4e/c8d 0 2024-03-22T02:28:54.447 INFO:tasks.workunit.client.1.smithi122.stdout:2/504: dwrite df/d1f/d3d/f92 [0,4194304] 0 2024-03-22T02:28:54.449 INFO:tasks.workunit.client.1.smithi122.stdout:2/505: write df/d1f/d3a/f58 [1405404,27164] 0 2024-03-22T02:28:54.450 INFO:tasks.workunit.client.1.smithi122.stdout:2/506: chown df/d1f/d2a/c2c 0 1 2024-03-22T02:28:54.451 INFO:tasks.workunit.client.1.smithi122.stdout:2/507: chown df/l1c 607218 1 2024-03-22T02:28:54.454 INFO:tasks.workunit.client.1.smithi122.stdout:2/508: chown df/d1f/d2a/d39/f62 2271 1 2024-03-22T02:28:54.457 INFO:tasks.workunit.client.1.smithi122.stdout:4/432: dwrite d0/d25/f8 [4194304,4194304] 0 2024-03-22T02:28:54.457 INFO:tasks.workunit.client.1.smithi122.stdout:5/415: readlink d8/l36 0 2024-03-22T02:28:54.463 INFO:tasks.workunit.client.1.smithi122.stdout:4/433: write d0/d12/d15/d42/d4e/f4a [3716040,46908] 0 2024-03-22T02:28:54.468 INFO:tasks.workunit.client.1.smithi122.stdout:2/509: link df/l1c df/d1f/d3a/da2/lad 0 2024-03-22T02:28:54.471 INFO:tasks.workunit.client.1.smithi122.stdout:4/434: creat d0/d12/d15/d42/d57/f8e x:0 0 0 2024-03-22T02:28:54.473 INFO:tasks.workunit.client.1.smithi122.stdout:5/416: rename d8/da/d28/d35/d71/c8f to d8/d1e/d22/d3a/d39/c90 0 2024-03-22T02:28:54.474 INFO:tasks.workunit.client.1.smithi122.stdout:5/417: dread - d8/d1e/d22/d3a/f7d zero size 2024-03-22T02:28:54.475 INFO:tasks.workunit.client.1.smithi122.stdout:5/418: truncate d8/d1e/f5e 2098892 0 2024-03-22T02:28:54.478 INFO:tasks.workunit.client.1.smithi122.stdout:2/510: truncate df/d1f/d3a/d41/f47 1539834 0 2024-03-22T02:28:54.482 INFO:tasks.workunit.client.1.smithi122.stdout:5/419: mknod d8/d1e/d22/d3a/d46/d4f/c91 0 2024-03-22T02:28:54.487 INFO:tasks.workunit.client.1.smithi122.stdout:5/420: unlink d8/f30 0 2024-03-22T02:28:54.491 INFO:tasks.workunit.client.1.smithi122.stdout:5/421: unlink d8/l15 0 2024-03-22T02:28:54.495 INFO:tasks.workunit.client.1.smithi122.stdout:5/422: link d8/da/d28/d35/d71/f72 d8/da/d28/d42/d4e/f92 0 2024-03-22T02:28:54.500 INFO:tasks.workunit.client.1.smithi122.stdout:0/398: sync 2024-03-22T02:28:54.501 INFO:tasks.workunit.client.1.smithi122.stdout:0/399: chown d11/d47/l49 1595256 1 2024-03-22T02:28:54.502 INFO:tasks.workunit.client.1.smithi122.stdout:5/423: dwrite d8/da/d28/d42/f7f [0,4194304] 0 2024-03-22T02:28:54.505 INFO:tasks.workunit.client.1.smithi122.stdout:0/400: rename d11/d16/d44/d4c to d11/d16/d44/d89 0 2024-03-22T02:28:54.515 INFO:tasks.workunit.client.0.smithi103.stdout:9/974: sync 2024-03-22T02:28:54.516 INFO:tasks.workunit.client.0.smithi103.stdout:0/999: sync 2024-03-22T02:28:54.521 INFO:tasks.workunit.client.0.smithi103.stdout:9/975: dwrite d2/d10/d43/d7e/d96/d9d/f7c [0,4194304] 0 2024-03-22T02:28:54.523 INFO:tasks.workunit.client.0.smithi103.stdout:9/976: chown d2/d10/d43/d9b/l10c 379324 1 2024-03-22T02:28:54.523 INFO:tasks.workunit.client.0.smithi103.stdout:9/977: stat d2/d5/d7b/cbb 0 2024-03-22T02:28:54.525 INFO:tasks.workunit.client.0.smithi103.stdout:9/978: mknod d2/da/d18/d1e/d77/d8b/c117 0 2024-03-22T02:28:54.526 INFO:tasks.workunit.client.0.smithi103.stdout:9/979: write d2/d10/d43/d7e/fc8 [3012441,51665] 0 2024-03-22T02:28:54.528 INFO:tasks.workunit.client.0.smithi103.stdout:9/980: mknod d2/da/d18/d1e/d34/d3d/d61/db7/de8/c118 0 2024-03-22T02:28:54.530 INFO:tasks.workunit.client.0.smithi103.stdout:9/981: unlink d2/d10/d59/da5/f104 0 2024-03-22T02:28:54.533 INFO:tasks.workunit.client.0.smithi103.stdout:9/982: getdents d2/d10/d43/d7e/d96 0 2024-03-22T02:28:54.553 INFO:tasks.workunit.client.0.smithi103.stdout:9/983: dwrite d2/d10/d43/d7e/f9c [0,4194304] 0 2024-03-22T02:28:54.554 INFO:tasks.workunit.client.0.smithi103.stdout:9/984: write d2/d10/d43/d7e/fc8 [1955174,51185] 0 2024-03-22T02:28:54.557 INFO:tasks.workunit.client.0.smithi103.stdout:9/985: rename d2/d5/c69 to d2/d10/d59/da5/c119 0 2024-03-22T02:28:54.565 INFO:tasks.workunit.client.1.smithi122.stdout:2/511: symlink df/d1f/d2a/lae 0 2024-03-22T02:28:54.572 INFO:tasks.workunit.client.1.smithi122.stdout:2/512: dwrite df/d1f/d9b/fa3 [0,4194304] 0 2024-03-22T02:28:54.576 INFO:tasks.workunit.client.1.smithi122.stdout:2/513: creat df/d1f/d23/faf x:0 0 0 2024-03-22T02:28:54.581 INFO:tasks.workunit.client.1.smithi122.stdout:5/424: symlink d8/d1e/d22/d3a/l93 0 2024-03-22T02:28:54.582 INFO:tasks.workunit.client.1.smithi122.stdout:0/401: write d11/d14/d1d/d27/d2f/d4e/f66 [1416658,117581] 0 2024-03-22T02:28:54.584 INFO:tasks.workunit.client.1.smithi122.stdout:5/425: truncate d8/da/d28/d35/f68 5220970 0 2024-03-22T02:28:54.584 INFO:tasks.workunit.client.1.smithi122.stdout:0/402: chown d11/d16/d44/d89/c36 731 1 2024-03-22T02:28:54.584 INFO:tasks.workunit.client.1.smithi122.stdout:0/403: stat fb 0 2024-03-22T02:28:54.587 INFO:tasks.workunit.client.1.smithi122.stdout:0/404: write d11/d14/d1d/d27/d2f/f3a [223300,12148] 0 2024-03-22T02:28:54.591 INFO:tasks.workunit.client.1.smithi122.stdout:0/405: rename d11/d47/d5d to d11/d14/d1d/d8a 0 2024-03-22T02:28:54.594 INFO:tasks.workunit.client.1.smithi122.stdout:2/514: dwrite df/d1f/f3c [4194304,4194304] 0 2024-03-22T02:28:54.595 INFO:tasks.workunit.client.0.smithi103.stdout:9/986: dread d2/d5/d4c/f5c [0,4194304] 0 2024-03-22T02:28:54.596 INFO:tasks.workunit.client.0.smithi103.stdout:9/987: chown d2/d10/d43/d7e/d96/d9d 1100 1 2024-03-22T02:28:54.597 INFO:tasks.workunit.client.1.smithi122.stdout:0/406: rename d11/d14/d23/d24/d33/d67/d85 to d11/d14/d23/d24/d8b 0 2024-03-22T02:28:54.602 INFO:tasks.workunit.client.0.smithi103.stdout:6/858: sync 2024-03-22T02:28:54.602 INFO:tasks.workunit.client.1.smithi122.stdout:2/515: dwrite df/d1f/f9c [0,4194304] 0 2024-03-22T02:28:54.605 INFO:tasks.workunit.client.0.smithi103.stdout:6/859: mknod d0/d3/d22/d58/dd8/c11b 0 2024-03-22T02:28:54.609 INFO:tasks.workunit.client.0.smithi103.stdout:6/860: rename d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/l98 to d0/d14/dc7/dfa/l11c 0 2024-03-22T02:28:54.609 INFO:tasks.workunit.client.1.smithi122.stdout:2/516: dread df/d1f/d2a/d51/d67/f6e [4194304,4194304] 0 2024-03-22T02:28:54.610 INFO:tasks.workunit.client.1.smithi122.stdout:5/426: dread d8/da/f10 [0,4194304] 0 2024-03-22T02:28:54.610 INFO:tasks.workunit.client.1.smithi122.stdout:2/517: write df/d1f/d3d/f92 [2436606,37471] 0 2024-03-22T02:28:54.611 INFO:tasks.workunit.client.0.smithi103.stdout:6/861: mkdir d0/d3/d33/d11d 0 2024-03-22T02:28:54.612 INFO:tasks.workunit.client.1.smithi122.stdout:5/427: write d8/d1a/f48 [1021453,87068] 0 2024-03-22T02:28:54.612 INFO:tasks.workunit.client.1.smithi122.stdout:2/518: fdatasync df/d1f/d33/d34/f43 0 2024-03-22T02:28:54.618 INFO:tasks.workunit.client.1.smithi122.stdout:5/428: dwrite d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:28:54.621 INFO:tasks.workunit.client.1.smithi122.stdout:5/429: creat d8/da/d28/d35/f94 x:0 0 0 2024-03-22T02:28:54.623 INFO:tasks.workunit.client.1.smithi122.stdout:5/430: unlink d8/da/d28/d42/c87 0 2024-03-22T02:28:54.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:54.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:54.625 INFO:tasks.workunit.client.1.smithi122.stdout:5/431: getdents d8/da/d28 0 2024-03-22T02:28:54.626 INFO:tasks.workunit.client.1.smithi122.stdout:5/432: read - d8/d1e/d22/d3a/d46/d4f/f63 zero size 2024-03-22T02:28:54.627 INFO:tasks.workunit.client.0.smithi103.stdout:6/862: dread d0/d88/d8e/dae/f101 [0,4194304] 0 2024-03-22T02:28:54.630 INFO:tasks.workunit.client.0.smithi103.stdout:6/863: getdents d0/d3/d22/d58/d6f/d75 0 2024-03-22T02:28:54.631 INFO:tasks.workunit.client.1.smithi122.stdout:5/433: rename d8/d1a/f48 to d8/da/d28/d42/f95 0 2024-03-22T02:28:54.631 INFO:tasks.workunit.client.0.smithi103.stdout:6/864: dread - d0/d3/d22/d58/de0/f105 zero size 2024-03-22T02:28:54.633 INFO:tasks.workunit.client.0.smithi103.stdout:6/865: mknod d0/d3/d22/d58/d6f/d75/daa/c11e 0 2024-03-22T02:28:54.635 INFO:tasks.workunit.client.0.smithi103.stdout:6/866: dread - d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/fff zero size 2024-03-22T02:28:54.635 INFO:tasks.workunit.client.1.smithi122.stdout:5/434: rename d8/da/d28/f2b to d8/d1e/d22/d3a/d39/f96 0 2024-03-22T02:28:54.635 INFO:tasks.workunit.client.0.smithi103.stdout:6/867: fsync d0/dbd/fea 0 2024-03-22T02:28:54.637 INFO:tasks.workunit.client.1.smithi122.stdout:5/435: rename d8/d1e/d22/d3a/l93 to d8/d1e/d51/l97 0 2024-03-22T02:28:54.638 INFO:tasks.workunit.client.0.smithi103.stdout:6/868: symlink d0/d3/d22/l11f 0 2024-03-22T02:28:54.640 INFO:tasks.workunit.client.0.smithi103.stdout:6/869: creat d0/d3/d22/d58/db4/deb/f120 x:0 0 0 2024-03-22T02:28:54.640 INFO:tasks.workunit.client.1.smithi122.stdout:0/407: dread d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:54.642 INFO:tasks.workunit.client.1.smithi122.stdout:0/408: fdatasync d11/d14/d23/d24/d33/f3f 0 2024-03-22T02:28:54.642 INFO:tasks.workunit.client.1.smithi122.stdout:0/409: fsync d11/d16/f55 0 2024-03-22T02:28:54.643 INFO:tasks.workunit.client.0.smithi103.stdout:6/870: dread d0/d3/f42 [0,4194304] 0 2024-03-22T02:28:54.644 INFO:tasks.workunit.client.1.smithi122.stdout:0/410: symlink d11/d4f/l8c 0 2024-03-22T02:28:54.645 INFO:tasks.workunit.client.1.smithi122.stdout:0/411: truncate d11/d14/d1d/f6a 4304553 0 2024-03-22T02:28:54.645 INFO:tasks.workunit.client.0.smithi103.stdout:6/871: unlink d0/d3/d22/d58/dd8/cef 0 2024-03-22T02:28:54.646 INFO:tasks.workunit.client.0.smithi103.stdout:6/872: stat d0/d3/d22/fab 0 2024-03-22T02:28:54.648 INFO:tasks.workunit.client.0.smithi103.stdout:6/873: unlink d0/d3/d33/l4d 0 2024-03-22T02:28:54.653 INFO:tasks.workunit.client.0.smithi103.stdout:6/874: link d0/d88/d8e/dae/lc0 d0/d14/d9a/d119/l121 0 2024-03-22T02:28:54.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/412: dwrite d11/d14/d23/d24/d33/f79 [0,4194304] 0 2024-03-22T02:28:54.654 INFO:tasks.workunit.client.0.smithi103.stdout:6/875: mknod d0/d3/d33/d11d/c122 0 2024-03-22T02:28:54.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/413: stat d11/d4f/d5a/f5c 0 2024-03-22T02:28:54.656 INFO:tasks.workunit.client.0.smithi103.stdout:6/876: creat d0/d14/dd2/f123 x:0 0 0 2024-03-22T02:28:54.656 INFO:tasks.workunit.client.1.smithi122.stdout:5/436: chown d8/da/d28/d42/d4e 234853215 1 2024-03-22T02:28:54.656 INFO:tasks.workunit.client.1.smithi122.stdout:2/519: dread df/d1f/d3a/d41/f47 [0,4194304] 0 2024-03-22T02:28:54.657 INFO:tasks.workunit.client.0.smithi103.stdout:6/877: fdatasync d0/d3/fbc 0 2024-03-22T02:28:54.657 INFO:tasks.workunit.client.1.smithi122.stdout:2/520: readlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:54.658 INFO:tasks.workunit.client.1.smithi122.stdout:0/414: dread d11/d14/d1d/f80 [0,4194304] 0 2024-03-22T02:28:54.660 INFO:tasks.workunit.client.0.smithi103.stdout:6/878: creat d0/d14/dc7/dce/f124 x:0 0 0 2024-03-22T02:28:54.660 INFO:tasks.workunit.client.1.smithi122.stdout:2/521: mkdir df/d1f/d2a/d51/d67/db0 0 2024-03-22T02:28:54.661 INFO:tasks.workunit.client.1.smithi122.stdout:0/415: dread d11/d14/d1b/f3c [0,4194304] 0 2024-03-22T02:28:54.661 INFO:tasks.workunit.client.0.smithi103.stdout:6/879: creat d0/d14/dc7/f125 x:0 0 0 2024-03-22T02:28:54.662 INFO:tasks.workunit.client.0.smithi103.stdout:6/880: fsync d0/d3/d22/d58/d6f/f8b 0 2024-03-22T02:28:54.665 INFO:tasks.workunit.client.0.smithi103.stdout:6/881: creat d0/d14/f126 x:0 0 0 2024-03-22T02:28:54.665 INFO:tasks.workunit.client.1.smithi122.stdout:2/522: dwrite df/d1f/f5c [4194304,4194304] 0 2024-03-22T02:28:54.666 INFO:tasks.workunit.client.0.smithi103.stdout:6/882: write d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/fff [1037994,27765] 0 2024-03-22T02:28:54.667 INFO:tasks.workunit.client.1.smithi122.stdout:2/523: readlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:54.670 INFO:tasks.workunit.client.1.smithi122.stdout:2/524: dread df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:54.676 INFO:tasks.workunit.client.1.smithi122.stdout:2/525: dwrite df/d1f/d33/d4d/f78 [0,4194304] 0 2024-03-22T02:28:54.677 INFO:tasks.workunit.client.1.smithi122.stdout:2/526: dread - df/d1f/d23/f86 zero size 2024-03-22T02:28:54.682 INFO:tasks.workunit.client.1.smithi122.stdout:2/527: rename df/d1f/d23/c45 to df/d1f/d2a/d39/cb1 0 2024-03-22T02:28:54.683 INFO:tasks.workunit.client.1.smithi122.stdout:2/528: write df/d1f/d33/f54 [2125556,91087] 0 2024-03-22T02:28:54.684 INFO:tasks.workunit.client.1.smithi122.stdout:2/529: stat df/d1f/d2a/d51/d67/db0 0 2024-03-22T02:28:54.687 INFO:tasks.workunit.client.1.smithi122.stdout:2/530: getdents df/d1f/d3d 0 2024-03-22T02:28:54.688 INFO:tasks.workunit.client.1.smithi122.stdout:2/531: write df/d1f/f5c [1999669,108149] 0 2024-03-22T02:28:54.692 INFO:tasks.workunit.client.0.smithi103.stdout:6/883: dread d0/d3/d22/fab [0,4194304] 0 2024-03-22T02:28:54.694 INFO:tasks.workunit.client.1.smithi122.stdout:2/532: dwrite df/f13 [0,4194304] 0 2024-03-22T02:28:54.697 INFO:tasks.workunit.client.1.smithi122.stdout:2/533: creat df/d1f/d33/d34/fb2 x:0 0 0 2024-03-22T02:28:54.698 INFO:tasks.workunit.client.0.smithi103.stdout:6/884: dwrite d0/d3/d22/d58/db4/deb/f120 [0,4194304] 0 2024-03-22T02:28:54.703 INFO:tasks.workunit.client.1.smithi122.stdout:2/534: dread fd [0,4194304] 0 2024-03-22T02:28:54.703 INFO:tasks.workunit.client.1.smithi122.stdout:2/535: dread - df/d1f/d33/d4d/f77 zero size 2024-03-22T02:28:54.703 INFO:tasks.workunit.client.0.smithi103.stdout:6/885: mkdir d0/d88/d8e/dae/d127 0 2024-03-22T02:28:54.705 INFO:tasks.workunit.client.0.smithi103.stdout:6/886: truncate d0/d14/dd2/f114 46589 0 2024-03-22T02:28:54.706 INFO:tasks.workunit.client.0.smithi103.stdout:9/988: sync 2024-03-22T02:28:54.708 INFO:tasks.workunit.client.0.smithi103.stdout:6/887: symlink d0/d88/d8e/dae/d109/l128 0 2024-03-22T02:28:54.708 INFO:tasks.workunit.client.1.smithi122.stdout:2/536: dread df/d1f/d33/f54 [0,4194304] 0 2024-03-22T02:28:54.709 INFO:tasks.workunit.client.0.smithi103.stdout:6/888: chown d0/d88/d8e/dae/lcd 2010 1 2024-03-22T02:28:54.710 INFO:tasks.workunit.client.1.smithi122.stdout:2/537: readlink df/d1f/d3a/d41/l4c 0 2024-03-22T02:28:54.710 INFO:tasks.workunit.client.0.smithi103.stdout:9/989: symlink d2/d10/d43/d7e/da4/dcd/l11a 0 2024-03-22T02:28:54.711 INFO:tasks.workunit.client.0.smithi103.stdout:9/990: readlink d2/la7 0 2024-03-22T02:28:54.711 INFO:tasks.workunit.client.0.smithi103.stdout:9/991: chown d2/d5/d4c 3282270 1 2024-03-22T02:28:54.711 INFO:tasks.workunit.client.0.smithi103.stdout:6/889: symlink d0/d88/l129 0 2024-03-22T02:28:54.712 INFO:tasks.workunit.client.1.smithi122.stdout:2/538: dread df/d1f/d33/d6a/f93 [0,4194304] 0 2024-03-22T02:28:54.715 INFO:tasks.workunit.client.1.smithi122.stdout:2/539: rename df/d1f/d3d/c55 to df/d1f/d3a/d4f/d84/cb3 0 2024-03-22T02:28:54.717 INFO:tasks.workunit.client.1.smithi122.stdout:2/540: symlink df/d1f/d2a/d51/d67/lb4 0 2024-03-22T02:28:54.719 INFO:tasks.workunit.client.0.smithi103.stdout:6/890: dwrite d0/d3/d22/d58/d6f/f8b [0,4194304] 0 2024-03-22T02:28:54.719 INFO:tasks.workunit.client.1.smithi122.stdout:2/541: rename df/d1f/d2a/d39 to df/d1f/d2a/d51/d67/db5 0 2024-03-22T02:28:54.720 INFO:tasks.workunit.client.1.smithi122.stdout:2/542: mknod df/d1f/d2a/d51/cb6 0 2024-03-22T02:28:54.721 INFO:tasks.workunit.client.1.smithi122.stdout:2/543: stat df/d1f/f20 0 2024-03-22T02:28:54.722 INFO:tasks.workunit.client.0.smithi103.stdout:6/891: symlink d0/d3/d22/d58/d6f/d75/db0/db1/l12a 0 2024-03-22T02:28:54.725 INFO:tasks.workunit.client.0.smithi103.stdout:6/892: mknod d0/c12b 0 2024-03-22T02:28:54.726 INFO:tasks.workunit.client.0.smithi103.stdout:6/893: write d0/d3/d22/d58/f80 [372190,4068] 0 2024-03-22T02:28:54.727 INFO:tasks.workunit.client.1.smithi122.stdout:2/544: dwrite df/d1f/f5c [4194304,4194304] 0 2024-03-22T02:28:54.729 INFO:tasks.workunit.client.1.smithi122.stdout:2/545: fsync df/f36 0 2024-03-22T02:28:54.731 INFO:tasks.workunit.client.0.smithi103.stdout:6/894: creat d0/d3/d22/d58/d6f/d75/db0/db1/f12c x:0 0 0 2024-03-22T02:28:54.731 INFO:tasks.workunit.client.1.smithi122.stdout:2/546: creat df/d1f/d3a/d4f/d84/fb7 x:0 0 0 2024-03-22T02:28:54.733 INFO:tasks.workunit.client.1.smithi122.stdout:2/547: mknod df/d1f/d2a/d51/cb8 0 2024-03-22T02:28:54.734 INFO:tasks.workunit.client.0.smithi103.stdout:6/895: mkdir d0/dbd/d103/d12d 0 2024-03-22T02:28:54.736 INFO:tasks.workunit.client.0.smithi103.stdout:6/896: creat d0/dbd/f12e x:0 0 0 2024-03-22T02:28:54.736 INFO:tasks.workunit.client.0.smithi103.stdout:9/992: dread d2/da/d18/fb0 [0,4194304] 0 2024-03-22T02:28:54.738 INFO:tasks.workunit.client.0.smithi103.stdout:9/993: dread - d2/da/d18/d1e/d77/d8b/dae/f114 zero size 2024-03-22T02:28:54.738 INFO:tasks.workunit.client.1.smithi122.stdout:2/548: dwrite df/d1f/f5c [4194304,4194304] 0 2024-03-22T02:28:54.741 INFO:tasks.workunit.client.0.smithi103.stdout:6/897: dread d0/d3/f42 [0,4194304] 0 2024-03-22T02:28:54.746 INFO:tasks.workunit.client.1.smithi122.stdout:2/549: dwrite df/d1f/d3a/d4f/f5e [0,4194304] 0 2024-03-22T02:28:54.748 INFO:tasks.workunit.client.0.smithi103.stdout:6/898: truncate d0/d3/f39 3494966 0 2024-03-22T02:28:54.749 INFO:tasks.workunit.client.1.smithi122.stdout:2/550: mknod df/d1f/d3d/d52/cb9 0 2024-03-22T02:28:54.751 INFO:tasks.workunit.client.0.smithi103.stdout:6/899: dread - d0/d3/d22/d58/d6f/d75/db0/db1/d113/f99 zero size 2024-03-22T02:28:54.752 INFO:tasks.workunit.client.1.smithi122.stdout:2/551: creat df/d1f/d2a/d72/fba x:0 0 0 2024-03-22T02:28:54.754 INFO:tasks.workunit.client.1.smithi122.stdout:2/552: symlink df/d1f/d33/d6a/d7c/lbb 0 2024-03-22T02:28:54.757 INFO:tasks.workunit.client.1.smithi122.stdout:2/553: rename df/d1f/d2a/d51/l85 to df/d1f/d3d/d52/lbc 0 2024-03-22T02:28:54.759 INFO:tasks.workunit.client.1.smithi122.stdout:3/375: sync 2024-03-22T02:28:54.759 INFO:tasks.workunit.client.1.smithi122.stdout:2/554: write df/d1f/d2a/d51/d67/f6e [1123574,74122] 0 2024-03-22T02:28:54.761 INFO:tasks.workunit.client.1.smithi122.stdout:2/555: truncate df/d1f/f20 5228475 0 2024-03-22T02:28:54.761 INFO:tasks.workunit.client.1.smithi122.stdout:3/376: chown d2/d9/dc/df/d3c/f41 189356 1 2024-03-22T02:28:54.762 INFO:tasks.workunit.client.1.smithi122.stdout:3/377: write d2/f5e [1058345,31424] 0 2024-03-22T02:28:54.763 INFO:tasks.workunit.client.1.smithi122.stdout:2/556: creat df/d1f/d3a/d4f/fbd x:0 0 0 2024-03-22T02:28:54.764 INFO:tasks.workunit.client.1.smithi122.stdout:3/378: chown d2/d9/dc/f2f 0 1 2024-03-22T02:28:54.765 INFO:tasks.workunit.client.1.smithi122.stdout:2/557: symlink df/d1f/d33/d6a/lbe 0 2024-03-22T02:28:54.767 INFO:tasks.workunit.client.1.smithi122.stdout:3/379: mknod d2/d9/c6d 0 2024-03-22T02:28:54.770 INFO:tasks.workunit.client.0.smithi103.stdout:9/994: dread d2/d5/f63 [4194304,4194304] 0 2024-03-22T02:28:54.770 INFO:tasks.workunit.client.1.smithi122.stdout:3/380: mknod d2/d9/dc/d17/d2b/d6b/c6e 0 2024-03-22T02:28:54.772 INFO:tasks.workunit.client.1.smithi122.stdout:3/381: creat d2/d9/dc/df/d38/f6f x:0 0 0 2024-03-22T02:28:54.774 INFO:tasks.workunit.client.1.smithi122.stdout:2/558: truncate df/d1f/d3d/f92 959226 0 2024-03-22T02:28:54.774 INFO:tasks.workunit.client.0.smithi103.stdout:9/995: dwrite d2/d10/d43/f7a [0,4194304] 0 2024-03-22T02:28:54.776 INFO:tasks.workunit.client.1.smithi122.stdout:2/559: chown df/d1f/d3a/da2/lad 15088 1 2024-03-22T02:28:54.782 INFO:tasks.workunit.client.1.smithi122.stdout:3/382: truncate d2/d9/dc/df/f53 4787602 0 2024-03-22T02:28:54.784 INFO:tasks.workunit.client.0.smithi103.stdout:9/996: truncate d2/da/d18/d1e/f88 1567539 0 2024-03-22T02:28:54.785 INFO:tasks.workunit.client.1.smithi122.stdout:2/560: dwrite df/d1f/d33/d4d/f78 [0,4194304] 0 2024-03-22T02:28:54.797 INFO:tasks.workunit.client.0.smithi103.stdout:9/997: write d2/da/d109/f10d [1429653,6529] 0 2024-03-22T02:28:54.803 INFO:tasks.workunit.client.0.smithi103.stdout:9/998: rename d2/d10/l16 to d2/da/d18/d1e/d77/l11b 0 2024-03-22T02:28:54.804 INFO:tasks.workunit.client.0.smithi103.stdout:9/999: chown d2/da/d18/d1e/fcc 945924 1 2024-03-22T02:28:54.806 INFO:tasks.workunit.client.1.smithi122.stdout:2/561: write df/d1f/d3d/f92 [432804,56804] 0 2024-03-22T02:28:54.809 INFO:tasks.workunit.client.1.smithi122.stdout:3/383: dread d2/d9/dc/d17/d28/f31 [4194304,4194304] 0 2024-03-22T02:28:54.812 INFO:tasks.workunit.client.1.smithi122.stdout:3/384: creat d2/d9/dc/d17/d28/d4c/f70 x:0 0 0 2024-03-22T02:28:54.812 INFO:tasks.workunit.client.1.smithi122.stdout:2/562: truncate df/d1f/f5c 6952576 0 2024-03-22T02:28:54.813 INFO:tasks.workunit.client.1.smithi122.stdout:3/385: mknod d2/d9/dc/df/d47/c71 0 2024-03-22T02:28:54.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/386: chown d2/l14 2539 1 2024-03-22T02:28:54.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/563: creat df/d1f/d2a/d51/d67/fbf x:0 0 0 2024-03-22T02:28:54.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/564: readlink df/d1f/d33/d6a/laa 0 2024-03-22T02:28:54.815 INFO:tasks.workunit.client.1.smithi122.stdout:3/387: chown d2/d9/dc/df/d47/c66 132584501 1 2024-03-22T02:28:54.819 INFO:tasks.workunit.client.1.smithi122.stdout:3/388: fsync d2/d9/dc/df/f53 0 2024-03-22T02:28:54.821 INFO:tasks.workunit.client.1.smithi122.stdout:3/389: write d2/d9/dc/d17/d2b/f67 [213829,74456] 0 2024-03-22T02:28:54.822 INFO:tasks.workunit.client.1.smithi122.stdout:2/565: creat df/d1f/d33/d4d/d7a/fc0 x:0 0 0 2024-03-22T02:28:54.822 INFO:tasks.workunit.client.1.smithi122.stdout:3/390: truncate d2/d9/dc/d17/f63 308969 0 2024-03-22T02:28:54.825 INFO:tasks.workunit.client.1.smithi122.stdout:2/566: creat df/d1f/d33/d4d/d94/fc1 x:0 0 0 2024-03-22T02:28:54.826 INFO:tasks.workunit.client.1.smithi122.stdout:3/391: symlink d2/d9/dc/df/d3c/d59/l72 0 2024-03-22T02:28:54.827 INFO:tasks.workunit.client.1.smithi122.stdout:2/567: rmdir df/d1f/d3a 39 2024-03-22T02:28:54.828 INFO:tasks.workunit.client.1.smithi122.stdout:2/568: write df/d1f/d9b/fa3 [429362,109846] 0 2024-03-22T02:28:54.828 INFO:tasks.workunit.client.1.smithi122.stdout:3/392: rmdir d2/d9/dc/df/d3c/d59 39 2024-03-22T02:28:54.829 INFO:tasks.workunit.client.1.smithi122.stdout:2/569: write df/d1f/d33/d34/f5f [4107105,101883] 0 2024-03-22T02:28:54.832 INFO:tasks.workunit.client.1.smithi122.stdout:3/393: symlink d2/l73 0 2024-03-22T02:28:54.833 INFO:tasks.workunit.client.1.smithi122.stdout:2/570: dread df/f37 [0,4194304] 0 2024-03-22T02:28:54.834 INFO:tasks.workunit.client.1.smithi122.stdout:3/394: write d2/d9/dc/d17/f63 [1326689,58287] 0 2024-03-22T02:28:54.836 INFO:tasks.workunit.client.1.smithi122.stdout:3/395: symlink d2/d9/dc/d17/d2b/d3f/l74 0 2024-03-22T02:28:54.838 INFO:tasks.workunit.client.1.smithi122.stdout:3/396: mknod d2/d9/dc/d17/d2b/d3f/c75 0 2024-03-22T02:28:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:54.849 INFO:tasks.workunit.client.1.smithi122.stdout:2/571: dread df/d1f/d2a/d51/d67/db5/f96 [0,4194304] 0 2024-03-22T02:28:54.849 INFO:tasks.workunit.client.1.smithi122.stdout:2/572: chown df 75757437 1 2024-03-22T02:28:54.852 INFO:tasks.workunit.client.1.smithi122.stdout:2/573: rename df/d1f/d2a to df/d1f/d33/d4d/d94/dc2 0 2024-03-22T02:28:54.854 INFO:tasks.workunit.client.1.smithi122.stdout:2/574: symlink df/d1f/d33/lc3 0 2024-03-22T02:28:54.856 INFO:tasks.workunit.client.1.smithi122.stdout:2/575: fsync df/d1f/d33/d4d/d94/dc2/d51/d67/f71 0 2024-03-22T02:28:54.858 INFO:tasks.workunit.client.1.smithi122.stdout:2/576: rename df/d1f/d23/l56 to df/d1f/d3d/lc4 0 2024-03-22T02:28:54.863 INFO:tasks.workunit.client.1.smithi122.stdout:2/577: write df/f37 [1488457,48466] 0 2024-03-22T02:28:54.866 INFO:tasks.workunit.client.1.smithi122.stdout:2/578: getdents df/d1f/d33/d4d/d94/dc2/d51/d67 0 2024-03-22T02:28:54.867 INFO:tasks.workunit.client.1.smithi122.stdout:2/579: dread - df/d1f/d33/d4d/d94/fc1 zero size 2024-03-22T02:28:54.868 INFO:tasks.workunit.client.1.smithi122.stdout:2/580: dread - df/d1f/d33/f46 zero size 2024-03-22T02:28:54.872 INFO:tasks.workunit.client.0.smithi103.stdout:6/900: sync 2024-03-22T02:28:54.875 INFO:tasks.workunit.client.0.smithi103.stdout:6/901: dread d0/d3/d33/fad [0,4194304] 0 2024-03-22T02:28:54.878 INFO:tasks.workunit.client.0.smithi103.stdout:6/902: mkdir d0/d3/d22/d58/db4/deb/d12f 0 2024-03-22T02:28:54.879 INFO:tasks.workunit.client.0.smithi103.stdout:6/903: creat d0/d3/d33/d11d/f130 x:0 0 0 2024-03-22T02:28:54.891 INFO:tasks.workunit.client.0.smithi103.stdout:6/904: dread d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f6b [0,4194304] 0 2024-03-22T02:28:54.893 INFO:tasks.workunit.client.0.smithi103.stdout:6/905: symlink d0/d3/d22/d58/d6f/l131 0 2024-03-22T02:28:54.894 INFO:tasks.workunit.client.0.smithi103.stdout:6/906: truncate d0/f5b 1453835 0 2024-03-22T02:28:54.895 INFO:tasks.workunit.client.0.smithi103.stdout:6/907: dread - d0/d88/d8e/ffb zero size 2024-03-22T02:28:54.895 INFO:tasks.workunit.client.0.smithi103.stdout:6/908: dread - d0/d3/d33/d11d/f130 zero size 2024-03-22T02:28:54.899 INFO:tasks.workunit.client.0.smithi103.stdout:6/909: write d0/d3/d22/d58/f8c [442293,31115] 0 2024-03-22T02:28:54.901 INFO:tasks.workunit.client.0.smithi103.stdout:6/910: mkdir d0/d3/d33/d132 0 2024-03-22T02:28:54.904 INFO:tasks.workunit.client.0.smithi103.stdout:6/911: symlink d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/l133 0 2024-03-22T02:28:54.905 INFO:tasks.workunit.client.0.smithi103.stdout:6/912: write d0/d3/fbc [278992,107487] 0 2024-03-22T02:28:54.911 INFO:tasks.workunit.client.0.smithi103.stdout:6/913: dwrite d0/d3/d22/f15 [0,4194304] 0 2024-03-22T02:28:54.911 INFO:tasks.workunit.client.0.smithi103.stdout:6/914: readlink d0/d14/dd2/lf9 0 2024-03-22T02:28:54.914 INFO:tasks.workunit.client.0.smithi103.stdout:6/915: mknod d0/d3/d22/c134 0 2024-03-22T02:28:54.917 INFO:tasks.workunit.client.0.smithi103.stdout:6/916: getdents d0/dbd/d103/d12d 0 2024-03-22T02:28:54.918 INFO:tasks.workunit.client.0.smithi103.stdout:6/917: truncate d0/f2c 1750934 0 2024-03-22T02:28:54.936 INFO:tasks.workunit.client.0.smithi103.stdout:6/918: truncate d0/f2c 1036504 0 2024-03-22T02:28:54.939 INFO:tasks.workunit.client.0.smithi103.stdout:6/919: creat d0/d3/d22/d58/db4/deb/d12f/f135 x:0 0 0 2024-03-22T02:28:54.945 INFO:tasks.workunit.client.0.smithi103.stdout:6/920: dwrite d0/d3/d22/d58/d6f/f8b [0,4194304] 0 2024-03-22T02:28:54.949 INFO:tasks.workunit.client.0.smithi103.stdout:6/921: getdents d0/d14/dc7/dfa 0 2024-03-22T02:28:54.955 INFO:tasks.workunit.client.0.smithi103.stdout:6/922: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f62 [0,4194304] 0 2024-03-22T02:28:54.957 INFO:tasks.workunit.client.0.smithi103.stdout:6/923: read d0/f1e [6758,83663] 0 2024-03-22T02:28:54.962 INFO:tasks.workunit.client.0.smithi103.stdout:6/924: rename d0/d88/d8e/dae/f101 to d0/f136 0 2024-03-22T02:28:54.963 INFO:tasks.workunit.client.0.smithi103.stdout:6/925: write d0/d88/d8e/fd1 [2040454,54075] 0 2024-03-22T02:28:54.976 INFO:tasks.workunit.client.1.smithi122.stdout:1/491: sync 2024-03-22T02:28:54.977 INFO:tasks.workunit.client.1.smithi122.stdout:1/492: write dc/f2c [3365695,130690] 0 2024-03-22T02:28:54.979 INFO:tasks.workunit.client.1.smithi122.stdout:1/493: mkdir dc/d3b/da2 0 2024-03-22T02:28:54.981 INFO:tasks.workunit.client.0.smithi103.stdout:6/926: dwrite d0/d3/d22/d58/d6f/d75/f7b [0,4194304] 0 2024-03-22T02:28:54.981 INFO:tasks.workunit.client.1.smithi122.stdout:1/494: mknod dc/d3b/d46/d4d/d67/ca3 0 2024-03-22T02:28:54.981 INFO:tasks.workunit.client.1.smithi122.stdout:1/495: chown dc/d3b/d46/d4d/f5c 80440 1 2024-03-22T02:28:54.982 INFO:tasks.workunit.client.1.smithi122.stdout:1/496: symlink dc/d4c/la4 0 2024-03-22T02:28:54.986 INFO:tasks.workunit.client.0.smithi103.stdout:6/927: dread d0/d14/f44 [0,4194304] 0 2024-03-22T02:28:54.995 INFO:tasks.workunit.client.0.smithi103.stdout:6/928: dwrite d0/d3/d22/f96 [0,4194304] 0 2024-03-22T02:28:54.996 INFO:tasks.workunit.client.0.smithi103.stdout:6/929: chown d0/d14/dc7/l115 97 1 2024-03-22T02:28:54.999 INFO:tasks.workunit.client.0.smithi103.stdout:6/930: mknod d0/d3/d33/c137 0 2024-03-22T02:28:55.001 INFO:tasks.workunit.client.0.smithi103.stdout:6/931: read d0/d3/f49 [3959622,50765] 0 2024-03-22T02:28:55.004 INFO:tasks.workunit.client.0.smithi103.stdout:6/932: mknod d0/d3/d33/c138 0 2024-03-22T02:28:55.005 INFO:tasks.workunit.client.0.smithi103.stdout:6/933: write d0/dbd/f10d [1581864,30660] 0 2024-03-22T02:28:55.008 INFO:tasks.workunit.client.0.smithi103.stdout:6/934: mknod d0/d14/d9a/d11a/c139 0 2024-03-22T02:28:55.024 INFO:tasks.workunit.client.0.smithi103.stdout:6/935: truncate d0/f41 130043 0 2024-03-22T02:28:55.024 INFO:tasks.workunit.client.1.smithi122.stdout:1/497: dread dc/fd [0,4194304] 0 2024-03-22T02:28:55.025 INFO:tasks.workunit.client.0.smithi103.stdout:6/936: chown d0/d88/d8e/dae/f111 231 1 2024-03-22T02:28:55.025 INFO:tasks.workunit.client.1.smithi122.stdout:1/498: fdatasync dc/f1a 0 2024-03-22T02:28:55.026 INFO:tasks.workunit.client.0.smithi103.stdout:6/937: mkdir d0/d3/d33/d132/d13a 0 2024-03-22T02:28:55.028 INFO:tasks.workunit.client.0.smithi103.stdout:6/938: chown d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/c6e 3065522 1 2024-03-22T02:28:55.033 INFO:tasks.workunit.client.0.smithi103.stdout:6/939: write d0/d3/fb8 [1427873,104077] 0 2024-03-22T02:28:55.035 INFO:tasks.workunit.client.0.smithi103.stdout:6/940: truncate d0/d88/d8e/dae/f111 19853 0 2024-03-22T02:28:55.042 INFO:tasks.workunit.client.1.smithi122.stdout:1/499: dread dc/d2b/f33 [0,4194304] 0 2024-03-22T02:28:55.043 INFO:tasks.workunit.client.0.smithi103.stdout:6/941: rmdir d0/d3/d22 39 2024-03-22T02:28:55.044 INFO:tasks.workunit.client.0.smithi103.stdout:6/942: write d0/f6a [4623147,14050] 0 2024-03-22T02:28:55.045 INFO:tasks.workunit.client.1.smithi122.stdout:1/500: mkdir dc/d3b/d46/d47/da5 0 2024-03-22T02:28:55.045 INFO:tasks.workunit.client.1.smithi122.stdout:1/501: stat dc/d55/l5d 0 2024-03-22T02:28:55.046 INFO:tasks.workunit.client.1.smithi122.stdout:1/502: write dc/d4c/f63 [4450315,63908] 0 2024-03-22T02:28:55.047 INFO:tasks.workunit.client.0.smithi103.stdout:6/943: mkdir d0/dbd/d103/d13b 0 2024-03-22T02:28:55.049 INFO:tasks.workunit.client.0.smithi103.stdout:6/944: unlink d0/la0 0 2024-03-22T02:28:55.049 INFO:tasks.workunit.client.0.smithi103.stdout:6/945: stat d0/d3/d33/d11d/c122 0 2024-03-22T02:28:55.050 INFO:tasks.workunit.client.0.smithi103.stdout:6/946: rmdir d0/d88 39 2024-03-22T02:28:55.052 INFO:tasks.workunit.client.0.smithi103.stdout:6/947: unlink d0/f41 0 2024-03-22T02:28:55.052 INFO:tasks.workunit.client.1.smithi122.stdout:1/503: dwrite dc/d2b/f57 [0,4194304] 0 2024-03-22T02:28:55.054 INFO:tasks.workunit.client.1.smithi122.stdout:1/504: unlink cb 0 2024-03-22T02:28:55.055 INFO:tasks.workunit.client.0.smithi103.stdout:6/948: symlink d0/d3/d22/d58/d6f/d75/daa/l13c 0 2024-03-22T02:28:55.058 INFO:tasks.workunit.client.0.smithi103.stdout:6/949: rename d0/d14/d9a/lba to d0/d3/d33/l13d 0 2024-03-22T02:28:55.058 INFO:tasks.workunit.client.1.smithi122.stdout:1/505: rmdir dc/d3b/da2 0 2024-03-22T02:28:55.059 INFO:tasks.workunit.client.0.smithi103.stdout:6/950: mknod d0/dbd/d103/d12d/c13e 0 2024-03-22T02:28:55.060 INFO:tasks.workunit.client.1.smithi122.stdout:1/506: write dc/d3b/d46/d4d/d90/f93 [4286985,84771] 0 2024-03-22T02:28:55.060 INFO:tasks.workunit.client.0.smithi103.stdout:6/951: dread - d0/d3/d22/d58/de0/f105 zero size 2024-03-22T02:28:55.065 INFO:tasks.workunit.client.1.smithi122.stdout:1/507: dwrite dc/f2c [0,4194304] 0 2024-03-22T02:28:55.066 INFO:tasks.workunit.client.0.smithi103.stdout:6/952: dwrite d0/d3/fb8 [0,4194304] 0 2024-03-22T02:28:55.066 INFO:tasks.workunit.client.1.smithi122.stdout:8/540: sync 2024-03-22T02:28:55.067 INFO:tasks.workunit.client.1.smithi122.stdout:8/541: read - d1/d22/f64 zero size 2024-03-22T02:28:55.069 INFO:tasks.workunit.client.1.smithi122.stdout:1/508: dread dc/f39 [0,4194304] 0 2024-03-22T02:28:55.069 INFO:tasks.workunit.client.0.smithi103.stdout:6/953: truncate d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/fa8 5704462 0 2024-03-22T02:28:55.070 INFO:tasks.workunit.client.1.smithi122.stdout:1/509: chown dc/l6a 50 1 2024-03-22T02:28:55.095 INFO:tasks.workunit.client.1.smithi122.stdout:1/510: truncate dc/d3b/f68 3823196 0 2024-03-22T02:28:55.101 INFO:tasks.workunit.client.1.smithi122.stdout:1/511: dread dc/f2a [0,4194304] 0 2024-03-22T02:28:55.103 INFO:tasks.workunit.client.1.smithi122.stdout:1/512: chown dc/d2b/c3e 0 1 2024-03-22T02:28:55.108 INFO:tasks.workunit.client.1.smithi122.stdout:1/513: link dc/d4c/l9b dc/d3b/d46/d4d/d71/la6 0 2024-03-22T02:28:55.109 INFO:tasks.workunit.client.1.smithi122.stdout:8/542: dread d1/f3d [0,4194304] 0 2024-03-22T02:28:55.112 INFO:tasks.workunit.client.1.smithi122.stdout:8/543: symlink d1/d9/d50/db1/lbc 0 2024-03-22T02:28:55.115 INFO:tasks.workunit.client.1.smithi122.stdout:9/585: sync 2024-03-22T02:28:55.115 INFO:tasks.workunit.client.1.smithi122.stdout:8/544: rename d1/d40/f7a to d1/fbd 0 2024-03-22T02:28:55.116 INFO:tasks.workunit.client.1.smithi122.stdout:9/586: stat db/df/d3e/d59/db5/f93 0 2024-03-22T02:28:55.116 INFO:tasks.workunit.client.1.smithi122.stdout:1/514: write dc/d3b/d46/d47/f8a [2365833,19983] 0 2024-03-22T02:28:55.117 INFO:tasks.workunit.client.1.smithi122.stdout:1/515: fdatasync dc/d2b/f57 0 2024-03-22T02:28:55.118 INFO:tasks.workunit.client.1.smithi122.stdout:9/587: write db/df/d3e/d81/d3d/fa1 [89164,45953] 0 2024-03-22T02:28:55.118 INFO:tasks.workunit.client.1.smithi122.stdout:8/545: symlink d1/d60/d73/d90/lbe 0 2024-03-22T02:28:55.121 INFO:tasks.workunit.client.1.smithi122.stdout:1/516: rename dc/f2c to dc/d3b/d46/fa7 0 2024-03-22T02:28:55.123 INFO:tasks.workunit.client.1.smithi122.stdout:1/517: chown dc/d3b/d46/d47/c58 1679 1 2024-03-22T02:28:55.123 INFO:tasks.workunit.client.1.smithi122.stdout:8/546: write d1/d3f/d45/f4b [3910036,126880] 0 2024-03-22T02:28:55.123 INFO:tasks.workunit.client.1.smithi122.stdout:1/518: fsync dc/d2b/f2d 0 2024-03-22T02:28:55.124 INFO:tasks.workunit.client.1.smithi122.stdout:1/519: fsync dc/f22 0 2024-03-22T02:28:55.125 INFO:tasks.workunit.client.1.smithi122.stdout:1/520: write dc/f22 [2172464,2577] 0 2024-03-22T02:28:55.126 INFO:tasks.workunit.client.1.smithi122.stdout:9/588: mknod db/df/d3e/d59/cbb 0 2024-03-22T02:28:55.126 INFO:tasks.workunit.client.1.smithi122.stdout:1/521: fdatasync dc/d3b/d46/f9e 0 2024-03-22T02:28:55.130 INFO:tasks.workunit.client.1.smithi122.stdout:8/547: creat d1/d3f/d45/fbf x:0 0 0 2024-03-22T02:28:55.135 INFO:tasks.workunit.client.1.smithi122.stdout:8/548: dread d1/d8/f5c [0,4194304] 0 2024-03-22T02:28:55.136 INFO:tasks.workunit.client.1.smithi122.stdout:8/549: write d1/d9/d50/f59 [648360,129548] 0 2024-03-22T02:28:55.140 INFO:tasks.workunit.client.1.smithi122.stdout:8/550: rename d1/d8/d63/d88/dab to d1/d8/dba/dc0 0 2024-03-22T02:28:55.143 INFO:tasks.workunit.client.1.smithi122.stdout:8/551: mknod d1/d8/cc1 0 2024-03-22T02:28:55.144 INFO:tasks.workunit.client.1.smithi122.stdout:1/522: dwrite dc/d3b/d46/d4d/f5c [0,4194304] 0 2024-03-22T02:28:55.145 INFO:tasks.workunit.client.1.smithi122.stdout:8/552: chown d1/d22/f55 8034 1 2024-03-22T02:28:55.145 INFO:tasks.workunit.client.1.smithi122.stdout:1/523: dread - dc/d2b/d86/f9f zero size 2024-03-22T02:28:55.146 INFO:tasks.workunit.client.1.smithi122.stdout:1/524: readlink dc/l14 0 2024-03-22T02:28:55.151 INFO:tasks.workunit.client.1.smithi122.stdout:8/553: creat d1/d8/dba/dc0/db9/fc2 x:0 0 0 2024-03-22T02:28:55.155 INFO:tasks.workunit.client.1.smithi122.stdout:8/554: creat d1/d40/d71/fc3 x:0 0 0 2024-03-22T02:28:55.155 INFO:tasks.workunit.client.1.smithi122.stdout:1/525: dwrite dc/d3b/d8d/d94/f98 [0,4194304] 0 2024-03-22T02:28:55.156 INFO:tasks.workunit.client.1.smithi122.stdout:8/555: chown d1/d9/l39 4 1 2024-03-22T02:28:55.157 INFO:tasks.workunit.client.1.smithi122.stdout:1/526: rmdir dc/d55 39 2024-03-22T02:28:55.160 INFO:tasks.workunit.client.1.smithi122.stdout:8/556: mkdir d1/d22/d6a/dc4 0 2024-03-22T02:28:55.162 INFO:tasks.workunit.client.1.smithi122.stdout:8/557: truncate d1/d40/d71/fc3 453155 0 2024-03-22T02:28:55.163 INFO:tasks.workunit.client.1.smithi122.stdout:1/527: symlink dc/d55/d99/la8 0 2024-03-22T02:28:55.165 INFO:tasks.workunit.client.1.smithi122.stdout:1/528: truncate dc/d4c/f8b 1176765 0 2024-03-22T02:28:55.166 INFO:tasks.workunit.client.1.smithi122.stdout:8/558: getdents d1/d8/d5e 0 2024-03-22T02:28:55.166 INFO:tasks.workunit.client.1.smithi122.stdout:1/529: write dc/d2b/f36 [2260056,22442] 0 2024-03-22T02:28:55.168 INFO:tasks.workunit.client.1.smithi122.stdout:8/559: truncate d1/d22/f64 971777 0 2024-03-22T02:28:55.174 INFO:tasks.workunit.client.1.smithi122.stdout:1/530: dwrite dc/d4c/f64 [0,4194304] 0 2024-03-22T02:28:55.176 INFO:tasks.workunit.client.1.smithi122.stdout:8/560: dwrite d1/d3f/d45/f68 [0,4194304] 0 2024-03-22T02:28:55.180 INFO:tasks.workunit.client.1.smithi122.stdout:8/561: write d1/d9/f28 [3090100,114433] 0 2024-03-22T02:28:55.187 INFO:tasks.workunit.client.1.smithi122.stdout:1/531: dwrite dc/d3b/f81 [0,4194304] 0 2024-03-22T02:28:55.192 INFO:tasks.workunit.client.1.smithi122.stdout:8/562: getdents d1/d3f/d45 0 2024-03-22T02:28:55.192 INFO:tasks.workunit.client.1.smithi122.stdout:1/532: symlink dc/d3b/d46/d4d/d67/la9 0 2024-03-22T02:28:55.193 INFO:tasks.workunit.client.1.smithi122.stdout:8/563: symlink d1/d9/d50/lc5 0 2024-03-22T02:28:55.231 INFO:tasks.workunit.client.0.smithi103.stdout:6/954: sync 2024-03-22T02:28:55.234 INFO:tasks.workunit.client.0.smithi103.stdout:6/955: dread d0/d3/d22/f3c [0,4194304] 0 2024-03-22T02:28:55.235 INFO:tasks.workunit.client.0.smithi103.stdout:6/956: chown d0/d3/d33/lc4 1394307 1 2024-03-22T02:28:55.237 INFO:tasks.workunit.client.1.smithi122.stdout:5/437: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:0/416: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:4/435: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:2/581: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:3/397: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:6/547: sync 2024-03-22T02:28:55.238 INFO:tasks.workunit.client.1.smithi122.stdout:7/430: sync 2024-03-22T02:28:55.240 INFO:tasks.workunit.client.1.smithi122.stdout:0/417: write d11/d4f/f82 [1920019,96286] 0 2024-03-22T02:28:55.241 INFO:tasks.workunit.client.1.smithi122.stdout:7/431: readlink d0/d4/d22/d5f/l78 0 2024-03-22T02:28:55.245 INFO:tasks.workunit.client.1.smithi122.stdout:0/418: fsync d11/d16/d44/f76 0 2024-03-22T02:28:55.245 INFO:tasks.workunit.client.1.smithi122.stdout:2/582: write df/d1f/d33/d6a/f93 [1066653,4306] 0 2024-03-22T02:28:55.246 INFO:tasks.workunit.client.1.smithi122.stdout:6/548: dread d7/de/dd/d20/f81 [0,4194304] 0 2024-03-22T02:28:55.246 INFO:tasks.workunit.client.1.smithi122.stdout:5/438: creat d8/d1e/d22/d78/f98 x:0 0 0 2024-03-22T02:28:55.246 INFO:tasks.workunit.client.1.smithi122.stdout:7/432: chown d0/d4/d22/d5f/d5e/f77 7 1 2024-03-22T02:28:55.247 INFO:tasks.workunit.client.1.smithi122.stdout:4/436: symlink d0/d7a/l8f 0 2024-03-22T02:28:55.247 INFO:tasks.workunit.client.1.smithi122.stdout:3/398: dread d2/d9/dc/d17/d28/f31 [4194304,4194304] 0 2024-03-22T02:28:55.248 INFO:tasks.workunit.client.1.smithi122.stdout:7/433: write d0/d4/f2d [1035920,84302] 0 2024-03-22T02:28:55.249 INFO:tasks.workunit.client.1.smithi122.stdout:0/419: creat d11/d14/f8d x:0 0 0 2024-03-22T02:28:55.250 INFO:tasks.workunit.client.1.smithi122.stdout:3/399: chown d2/f15 32 1 2024-03-22T02:28:55.251 INFO:tasks.workunit.client.1.smithi122.stdout:4/437: stat d0/d12/d15/d42/d4e/de/c3f 0 2024-03-22T02:28:55.252 INFO:tasks.workunit.client.1.smithi122.stdout:7/434: readlink d0/d25/l86 0 2024-03-22T02:28:55.252 INFO:tasks.workunit.client.1.smithi122.stdout:0/420: readlink d11/d14/d1b/l63 0 2024-03-22T02:28:55.253 INFO:tasks.workunit.client.1.smithi122.stdout:6/549: creat d7/fab x:0 0 0 2024-03-22T02:28:55.254 INFO:tasks.workunit.client.1.smithi122.stdout:0/421: stat d11/d14/d1d/d27/c62 0 2024-03-22T02:28:55.255 INFO:tasks.workunit.client.1.smithi122.stdout:5/439: rmdir d8/da/d28/d42/d8e 0 2024-03-22T02:28:55.258 INFO:tasks.workunit.client.1.smithi122.stdout:2/583: dread df/d1f/d33/d6a/f8d [0,4194304] 0 2024-03-22T02:28:55.258 INFO:tasks.workunit.client.1.smithi122.stdout:4/438: rename d0/d41 to d0/d12/d15/d3d/d90 0 2024-03-22T02:28:55.262 INFO:tasks.workunit.client.1.smithi122.stdout:6/550: creat d7/de/d27/d52/fac x:0 0 0 2024-03-22T02:28:55.268 INFO:tasks.workunit.client.1.smithi122.stdout:4/439: mkdir d0/d12/d15/d42/d4e/de/d91 0 2024-03-22T02:28:55.269 INFO:tasks.workunit.client.1.smithi122.stdout:2/584: mknod df/d1f/d33/d6a/cc5 0 2024-03-22T02:28:55.271 INFO:tasks.workunit.client.1.smithi122.stdout:2/585: chown df/d1f/f9c 1 1 2024-03-22T02:28:55.275 INFO:tasks.workunit.client.1.smithi122.stdout:3/400: link d2/d9/dc/d17/d45/f46 d2/d9/dc/df/d47/f76 0 2024-03-22T02:28:55.275 INFO:tasks.workunit.client.1.smithi122.stdout:5/440: dwrite d8/da/d28/d35/f94 [0,4194304] 0 2024-03-22T02:28:55.276 INFO:tasks.workunit.client.1.smithi122.stdout:6/551: dwrite d7/de/d27/d52/f7d [0,4194304] 0 2024-03-22T02:28:55.285 INFO:tasks.workunit.client.1.smithi122.stdout:4/440: creat d0/d12/d61/f92 x:0 0 0 2024-03-22T02:28:55.285 INFO:tasks.workunit.client.1.smithi122.stdout:7/435: getdents d0/d4/d20/d57 0 2024-03-22T02:28:55.285 INFO:tasks.workunit.client.1.smithi122.stdout:2/586: write df/d1f/f2d [3104946,48992] 0 2024-03-22T02:28:55.288 INFO:tasks.workunit.client.1.smithi122.stdout:4/441: write d0/d12/f21 [2391770,64642] 0 2024-03-22T02:28:55.290 INFO:tasks.workunit.client.1.smithi122.stdout:0/422: dwrite d11/d14/d1b/f3c [0,4194304] 0 2024-03-22T02:28:55.290 INFO:tasks.workunit.client.1.smithi122.stdout:4/442: chown d0/d12/d15/d3d/f68 4 1 2024-03-22T02:28:55.299 INFO:tasks.workunit.client.1.smithi122.stdout:5/441: mknod d8/d1e/d22/d3a/d46/c99 0 2024-03-22T02:28:55.300 INFO:tasks.workunit.client.1.smithi122.stdout:5/442: fdatasync d8/da/d28/d42/f7f 0 2024-03-22T02:28:55.303 INFO:tasks.workunit.client.1.smithi122.stdout:2/587: rmdir df/d1f/d33/d4d 39 2024-03-22T02:28:55.304 INFO:tasks.workunit.client.1.smithi122.stdout:0/423: unlink d11/d14/l34 0 2024-03-22T02:28:55.308 INFO:tasks.workunit.client.1.smithi122.stdout:5/443: readlink d8/d1e/d51/l97 0 2024-03-22T02:28:55.312 INFO:tasks.workunit.client.1.smithi122.stdout:2/588: unlink df/d1f/d33/d34/l4e 0 2024-03-22T02:28:55.315 INFO:tasks.workunit.client.1.smithi122.stdout:3/401: creat d2/d9/dc/d17/d2b/f77 x:0 0 0 2024-03-22T02:28:55.316 INFO:tasks.workunit.client.1.smithi122.stdout:2/589: symlink df/d1f/d33/d6a/d7c/lc6 0 2024-03-22T02:28:55.318 INFO:tasks.workunit.client.1.smithi122.stdout:2/590: write df/d1f/d9b/fa3 [1934675,105327] 0 2024-03-22T02:28:55.320 INFO:tasks.workunit.client.1.smithi122.stdout:3/402: mkdir d2/d9/dc/df/d3c/d78 0 2024-03-22T02:28:55.323 INFO:tasks.workunit.client.1.smithi122.stdout:2/591: mkdir df/d1f/dc7 0 2024-03-22T02:28:55.324 INFO:tasks.workunit.client.1.smithi122.stdout:5/444: getdents d8/d3b 0 2024-03-22T02:28:55.325 INFO:tasks.workunit.client.1.smithi122.stdout:3/403: mkdir d2/d9/dc/df/d47/d79 0 2024-03-22T02:28:55.326 INFO:tasks.workunit.client.1.smithi122.stdout:3/404: write d2/f62 [1776444,4047] 0 2024-03-22T02:28:55.328 INFO:tasks.workunit.client.1.smithi122.stdout:2/592: stat df/d1f/d33/d4d/d94/dc2/d51/d67/db5 0 2024-03-22T02:28:55.328 INFO:tasks.workunit.client.1.smithi122.stdout:3/405: chown d2/d9/dc/d17/d2b/d6b/c6e 11 1 2024-03-22T02:28:55.331 INFO:tasks.workunit.client.1.smithi122.stdout:2/593: creat df/d1f/d33/d4d/d94/fc8 x:0 0 0 2024-03-22T02:28:55.333 INFO:tasks.workunit.client.1.smithi122.stdout:6/552: write d7/fa [800950,78800] 0 2024-03-22T02:28:55.334 INFO:tasks.workunit.client.1.smithi122.stdout:7/436: write d0/d25/d46/f85 [3052904,34857] 0 2024-03-22T02:28:55.337 INFO:tasks.workunit.client.1.smithi122.stdout:2/594: chown df/d1f/d33/d4d/d94/fc1 5285 1 2024-03-22T02:28:55.337 INFO:tasks.workunit.client.1.smithi122.stdout:6/553: stat d7/de/d27/l11 0 2024-03-22T02:28:55.339 INFO:tasks.workunit.client.1.smithi122.stdout:7/437: dread - d0/d4/d20/d57/f6a zero size 2024-03-22T02:28:55.340 INFO:tasks.workunit.client.1.smithi122.stdout:3/406: dwrite d2/d9/dc/df/d38/f3b [0,4194304] 0 2024-03-22T02:28:55.340 INFO:tasks.workunit.client.1.smithi122.stdout:4/443: dwrite d0/d25/f13 [0,4194304] 0 2024-03-22T02:28:55.351 INFO:tasks.workunit.client.1.smithi122.stdout:6/554: write d7/de/dd/f56 [940496,42430] 0 2024-03-22T02:28:55.354 INFO:tasks.workunit.client.1.smithi122.stdout:2/595: creat df/d1f/d33/d4d/d94/dc2/d51/d67/db0/fc9 x:0 0 0 2024-03-22T02:28:55.359 INFO:tasks.workunit.client.1.smithi122.stdout:3/407: dwrite d2/d9/dc/d17/f42 [0,4194304] 0 2024-03-22T02:28:55.362 INFO:tasks.workunit.client.1.smithi122.stdout:2/596: rmdir df/d1f/dc7 0 2024-03-22T02:28:55.362 INFO:tasks.workunit.client.1.smithi122.stdout:3/408: readlink d2/d9/dc/d17/d28/l32 0 2024-03-22T02:28:55.366 INFO:tasks.workunit.client.0.smithi103.stdout:6/957: sync 2024-03-22T02:28:55.366 INFO:tasks.workunit.client.1.smithi122.stdout:6/555: dwrite d7/de/dd/f56 [0,4194304] 0 2024-03-22T02:28:55.373 INFO:tasks.workunit.client.0.smithi103.stdout:6/958: dwrite d0/d3/d22/d58/de0/ff0 [0,4194304] 0 2024-03-22T02:28:55.374 INFO:tasks.workunit.client.1.smithi122.stdout:2/597: getdents df/d1f/d33/d6a 0 2024-03-22T02:28:55.377 INFO:tasks.workunit.client.1.smithi122.stdout:3/409: rename d2/d9/dc/d17/d2b/f77 to d2/d9/dc/df/d3c/f7a 0 2024-03-22T02:28:55.378 INFO:tasks.workunit.client.1.smithi122.stdout:6/556: dwrite d7/de/d27/f36 [0,4194304] 0 2024-03-22T02:28:55.383 INFO:tasks.workunit.client.1.smithi122.stdout:2/598: unlink df/d1f/d33/d4d/d94/dc2/d51/cab 0 2024-03-22T02:28:55.384 INFO:tasks.workunit.client.1.smithi122.stdout:6/557: mkdir d7/de/dd/d20/d2d/dad 0 2024-03-22T02:28:55.385 INFO:tasks.workunit.client.1.smithi122.stdout:6/558: fsync f5 0 2024-03-22T02:28:55.386 INFO:tasks.workunit.client.1.smithi122.stdout:2/599: fsync df/d1f/d33/f80 0 2024-03-22T02:28:55.388 INFO:tasks.workunit.client.1.smithi122.stdout:3/410: link d2/d9/dc/d17/f63 d2/d9/dc/df/d3c/f7b 0 2024-03-22T02:28:55.391 INFO:tasks.workunit.client.1.smithi122.stdout:2/600: mkdir df/d1f/d33/d4d/d94/dc2/d51/d67/db5/dca 0 2024-03-22T02:28:55.394 INFO:tasks.workunit.client.1.smithi122.stdout:2/601: symlink df/d1f/lcb 0 2024-03-22T02:28:55.396 INFO:tasks.workunit.client.1.smithi122.stdout:4/444: dread d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:55.397 INFO:tasks.workunit.client.1.smithi122.stdout:6/559: dread d7/de/dd/d20/f3f [0,4194304] 0 2024-03-22T02:28:55.399 INFO:tasks.workunit.client.1.smithi122.stdout:7/438: dread d0/d4/f9 [0,4194304] 0 2024-03-22T02:28:55.399 INFO:tasks.workunit.client.1.smithi122.stdout:2/602: write df/d1f/d33/d4d/d7a/f8a [130479,79446] 0 2024-03-22T02:28:55.401 INFO:tasks.workunit.client.1.smithi122.stdout:6/560: mkdir d7/de/dd/d20/d2d/dae 0 2024-03-22T02:28:55.401 INFO:tasks.workunit.client.1.smithi122.stdout:6/561: read d7/fa [876771,43282] 0 2024-03-22T02:28:55.402 INFO:tasks.workunit.client.1.smithi122.stdout:4/445: rmdir d0/d12/d15/d2d/d77 0 2024-03-22T02:28:55.403 INFO:tasks.workunit.client.1.smithi122.stdout:9/589: sync 2024-03-22T02:28:55.403 INFO:tasks.workunit.client.1.smithi122.stdout:4/446: write d0/f6a [350156,126952] 0 2024-03-22T02:28:55.407 INFO:tasks.workunit.client.0.smithi103.stdout:6/959: dread d0/d88/d8e/fd1 [0,4194304] 0 2024-03-22T02:28:55.410 INFO:tasks.workunit.client.0.smithi103.stdout:6/960: dread d0/f69 [0,4194304] 0 2024-03-22T02:28:55.411 INFO:tasks.workunit.client.0.smithi103.stdout:6/961: readlink d0/d3/d22/l11f 0 2024-03-22T02:28:55.412 INFO:tasks.workunit.client.1.smithi122.stdout:9/590: dwrite db/d23/d1f/d57/d66/f7a [0,4194304] 0 2024-03-22T02:28:55.413 INFO:tasks.workunit.client.0.smithi103.stdout:6/962: mknod d0/d3/d22/d58/d6f/d75/db0/db1/c13f 0 2024-03-22T02:28:55.417 INFO:tasks.workunit.client.1.smithi122.stdout:6/562: rename d7/de/dd/df/c83 to d7/de/dd/d20/d2d/d54/d76/d86/d96/caf 0 2024-03-22T02:28:55.419 INFO:tasks.workunit.client.0.smithi103.stdout:6/963: dwrite d0/d14/dd2/fd6 [0,4194304] 0 2024-03-22T02:28:55.423 INFO:tasks.workunit.client.1.smithi122.stdout:4/447: getdents d0/d12/d15/d42 0 2024-03-22T02:28:55.423 INFO:tasks.workunit.client.0.smithi103.stdout:6/964: unlink d0/d88/d8e/l91 0 2024-03-22T02:28:55.424 INFO:tasks.workunit.client.1.smithi122.stdout:3/411: write d2/d9/dc/d17/d28/f4d [75259,105862] 0 2024-03-22T02:28:55.426 INFO:tasks.workunit.client.0.smithi103.stdout:6/965: mknod d0/d14/dc7/dce/c140 0 2024-03-22T02:28:55.427 INFO:tasks.workunit.client.0.smithi103.stdout:6/966: dread - d0/d14/dd2/f123 zero size 2024-03-22T02:28:55.430 INFO:tasks.workunit.client.1.smithi122.stdout:7/439: dread d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:55.430 INFO:tasks.workunit.client.1.smithi122.stdout:2/603: dread df/d1f/d33/d34/f43 [0,4194304] 0 2024-03-22T02:28:55.430 INFO:tasks.workunit.client.1.smithi122.stdout:3/412: write d2/f39 [1569197,5221] 0 2024-03-22T02:28:55.431 INFO:tasks.workunit.client.1.smithi122.stdout:6/563: dwrite d7/de/d27/d52/f57 [0,4194304] 0 2024-03-22T02:28:55.432 INFO:tasks.workunit.client.0.smithi103.stdout:6/967: dwrite d0/d14/fe9 [0,4194304] 0 2024-03-22T02:28:55.433 INFO:tasks.workunit.client.1.smithi122.stdout:7/440: fsync d0/d4e/f73 0 2024-03-22T02:28:55.434 INFO:tasks.workunit.client.1.smithi122.stdout:3/413: write d2/f15 [6044699,113406] 0 2024-03-22T02:28:55.436 INFO:tasks.workunit.client.1.smithi122.stdout:6/564: write d7/de/dd/d39/f5b [346987,94861] 0 2024-03-22T02:28:55.437 INFO:tasks.workunit.client.1.smithi122.stdout:7/441: truncate d0/d30/f52 794767 0 2024-03-22T02:28:55.439 INFO:tasks.workunit.client.1.smithi122.stdout:4/448: getdents d0/d12/d15/d42/d57 0 2024-03-22T02:28:55.440 INFO:tasks.workunit.client.1.smithi122.stdout:7/442: fsync d0/d4e/d5d/f6e 0 2024-03-22T02:28:55.441 INFO:tasks.workunit.client.0.smithi103.stdout:6/968: dwrite d0/dbd/f10d [0,4194304] 0 2024-03-22T02:28:55.445 INFO:tasks.workunit.client.1.smithi122.stdout:3/414: creat d2/d9/dc/d17/d51/f7c x:0 0 0 2024-03-22T02:28:55.449 INFO:tasks.workunit.client.1.smithi122.stdout:4/449: mknod d0/d12/d15/d42/d4e/de/d40/c93 0 2024-03-22T02:28:55.453 INFO:tasks.workunit.client.1.smithi122.stdout:9/591: dread db/df/f7d [0,4194304] 0 2024-03-22T02:28:55.456 INFO:tasks.workunit.client.1.smithi122.stdout:2/604: rename df/f15 to df/d1f/d33/d4d/d94/dc2/d72/fcc 0 2024-03-22T02:28:55.457 INFO:tasks.workunit.client.1.smithi122.stdout:3/415: dwrite d2/d9/dc/d17/d28/f69 [0,4194304] 0 2024-03-22T02:28:55.458 INFO:tasks.workunit.client.1.smithi122.stdout:7/443: write d0/d4/d22/f55 [877445,63176] 0 2024-03-22T02:28:55.459 INFO:tasks.workunit.client.1.smithi122.stdout:7/444: chown d0/d25 3 1 2024-03-22T02:28:55.459 INFO:tasks.workunit.client.1.smithi122.stdout:4/450: mknod d0/d12/d15/d42/c94 0 2024-03-22T02:28:55.465 INFO:tasks.workunit.client.1.smithi122.stdout:7/445: chown d0/d4/d20/d21/d28/l4d 359705 1 2024-03-22T02:28:55.466 INFO:tasks.workunit.client.1.smithi122.stdout:4/451: mknod d0/d12/d15/d42/c95 0 2024-03-22T02:28:55.468 INFO:tasks.workunit.client.1.smithi122.stdout:6/565: getdents d7/de/dd/d20/d2d/d54 0 2024-03-22T02:28:55.468 INFO:tasks.workunit.client.1.smithi122.stdout:4/452: readlink d0/d12/l47 0 2024-03-22T02:28:55.468 INFO:tasks.workunit.client.1.smithi122.stdout:3/416: creat d2/d9/dc/d17/d2b/d6a/f7d x:0 0 0 2024-03-22T02:28:55.469 INFO:tasks.workunit.client.1.smithi122.stdout:3/417: fdatasync d2/f7 0 2024-03-22T02:28:55.469 INFO:tasks.workunit.client.1.smithi122.stdout:7/446: rename d0/c2c to d0/d4/d22/c8b 0 2024-03-22T02:28:55.470 INFO:tasks.workunit.client.1.smithi122.stdout:4/453: chown d0/d12/d15/d42/d4e/de/f44 3559893 1 2024-03-22T02:28:55.473 INFO:tasks.workunit.client.1.smithi122.stdout:6/566: creat d7/de/dd/d20/d8f/fb0 x:0 0 0 2024-03-22T02:28:55.478 INFO:tasks.workunit.client.1.smithi122.stdout:4/454: dread d0/d25/fa [0,4194304] 0 2024-03-22T02:28:55.479 INFO:tasks.workunit.client.1.smithi122.stdout:3/418: mkdir d2/d9/d7e 0 2024-03-22T02:28:55.483 INFO:tasks.workunit.client.1.smithi122.stdout:6/567: dwrite d7/de/dd/d20/fa5 [0,4194304] 0 2024-03-22T02:28:55.483 INFO:tasks.workunit.client.1.smithi122.stdout:4/455: chown d0/d12/d15/d42/c7f 1217935508 1 2024-03-22T02:28:55.486 INFO:tasks.workunit.client.1.smithi122.stdout:9/592: dread f8 [0,4194304] 0 2024-03-22T02:28:55.489 INFO:tasks.workunit.client.1.smithi122.stdout:3/419: symlink d2/d9/dc/df/d47/l7f 0 2024-03-22T02:28:55.495 INFO:tasks.workunit.client.1.smithi122.stdout:9/593: dwrite db/d23/d1f/f2b [0,4194304] 0 2024-03-22T02:28:55.497 INFO:tasks.workunit.client.1.smithi122.stdout:9/594: dread - db/d23/d1f/d57/f87 zero size 2024-03-22T02:28:55.503 INFO:tasks.workunit.client.1.smithi122.stdout:9/595: creat db/df/d3e/d81/d25/d41/fbc x:0 0 0 2024-03-22T02:28:55.504 INFO:tasks.workunit.client.1.smithi122.stdout:9/596: chown db/d23/d1f/fa0 29426006 1 2024-03-22T02:28:55.508 INFO:tasks.workunit.client.1.smithi122.stdout:3/420: getdents d2/d9/dc/df/d3c/d59 0 2024-03-22T02:28:55.509 INFO:tasks.workunit.client.1.smithi122.stdout:9/597: creat db/d1b/da8/dba/fbd x:0 0 0 2024-03-22T02:28:55.511 INFO:tasks.workunit.client.1.smithi122.stdout:9/598: readlink db/df/l7c 0 2024-03-22T02:28:55.513 INFO:tasks.workunit.client.1.smithi122.stdout:3/421: rmdir d2/d9/dc/df/d38 39 2024-03-22T02:28:55.514 INFO:tasks.workunit.client.1.smithi122.stdout:7/447: dwrite d0/d4/d22/f32 [0,4194304] 0 2024-03-22T02:28:55.516 INFO:tasks.workunit.client.1.smithi122.stdout:9/599: write db/fb3 [458962,30696] 0 2024-03-22T02:28:55.520 INFO:tasks.workunit.client.1.smithi122.stdout:3/422: symlink d2/d9/dc/d17/d2b/d6b/l80 0 2024-03-22T02:28:55.522 INFO:tasks.workunit.client.1.smithi122.stdout:7/448: chown d0/d4/d22/d5f/d5e/f77 25661596 1 2024-03-22T02:28:55.522 INFO:tasks.workunit.client.1.smithi122.stdout:9/600: truncate db/d1b/d6c/f4f 4560421 0 2024-03-22T02:28:55.523 INFO:tasks.workunit.client.1.smithi122.stdout:4/456: dwrite d0/d12/d15/d42/d4e/de/f8b [0,4194304] 0 2024-03-22T02:28:55.525 INFO:tasks.workunit.client.1.smithi122.stdout:6/568: dwrite d7/de/dd/d20/f45 [0,4194304] 0 2024-03-22T02:28:55.526 INFO:tasks.workunit.client.1.smithi122.stdout:7/449: fsync d0/d30/f5b 0 2024-03-22T02:28:55.530 INFO:tasks.workunit.client.1.smithi122.stdout:4/457: truncate d0/d12/d15/f8c 4816081 0 2024-03-22T02:28:55.533 INFO:tasks.workunit.client.1.smithi122.stdout:6/569: read d7/de/dd/d39/f4f [1489006,48927] 0 2024-03-22T02:28:55.534 INFO:tasks.workunit.client.1.smithi122.stdout:4/458: creat d0/d12/d15/d42/d57/f96 x:0 0 0 2024-03-22T02:28:55.537 INFO:tasks.workunit.client.1.smithi122.stdout:7/450: dread d0/d4/f12 [0,4194304] 0 2024-03-22T02:28:55.538 INFO:tasks.workunit.client.1.smithi122.stdout:3/423: write d2/d9/dc/df/d3c/f41 [325912,8811] 0 2024-03-22T02:28:55.541 INFO:tasks.workunit.client.1.smithi122.stdout:7/451: rmdir d0/d4/d22/d5f 39 2024-03-22T02:28:55.542 INFO:tasks.workunit.client.1.smithi122.stdout:7/452: readlink d0/d25/d5c/l82 0 2024-03-22T02:28:55.542 INFO:tasks.workunit.client.1.smithi122.stdout:3/424: read d2/f55 [1516888,120358] 0 2024-03-22T02:28:55.545 INFO:tasks.workunit.client.1.smithi122.stdout:7/453: creat d0/d30/f8c x:0 0 0 2024-03-22T02:28:55.546 INFO:tasks.workunit.client.1.smithi122.stdout:4/459: dwrite d0/d12/f82 [0,4194304] 0 2024-03-22T02:28:55.546 INFO:tasks.workunit.client.1.smithi122.stdout:3/425: creat d2/d9/dc/d17/d45/f81 x:0 0 0 2024-03-22T02:28:55.547 INFO:tasks.workunit.client.1.smithi122.stdout:7/454: fsync d0/f16 0 2024-03-22T02:28:55.549 INFO:tasks.workunit.client.1.smithi122.stdout:4/460: dread - d0/d12/d15/d42/f7b zero size 2024-03-22T02:28:55.550 INFO:tasks.workunit.client.1.smithi122.stdout:6/570: truncate d7/de/dd/d23/f63 4424283 0 2024-03-22T02:28:55.554 INFO:tasks.workunit.client.1.smithi122.stdout:7/455: symlink d0/d4/d22/d5f/l8d 0 2024-03-22T02:28:55.559 INFO:tasks.workunit.client.1.smithi122.stdout:7/456: creat d0/d25/d5c/f8e x:0 0 0 2024-03-22T02:28:55.560 INFO:tasks.workunit.client.1.smithi122.stdout:4/461: getdents d0/d12/d15 0 2024-03-22T02:28:55.563 INFO:tasks.workunit.client.1.smithi122.stdout:6/571: dwrite d7/de/dd/d39/f4f [8388608,4194304] 0 2024-03-22T02:28:55.564 INFO:tasks.workunit.client.1.smithi122.stdout:3/426: dread d2/d9/dc/df/d49/f54 [4194304,4194304] 0 2024-03-22T02:28:55.564 INFO:tasks.workunit.client.1.smithi122.stdout:6/572: chown d7/de/dd/d20/f81 1348 1 2024-03-22T02:28:55.568 INFO:tasks.workunit.client.1.smithi122.stdout:4/462: chown d0/d12/d15/d2d/c6b 2 1 2024-03-22T02:28:55.569 INFO:tasks.workunit.client.1.smithi122.stdout:4/463: read d0/d12/d15/d42/d57/f64 [510225,55153] 0 2024-03-22T02:28:55.569 INFO:tasks.workunit.client.0.smithi103.stdout:6/969: sync 2024-03-22T02:28:55.571 INFO:tasks.workunit.client.1.smithi122.stdout:3/427: chown d2/d9/dc/df/d38/f3b 9267689 1 2024-03-22T02:28:55.571 INFO:tasks.workunit.client.0.smithi103.stdout:6/970: symlink d0/d3/d33/d132/d13a/l141 0 2024-03-22T02:28:55.572 INFO:tasks.workunit.client.1.smithi122.stdout:6/573: mknod d7/de/dd/d20/d2d/d54/d76/cb1 0 2024-03-22T02:28:55.572 INFO:tasks.workunit.client.1.smithi122.stdout:3/428: chown d2/l14 1316894 1 2024-03-22T02:28:55.572 INFO:tasks.workunit.client.1.smithi122.stdout:7/457: link d0/d4/l23 d0/d30/l8f 0 2024-03-22T02:28:55.573 INFO:tasks.workunit.client.0.smithi103.stdout:6/971: read d0/d3/d22/d58/d6f/d75/daa/fcf [208768,34934] 0 2024-03-22T02:28:55.574 INFO:tasks.workunit.client.1.smithi122.stdout:7/458: stat d0/d4/d22/d5f/d5e/f77 0 2024-03-22T02:28:55.575 INFO:tasks.workunit.client.0.smithi103.stdout:6/972: mknod d0/d14/d9a/c142 0 2024-03-22T02:28:55.575 INFO:tasks.workunit.client.1.smithi122.stdout:6/574: mkdir d7/de/dd/d20/d2d/d54/d76/d86/d96/db2 0 2024-03-22T02:28:55.578 INFO:tasks.workunit.client.1.smithi122.stdout:3/429: unlink d2/d9/dc/df/d3c/f7a 0 2024-03-22T02:28:55.578 INFO:tasks.workunit.client.0.smithi103.stdout:6/973: link d0/d3/d33/lc4 d0/d14/l143 0 2024-03-22T02:28:55.579 INFO:tasks.workunit.client.1.smithi122.stdout:3/430: fsync d2/f7 0 2024-03-22T02:28:55.580 INFO:tasks.workunit.client.1.smithi122.stdout:7/459: dwrite d0/d4/d20/d21/f75 [0,4194304] 0 2024-03-22T02:28:55.582 INFO:tasks.workunit.client.1.smithi122.stdout:6/575: creat d7/de/dd/d39/fb3 x:0 0 0 2024-03-22T02:28:55.583 INFO:tasks.workunit.client.1.smithi122.stdout:7/460: chown d0/d25/d5c/l82 2413 1 2024-03-22T02:28:55.585 INFO:tasks.workunit.client.0.smithi103.stdout:6/974: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f55 [0,4194304] 0 2024-03-22T02:28:55.587 INFO:tasks.workunit.client.0.smithi103.stdout:6/975: chown d0/d3/d22/d58/db4/deb/d12f/f135 1363 1 2024-03-22T02:28:55.588 INFO:tasks.workunit.client.0.smithi103.stdout:6/976: stat d0/d3/d22/d58/de0/ff0 0 2024-03-22T02:28:55.591 INFO:tasks.workunit.client.0.smithi103.stdout:6/977: mkdir d0/d3/d144 0 2024-03-22T02:28:55.595 INFO:tasks.workunit.client.1.smithi122.stdout:1/533: sync 2024-03-22T02:28:55.597 INFO:tasks.workunit.client.0.smithi103.stdout:6/978: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f62 [0,4194304] 0 2024-03-22T02:28:55.598 INFO:tasks.workunit.client.1.smithi122.stdout:4/464: write d0/d25/f28 [2658797,43634] 0 2024-03-22T02:28:55.600 INFO:tasks.workunit.client.1.smithi122.stdout:4/465: write d0/d25/f3b [1512901,14800] 0 2024-03-22T02:28:55.602 INFO:tasks.workunit.client.1.smithi122.stdout:3/431: rename d2/d9/dc/d57/f5f to d2/d9/dc/d17/d2b/f82 0 2024-03-22T02:28:55.604 INFO:tasks.workunit.client.1.smithi122.stdout:1/534: creat dc/d55/faa x:0 0 0 2024-03-22T02:28:55.604 INFO:tasks.workunit.client.1.smithi122.stdout:4/466: write d0/d12/d15/d42/d57/f8e [782854,77653] 0 2024-03-22T02:28:55.607 INFO:tasks.workunit.client.1.smithi122.stdout:3/432: symlink d2/d9/dc/df/d47/d79/l83 0 2024-03-22T02:28:55.610 INFO:tasks.workunit.client.1.smithi122.stdout:3/433: chown d2/d9/dc/df/f48 1399 1 2024-03-22T02:28:55.611 INFO:tasks.workunit.client.0.smithi103.stdout:6/979: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f40 [0,4194304] 0 2024-03-22T02:28:55.614 INFO:tasks.workunit.client.0.smithi103.stdout:6/980: symlink d0/dbd/d103/l145 0 2024-03-22T02:28:55.616 INFO:tasks.workunit.client.1.smithi122.stdout:6/576: dwrite d7/de/dd/d23/f48 [4194304,4194304] 0 2024-03-22T02:28:55.622 INFO:tasks.workunit.client.0.smithi103.stdout:6/981: dwrite d0/d3/d22/d58/d6f/d75/db0/db1/f12c [0,4194304] 0 2024-03-22T02:28:55.623 INFO:tasks.workunit.client.1.smithi122.stdout:1/535: truncate dc/d3b/f81 1849575 0 2024-03-22T02:28:55.624 INFO:tasks.workunit.client.1.smithi122.stdout:4/467: truncate d0/d12/d15/d3d/f68 2585871 0 2024-03-22T02:28:55.625 INFO:tasks.workunit.client.1.smithi122.stdout:4/468: write d0/d25/f3a [1094705,29454] 0 2024-03-22T02:28:55.625 INFO:tasks.workunit.client.1.smithi122.stdout:3/434: write d2/d9/dc/df/d49/f54 [7596418,64946] 0 2024-03-22T02:28:55.630 INFO:tasks.workunit.client.0.smithi103.stdout:6/982: dread d0/d88/d8e/dae/f10b [0,4194304] 0 2024-03-22T02:28:55.631 INFO:tasks.workunit.client.0.smithi103.stdout:6/983: chown d0/d3/d22/d58/f79 60945256 1 2024-03-22T02:28:55.632 INFO:tasks.workunit.client.0.smithi103.stdout:6/984: truncate d0/d14/dc7/dce/f124 805062 0 2024-03-22T02:28:55.633 INFO:tasks.workunit.client.1.smithi122.stdout:6/577: dwrite d7/de/dd/d23/f47 [0,4194304] 0 2024-03-22T02:28:55.634 INFO:tasks.workunit.client.0.smithi103.stdout:6/985: mkdir d0/dbd/d103/d12d/d146 0 2024-03-22T02:28:55.636 INFO:tasks.workunit.client.0.smithi103.stdout:6/986: stat d0/d3/d22/d58/d6f/d75/db0/db1/d113/ff5 0 2024-03-22T02:28:55.636 INFO:tasks.workunit.client.1.smithi122.stdout:3/435: unlink d2/d9/dc/d17/l43 0 2024-03-22T02:28:55.636 INFO:tasks.workunit.client.1.smithi122.stdout:4/469: creat d0/d12/d61/f97 x:0 0 0 2024-03-22T02:28:55.636 INFO:tasks.workunit.client.1.smithi122.stdout:1/536: dwrite dc/f1a [0,4194304] 0 2024-03-22T02:28:55.638 INFO:tasks.workunit.client.1.smithi122.stdout:7/461: dread d0/d30/f53 [0,4194304] 0 2024-03-22T02:28:55.641 INFO:tasks.workunit.client.0.smithi103.stdout:6/987: rename d0/d3/f42 to d0/d3/d22/d58/d6f/d75/f147 0 2024-03-22T02:28:55.642 INFO:tasks.workunit.client.1.smithi122.stdout:6/578: read d7/de/dd/d20/d2d/d54/d76/d86/f87 [4028662,100098] 0 2024-03-22T02:28:55.642 INFO:tasks.workunit.client.1.smithi122.stdout:4/470: creat d0/d12/d15/d3d/d90/f98 x:0 0 0 2024-03-22T02:28:55.644 INFO:tasks.workunit.client.1.smithi122.stdout:1/537: dread - dc/d3b/d46/d4d/d67/f6c zero size 2024-03-22T02:28:55.644 INFO:tasks.workunit.client.0.smithi103.stdout:6/988: unlink d0/d3/d22/fab 0 2024-03-22T02:28:55.646 INFO:tasks.workunit.client.1.smithi122.stdout:3/436: read d2/d9/dc/df/d49/f54 [1022509,4682] 0 2024-03-22T02:28:55.646 INFO:tasks.workunit.client.1.smithi122.stdout:4/471: mknod d0/d12/c99 0 2024-03-22T02:28:55.648 INFO:tasks.workunit.client.1.smithi122.stdout:7/462: dread d0/f8 [0,4194304] 0 2024-03-22T02:28:55.651 INFO:tasks.workunit.client.1.smithi122.stdout:1/538: rename dc/d55/d99 to dc/d3b/d46/d4d/dab 0 2024-03-22T02:28:55.655 INFO:tasks.workunit.client.1.smithi122.stdout:3/437: symlink d2/d9/dc/df/d3c/d59/l84 0 2024-03-22T02:28:55.657 INFO:tasks.workunit.client.0.smithi103.stdout:6/989: chown d0/d3/d22/d58/d6f/d75/daa 100 1 2024-03-22T02:28:55.658 INFO:tasks.workunit.client.1.smithi122.stdout:1/539: rmdir dc/d4c 39 2024-03-22T02:28:55.662 INFO:tasks.workunit.client.1.smithi122.stdout:4/472: getdents d0/d7a 0 2024-03-22T02:28:55.662 INFO:tasks.workunit.client.0.smithi103.stdout:6/990: write d0/f2c [103369,5553] 0 2024-03-22T02:28:55.664 INFO:tasks.workunit.client.0.smithi103.stdout:6/991: creat d0/d14/d9a/d11a/f148 x:0 0 0 2024-03-22T02:28:55.667 INFO:tasks.workunit.client.0.smithi103.stdout:6/992: mkdir d0/d3/d22/d58/d149 0 2024-03-22T02:28:55.668 INFO:tasks.workunit.client.1.smithi122.stdout:7/463: getdents d0/d25 0 2024-03-22T02:28:55.669 INFO:tasks.workunit.client.0.smithi103.stdout:6/993: fsync d0/d3/d33/f5d 0 2024-03-22T02:28:55.670 INFO:tasks.workunit.client.0.smithi103.stdout:6/994: write d0/d3/d33/f60 [930262,20184] 0 2024-03-22T02:28:55.671 INFO:tasks.workunit.client.0.smithi103.stdout:6/995: dread - d0/dbd/d103/f112 zero size 2024-03-22T02:28:55.672 INFO:tasks.workunit.client.1.smithi122.stdout:4/473: dwrite d0/d25/f3b [0,4194304] 0 2024-03-22T02:28:55.676 INFO:tasks.workunit.client.1.smithi122.stdout:6/579: dread d7/de/dd/d20/d2d/d54/d76/f6b [0,4194304] 0 2024-03-22T02:28:55.677 INFO:tasks.workunit.client.0.smithi103.stdout:6/996: creat d0/d3/d22/d58/d6f/d75/db0/db1/d113/d68/f14a x:0 0 0 2024-03-22T02:28:55.677 INFO:tasks.workunit.client.1.smithi122.stdout:7/464: write d0/f1b [5055249,1612] 0 2024-03-22T02:28:55.679 INFO:tasks.workunit.client.0.smithi103.stdout:6/997: unlink d0/d3/d22/l11f 0 2024-03-22T02:28:55.681 INFO:tasks.workunit.client.1.smithi122.stdout:3/438: dwrite d2/d9/dc/df/d47/f76 [0,4194304] 0 2024-03-22T02:28:55.682 INFO:tasks.workunit.client.1.smithi122.stdout:7/465: fsync d0/d4/d22/d5f/d5e/f77 0 2024-03-22T02:28:55.682 INFO:tasks.workunit.client.1.smithi122.stdout:3/439: readlink d2/d9/dc/l37 0 2024-03-22T02:28:55.683 INFO:tasks.workunit.client.0.smithi103.stdout:6/998: dread d0/d3/d22/d58/d6f/f8b [0,4194304] 0 2024-03-22T02:28:55.686 INFO:tasks.workunit.client.1.smithi122.stdout:6/580: dread d7/de/dd/d39/f77 [0,4194304] 0 2024-03-22T02:28:55.687 INFO:tasks.workunit.client.0.smithi103.stdout:6/999: dread d0/d3/fb8 [0,4194304] 0 2024-03-22T02:28:55.687 INFO:tasks.workunit.client.1.smithi122.stdout:3/440: mkdir d2/d9/dc/df/d49/d85 0 2024-03-22T02:28:55.690 INFO:tasks.workunit.client.0.smithi103.stderr:+ rm -rf -- ./tmp.6FLcEpU5kx 2024-03-22T02:28:55.691 INFO:tasks.workunit.client.1.smithi122.stdout:3/441: symlink d2/d9/dc/d17/d2b/l86 0 2024-03-22T02:28:55.694 INFO:tasks.workunit.client.1.smithi122.stdout:3/442: mkdir d2/d9/dc/df/d38/d87 0 2024-03-22T02:28:55.694 INFO:tasks.workunit.client.1.smithi122.stdout:3/443: fdatasync d2/f15 0 2024-03-22T02:28:55.695 INFO:tasks.workunit.client.1.smithi122.stdout:3/444: chown d2/d9/dc/d17/d2b/d6b/c6e 122 1 2024-03-22T02:28:55.697 INFO:tasks.workunit.client.1.smithi122.stdout:4/474: truncate d0/d12/d15/d42/d4e/de/f5d 3508778 0 2024-03-22T02:28:55.700 INFO:tasks.workunit.client.1.smithi122.stdout:3/445: mknod d2/d9/dc/d17/d2b/c88 0 2024-03-22T02:28:55.701 INFO:tasks.workunit.client.1.smithi122.stdout:7/466: dwrite d0/d4/d20/f27 [0,4194304] 0 2024-03-22T02:28:55.703 INFO:tasks.workunit.client.1.smithi122.stdout:3/446: chown d2/d9/dc/l37 127534 1 2024-03-22T02:28:55.703 INFO:tasks.workunit.client.1.smithi122.stdout:4/475: creat d0/d12/d15/d42/d57/f9a x:0 0 0 2024-03-22T02:28:55.703 INFO:tasks.workunit.client.1.smithi122.stdout:6/581: truncate d7/de/dd/d23/f48 3525595 0 2024-03-22T02:28:55.704 INFO:tasks.workunit.client.1.smithi122.stdout:4/476: chown d0/d12/d15/d3d/d90/c56 113 1 2024-03-22T02:28:55.705 INFO:tasks.workunit.client.1.smithi122.stdout:7/467: creat d0/d4/d20/d21/d28/f90 x:0 0 0 2024-03-22T02:28:55.706 INFO:tasks.workunit.client.1.smithi122.stdout:3/447: mkdir d2/d9/dc/d57/d89 0 2024-03-22T02:28:55.707 INFO:tasks.workunit.client.1.smithi122.stdout:6/582: creat d7/de/dd/d20/d2d/d54/fb4 x:0 0 0 2024-03-22T02:28:55.709 INFO:tasks.workunit.client.1.smithi122.stdout:3/448: mkdir d2/d8a 0 2024-03-22T02:28:55.713 INFO:tasks.workunit.client.1.smithi122.stdout:6/583: truncate d7/de/dd/d20/f3f 913530 0 2024-03-22T02:28:55.714 INFO:tasks.workunit.client.1.smithi122.stdout:7/468: dwrite d0/d30/f5b [0,4194304] 0 2024-03-22T02:28:55.715 INFO:tasks.workunit.client.1.smithi122.stdout:6/584: stat d7/de/dd/d20/d2d/d54/d76 0 2024-03-22T02:28:55.719 INFO:tasks.workunit.client.1.smithi122.stdout:3/449: truncate d2/d9/dc/d17/d2b/f67 683474 0 2024-03-22T02:28:55.723 INFO:tasks.workunit.client.1.smithi122.stdout:3/450: write d2/d9/dc/df/d47/f56 [158083,11972] 0 2024-03-22T02:28:55.727 INFO:tasks.workunit.client.1.smithi122.stdout:7/469: dwrite d0/d4/d22/d5f/d5e/f77 [0,4194304] 0 2024-03-22T02:28:55.730 INFO:tasks.workunit.client.1.smithi122.stdout:3/451: symlink d2/d9/dc/df/l8b 0 2024-03-22T02:28:55.732 INFO:tasks.workunit.client.1.smithi122.stdout:3/452: write d2/d9/dc/d17/d2b/d6a/f7d [832923,115334] 0 2024-03-22T02:28:55.735 INFO:tasks.workunit.client.1.smithi122.stdout:3/453: write d2/d9/dc/d17/d28/f35 [2413013,48340] 0 2024-03-22T02:28:55.736 INFO:tasks.workunit.client.1.smithi122.stdout:7/470: dread d0/d4/d20/d21/f2b [0,4194304] 0 2024-03-22T02:28:55.740 INFO:tasks.workunit.client.1.smithi122.stdout:7/471: symlink d0/d4e/d5d/l91 0 2024-03-22T02:28:55.741 INFO:tasks.workunit.client.1.smithi122.stdout:6/585: dwrite d7/de/dd/f13 [4194304,4194304] 0 2024-03-22T02:28:55.742 INFO:tasks.workunit.client.1.smithi122.stdout:7/472: rmdir d0/d4/d22/d5f/d5e 39 2024-03-22T02:28:55.743 INFO:tasks.workunit.client.1.smithi122.stdout:3/454: getdents d2/d9/dc/d17/d51 0 2024-03-22T02:28:55.744 INFO:tasks.workunit.client.1.smithi122.stdout:6/586: chown d7/de/dd/d20/d2d/d54/d76/cb1 15 1 2024-03-22T02:28:55.746 INFO:tasks.workunit.client.1.smithi122.stdout:7/473: creat d0/d4/d80/f92 x:0 0 0 2024-03-22T02:28:55.746 INFO:tasks.workunit.client.1.smithi122.stdout:3/455: dread - d2/d9/dc/df/d38/f6f zero size 2024-03-22T02:28:55.831 INFO:tasks.workunit.client.1.smithi122.stdout:3/456: creat d2/d9/dc/d17/d2b/d6b/f8c x:0 0 0 2024-03-22T02:28:55.832 INFO:tasks.workunit.client.1.smithi122.stdout:6/587: getdents d7/de/dd/d20/d2d/dad 0 2024-03-22T02:28:55.835 INFO:tasks.workunit.client.1.smithi122.stdout:6/588: creat d7/de/dd/d20/d2d/d54/d76/d86/d96/fb5 x:0 0 0 2024-03-22T02:28:55.835 INFO:tasks.workunit.client.1.smithi122.stdout:7/474: write d0/d4e/d5d/f6e [53160,75358] 0 2024-03-22T02:28:55.836 INFO:tasks.workunit.client.1.smithi122.stdout:3/457: creat d2/d9/f8d x:0 0 0 2024-03-22T02:28:55.837 INFO:tasks.workunit.client.1.smithi122.stdout:6/589: stat d7/de/dd/d20/d2d/d54/d76/l50 0 2024-03-22T02:28:55.838 INFO:tasks.workunit.client.1.smithi122.stdout:3/458: fdatasync d2/d9/dc/d17/d51/f7c 0 2024-03-22T02:28:55.838 INFO:tasks.workunit.client.1.smithi122.stdout:7/475: read d0/f1b [1696393,100009] 0 2024-03-22T02:28:55.838 INFO:tasks.workunit.client.1.smithi122.stdout:6/590: readlink d7/de/l2f 0 2024-03-22T02:28:55.839 INFO:tasks.workunit.client.1.smithi122.stdout:7/476: stat d0/d25/d46/f85 0 2024-03-22T02:28:55.839 INFO:tasks.workunit.client.1.smithi122.stdout:8/564: sync 2024-03-22T02:28:55.840 INFO:tasks.workunit.client.1.smithi122.stdout:6/591: truncate d7/de/fa8 123398 0 2024-03-22T02:28:55.841 INFO:tasks.workunit.client.1.smithi122.stdout:3/459: write d2/d9/dc/d17/d2b/f82 [983852,66313] 0 2024-03-22T02:28:55.845 INFO:tasks.workunit.client.1.smithi122.stdout:6/592: symlink d7/de/dd/d20/d2d/d54/d76/lb6 0 2024-03-22T02:28:55.846 INFO:tasks.workunit.client.1.smithi122.stdout:6/593: stat d7/de/f6c 0 2024-03-22T02:28:55.848 INFO:tasks.workunit.client.1.smithi122.stdout:8/565: rename d1/d8/d8a/d78/f82 to d1/d3f/fc6 0 2024-03-22T02:28:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: pgmap v21: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 64 MiB/s rd, 216 MiB/s wr, 390 op/s 2024-03-22T02:28:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:55.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:55 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:55.849 INFO:tasks.workunit.client.1.smithi122.stdout:7/477: dwrite d0/d4/d22/d5f/d5e/f77 [0,4194304] 0 2024-03-22T02:28:55.850 INFO:tasks.workunit.client.1.smithi122.stdout:7/478: stat d0/d4/d22/d5f/d7d 0 2024-03-22T02:28:55.850 INFO:tasks.workunit.client.1.smithi122.stdout:6/594: chown d7/de/dd/c1f 1747449 1 2024-03-22T02:28:55.852 INFO:tasks.workunit.client.1.smithi122.stdout:8/566: rmdir d1/d81 39 2024-03-22T02:28:55.854 INFO:tasks.workunit.client.1.smithi122.stdout:8/567: dread - d1/d8/d2f/f91 zero size 2024-03-22T02:28:55.858 INFO:tasks.workunit.client.1.smithi122.stdout:7/479: dwrite d0/f84 [0,4194304] 0 2024-03-22T02:28:55.860 INFO:tasks.workunit.client.1.smithi122.stdout:7/480: readlink d0/d4/d22/d5f/d5e/l67 0 2024-03-22T02:28:55.864 INFO:tasks.workunit.client.1.smithi122.stdout:7/481: unlink d0/d4/d22/d5f/c68 0 2024-03-22T02:28:55.865 INFO:tasks.workunit.client.1.smithi122.stdout:7/482: chown d0/d4e/l61 33509195 1 2024-03-22T02:28:55.866 INFO:tasks.workunit.client.1.smithi122.stdout:6/595: truncate d7/de/f43 405975 0 2024-03-22T02:28:55.867 INFO:tasks.workunit.client.1.smithi122.stdout:8/568: truncate d1/d8/d2f/f38 3185534 0 2024-03-22T02:28:55.868 INFO:tasks.workunit.client.1.smithi122.stdout:7/483: dread d0/d25/d5c/f6d [0,4194304] 0 2024-03-22T02:28:55.872 INFO:tasks.workunit.client.1.smithi122.stdout:8/569: symlink d1/d8/d2f/lc7 0 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: pgmap v21: 65 pgs: 65 active+clean; 2.4 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 64 MiB/s rd, 216 MiB/s wr, 390 op/s 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:28:55.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:55.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:55.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:55 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:55.875 INFO:tasks.workunit.client.1.smithi122.stdout:6/596: dread d7/de/dd/d20/fa5 [0,4194304] 0 2024-03-22T02:28:55.875 INFO:tasks.workunit.client.1.smithi122.stdout:7/484: link d0/d4/d20/d57/f6a d0/d4e/d5d/f93 0 2024-03-22T02:28:55.876 INFO:tasks.workunit.client.1.smithi122.stdout:7/485: stat d0/d25/d5c/l82 0 2024-03-22T02:28:55.876 INFO:tasks.workunit.client.1.smithi122.stdout:8/570: creat d1/d60/d73/da3/fc8 x:0 0 0 2024-03-22T02:28:55.877 INFO:tasks.workunit.client.1.smithi122.stdout:7/486: read d0/f8 [1373247,63473] 0 2024-03-22T02:28:55.877 INFO:tasks.workunit.client.1.smithi122.stdout:7/487: readlink d0/d4/d20/l40 0 2024-03-22T02:28:55.881 INFO:tasks.workunit.client.1.smithi122.stdout:8/571: rename d1/d8/d5e to d1/d40/d71/dc9 0 2024-03-22T02:28:55.888 INFO:tasks.workunit.client.1.smithi122.stdout:6/597: write d7/de/dd/d20/f81 [115089,68758] 0 2024-03-22T02:28:55.890 INFO:tasks.workunit.client.1.smithi122.stdout:8/572: dwrite d1/f3d [0,4194304] 0 2024-03-22T02:28:55.897 INFO:tasks.workunit.client.1.smithi122.stdout:8/573: creat d1/d40/d48/d8d/fca x:0 0 0 2024-03-22T02:28:55.897 INFO:tasks.workunit.client.1.smithi122.stdout:6/598: dwrite d7/f65 [0,4194304] 0 2024-03-22T02:28:55.901 INFO:tasks.workunit.client.1.smithi122.stdout:7/488: dread d0/d4/fb [0,4194304] 0 2024-03-22T02:28:55.901 INFO:tasks.workunit.client.1.smithi122.stdout:8/574: unlink d1/d9/d50/f59 0 2024-03-22T02:28:55.905 INFO:tasks.workunit.client.1.smithi122.stdout:8/575: symlink d1/d8/d8a/d78/lcb 0 2024-03-22T02:28:55.907 INFO:tasks.workunit.client.1.smithi122.stdout:7/489: mkdir d0/d4/d94 0 2024-03-22T02:28:55.909 INFO:tasks.workunit.client.1.smithi122.stdout:8/576: creat d1/d40/d71/d97/fcc x:0 0 0 2024-03-22T02:28:55.910 INFO:tasks.workunit.client.1.smithi122.stdout:6/599: dwrite d7/de/d27/f6a [0,4194304] 0 2024-03-22T02:28:55.911 INFO:tasks.workunit.client.1.smithi122.stdout:7/490: creat d0/d4/d20/d21/f95 x:0 0 0 2024-03-22T02:28:55.914 INFO:tasks.workunit.client.1.smithi122.stdout:8/577: readlink d1/d8/d63/l85 0 2024-03-22T02:28:55.916 INFO:tasks.workunit.client.1.smithi122.stdout:7/491: symlink d0/d4/d20/l96 0 2024-03-22T02:28:55.917 INFO:tasks.workunit.client.1.smithi122.stdout:6/600: mknod d7/de/d27/d52/cb7 0 2024-03-22T02:28:55.920 INFO:tasks.workunit.client.1.smithi122.stdout:8/578: rename d1/d40/d48/c66 to d1/d40/d71/ccd 0 2024-03-22T02:28:55.921 INFO:tasks.workunit.client.1.smithi122.stdout:8/579: dread - d1/d4d/f56 zero size 2024-03-22T02:28:55.923 INFO:tasks.workunit.client.1.smithi122.stdout:7/492: symlink d0/d4/d22/d5f/l97 0 2024-03-22T02:28:55.925 INFO:tasks.workunit.client.1.smithi122.stdout:8/580: dread d1/f3d [0,4194304] 0 2024-03-22T02:28:55.926 INFO:tasks.workunit.client.1.smithi122.stdout:6/601: symlink d7/de/dd/d20/d2d/dad/lb8 0 2024-03-22T02:28:55.928 INFO:tasks.workunit.client.1.smithi122.stdout:6/602: chown d7/de/dd/d20/fa5 13 1 2024-03-22T02:28:55.928 INFO:tasks.workunit.client.1.smithi122.stdout:8/581: symlink d1/d60/d73/da3/lce 0 2024-03-22T02:28:55.929 INFO:tasks.workunit.client.1.smithi122.stdout:7/493: dread d0/d4/d20/f27 [0,4194304] 0 2024-03-22T02:28:55.930 INFO:tasks.workunit.client.1.smithi122.stdout:6/603: dread d7/de/fa8 [0,4194304] 0 2024-03-22T02:28:55.932 INFO:tasks.workunit.client.1.smithi122.stdout:6/604: write d7/de/dd/d20/d2d/f5f [314512,71864] 0 2024-03-22T02:28:55.933 INFO:tasks.workunit.client.1.smithi122.stdout:7/494: mkdir d0/d25/d46/d98 0 2024-03-22T02:28:55.934 INFO:tasks.workunit.client.1.smithi122.stdout:7/495: write d0/d30/f5b [1419212,46708] 0 2024-03-22T02:28:55.935 INFO:tasks.workunit.client.1.smithi122.stdout:8/582: getdents d1/d8/d8a 0 2024-03-22T02:28:55.939 INFO:tasks.workunit.client.1.smithi122.stdout:8/583: unlink d1/d9/f26 0 2024-03-22T02:28:55.941 INFO:tasks.workunit.client.1.smithi122.stdout:7/496: creat d0/d4/d22/d5f/d5e/f99 x:0 0 0 2024-03-22T02:28:55.948 INFO:tasks.workunit.client.1.smithi122.stdout:7/497: mkdir d0/d25/d46/d9a 0 2024-03-22T02:28:55.949 INFO:tasks.workunit.client.1.smithi122.stdout:6/605: dread d7/de/dd/f13 [0,4194304] 0 2024-03-22T02:28:55.955 INFO:tasks.workunit.client.1.smithi122.stdout:6/606: mknod d7/de/dd/d20/cb9 0 2024-03-22T02:28:55.956 INFO:tasks.workunit.client.1.smithi122.stdout:7/498: rename d0/d4/d22/d5f to d0/d4/d20/d21/d9b 0 2024-03-22T02:28:55.958 INFO:tasks.workunit.client.1.smithi122.stdout:8/584: write d1/d9/f13 [1256861,117178] 0 2024-03-22T02:28:55.960 INFO:tasks.workunit.client.1.smithi122.stdout:6/607: getdents d7/de/dd/d20/d2d/dae 0 2024-03-22T02:28:55.963 INFO:tasks.workunit.client.1.smithi122.stdout:6/608: unlink d7/de/l2f 0 2024-03-22T02:28:55.964 INFO:tasks.workunit.client.1.smithi122.stdout:6/609: truncate f5 4506502 0 2024-03-22T02:28:55.965 INFO:tasks.workunit.client.1.smithi122.stdout:7/499: getdents d0/d4e 0 2024-03-22T02:28:55.966 INFO:tasks.workunit.client.1.smithi122.stdout:7/500: truncate d0/d30/f8c 24980 0 2024-03-22T02:28:55.967 INFO:tasks.workunit.client.1.smithi122.stdout:6/610: fdatasync d7/de/f3e 0 2024-03-22T02:28:55.969 INFO:tasks.workunit.client.1.smithi122.stdout:6/611: readlink d7/de/dd/d20/d2d/la1 0 2024-03-22T02:28:55.969 INFO:tasks.workunit.client.1.smithi122.stdout:7/501: readlink d0/d25/d46/d48/l7a 0 2024-03-22T02:28:55.972 INFO:tasks.workunit.client.1.smithi122.stdout:7/502: symlink d0/d25/d5c/l9c 0 2024-03-22T02:28:55.974 INFO:tasks.workunit.client.1.smithi122.stdout:8/585: dread d1/d22/f55 [0,4194304] 0 2024-03-22T02:28:55.976 INFO:tasks.workunit.client.1.smithi122.stdout:8/586: creat d1/d9/d50/da6/fcf x:0 0 0 2024-03-22T02:28:55.981 INFO:tasks.workunit.client.1.smithi122.stdout:8/587: write d1/d8/f33 [6926675,105820] 0 2024-03-22T02:28:55.983 INFO:tasks.workunit.client.1.smithi122.stdout:8/588: symlink d1/d60/ld0 0 2024-03-22T02:28:55.988 INFO:tasks.workunit.client.1.smithi122.stdout:6/612: dread d7/de/dd/d20/d2d/f9a [0,4194304] 0 2024-03-22T02:28:55.991 INFO:tasks.workunit.client.1.smithi122.stdout:6/613: mkdir d7/de/dd/d23/dba 0 2024-03-22T02:28:55.991 INFO:tasks.workunit.client.1.smithi122.stdout:8/589: dwrite d1/d40/d48/f6d [4194304,4194304] 0 2024-03-22T02:28:55.992 INFO:tasks.workunit.client.1.smithi122.stdout:8/590: chown d1/d8/d2f/l54 7200078 1 2024-03-22T02:28:55.993 INFO:tasks.workunit.client.1.smithi122.stdout:8/591: stat d1/d22/f64 0 2024-03-22T02:28:55.995 INFO:tasks.workunit.client.1.smithi122.stdout:6/614: creat d7/de/dd/d23/d7f/fbb x:0 0 0 2024-03-22T02:28:55.997 INFO:tasks.workunit.client.1.smithi122.stdout:8/592: truncate d1/fbd 515891 0 2024-03-22T02:28:56.040 INFO:tasks.workunit.client.1.smithi122.stdout:6/615: symlink d7/de/dd/d23/d7f/lbc 0 2024-03-22T02:28:56.041 INFO:tasks.workunit.client.1.smithi122.stdout:6/616: chown d7/f15 1059810454 1 2024-03-22T02:28:56.042 INFO:tasks.workunit.client.1.smithi122.stdout:8/593: mknod d1/d3f/cd1 0 2024-03-22T02:28:56.043 INFO:tasks.workunit.client.1.smithi122.stdout:0/424: sync 2024-03-22T02:28:56.043 INFO:tasks.workunit.client.1.smithi122.stdout:5/445: sync 2024-03-22T02:28:56.044 INFO:tasks.workunit.client.1.smithi122.stdout:0/425: write d11/d14/f56 [4994317,58774] 0 2024-03-22T02:28:56.047 INFO:tasks.workunit.client.1.smithi122.stdout:8/594: unlink d1/d22/f52 0 2024-03-22T02:28:56.051 INFO:tasks.workunit.client.1.smithi122.stdout:5/446: rename d8/d1e/d22/d3a/d46/d4f/c91 to d8/d1e/d22/d78/c9a 0 2024-03-22T02:28:56.052 INFO:tasks.workunit.client.1.smithi122.stdout:6/617: rmdir d7 39 2024-03-22T02:28:56.053 INFO:tasks.workunit.client.1.smithi122.stdout:0/426: creat d11/d14/d23/d24/d33/f8e x:0 0 0 2024-03-22T02:28:56.054 INFO:tasks.workunit.client.1.smithi122.stdout:8/595: dwrite d1/d9/f77 [0,4194304] 0 2024-03-22T02:28:56.059 INFO:tasks.workunit.client.1.smithi122.stdout:6/618: dread - d7/de/dd/d9b/f9e zero size 2024-03-22T02:28:56.060 INFO:tasks.workunit.client.1.smithi122.stdout:0/427: write d11/d16/d44/d89/f3e [963750,15370] 0 2024-03-22T02:28:56.062 INFO:tasks.workunit.client.1.smithi122.stdout:5/447: dwrite d8/d1e/d51/f75 [0,4194304] 0 2024-03-22T02:28:56.066 INFO:tasks.workunit.client.1.smithi122.stdout:6/619: write d7/de/dd/d20/d2d/d54/fb4 [163292,76328] 0 2024-03-22T02:28:56.066 INFO:tasks.workunit.client.1.smithi122.stdout:0/428: read d11/d14/d23/f77 [3914127,25900] 0 2024-03-22T02:28:56.069 INFO:tasks.workunit.client.1.smithi122.stdout:5/448: read d8/d1e/d22/d3a/d46/d4f/f54 [2827056,80945] 0 2024-03-22T02:28:56.070 INFO:tasks.workunit.client.1.smithi122.stdout:0/429: write d11/d14/d1d/f6a [4472287,82642] 0 2024-03-22T02:28:56.071 INFO:tasks.workunit.client.1.smithi122.stdout:8/596: link d1/d40/d51/l8e d1/d3f/d45/ld2 0 2024-03-22T02:28:56.075 INFO:tasks.workunit.client.1.smithi122.stdout:5/449: mkdir d8/d1e/d22/d3a/d46/d4f/d9b 0 2024-03-22T02:28:56.077 INFO:tasks.workunit.client.1.smithi122.stdout:6/620: dwrite d7/de/d27/f36 [0,4194304] 0 2024-03-22T02:28:56.079 INFO:tasks.workunit.client.1.smithi122.stdout:8/597: link d1/d40/d71/dc9/f65 d1/fd3 0 2024-03-22T02:28:56.081 INFO:tasks.workunit.client.1.smithi122.stdout:8/598: chown d1/d9/d50/lb0 248 1 2024-03-22T02:28:56.082 INFO:tasks.workunit.client.1.smithi122.stdout:5/450: symlink d8/l9c 0 2024-03-22T02:28:56.082 INFO:tasks.workunit.client.1.smithi122.stdout:6/621: creat d7/de/dd/d20/d2d/d54/d76/fbd x:0 0 0 2024-03-22T02:28:56.085 INFO:tasks.workunit.client.1.smithi122.stdout:5/451: unlink d8/da/c79 0 2024-03-22T02:28:56.088 INFO:tasks.workunit.client.1.smithi122.stdout:6/622: creat d7/de/dd/d20/d2d/dae/fbe x:0 0 0 2024-03-22T02:28:56.090 INFO:tasks.workunit.client.1.smithi122.stdout:5/452: creat d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f9d x:0 0 0 2024-03-22T02:28:56.093 INFO:tasks.workunit.client.1.smithi122.stdout:6/623: rmdir d7/de/dd/d23/dba 0 2024-03-22T02:28:56.094 INFO:tasks.workunit.client.1.smithi122.stdout:6/624: chown d7/de/d27/l2e 1337573 1 2024-03-22T02:28:56.100 INFO:tasks.workunit.client.1.smithi122.stdout:5/453: dread d8/d1e/f5e [0,4194304] 0 2024-03-22T02:28:56.101 INFO:tasks.workunit.client.1.smithi122.stdout:6/625: link d7/de/dd/d39/l95 d7/de/dd/df/lbf 0 2024-03-22T02:28:56.104 INFO:tasks.workunit.client.1.smithi122.stdout:6/626: mkdir d7/de/dd/d20/d2d/dc0 0 2024-03-22T02:28:56.105 INFO:tasks.workunit.client.1.smithi122.stdout:8/599: truncate d1/f53 925847 0 2024-03-22T02:28:56.105 INFO:tasks.workunit.client.1.smithi122.stdout:6/627: fdatasync d7/de/dd/d23/d7f/fbb 0 2024-03-22T02:28:56.106 INFO:tasks.workunit.client.1.smithi122.stdout:5/454: mknod d8/d1e/d22/d3a/d46/d4f/d55/d6a/c9e 0 2024-03-22T02:28:56.107 INFO:tasks.workunit.client.1.smithi122.stdout:6/628: creat d7/de/d27/fc1 x:0 0 0 2024-03-22T02:28:56.109 INFO:tasks.workunit.client.1.smithi122.stdout:5/455: symlink d8/d1e/d22/d3a/l9f 0 2024-03-22T02:28:56.110 INFO:tasks.workunit.client.1.smithi122.stdout:5/456: write d8/d1a/f70 [412086,81819] 0 2024-03-22T02:28:56.111 INFO:tasks.workunit.client.1.smithi122.stdout:6/629: mkdir d7/de/dd/d39/dc2 0 2024-03-22T02:28:56.112 INFO:tasks.workunit.client.1.smithi122.stdout:5/457: read d8/d1e/d22/d3a/d39/f96 [3874151,10925] 0 2024-03-22T02:28:56.114 INFO:tasks.workunit.client.1.smithi122.stdout:5/458: write d8/d1e/d22/d3a/d46/f4b [1441094,101224] 0 2024-03-22T02:28:56.114 INFO:tasks.workunit.client.1.smithi122.stdout:5/459: read - d8/d1e/d22/d78/f98 zero size 2024-03-22T02:28:56.116 INFO:tasks.workunit.client.1.smithi122.stdout:6/630: rename d7/de/dd/d20/d2d/d54/f5a to d7/de/dd/d20/d2d/d54/d76/fc3 0 2024-03-22T02:28:56.117 INFO:tasks.workunit.client.1.smithi122.stdout:5/460: symlink d8/d1a/la0 0 2024-03-22T02:28:56.118 INFO:tasks.workunit.client.1.smithi122.stdout:6/631: creat d7/de/dd/df/fc4 x:0 0 0 2024-03-22T02:28:56.120 INFO:tasks.workunit.client.1.smithi122.stdout:5/461: unlink d8/d1e/d22/d3a/l9f 0 2024-03-22T02:28:56.122 INFO:tasks.workunit.client.1.smithi122.stdout:5/462: rmdir d8/d1e/d22/d3a/d46 39 2024-03-22T02:28:56.124 INFO:tasks.workunit.client.1.smithi122.stdout:6/632: getdents d7/de/dd/d20/d2d/dae 0 2024-03-22T02:28:56.125 INFO:tasks.workunit.client.1.smithi122.stdout:5/463: stat d8/d1e/d22/d3a/d46/d4f/d9b 0 2024-03-22T02:28:56.126 INFO:tasks.workunit.client.1.smithi122.stdout:5/464: read d8/d3b/f62 [69120,8862] 0 2024-03-22T02:28:56.127 INFO:tasks.workunit.client.1.smithi122.stdout:5/465: chown d8/da/f10 2205 1 2024-03-22T02:28:56.130 INFO:tasks.workunit.client.1.smithi122.stdout:5/466: mknod d8/d1a/ca1 0 2024-03-22T02:28:56.135 INFO:tasks.workunit.client.1.smithi122.stdout:5/467: dread d8/d1e/d51/f75 [0,4194304] 0 2024-03-22T02:28:56.138 INFO:tasks.workunit.client.1.smithi122.stdout:6/633: dread d7/de/dd/d23/f48 [0,4194304] 0 2024-03-22T02:28:56.139 INFO:tasks.workunit.client.1.smithi122.stdout:5/468: creat d8/d1e/d22/d3a/d46/d4f/fa2 x:0 0 0 2024-03-22T02:28:56.145 INFO:tasks.workunit.client.1.smithi122.stdout:5/469: dread d8/f26 [0,4194304] 0 2024-03-22T02:28:56.148 INFO:tasks.workunit.client.1.smithi122.stdout:6/634: dwrite d7/de/dd/df/fc4 [0,4194304] 0 2024-03-22T02:28:56.148 INFO:tasks.workunit.client.1.smithi122.stdout:5/470: mkdir d8/d1e/d22/da3 0 2024-03-22T02:28:56.149 INFO:tasks.workunit.client.1.smithi122.stdout:6/635: chown d7/de/fa8 3581309 1 2024-03-22T02:28:56.149 INFO:tasks.workunit.client.1.smithi122.stdout:5/471: read d8/f26 [706490,45463] 0 2024-03-22T02:28:56.152 INFO:tasks.workunit.client.1.smithi122.stdout:5/472: unlink d8/f20 0 2024-03-22T02:28:56.153 INFO:tasks.workunit.client.1.smithi122.stdout:6/636: creat d7/de/dd/d20/d2d/dae/fc5 x:0 0 0 2024-03-22T02:28:56.157 INFO:tasks.workunit.client.1.smithi122.stdout:5/473: rename d8/da/d28/d35/d71/f72 to d8/d1a/fa4 0 2024-03-22T02:28:56.157 INFO:tasks.workunit.client.1.smithi122.stdout:6/637: rmdir d7/de/dd/d23 39 2024-03-22T02:28:56.159 INFO:tasks.workunit.client.1.smithi122.stdout:5/474: dread - d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f9d zero size 2024-03-22T02:28:56.161 INFO:tasks.workunit.client.1.smithi122.stdout:5/475: rmdir d8/da 39 2024-03-22T02:28:56.164 INFO:tasks.workunit.client.1.smithi122.stdout:6/638: link d7/de/dd/d20/d2d/d54/d76/f4e d7/de/dd/d20/d2d/dad/fc6 0 2024-03-22T02:28:56.164 INFO:tasks.workunit.client.1.smithi122.stdout:6/639: write f5 [3243767,5780] 0 2024-03-22T02:28:56.165 INFO:tasks.workunit.client.1.smithi122.stdout:5/476: rename d8/d1e/f5e to d8/d1e/d22/da3/fa5 0 2024-03-22T02:28:56.167 INFO:tasks.workunit.client.1.smithi122.stdout:6/640: unlink d7/de/dd/d20/d8f/fb0 0 2024-03-22T02:28:56.168 INFO:tasks.workunit.client.1.smithi122.stdout:6/641: stat d7/de/dd/d20/d2d/f30 0 2024-03-22T02:28:56.169 INFO:tasks.workunit.client.1.smithi122.stdout:5/477: rename d8/c9 to d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/ca6 0 2024-03-22T02:28:56.170 INFO:tasks.workunit.client.1.smithi122.stdout:6/642: creat d7/de/dd/df/fc7 x:0 0 0 2024-03-22T02:28:56.171 INFO:tasks.workunit.client.1.smithi122.stdout:5/478: dread - d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d zero size 2024-03-22T02:28:56.173 INFO:tasks.workunit.client.1.smithi122.stdout:6/643: symlink d7/de/dd/d20/d8f/lc8 0 2024-03-22T02:28:56.175 INFO:tasks.workunit.client.1.smithi122.stdout:5/479: rename d8/d1e/d22/d3a/d46/d4f/c50 to d8/d1e/d22/d3a/d46/d4f/ca7 0 2024-03-22T02:28:56.178 INFO:tasks.workunit.client.1.smithi122.stdout:6/644: creat d7/de/dd/fc9 x:0 0 0 2024-03-22T02:28:56.180 INFO:tasks.workunit.client.1.smithi122.stdout:6/645: creat d7/de/dd/d20/d2d/d54/fca x:0 0 0 2024-03-22T02:28:56.183 INFO:tasks.workunit.client.1.smithi122.stdout:5/480: read d8/d1e/d22/da3/fa5 [930029,43536] 0 2024-03-22T02:28:56.186 INFO:tasks.workunit.client.1.smithi122.stdout:6/646: dread d7/de/dd/f56 [0,4194304] 0 2024-03-22T02:28:56.189 INFO:tasks.workunit.client.1.smithi122.stdout:5/481: dread d8/da/d28/d42/d4e/f58 [0,4194304] 0 2024-03-22T02:28:56.191 INFO:tasks.workunit.client.1.smithi122.stdout:6/647: getdents d7/de/dd/d20/d2d/d54/d76/d86/d96 0 2024-03-22T02:28:56.193 INFO:tasks.workunit.client.1.smithi122.stdout:5/482: creat d8/d1e/d22/d3a/d39/fa8 x:0 0 0 2024-03-22T02:28:56.196 INFO:tasks.workunit.client.1.smithi122.stdout:6/648: dread d7/fa [0,4194304] 0 2024-03-22T02:28:56.200 INFO:tasks.workunit.client.1.smithi122.stdout:5/483: dwrite d8/da/d28/d42/f95 [0,4194304] 0 2024-03-22T02:28:56.200 INFO:tasks.workunit.client.1.smithi122.stdout:6/649: dread d7/de/dd/d39/f77 [0,4194304] 0 2024-03-22T02:28:56.204 INFO:tasks.workunit.client.1.smithi122.stdout:5/484: mkdir d8/d1e/d22/d3a/d46/d4f/d9b/da9 0 2024-03-22T02:28:56.204 INFO:tasks.workunit.client.1.smithi122.stdout:6/650: getdents d7/de/dd/d20/d2d/dc0 0 2024-03-22T02:28:56.206 INFO:tasks.workunit.client.1.smithi122.stdout:5/485: write d8/da/d28/d35/f68 [5951649,2557] 0 2024-03-22T02:28:56.207 INFO:tasks.workunit.client.1.smithi122.stdout:6/651: symlink d7/lcb 0 2024-03-22T02:28:56.208 INFO:tasks.workunit.client.1.smithi122.stdout:6/652: write d7/de/d27/f6a [776900,113171] 0 2024-03-22T02:28:56.211 INFO:tasks.workunit.client.1.smithi122.stdout:6/653: write d7/de/dd/d39/f73 [980005,118725] 0 2024-03-22T02:28:56.212 INFO:tasks.workunit.client.1.smithi122.stdout:6/654: chown d7/de/f92 476578440 1 2024-03-22T02:28:56.215 INFO:tasks.workunit.client.1.smithi122.stdout:6/655: mknod d7/de/dd/d20/d2d/d54/d76/ccc 0 2024-03-22T02:28:56.216 INFO:tasks.workunit.client.1.smithi122.stdout:6/656: creat d7/de/dd/d20/d2d/dae/fcd x:0 0 0 2024-03-22T02:28:56.218 INFO:tasks.workunit.client.1.smithi122.stdout:6/657: write d7/de/dd/d39/fa2 [5114432,55183] 0 2024-03-22T02:28:56.223 INFO:tasks.workunit.client.1.smithi122.stdout:5/486: dread d8/d1a/f1c [0,4194304] 0 2024-03-22T02:28:56.295 INFO:tasks.workunit.client.1.smithi122.stdout:6/658: dread d7/de/f92 [0,4194304] 0 2024-03-22T02:28:56.295 INFO:tasks.workunit.client.1.smithi122.stdout:5/487: dwrite d8/d1e/d22/d3a/d39/fa8 [0,4194304] 0 2024-03-22T02:28:56.295 INFO:tasks.workunit.client.1.smithi122.stdout:6/659: mknod d7/de/d27/cce 0 2024-03-22T02:28:56.302 INFO:tasks.workunit.client.1.smithi122.stdout:6/660: dwrite d7/de/dd/fc9 [0,4194304] 0 2024-03-22T02:28:56.307 INFO:tasks.workunit.client.1.smithi122.stdout:6/661: dread d7/de/dd/d20/d2d/f9a [0,4194304] 0 2024-03-22T02:28:56.311 INFO:tasks.workunit.client.1.smithi122.stdout:6/662: creat d7/de/dd/d20/d2d/d54/d76/d86/d96/db2/fcf x:0 0 0 2024-03-22T02:28:56.313 INFO:tasks.workunit.client.1.smithi122.stdout:6/663: creat d7/de/dd/d20/d8f/fd0 x:0 0 0 2024-03-22T02:28:56.316 INFO:tasks.workunit.client.1.smithi122.stdout:6/664: rename d7/de/dd/d20/d2d/d54/l82 to d7/de/dd/df/ld1 0 2024-03-22T02:28:56.320 INFO:tasks.workunit.client.1.smithi122.stdout:6/665: link d7/de/dd/d20/d2d/d54/d76/l91 d7/de/d27/ld2 0 2024-03-22T02:28:56.322 INFO:tasks.workunit.client.1.smithi122.stdout:6/666: write d7/de/dd/d20/f81 [1655542,40387] 0 2024-03-22T02:28:56.322 INFO:tasks.workunit.client.1.smithi122.stdout:6/667: chown d7/f1b 304 1 2024-03-22T02:28:56.323 INFO:tasks.workunit.client.1.smithi122.stdout:6/668: read d7/de/dd/f3a [549584,68089] 0 2024-03-22T02:28:56.325 INFO:tasks.workunit.client.1.smithi122.stdout:6/669: write d7/de/dd/f13 [4453430,23400] 0 2024-03-22T02:28:56.328 INFO:tasks.workunit.client.1.smithi122.stdout:6/670: symlink d7/de/dd/d20/d2d/ld3 0 2024-03-22T02:28:56.329 INFO:tasks.workunit.client.1.smithi122.stdout:6/671: chown d7/c41 0 1 2024-03-22T02:28:56.330 INFO:tasks.workunit.client.1.smithi122.stdout:6/672: chown d7/de/d27/f64 2811 1 2024-03-22T02:28:56.336 INFO:tasks.workunit.client.1.smithi122.stdout:6/673: dwrite d7/f65 [0,4194304] 0 2024-03-22T02:28:56.340 INFO:tasks.workunit.client.1.smithi122.stdout:6/674: rename d7/de/d27/c55 to d7/de/dd/d23/cd4 0 2024-03-22T02:28:56.343 INFO:tasks.workunit.client.1.smithi122.stdout:6/675: dread d7/de/dd/d20/d2d/d54/fb4 [0,4194304] 0 2024-03-22T02:28:56.345 INFO:tasks.workunit.client.1.smithi122.stdout:6/676: dread d7/de/fa8 [0,4194304] 0 2024-03-22T02:28:56.348 INFO:tasks.workunit.client.1.smithi122.stdout:6/677: creat d7/de/dd/d20/d2d/dad/fd5 x:0 0 0 2024-03-22T02:28:56.364 INFO:tasks.workunit.client.1.smithi122.stdout:6/678: dread d7/de/dd/d20/f21 [0,4194304] 0 2024-03-22T02:28:56.366 INFO:tasks.workunit.client.1.smithi122.stdout:6/679: dread d7/de/dd/d23/f47 [0,4194304] 0 2024-03-22T02:28:56.379 INFO:tasks.workunit.client.1.smithi122.stdout:6/680: dread d7/de/dd/d20/d2d/d54/d76/f89 [0,4194304] 0 2024-03-22T02:28:56.380 INFO:tasks.workunit.client.1.smithi122.stdout:6/681: chown d7/de/dd/d20/d2d/d54/f9d 28173924 1 2024-03-22T02:28:56.381 INFO:tasks.workunit.client.1.smithi122.stdout:6/682: dread d7/de/f43 [0,4194304] 0 2024-03-22T02:28:56.385 INFO:tasks.workunit.client.1.smithi122.stdout:6/683: creat d7/de/dd/d20/fd6 x:0 0 0 2024-03-22T02:28:56.386 INFO:tasks.workunit.client.1.smithi122.stdout:6/684: chown d7/de/dd/d20/d2d/dad/lb8 6 1 2024-03-22T02:28:56.387 INFO:tasks.workunit.client.1.smithi122.stdout:6/685: write d7/de/dd/df/fc4 [3162200,106624] 0 2024-03-22T02:28:56.389 INFO:tasks.workunit.client.1.smithi122.stdout:6/686: symlink d7/de/dd/d20/d2d/dc0/ld7 0 2024-03-22T02:28:56.395 INFO:tasks.workunit.client.1.smithi122.stdout:6/687: dwrite d7/f65 [0,4194304] 0 2024-03-22T02:28:56.398 INFO:tasks.workunit.client.1.smithi122.stdout:6/688: mkdir d7/de/dd/d20/d2d/d54/d76/d86/d96/db2/dd8 0 2024-03-22T02:28:56.399 INFO:tasks.workunit.client.1.smithi122.stdout:6/689: fdatasync d7/de/dd/d20/d2d/dae/fbe 0 2024-03-22T02:28:56.401 INFO:tasks.workunit.client.1.smithi122.stdout:6/690: mknod d7/de/dd/d23/d7f/cd9 0 2024-03-22T02:28:56.404 INFO:tasks.workunit.client.1.smithi122.stdout:6/691: symlink d7/de/dd/d20/d2d/d54/d76/d86/d96/db2/lda 0 2024-03-22T02:28:56.405 INFO:tasks.workunit.client.1.smithi122.stdout:6/692: write d7/de/dd/d20/d8f/fd0 [375821,120307] 0 2024-03-22T02:28:56.408 INFO:tasks.workunit.client.1.smithi122.stdout:6/693: creat d7/de/dd/d23/d7f/fdb x:0 0 0 2024-03-22T02:28:56.411 INFO:tasks.workunit.client.1.smithi122.stdout:6/694: rename d7/de/dd/df/fc4 to d7/fdc 0 2024-03-22T02:28:56.418 INFO:tasks.workunit.client.1.smithi122.stdout:6/695: write d7/de/dd/d20/f3f [1363276,33124] 0 2024-03-22T02:28:56.421 INFO:tasks.workunit.client.1.smithi122.stdout:6/696: unlink d7/de/dd/d39/f5b 0 2024-03-22T02:28:56.422 INFO:tasks.workunit.client.1.smithi122.stdout:6/697: chown d7/de/dd/d20/f21 15811499 1 2024-03-22T02:28:56.435 INFO:tasks.workunit.client.1.smithi122.stdout:6/698: dread d7/de/dd/d20/f51 [0,4194304] 0 2024-03-22T02:28:56.437 INFO:tasks.workunit.client.1.smithi122.stdout:6/699: creat d7/de/dd/d20/d2d/d54/d76/d86/d96/db2/fdd x:0 0 0 2024-03-22T02:28:56.442 INFO:tasks.workunit.client.1.smithi122.stdout:6/700: write d7/de/dd/d23/f48 [4295579,78282] 0 2024-03-22T02:28:56.445 INFO:tasks.workunit.client.1.smithi122.stdout:6/701: creat d7/de/dd/d20/d2d/d54/d76/fde x:0 0 0 2024-03-22T02:28:56.449 INFO:tasks.workunit.client.1.smithi122.stdout:6/702: unlink d7/de/dd/d20/d2d/d54/d76/d86/d96/fb5 0 2024-03-22T02:28:56.450 INFO:tasks.workunit.client.1.smithi122.stdout:6/703: rmdir d7/de/dd/d9b 39 2024-03-22T02:28:56.453 INFO:tasks.workunit.client.1.smithi122.stdout:6/704: symlink d7/de/dd/d20/d2d/dad/ldf 0 2024-03-22T02:28:56.461 INFO:tasks.workunit.client.1.smithi122.stdout:6/705: rename d7/de/dd/d20/d2d to d7/de/dd/d20/de0 0 2024-03-22T02:28:56.461 INFO:tasks.workunit.client.1.smithi122.stdout:6/706: chown d7/de/d27/f36 2782 1 2024-03-22T02:28:56.463 INFO:tasks.workunit.client.1.smithi122.stdout:6/707: fsync d7/de/dd/d9b/f9e 0 2024-03-22T02:28:56.465 INFO:tasks.workunit.client.1.smithi122.stdout:6/708: creat d7/de/d27/d52/fe1 x:0 0 0 2024-03-22T02:28:56.467 INFO:tasks.workunit.client.1.smithi122.stdout:6/709: creat d7/fe2 x:0 0 0 2024-03-22T02:28:56.468 INFO:tasks.workunit.client.1.smithi122.stdout:6/710: write d7/de/d27/f64 [852977,54435] 0 2024-03-22T02:28:56.470 INFO:tasks.workunit.client.1.smithi122.stdout:6/711: creat d7/de/dd/d23/d7f/fe3 x:0 0 0 2024-03-22T02:28:56.471 INFO:tasks.workunit.client.1.smithi122.stdout:6/712: truncate d7/de/dd/d20/de0/d54/f9d 378352 0 2024-03-22T02:28:56.472 INFO:tasks.workunit.client.1.smithi122.stdout:6/713: dread - d7/de/dd/d20/de0/d54/d76/d86/d96/db2/fdd zero size 2024-03-22T02:28:56.474 INFO:tasks.workunit.client.1.smithi122.stdout:6/714: mknod d7/de/dd/d23/ce4 0 2024-03-22T02:28:56.477 INFO:tasks.workunit.client.1.smithi122.stdout:6/715: unlink d7/de/dd/d20/de0/d54/fca 0 2024-03-22T02:28:56.478 INFO:tasks.workunit.client.1.smithi122.stdout:6/716: chown d7/de/dd/d23/f63 41 1 2024-03-22T02:28:56.490 INFO:tasks.workunit.client.1.smithi122.stdout:6/717: dwrite d7/de/dd/d20/de0/d54/d76/fc3 [0,4194304] 0 2024-03-22T02:28:56.492 INFO:tasks.workunit.client.1.smithi122.stdout:2/605: sync 2024-03-22T02:28:56.496 INFO:tasks.workunit.client.1.smithi122.stdout:6/718: creat d7/de/dd/d20/de0/d54/d76/fe5 x:0 0 0 2024-03-22T02:28:56.500 INFO:tasks.workunit.client.1.smithi122.stdout:6/719: truncate d7/de/dd/f42 3508887 0 2024-03-22T02:28:56.501 INFO:tasks.workunit.client.1.smithi122.stdout:6/720: dread - d7/de/d27/d52/fe1 zero size 2024-03-22T02:28:56.504 INFO:tasks.workunit.client.1.smithi122.stdout:2/606: write df/d1f/d33/d4d/d94/dc2/d72/fcc [2059636,64320] 0 2024-03-22T02:28:56.505 INFO:tasks.workunit.client.1.smithi122.stdout:2/607: fdatasync df/d1f/d33/f80 0 2024-03-22T02:28:56.506 INFO:tasks.workunit.client.1.smithi122.stdout:6/721: link d7/de/dd/d20/de0/d54/d76/f89 d7/de/dd/d20/de0/dae/fe6 0 2024-03-22T02:28:56.508 INFO:tasks.workunit.client.1.smithi122.stdout:6/722: write d7/de/dd/df/fc7 [375308,34013] 0 2024-03-22T02:28:56.508 INFO:tasks.workunit.client.1.smithi122.stdout:2/608: creat df/d1f/d33/d4d/d94/dc2/d51/fcd x:0 0 0 2024-03-22T02:28:56.509 INFO:tasks.workunit.client.1.smithi122.stdout:2/609: dread - df/d1f/d33/d6a/f90 zero size 2024-03-22T02:28:56.520 INFO:tasks.workunit.client.1.smithi122.stdout:6/723: dread d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:28:56.522 INFO:tasks.workunit.client.1.smithi122.stdout:6/724: creat d7/de/dd/d20/d8f/fe7 x:0 0 0 2024-03-22T02:28:56.524 INFO:tasks.workunit.client.1.smithi122.stdout:6/725: symlink d7/de/dd/le8 0 2024-03-22T02:28:56.651 INFO:tasks.workunit.client.1.smithi122.stdout:7/503: sync 2024-03-22T02:28:56.651 INFO:tasks.workunit.client.1.smithi122.stdout:1/540: sync 2024-03-22T02:28:56.651 INFO:tasks.workunit.client.1.smithi122.stdout:9/601: sync 2024-03-22T02:28:56.651 INFO:tasks.workunit.client.1.smithi122.stdout:0/430: sync 2024-03-22T02:28:56.652 INFO:tasks.workunit.client.1.smithi122.stdout:5/488: sync 2024-03-22T02:28:56.652 INFO:tasks.workunit.client.1.smithi122.stdout:8/600: sync 2024-03-22T02:28:56.652 INFO:tasks.workunit.client.1.smithi122.stdout:3/460: sync 2024-03-22T02:28:56.652 INFO:tasks.workunit.client.1.smithi122.stdout:4/477: sync 2024-03-22T02:28:56.654 INFO:tasks.workunit.client.1.smithi122.stdout:7/504: read d0/d30/f53 [940433,97907] 0 2024-03-22T02:28:56.654 INFO:tasks.workunit.client.1.smithi122.stdout:9/602: read db/d1b/f6a [242988,71526] 0 2024-03-22T02:28:56.655 INFO:tasks.workunit.client.1.smithi122.stdout:0/431: read d11/f40 [2782118,111179] 0 2024-03-22T02:28:56.656 INFO:tasks.workunit.client.1.smithi122.stdout:5/489: chown d8/da/d28/f3d 11 1 2024-03-22T02:28:56.656 INFO:tasks.workunit.client.1.smithi122.stdout:9/603: chown db/d23/l12 46197 1 2024-03-22T02:28:56.657 INFO:tasks.workunit.client.1.smithi122.stdout:1/541: unlink dc/f4a 0 2024-03-22T02:28:56.658 INFO:tasks.workunit.client.1.smithi122.stdout:8/601: creat d1/d9/d50/da6/fd4 x:0 0 0 2024-03-22T02:28:56.661 INFO:tasks.workunit.client.1.smithi122.stdout:0/432: creat d11/d14/d1b/f8f x:0 0 0 2024-03-22T02:28:56.662 INFO:tasks.workunit.client.1.smithi122.stdout:1/542: mknod dc/d3b/d46/cac 0 2024-03-22T02:28:56.664 INFO:tasks.workunit.client.1.smithi122.stdout:4/478: fdatasync d0/d12/d15/d42/d4e/de/f5d 0 2024-03-22T02:28:56.666 INFO:tasks.workunit.client.1.smithi122.stdout:9/604: rename db/d1b/d6c/l3c to db/d23/d1f/d57/lbe 0 2024-03-22T02:28:56.678 INFO:tasks.workunit.client.1.smithi122.stdout:1/543: dwrite dc/f80 [0,4194304] 0 2024-03-22T02:28:56.678 INFO:tasks.workunit.client.1.smithi122.stdout:0/433: rename d11/d14/f56 to d11/d16/d44/f90 0 2024-03-22T02:28:56.679 INFO:tasks.workunit.client.1.smithi122.stdout:8/602: mknod d1/d22/d6a/dc4/cd5 0 2024-03-22T02:28:56.680 INFO:tasks.workunit.client.1.smithi122.stdout:3/461: link d2/d9/dc/d17/d2b/d6b/l80 d2/d9/d7e/l8e 0 2024-03-22T02:28:56.687 INFO:tasks.workunit.client.1.smithi122.stdout:9/605: rename db/df/l7c to db/df/d3e/d59/lbf 0 2024-03-22T02:28:56.690 INFO:tasks.workunit.client.1.smithi122.stdout:8/603: mknod d1/d22/d6a/dc4/cd6 0 2024-03-22T02:28:56.691 INFO:tasks.workunit.client.1.smithi122.stdout:7/505: truncate d0/d30/f5b 3000354 0 2024-03-22T02:28:56.694 INFO:tasks.workunit.client.1.smithi122.stdout:0/434: rename d11/d83 to d11/d14/d1b/d91 0 2024-03-22T02:28:56.695 INFO:tasks.workunit.client.1.smithi122.stdout:4/479: write d0/d12/d15/d42/d4e/de/d40/f60 [692939,59774] 0 2024-03-22T02:28:56.696 INFO:tasks.workunit.client.1.smithi122.stdout:8/604: creat d1/d40/d71/db4/fd7 x:0 0 0 2024-03-22T02:28:56.698 INFO:tasks.workunit.client.1.smithi122.stdout:0/435: fsync d11/d4f/d5a/f5c 0 2024-03-22T02:28:56.698 INFO:tasks.workunit.client.1.smithi122.stdout:3/462: rename d2/d9/dc/df/l8b to d2/d9/dc/df/d3c/d78/l8f 0 2024-03-22T02:28:56.700 INFO:tasks.workunit.client.1.smithi122.stdout:7/506: dwrite d0/d30/f52 [0,4194304] 0 2024-03-22T02:28:56.701 INFO:tasks.workunit.client.1.smithi122.stdout:3/463: write d2/d9/dc/d17/d28/f6c [808171,16252] 0 2024-03-22T02:28:56.703 INFO:tasks.workunit.client.1.smithi122.stdout:3/464: chown d2/d9/dc/df/d38/f6f 15 1 2024-03-22T02:28:56.704 INFO:tasks.workunit.client.1.smithi122.stdout:3/465: chown d2/d9/dc/d17/d28/f6c 75360 1 2024-03-22T02:28:56.705 INFO:tasks.workunit.client.1.smithi122.stdout:4/480: creat d0/d12/d15/d42/d4e/de/f9b x:0 0 0 2024-03-22T02:28:56.706 INFO:tasks.workunit.client.1.smithi122.stdout:4/481: chown d0/d12/d15/d42/f89 45293 1 2024-03-22T02:28:56.706 INFO:tasks.workunit.client.1.smithi122.stdout:8/605: creat d1/d9/d50/db1/fd8 x:0 0 0 2024-03-22T02:28:56.708 INFO:tasks.workunit.client.1.smithi122.stdout:9/606: rename db/d23/d1f to db/d23/d1f/d57/d66/dc0 22 2024-03-22T02:28:56.709 INFO:tasks.workunit.client.1.smithi122.stdout:4/482: read d0/d25/f28 [3631824,126798] 0 2024-03-22T02:28:56.709 INFO:tasks.workunit.client.1.smithi122.stdout:1/544: write dc/d55/f77 [1041276,99987] 0 2024-03-22T02:28:56.712 INFO:tasks.workunit.client.1.smithi122.stdout:0/436: symlink d11/d14/l92 0 2024-03-22T02:28:56.712 INFO:tasks.workunit.client.1.smithi122.stdout:7/507: mknod d0/d25/d5c/c9d 0 2024-03-22T02:28:56.722 INFO:tasks.workunit.client.1.smithi122.stdout:0/437: symlink d11/d4f/d5a/l93 0 2024-03-22T02:28:56.723 INFO:tasks.workunit.client.1.smithi122.stdout:9/607: dwrite db/df/d3e/d81/fb6 [0,4194304] 0 2024-03-22T02:28:56.730 INFO:tasks.workunit.client.1.smithi122.stdout:1/545: dwrite dc/d3b/d46/d4d/f7d [0,4194304] 0 2024-03-22T02:28:56.731 INFO:tasks.workunit.client.1.smithi122.stdout:7/508: dwrite d0/d30/f8c [0,4194304] 0 2024-03-22T02:28:56.734 INFO:tasks.workunit.client.1.smithi122.stdout:1/546: chown dc/d3b/d46/d4d/d67/f6c 305360 1 2024-03-22T02:28:56.739 INFO:tasks.workunit.client.1.smithi122.stdout:4/483: write d0/d12/d15/d42/d4e/de/f5d [2833491,100737] 0 2024-03-22T02:28:56.741 INFO:tasks.workunit.client.1.smithi122.stdout:9/608: write db/df/d3e/d59/db5/f8a [1146182,67319] 0 2024-03-22T02:28:56.747 INFO:tasks.workunit.client.1.smithi122.stdout:1/547: dread dc/f10 [0,4194304] 0 2024-03-22T02:28:56.747 INFO:tasks.workunit.client.1.smithi122.stdout:9/609: write db/d23/d1f/d57/d66/d73/f91 [243133,78548] 0 2024-03-22T02:28:56.748 INFO:tasks.workunit.client.1.smithi122.stdout:7/509: dwrite d0/d4/f2d [0,4194304] 0 2024-03-22T02:28:56.750 INFO:tasks.workunit.client.1.smithi122.stdout:1/548: fdatasync dc/f1a 0 2024-03-22T02:28:56.752 INFO:tasks.workunit.client.1.smithi122.stdout:3/466: dwrite d2/d9/dc/df/d3c/f7b [0,4194304] 0 2024-03-22T02:28:56.753 INFO:tasks.workunit.client.1.smithi122.stdout:0/438: getdents d11/d14/d1d 0 2024-03-22T02:28:56.760 INFO:tasks.workunit.client.1.smithi122.stdout:7/510: dread d0/d4e/d5d/f6e [0,4194304] 0 2024-03-22T02:28:56.760 INFO:tasks.workunit.client.1.smithi122.stdout:9/610: rename db/df/d3e/d81/d29 to db/d1b/d28/dc1 0 2024-03-22T02:28:56.761 INFO:tasks.workunit.client.1.smithi122.stdout:3/467: write d2/d9/dc/df/d38/f3b [386670,71003] 0 2024-03-22T02:28:56.763 INFO:tasks.workunit.client.1.smithi122.stdout:8/606: dwrite d1/d40/f4a [0,4194304] 0 2024-03-22T02:28:56.766 INFO:tasks.workunit.client.1.smithi122.stdout:0/439: fsync d11/f3d 0 2024-03-22T02:28:56.766 INFO:tasks.workunit.client.1.smithi122.stdout:9/611: fsync db/d23/d1f/fa0 0 2024-03-22T02:28:56.769 INFO:tasks.workunit.client.1.smithi122.stdout:7/511: creat d0/d4e/f9e x:0 0 0 2024-03-22T02:28:56.771 INFO:tasks.workunit.client.1.smithi122.stdout:7/512: write d0/d25/d46/f85 [1882021,54060] 0 2024-03-22T02:28:56.772 INFO:tasks.workunit.client.1.smithi122.stdout:1/549: dread dc/d4c/f8b [0,4194304] 0 2024-03-22T02:28:56.772 INFO:tasks.workunit.client.1.smithi122.stdout:0/440: symlink d11/d14/d1b/l94 0 2024-03-22T02:28:56.778 INFO:tasks.workunit.client.1.smithi122.stdout:7/513: dread d0/d25/d5c/f6d [0,4194304] 0 2024-03-22T02:28:56.780 INFO:tasks.workunit.client.1.smithi122.stdout:3/468: write d2/d9/dc/d17/f63 [5091576,15618] 0 2024-03-22T02:28:56.781 INFO:tasks.workunit.client.1.smithi122.stdout:1/550: symlink dc/d3b/lad 0 2024-03-22T02:28:56.783 INFO:tasks.workunit.client.1.smithi122.stdout:3/469: chown d2/d9/dc/d17/d28/d4c/c52 147 1 2024-03-22T02:28:56.783 INFO:tasks.workunit.client.1.smithi122.stdout:0/441: dwrite d11/d14/d1d/d27/d32/f6c [0,4194304] 0 2024-03-22T02:28:56.785 INFO:tasks.workunit.client.1.smithi122.stdout:7/514: rename d0/l1c to d0/d25/d46/d9a/l9f 0 2024-03-22T02:28:56.789 INFO:tasks.workunit.client.1.smithi122.stdout:1/551: write dc/d3b/d46/fa7 [3062928,95868] 0 2024-03-22T02:28:56.793 INFO:tasks.workunit.client.1.smithi122.stdout:3/470: link d2/d9/dc/df/d38/f5d d2/d9/dc/df/d38/d87/f90 0 2024-03-22T02:28:56.794 INFO:tasks.workunit.client.1.smithi122.stdout:3/471: chown d2/d9/dc/l37 0 1 2024-03-22T02:28:56.795 INFO:tasks.workunit.client.1.smithi122.stdout:7/515: symlink d0/d4/d94/la0 0 2024-03-22T02:28:56.796 INFO:tasks.workunit.client.1.smithi122.stdout:7/516: write d0/d4/d20/d21/d9b/d5e/f77 [736042,97962] 0 2024-03-22T02:28:56.796 INFO:tasks.workunit.client.1.smithi122.stdout:7/517: chown d0/d25/d46/d9a 13482 1 2024-03-22T02:28:56.798 INFO:tasks.workunit.client.1.smithi122.stdout:1/552: rename dc/f49 to dc/d3b/d8d/d72/d9c/fae 0 2024-03-22T02:28:56.798 INFO:tasks.workunit.client.1.smithi122.stdout:0/442: fdatasync d11/d14/d1b/f38 0 2024-03-22T02:28:56.799 INFO:tasks.workunit.client.1.smithi122.stdout:4/484: dread d0/d12/d61/f74 [0,4194304] 0 2024-03-22T02:28:56.800 INFO:tasks.workunit.client.1.smithi122.stdout:0/443: chown d11/d16/f45 205111 1 2024-03-22T02:28:56.800 INFO:tasks.workunit.client.1.smithi122.stdout:3/472: creat d2/d9/dc/d57/f91 x:0 0 0 2024-03-22T02:28:56.802 INFO:tasks.workunit.client.1.smithi122.stdout:7/518: mkdir d0/d25/da1 0 2024-03-22T02:28:56.803 INFO:tasks.workunit.client.1.smithi122.stdout:1/553: creat dc/d3b/d46/d4d/dab/faf x:0 0 0 2024-03-22T02:28:56.806 INFO:tasks.workunit.client.1.smithi122.stdout:0/444: creat d11/d14/d23/d24/d33/f95 x:0 0 0 2024-03-22T02:28:56.806 INFO:tasks.workunit.client.1.smithi122.stdout:4/485: chown d0/d12/d15/d42/d4e/f5a 71 1 2024-03-22T02:28:56.807 INFO:tasks.workunit.client.1.smithi122.stdout:9/612: truncate db/d1b/d6c/f4f 2877402 0 2024-03-22T02:28:56.808 INFO:tasks.workunit.client.1.smithi122.stdout:3/473: rmdir d2/d9/dc/d17/d2b/d3f 39 2024-03-22T02:28:56.809 INFO:tasks.workunit.client.1.smithi122.stdout:7/519: rmdir d0/d4/d20/d21/d28 39 2024-03-22T02:28:56.811 INFO:tasks.workunit.client.1.smithi122.stdout:4/486: creat d0/d12/d15/d42/f9c x:0 0 0 2024-03-22T02:28:56.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/474: write d2/d9/dc/f36 [2418949,126364] 0 2024-03-22T02:28:56.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/613: dread db/df/d3e/d81/d3d/fa1 [0,4194304] 0 2024-03-22T02:28:56.818 INFO:tasks.workunit.client.1.smithi122.stdout:8/607: dwrite d1/d40/f47 [0,4194304] 0 2024-03-22T02:28:56.819 INFO:tasks.workunit.client.1.smithi122.stdout:1/554: dwrite dc/d4c/f62 [0,4194304] 0 2024-03-22T02:28:56.821 INFO:tasks.workunit.client.1.smithi122.stdout:7/520: read - d0/d25/d46/f71 zero size 2024-03-22T02:28:56.822 INFO:tasks.workunit.client.1.smithi122.stdout:7/521: chown d0/d4e 39 1 2024-03-22T02:28:56.831 INFO:tasks.workunit.client.1.smithi122.stdout:8/608: write d1/d22/f55 [1861614,113902] 0 2024-03-22T02:28:56.833 INFO:tasks.workunit.client.1.smithi122.stdout:7/522: dwrite d0/d25/d5c/f8e [0,4194304] 0 2024-03-22T02:28:56.837 INFO:tasks.workunit.client.1.smithi122.stdout:1/555: rename dc/d3b/d46/d4d to dc/d3b/d8d/d72/d9c/db0 0 2024-03-22T02:28:56.840 INFO:tasks.workunit.client.1.smithi122.stdout:1/556: readlink dc/d3b/d46/d47/la1 0 2024-03-22T02:28:56.843 INFO:tasks.workunit.client.1.smithi122.stdout:8/609: rmdir d1/d8/dba/dc0/db9 39 2024-03-22T02:28:56.845 INFO:tasks.workunit.client.1.smithi122.stdout:7/523: mknod d0/d25/d46/d9a/ca2 0 2024-03-22T02:28:56.846 INFO:tasks.workunit.client.1.smithi122.stdout:8/610: write d1/d60/d73/da3/fc8 [502903,32852] 0 2024-03-22T02:28:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:56 smithi122 ceph-mon[36403]: Adjusting osd_memory_target on smithi103 to 1557M 2024-03-22T02:28:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:56 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:28:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:56 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:28:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:56 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:28:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:56 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:28:56.849 INFO:tasks.workunit.client.1.smithi122.stdout:9/614: getdents db/d23/d1f/d57 0 2024-03-22T02:28:56.850 INFO:tasks.workunit.client.1.smithi122.stdout:8/611: readlink d1/d22/d6a/lac 0 2024-03-22T02:28:56.851 INFO:tasks.workunit.client.1.smithi122.stdout:1/557: link dc/d3b/d8d/d72/d9c/db0/d67/f6c dc/d3b/d8d/d72/d9c/db0/d71/fb1 0 2024-03-22T02:28:56.853 INFO:tasks.workunit.client.1.smithi122.stdout:7/524: link d0/d4/l7c d0/d30/la3 0 2024-03-22T02:28:56.855 INFO:tasks.workunit.client.1.smithi122.stdout:0/445: dwrite d11/d4f/d5a/f61 [4194304,4194304] 0 2024-03-22T02:28:56.859 INFO:tasks.workunit.client.1.smithi122.stdout:7/525: read d0/d25/d5c/f8e [3732639,69135] 0 2024-03-22T02:28:56.862 INFO:tasks.workunit.client.1.smithi122.stdout:7/526: fdatasync d0/d4/f2d 0 2024-03-22T02:28:56.862 INFO:tasks.workunit.client.1.smithi122.stdout:3/475: dread d2/d9/dc/d17/d28/f35 [0,4194304] 0 2024-03-22T02:28:56.862 INFO:tasks.workunit.client.1.smithi122.stdout:8/612: truncate d1/d40/f9e 1028603 0 2024-03-22T02:28:56.863 INFO:tasks.workunit.client.1.smithi122.stdout:9/615: fsync db/d1b/d6c/d58/d9d/fa6 0 2024-03-22T02:28:56.864 INFO:tasks.workunit.client.1.smithi122.stdout:4/487: dwrite d0/d25/f5e [0,4194304] 0 2024-03-22T02:28:56.869 INFO:tasks.workunit.client.1.smithi122.stdout:9/616: chown db/df/d3e/d81/d3d/d86 15 1 2024-03-22T02:28:56.871 INFO:tasks.workunit.client.1.smithi122.stdout:1/558: dwrite dc/f80 [0,4194304] 0 2024-03-22T02:28:56.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:56 smithi103 ceph-mon[29258]: Adjusting osd_memory_target on smithi103 to 1557M 2024-03-22T02:28:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:56 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:28:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:56 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:28:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:56 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:28:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:56 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:28:56.874 INFO:tasks.workunit.client.1.smithi122.stdout:8/613: truncate d1/d8/dba/dc0/db9/fc2 225675 0 2024-03-22T02:28:56.874 INFO:tasks.workunit.client.1.smithi122.stdout:7/527: unlink d0/d25/l60 0 2024-03-22T02:28:56.875 INFO:tasks.workunit.client.1.smithi122.stdout:4/488: creat d0/d12/d15/d42/d4e/de/d40/f9d x:0 0 0 2024-03-22T02:28:56.877 INFO:tasks.workunit.client.1.smithi122.stdout:3/476: mknod d2/d9/dc/df/d3c/d59/c92 0 2024-03-22T02:28:56.878 INFO:tasks.workunit.client.1.smithi122.stdout:1/559: write dc/f2f [2236599,40492] 0 2024-03-22T02:28:56.879 INFO:tasks.workunit.client.1.smithi122.stdout:4/489: mknod d0/d12/c9e 0 2024-03-22T02:28:56.880 INFO:tasks.workunit.client.1.smithi122.stdout:9/617: dread db/d23/d9a/fa3 [0,4194304] 0 2024-03-22T02:28:56.880 INFO:tasks.workunit.client.1.smithi122.stdout:8/614: dread d1/d40/d71/fc3 [0,4194304] 0 2024-03-22T02:28:56.882 INFO:tasks.workunit.client.1.smithi122.stdout:7/528: rename d0/d4/d20/d21/f75 to d0/d4/d94/fa4 0 2024-03-22T02:28:56.884 INFO:tasks.workunit.client.1.smithi122.stdout:9/618: creat db/d23/d1e/dad/fc2 x:0 0 0 2024-03-22T02:28:56.886 INFO:tasks.workunit.client.1.smithi122.stdout:8/615: symlink d1/d40/d71/ld9 0 2024-03-22T02:28:56.887 INFO:tasks.workunit.client.1.smithi122.stdout:8/616: chown d1/d40/f9e 3126 1 2024-03-22T02:28:56.888 INFO:tasks.workunit.client.1.smithi122.stdout:0/446: dread d11/d14/d23/f77 [0,4194304] 0 2024-03-22T02:28:56.888 INFO:tasks.workunit.client.1.smithi122.stdout:7/529: creat d0/d4/d94/fa5 x:0 0 0 2024-03-22T02:28:56.888 INFO:tasks.workunit.client.1.smithi122.stdout:8/617: dread - d1/d60/d73/f94 zero size 2024-03-22T02:28:56.889 INFO:tasks.workunit.client.1.smithi122.stdout:7/530: fdatasync d0/f87 0 2024-03-22T02:28:56.890 INFO:tasks.workunit.client.1.smithi122.stdout:9/619: mkdir db/df/d3e/d59/db5/dc3 0 2024-03-22T02:28:56.890 INFO:tasks.workunit.client.1.smithi122.stdout:0/447: write d11/d14/d23/f72 [378546,33084] 0 2024-03-22T02:28:56.895 INFO:tasks.workunit.client.1.smithi122.stdout:1/560: link dc/d2b/d86/f9f dc/d3b/d8d/d72/d9c/db0/d71/fb2 0 2024-03-22T02:28:56.896 INFO:tasks.workunit.client.1.smithi122.stdout:9/620: truncate db/d23/d1f/d57/d66/d73/f91 932299 0 2024-03-22T02:28:56.898 INFO:tasks.workunit.client.1.smithi122.stdout:2/610: sync 2024-03-22T02:28:56.899 INFO:tasks.workunit.client.1.smithi122.stdout:8/618: dwrite d1/d8/d63/d88/f9a [0,4194304] 0 2024-03-22T02:28:56.903 INFO:tasks.workunit.client.1.smithi122.stdout:1/561: truncate dc/d3b/d8d/d72/f96 690292 0 2024-03-22T02:28:56.905 INFO:tasks.workunit.client.1.smithi122.stdout:2/611: creat df/d1f/d33/d6a/d7c/fce x:0 0 0 2024-03-22T02:28:56.906 INFO:tasks.workunit.client.1.smithi122.stdout:0/448: dwrite d11/d14/d1d/f80 [0,4194304] 0 2024-03-22T02:28:56.907 INFO:tasks.workunit.client.1.smithi122.stdout:8/619: creat d1/d4d/fda x:0 0 0 2024-03-22T02:28:56.910 INFO:tasks.workunit.client.1.smithi122.stdout:1/562: symlink dc/d3b/d8d/d72/d9c/lb3 0 2024-03-22T02:28:56.913 INFO:tasks.workunit.client.1.smithi122.stdout:4/490: dread d0/d12/d15/d3d/d90/f78 [0,4194304] 0 2024-03-22T02:28:56.914 INFO:tasks.workunit.client.1.smithi122.stdout:9/621: unlink db/d23/d1f/d57/d66/c92 0 2024-03-22T02:28:56.919 INFO:tasks.workunit.client.1.smithi122.stdout:0/449: symlink d11/d14/d1b/d91/l96 0 2024-03-22T02:28:56.921 INFO:tasks.workunit.client.1.smithi122.stdout:3/477: truncate d2/d9/dc/d17/d28/f69 3356429 0 2024-03-22T02:28:56.924 INFO:tasks.workunit.client.1.smithi122.stdout:4/491: mknod d0/d12/d15/d2d/c9f 0 2024-03-22T02:28:56.926 INFO:tasks.workunit.client.1.smithi122.stdout:2/612: fdatasync df/d1f/d3a/d4f/f70 0 2024-03-22T02:28:56.928 INFO:tasks.workunit.client.1.smithi122.stdout:7/531: truncate d0/d30/f52 1676505 0 2024-03-22T02:28:56.929 INFO:tasks.workunit.client.1.smithi122.stdout:7/532: readlink d0/d25/l6c 0 2024-03-22T02:28:56.930 INFO:tasks.workunit.client.1.smithi122.stdout:9/622: dwrite fa [8388608,4194304] 0 2024-03-22T02:28:56.930 INFO:tasks.workunit.client.1.smithi122.stdout:3/478: creat d2/d9/dc/d57/f93 x:0 0 0 2024-03-22T02:28:56.935 INFO:tasks.workunit.client.1.smithi122.stdout:0/450: creat d11/d14/d1d/d27/d2f/d4e/f97 x:0 0 0 2024-03-22T02:28:56.937 INFO:tasks.workunit.client.1.smithi122.stdout:0/451: chown d11/f40 1 1 2024-03-22T02:28:56.937 INFO:tasks.workunit.client.1.smithi122.stdout:9/623: dwrite db/df/f7d [4194304,4194304] 0 2024-03-22T02:28:56.939 INFO:tasks.workunit.client.1.smithi122.stdout:0/452: dread - d11/d14/d1d/d27/d2f/d4e/f97 zero size 2024-03-22T02:28:56.941 INFO:tasks.workunit.client.1.smithi122.stdout:0/453: truncate d11/d16/d44/f76 932036 0 2024-03-22T02:28:56.944 INFO:tasks.workunit.client.1.smithi122.stdout:4/492: creat d0/d12/d15/d42/d4e/de/fa0 x:0 0 0 2024-03-22T02:28:56.944 INFO:tasks.workunit.client.1.smithi122.stdout:1/563: creat dc/d3b/fb4 x:0 0 0 2024-03-22T02:28:56.945 INFO:tasks.workunit.client.1.smithi122.stdout:8/620: write d1/fd3 [453597,54959] 0 2024-03-22T02:28:56.945 INFO:tasks.workunit.client.1.smithi122.stdout:2/613: rename df/d1f/d33/d4d/d94/dc2/c2c to df/d1f/d33/d34/ccf 0 2024-03-22T02:28:56.950 INFO:tasks.workunit.client.1.smithi122.stdout:7/533: creat d0/d4e/fa6 x:0 0 0 2024-03-22T02:28:56.952 INFO:tasks.workunit.client.1.smithi122.stdout:0/454: dwrite d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:56.953 INFO:tasks.workunit.client.1.smithi122.stdout:3/479: unlink d2/d9/dc/df/d3c/f41 0 2024-03-22T02:28:56.954 INFO:tasks.workunit.client.1.smithi122.stdout:9/624: rmdir db/d1b/da8/dba 39 2024-03-22T02:28:56.957 INFO:tasks.workunit.client.1.smithi122.stdout:1/564: rmdir dc/d3b/d8d/d94 39 2024-03-22T02:28:56.960 INFO:tasks.workunit.client.1.smithi122.stdout:8/621: unlink d1/d8/d63/d88/f9a 0 2024-03-22T02:28:56.962 INFO:tasks.workunit.client.1.smithi122.stdout:8/622: truncate d1/d40/d71/db4/fbb 308173 0 2024-03-22T02:28:56.962 INFO:tasks.workunit.client.1.smithi122.stdout:8/623: readlink d1/d9/d50/lb0 0 2024-03-22T02:28:56.963 INFO:tasks.workunit.client.1.smithi122.stdout:2/614: creat df/d1f/d3d/d52/fd0 x:0 0 0 2024-03-22T02:28:56.963 INFO:tasks.workunit.client.1.smithi122.stdout:8/624: dread - d1/d81/fa5 zero size 2024-03-22T02:28:56.967 INFO:tasks.workunit.client.1.smithi122.stdout:9/625: mkdir db/df/d3e/d59/db5/dc4 0 2024-03-22T02:28:56.968 INFO:tasks.workunit.client.1.smithi122.stdout:7/534: dwrite d0/f1b [0,4194304] 0 2024-03-22T02:28:56.968 INFO:tasks.workunit.client.1.smithi122.stdout:9/626: fsync db/d23/fa9 0 2024-03-22T02:28:56.970 INFO:tasks.workunit.client.1.smithi122.stdout:1/565: mkdir dc/d3b/d46/db5 0 2024-03-22T02:28:56.973 INFO:tasks.workunit.client.1.smithi122.stdout:1/566: truncate dc/d3b/d8d/d72/d9c/db0/f7a 838190 0 2024-03-22T02:28:56.979 INFO:tasks.workunit.client.1.smithi122.stdout:2/615: mknod df/d1f/d33/d4d/d94/dc2/d51/d67/cd1 0 2024-03-22T02:28:56.979 INFO:tasks.workunit.client.1.smithi122.stdout:5/490: sync 2024-03-22T02:28:56.979 INFO:tasks.workunit.client.1.smithi122.stdout:6/726: sync 2024-03-22T02:28:56.980 INFO:tasks.workunit.client.1.smithi122.stdout:0/455: dread d11/f40 [0,4194304] 0 2024-03-22T02:28:56.981 INFO:tasks.workunit.client.1.smithi122.stdout:8/625: mknod d1/d40/d71/db4/cdb 0 2024-03-22T02:28:56.982 INFO:tasks.workunit.client.1.smithi122.stdout:6/727: write d7/de/d27/d52/fac [652263,89440] 0 2024-03-22T02:28:56.987 INFO:tasks.workunit.client.1.smithi122.stdout:5/491: dwrite d8/da/d28/d35/f4c [0,4194304] 0 2024-03-22T02:28:56.989 INFO:tasks.workunit.client.1.smithi122.stdout:5/492: chown d8/da/d28/d42 65081728 1 2024-03-22T02:28:56.990 INFO:tasks.workunit.client.1.smithi122.stdout:4/493: truncate d0/d12/d15/d42/d4e/de/f8b 51218 0 2024-03-22T02:28:56.991 INFO:tasks.workunit.client.1.smithi122.stdout:4/494: write d0/d12/f79 [364050,35603] 0 2024-03-22T02:28:56.996 INFO:tasks.workunit.client.1.smithi122.stdout:0/456: symlink d11/d47/l98 0 2024-03-22T02:28:56.997 INFO:tasks.workunit.client.1.smithi122.stdout:3/480: truncate d2/d9/dc/df/d47/f76 1513577 0 2024-03-22T02:28:56.997 INFO:tasks.workunit.client.1.smithi122.stdout:0/457: stat d11/d14/d23/d24/d33/f3f 0 2024-03-22T02:28:56.999 INFO:tasks.workunit.client.1.smithi122.stdout:2/616: rename df/d1f/d33/d4d/d94/dc2/d51/d67/db5 to df/d1f/d23/dd2 0 2024-03-22T02:28:57.001 INFO:tasks.workunit.client.1.smithi122.stdout:8/626: mkdir d1/d8/ddc 0 2024-03-22T02:28:57.002 INFO:tasks.workunit.client.1.smithi122.stdout:6/728: symlink d7/de/dd/d20/de0/dc0/le9 0 2024-03-22T02:28:57.004 INFO:tasks.workunit.client.1.smithi122.stdout:7/535: mknod d0/d25/d46/ca7 0 2024-03-22T02:28:57.005 INFO:tasks.workunit.client.1.smithi122.stdout:7/536: read d0/d25/d5c/f6d [247878,10996] 0 2024-03-22T02:28:57.006 INFO:tasks.workunit.client.1.smithi122.stdout:7/537: fsync d0/d4e/f9e 0 2024-03-22T02:28:57.007 INFO:tasks.workunit.client.1.smithi122.stdout:7/538: write d0/d4/d22/f55 [647884,53063] 0 2024-03-22T02:28:57.007 INFO:tasks.workunit.client.1.smithi122.stdout:9/627: mkdir db/d1b/d6c/d58/d9d/dc5 0 2024-03-22T02:28:57.009 INFO:tasks.workunit.client.1.smithi122.stdout:4/495: stat d0/d12/d15/d42/d4e/l63 0 2024-03-22T02:28:57.009 INFO:tasks.workunit.client.1.smithi122.stdout:9/628: chown db/df/d3e/d81/d25/c43 179 1 2024-03-22T02:28:57.012 INFO:tasks.workunit.client.1.smithi122.stdout:4/496: write d0/d12/d15/d42/d4e/de/f44 [4325670,85724] 0 2024-03-22T02:28:57.013 INFO:tasks.workunit.client.1.smithi122.stdout:1/567: dread dc/ff [0,4194304] 0 2024-03-22T02:28:57.014 INFO:tasks.workunit.client.1.smithi122.stdout:5/493: dread d8/da/f10 [0,4194304] 0 2024-03-22T02:28:57.018 INFO:tasks.workunit.client.1.smithi122.stdout:8/627: mknod d1/d40/d71/d97/cdd 0 2024-03-22T02:28:57.019 INFO:tasks.workunit.client.1.smithi122.stdout:8/628: dread - d1/d8/d8a/fad zero size 2024-03-22T02:28:57.021 INFO:tasks.workunit.client.1.smithi122.stdout:8/629: write d1/d60/d73/d90/f9b [1712270,80821] 0 2024-03-22T02:28:57.022 INFO:tasks.workunit.client.1.smithi122.stdout:7/539: mknod d0/d4e/d5d/ca8 0 2024-03-22T02:28:57.023 INFO:tasks.workunit.client.1.smithi122.stdout:4/497: dread d0/d25/f3a [0,4194304] 0 2024-03-22T02:28:57.023 INFO:tasks.workunit.client.1.smithi122.stdout:8/630: write d1/d40/d48/faf [159902,68618] 0 2024-03-22T02:28:57.024 INFO:tasks.workunit.client.1.smithi122.stdout:0/458: dread d11/d14/d1d/d27/d32/f5f [0,4194304] 0 2024-03-22T02:28:57.024 INFO:tasks.workunit.client.1.smithi122.stdout:3/481: stat d2/d9/dc/df/d38/f3b 0 2024-03-22T02:28:57.025 INFO:tasks.workunit.client.1.smithi122.stdout:8/631: chown d1/d8/d2f/f98 254647 1 2024-03-22T02:28:57.026 INFO:tasks.workunit.client.1.smithi122.stdout:9/629: creat db/df/d3e/d59/db5/fc6 x:0 0 0 2024-03-22T02:28:57.027 INFO:tasks.workunit.client.1.smithi122.stdout:1/568: symlink dc/d2b/d86/lb6 0 2024-03-22T02:28:57.028 INFO:tasks.workunit.client.1.smithi122.stdout:2/617: mknod df/d1f/d23/dd2/dca/cd3 0 2024-03-22T02:28:57.029 INFO:tasks.workunit.client.1.smithi122.stdout:5/494: rename d8/d1e/d22/d3a/d46/f64 to d8/d1e/d22/d3a/d46/faa 0 2024-03-22T02:28:57.030 INFO:tasks.workunit.client.1.smithi122.stdout:6/729: mknod d7/de/dd/d39/cea 0 2024-03-22T02:28:57.031 INFO:tasks.workunit.client.1.smithi122.stdout:6/730: write d7/de/dd/d23/d7f/fdb [973646,69851] 0 2024-03-22T02:28:57.033 INFO:tasks.workunit.client.1.smithi122.stdout:7/540: write d0/d4/d20/d21/f2b [1015715,115150] 0 2024-03-22T02:28:57.034 INFO:tasks.workunit.client.1.smithi122.stdout:4/498: mkdir d0/d12/d15/d42/d4e/de/da1 0 2024-03-22T02:28:57.038 INFO:tasks.workunit.client.1.smithi122.stdout:8/632: rename d1/d22/d6a/dc4 to d1/d40/d71/d97/db8/dde 0 2024-03-22T02:28:57.040 INFO:tasks.workunit.client.1.smithi122.stdout:5/495: creat d8/da/d28/d35/d71/fab x:0 0 0 2024-03-22T02:28:57.041 INFO:tasks.workunit.client.1.smithi122.stdout:6/731: symlink d7/de/dd/d23/leb 0 2024-03-22T02:28:57.042 INFO:tasks.workunit.client.1.smithi122.stdout:4/499: mkdir d0/d12/da2 0 2024-03-22T02:28:57.045 INFO:tasks.workunit.client.1.smithi122.stdout:4/500: fdatasync d0/d12/d15/d42/d57/f9a 0 2024-03-22T02:28:57.045 INFO:tasks.workunit.client.1.smithi122.stdout:9/630: symlink db/d23/d1f/d57/d66/lc7 0 2024-03-22T02:28:57.049 INFO:tasks.workunit.client.1.smithi122.stdout:6/732: dwrite d7/de/dd/d23/d7f/fe3 [0,4194304] 0 2024-03-22T02:28:57.052 INFO:tasks.workunit.client.1.smithi122.stdout:2/618: rename df/d1f/d3a to df/d1f/d33/d6a/d7c/d8e/da0/dd4 0 2024-03-22T02:28:57.055 INFO:tasks.workunit.client.1.smithi122.stdout:9/631: dwrite db/d23/d1f/d57/fb8 [0,4194304] 0 2024-03-22T02:28:57.059 INFO:tasks.workunit.client.1.smithi122.stdout:0/459: getdents d11/d14/d1d/d27 0 2024-03-22T02:28:57.064 INFO:tasks.workunit.client.1.smithi122.stdout:7/541: rename d0/f3b to d0/d4/d94/fa9 0 2024-03-22T02:28:57.065 INFO:tasks.workunit.client.1.smithi122.stdout:2/619: dwrite df/d1f/d33/f46 [0,4194304] 0 2024-03-22T02:28:57.076 INFO:tasks.workunit.client.1.smithi122.stdout:8/633: rename d1/d40/d48/l79 to d1/d4d/ldf 0 2024-03-22T02:28:57.078 INFO:tasks.workunit.client.1.smithi122.stdout:1/569: dwrite dc/d2b/f8c [0,4194304] 0 2024-03-22T02:28:57.078 INFO:tasks.workunit.client.1.smithi122.stdout:5/496: link d8/d1a/la0 d8/da/d28/d42/d4e/lac 0 2024-03-22T02:28:57.079 INFO:tasks.workunit.client.1.smithi122.stdout:3/482: getdents d2/d9/dc/d17/d2b/d3f 0 2024-03-22T02:28:57.082 INFO:tasks.workunit.client.1.smithi122.stdout:9/632: unlink db/d23/c90 0 2024-03-22T02:28:57.084 INFO:tasks.workunit.client.1.smithi122.stdout:9/633: dread - db/d23/d1f/fa0 zero size 2024-03-22T02:28:57.085 INFO:tasks.workunit.client.1.smithi122.stdout:9/634: read - db/df/d3e/d81/d25/f53 zero size 2024-03-22T02:28:57.087 INFO:tasks.workunit.client.1.smithi122.stdout:8/634: creat d1/d9/d50/fe0 x:0 0 0 2024-03-22T02:28:57.089 INFO:tasks.workunit.client.1.smithi122.stdout:5/497: write d8/d3b/f69 [80820,22790] 0 2024-03-22T02:28:57.094 INFO:tasks.workunit.client.1.smithi122.stdout:4/501: getdents d0/d12/d15/d3d 0 2024-03-22T02:28:57.095 INFO:tasks.workunit.client.1.smithi122.stdout:6/733: getdents d7/de/dd/d20/de0/dae 0 2024-03-22T02:28:57.096 INFO:tasks.workunit.client.1.smithi122.stdout:0/460: link d11/d14/d1d/d27/l2b d11/d16/d37/d84/d87/l99 0 2024-03-22T02:28:57.098 INFO:tasks.workunit.client.1.smithi122.stdout:7/542: mkdir d0/d4/d20/d21/d9b/d5e/daa 0 2024-03-22T02:28:57.100 INFO:tasks.workunit.client.1.smithi122.stdout:1/570: creat dc/d3b/d46/d47/da5/fb7 x:0 0 0 2024-03-22T02:28:57.104 INFO:tasks.workunit.client.1.smithi122.stdout:2/620: link df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/lad df/d1f/d33/d4d/d94/ld5 0 2024-03-22T02:28:57.107 INFO:tasks.workunit.client.1.smithi122.stdout:0/461: rmdir d11/d14/d23/d24/d33 39 2024-03-22T02:28:57.107 INFO:tasks.workunit.client.1.smithi122.stdout:2/621: fsync df/d1f/d23/faf 0 2024-03-22T02:28:57.109 INFO:tasks.workunit.client.1.smithi122.stdout:7/543: dread d0/d4/d22/f55 [0,4194304] 0 2024-03-22T02:28:57.109 INFO:tasks.workunit.client.1.smithi122.stdout:8/635: mkdir d1/de1 0 2024-03-22T02:28:57.110 INFO:tasks.workunit.client.1.smithi122.stdout:3/483: dread d2/f8 [0,4194304] 0 2024-03-22T02:28:57.110 INFO:tasks.workunit.client.1.smithi122.stdout:9/635: dwrite db/df/d3e/d81/d25/fae [4194304,4194304] 0 2024-03-22T02:28:57.112 INFO:tasks.workunit.client.1.smithi122.stdout:5/498: truncate d8/da/d28/d35/f37 3678271 0 2024-03-22T02:28:57.112 INFO:tasks.workunit.client.1.smithi122.stdout:4/502: write d0/d25/f84 [421814,111446] 0 2024-03-22T02:28:57.116 INFO:tasks.workunit.client.1.smithi122.stdout:7/544: write d0/d25/d46/f71 [68297,36090] 0 2024-03-22T02:28:57.118 INFO:tasks.workunit.client.1.smithi122.stdout:6/734: creat d7/de/dd/d20/fec x:0 0 0 2024-03-22T02:28:57.120 INFO:tasks.workunit.client.1.smithi122.stdout:4/503: readlink d0/d12/d15/d42/d57/l88 0 2024-03-22T02:28:57.126 INFO:tasks.workunit.client.1.smithi122.stdout:0/462: dread d11/f3d [0,4194304] 0 2024-03-22T02:28:57.126 INFO:tasks.workunit.client.1.smithi122.stdout:1/571: dread dc/f51 [0,4194304] 0 2024-03-22T02:28:57.128 INFO:tasks.workunit.client.1.smithi122.stdout:2/622: link df/d1f/d33/d6a/cc5 df/d1f/d23/dd2/cd6 0 2024-03-22T02:28:57.128 INFO:tasks.workunit.client.1.smithi122.stdout:8/636: dwrite d1/d22/f8b [0,4194304] 0 2024-03-22T02:28:57.133 INFO:tasks.workunit.client.1.smithi122.stdout:4/504: rename d0/d12/d15/d3d/f62 to d0/d12/d61/fa3 0 2024-03-22T02:28:57.133 INFO:tasks.workunit.client.1.smithi122.stdout:0/463: write d11/d14/d23/f6d [1227803,74034] 0 2024-03-22T02:28:57.134 INFO:tasks.workunit.client.1.smithi122.stdout:9/636: dread f8 [0,4194304] 0 2024-03-22T02:28:57.134 INFO:tasks.workunit.client.1.smithi122.stdout:5/499: creat d8/d1e/d22/d3a/d7a/fad x:0 0 0 2024-03-22T02:28:57.134 INFO:tasks.workunit.client.1.smithi122.stdout:8/637: write d1/d22/f32 [4462956,34807] 0 2024-03-22T02:28:57.135 INFO:tasks.workunit.client.1.smithi122.stdout:6/735: mknod d7/de/dd/d20/ced 0 2024-03-22T02:28:57.136 INFO:tasks.workunit.client.1.smithi122.stdout:4/505: mknod d0/d12/ca4 0 2024-03-22T02:28:57.136 INFO:tasks.workunit.client.1.smithi122.stdout:3/484: write d2/d9/dc/d17/d45/f46 [532854,21620] 0 2024-03-22T02:28:57.139 INFO:tasks.workunit.client.1.smithi122.stdout:2/623: dread df/f2b [0,4194304] 0 2024-03-22T02:28:57.139 INFO:tasks.workunit.client.1.smithi122.stdout:1/572: symlink dc/d3b/d8d/d72/d9c/lb8 0 2024-03-22T02:28:57.142 INFO:tasks.workunit.client.1.smithi122.stdout:5/500: write d8/da/f10 [9320674,11158] 0 2024-03-22T02:28:57.142 INFO:tasks.workunit.client.1.smithi122.stdout:8/638: rmdir d1/d40 39 2024-03-22T02:28:57.142 INFO:tasks.workunit.client.1.smithi122.stdout:9/637: chown db/d23/l12 16 1 2024-03-22T02:28:57.143 INFO:tasks.workunit.client.1.smithi122.stdout:0/464: creat d11/d16/d37/f9a x:0 0 0 2024-03-22T02:28:57.147 INFO:tasks.workunit.client.1.smithi122.stdout:7/545: dwrite d0/d4/d20/d21/d9b/d5e/f99 [0,4194304] 0 2024-03-22T02:28:57.151 INFO:tasks.workunit.client.1.smithi122.stdout:8/639: creat d1/d81/fe2 x:0 0 0 2024-03-22T02:28:57.152 INFO:tasks.workunit.client.1.smithi122.stdout:2/624: unlink df/d1f/d33/d34/c9d 0 2024-03-22T02:28:57.153 INFO:tasks.workunit.client.1.smithi122.stdout:8/640: dread - d1/d60/d73/f94 zero size 2024-03-22T02:28:57.153 INFO:tasks.workunit.client.1.smithi122.stdout:8/641: stat d1/d8/f36 0 2024-03-22T02:28:57.155 INFO:tasks.workunit.client.1.smithi122.stdout:0/465: creat d11/d14/d1d/d27/d59/f9b x:0 0 0 2024-03-22T02:28:57.157 INFO:tasks.workunit.client.1.smithi122.stdout:9/638: mknod db/df/d3e/d59/db5/cc8 0 2024-03-22T02:28:57.158 INFO:tasks.workunit.client.1.smithi122.stdout:7/546: mkdir d0/d4e/d5d/dab 0 2024-03-22T02:28:57.158 INFO:tasks.workunit.client.1.smithi122.stdout:7/547: fsync d0/d30/f8c 0 2024-03-22T02:28:57.161 INFO:tasks.workunit.client.1.smithi122.stdout:8/642: creat d1/d3f/d45/fe3 x:0 0 0 2024-03-22T02:28:57.163 INFO:tasks.workunit.client.1.smithi122.stdout:7/548: dread d0/d30/f8c [0,4194304] 0 2024-03-22T02:28:57.163 INFO:tasks.workunit.client.1.smithi122.stdout:0/466: mknod d11/d14/d1d/d27/d32/c9c 0 2024-03-22T02:28:57.166 INFO:tasks.workunit.client.1.smithi122.stdout:5/501: link d8/d1e/d22/d3a/d39/fa8 d8/d1a/d83/fae 0 2024-03-22T02:28:57.168 INFO:tasks.workunit.client.1.smithi122.stdout:4/506: getdents d0/d12/d15/d42 0 2024-03-22T02:28:57.169 INFO:tasks.workunit.client.1.smithi122.stdout:6/736: getdents d7/de/dd/d20/de0/dc0 0 2024-03-22T02:28:57.170 INFO:tasks.workunit.client.1.smithi122.stdout:8/643: mknod d1/d8/dba/ce4 0 2024-03-22T02:28:57.172 INFO:tasks.workunit.client.1.smithi122.stdout:1/573: dread dc/d3b/d8d/d72/d9c/db0/f5c [0,4194304] 0 2024-03-22T02:28:57.174 INFO:tasks.workunit.client.1.smithi122.stdout:0/467: dwrite d11/d14/d1b/f38 [0,4194304] 0 2024-03-22T02:28:57.176 INFO:tasks.workunit.client.1.smithi122.stdout:4/507: unlink d0/d12/ca4 0 2024-03-22T02:28:57.181 INFO:tasks.workunit.client.1.smithi122.stdout:3/485: dwrite d2/d9/dc/df/d38/d87/f90 [0,4194304] 0 2024-03-22T02:28:57.182 INFO:tasks.workunit.client.1.smithi122.stdout:3/486: chown d2/d9/dc/d17/d28 4151247 1 2024-03-22T02:28:57.183 INFO:tasks.workunit.client.1.smithi122.stdout:3/487: read d2/d9/dc/df/d38/f5d [1160476,129260] 0 2024-03-22T02:28:57.184 INFO:tasks.workunit.client.1.smithi122.stdout:3/488: dread - d2/d9/dc/d17/d51/f7c zero size 2024-03-22T02:28:57.186 INFO:tasks.workunit.client.1.smithi122.stdout:3/489: stat d2/l73 0 2024-03-22T02:28:57.186 INFO:tasks.workunit.client.1.smithi122.stdout:7/549: creat d0/fac x:0 0 0 2024-03-22T02:28:57.187 INFO:tasks.workunit.client.1.smithi122.stdout:7/550: dread - d0/d4/d20/d21/f95 zero size 2024-03-22T02:28:57.187 INFO:tasks.workunit.client.1.smithi122.stdout:0/468: mkdir d11/d14/d1d/d27/d9d 0 2024-03-22T02:28:57.194 INFO:tasks.workunit.client.1.smithi122.stdout:8/644: dread d1/d9/f77 [0,4194304] 0 2024-03-22T02:28:57.195 INFO:tasks.workunit.client.1.smithi122.stdout:4/508: dread d0/d12/d15/d42/d57/f64 [0,4194304] 0 2024-03-22T02:28:57.196 INFO:tasks.workunit.client.1.smithi122.stdout:0/469: mknod d11/d14/d23/d24/d33/c9e 0 2024-03-22T02:28:57.198 INFO:tasks.workunit.client.1.smithi122.stdout:6/737: getdents d7/de/dd/d20/de0/d54 0 2024-03-22T02:28:57.199 INFO:tasks.workunit.client.1.smithi122.stdout:3/490: creat d2/d9/dc/df/d49/d85/f94 x:0 0 0 2024-03-22T02:28:57.200 INFO:tasks.workunit.client.1.smithi122.stdout:1/574: link dc/d3b/d8d/d72/d9c/db0/d67/ca3 dc/d2b/cb9 0 2024-03-22T02:28:57.202 INFO:tasks.workunit.client.1.smithi122.stdout:4/509: symlink d0/d12/la5 0 2024-03-22T02:28:57.205 INFO:tasks.workunit.client.1.smithi122.stdout:0/470: mkdir d11/d16/d37/d9f 0 2024-03-22T02:28:57.205 INFO:tasks.workunit.client.1.smithi122.stdout:8/645: dread d1/d40/d71/dc9/f65 [0,4194304] 0 2024-03-22T02:28:57.207 INFO:tasks.workunit.client.1.smithi122.stdout:3/491: mknod d2/d9/dc/d17/d51/c95 0 2024-03-22T02:28:57.212 INFO:tasks.workunit.client.1.smithi122.stdout:6/738: creat d7/de/dd/d39/dc2/fee x:0 0 0 2024-03-22T02:28:57.212 INFO:tasks.workunit.client.1.smithi122.stdout:4/510: write d0/d12/d15/d42/d4e/de/f44 [2058820,88167] 0 2024-03-22T02:28:57.213 INFO:tasks.workunit.client.1.smithi122.stdout:8/646: chown d1/d40/d48/c83 0 1 2024-03-22T02:28:57.214 INFO:tasks.workunit.client.1.smithi122.stdout:6/739: readlink d7/de/d27/l2e 0 2024-03-22T02:28:57.217 INFO:tasks.workunit.client.1.smithi122.stdout:5/502: dwrite d8/d1e/d22/d3a/d39/f8b [0,4194304] 0 2024-03-22T02:28:57.219 INFO:tasks.workunit.client.1.smithi122.stdout:1/575: dwrite dc/d4c/f64 [4194304,4194304] 0 2024-03-22T02:28:57.222 INFO:tasks.workunit.client.1.smithi122.stdout:6/740: readlink d7/de/l3d 0 2024-03-22T02:28:57.222 INFO:tasks.workunit.client.1.smithi122.stdout:3/492: symlink d2/d9/dc/df/d3c/d59/l96 0 2024-03-22T02:28:57.222 INFO:tasks.workunit.client.1.smithi122.stdout:7/551: dread d0/d4/d22/f64 [4194304,4194304] 0 2024-03-22T02:28:57.224 INFO:tasks.workunit.client.1.smithi122.stdout:4/511: unlink d0/d25/f3a 0 2024-03-22T02:28:57.225 INFO:tasks.workunit.client.1.smithi122.stdout:5/503: creat d8/d3b/faf x:0 0 0 2024-03-22T02:28:57.230 INFO:tasks.workunit.client.1.smithi122.stdout:0/471: dwrite d11/d16/f55 [4194304,4194304] 0 2024-03-22T02:28:57.230 INFO:tasks.workunit.client.1.smithi122.stdout:4/512: readlink d0/d12/l46 0 2024-03-22T02:28:57.230 INFO:tasks.workunit.client.1.smithi122.stdout:3/493: read d2/d9/dc/f36 [6960214,80540] 0 2024-03-22T02:28:57.235 INFO:tasks.workunit.client.1.smithi122.stdout:6/741: dread d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:28:57.238 INFO:tasks.workunit.client.1.smithi122.stdout:7/552: creat d0/d4/d20/d21/d28/fad x:0 0 0 2024-03-22T02:28:57.238 INFO:tasks.workunit.client.1.smithi122.stdout:3/494: dread d2/d9/dc/d17/d28/f6c [0,4194304] 0 2024-03-22T02:28:57.241 INFO:tasks.workunit.client.1.smithi122.stdout:0/472: mknod d11/d16/ca0 0 2024-03-22T02:28:57.242 INFO:tasks.workunit.client.1.smithi122.stdout:0/473: write d11/d4f/d5a/f61 [6861580,23278] 0 2024-03-22T02:28:57.244 INFO:tasks.workunit.client.1.smithi122.stdout:6/742: symlink d7/de/lef 0 2024-03-22T02:28:57.245 INFO:tasks.workunit.client.1.smithi122.stdout:7/553: write d0/d4/fb [1321466,114756] 0 2024-03-22T02:28:57.245 INFO:tasks.workunit.client.1.smithi122.stdout:3/495: symlink d2/d9/dc/d57/l97 0 2024-03-22T02:28:57.247 INFO:tasks.workunit.client.1.smithi122.stdout:6/743: chown d7/de/dd/d20/de0/dae/fbe 0 1 2024-03-22T02:28:57.250 INFO:tasks.workunit.client.1.smithi122.stdout:8/647: dwrite d1/d8/d2f/f44 [0,4194304] 0 2024-03-22T02:28:57.253 INFO:tasks.workunit.client.1.smithi122.stdout:5/504: write d8/f26 [2179980,59242] 0 2024-03-22T02:28:57.254 INFO:tasks.workunit.client.1.smithi122.stdout:7/554: unlink d0/d4/d20/d21/d9b/f81 0 2024-03-22T02:28:57.256 INFO:tasks.workunit.client.1.smithi122.stdout:0/474: getdents d11/d14/d23/d24/d33 0 2024-03-22T02:28:57.256 INFO:tasks.workunit.client.1.smithi122.stdout:5/505: stat d8/d1e/d22/d3a/d46/f6f 0 2024-03-22T02:28:57.259 INFO:tasks.workunit.client.1.smithi122.stdout:3/496: rename d2/d9/dc/d17/d28/l32 to d2/d9/dc/d17/d28/l98 0 2024-03-22T02:28:57.261 INFO:tasks.workunit.client.1.smithi122.stdout:5/506: dread - d8/d1e/f7e zero size 2024-03-22T02:28:57.264 INFO:tasks.workunit.client.1.smithi122.stdout:1/576: dwrite dc/d3b/d8d/d72/d9c/db0/d71/fb2 [0,4194304] 0 2024-03-22T02:28:57.265 INFO:tasks.workunit.client.1.smithi122.stdout:3/497: write d2/f5e [5220769,67505] 0 2024-03-22T02:28:57.265 INFO:tasks.workunit.client.1.smithi122.stdout:7/555: unlink d0/d4/d20/l76 0 2024-03-22T02:28:57.266 INFO:tasks.workunit.client.1.smithi122.stdout:0/475: write d11/d14/d1d/d27/d2f/f58 [4754742,36180] 0 2024-03-22T02:28:57.267 INFO:tasks.workunit.client.1.smithi122.stdout:5/507: stat d8/d1e/d22/d3a/f32 0 2024-03-22T02:28:57.270 INFO:tasks.workunit.client.1.smithi122.stdout:6/744: rename d7/de/dd/d20/de0/c60 to d7/de/dd/d23/cf0 0 2024-03-22T02:28:57.271 INFO:tasks.workunit.client.1.smithi122.stdout:8/648: dwrite d1/d40/d48/faf [0,4194304] 0 2024-03-22T02:28:57.272 INFO:tasks.workunit.client.1.smithi122.stdout:0/476: write d11/d14/d23/f2d [1200761,87989] 0 2024-03-22T02:28:57.273 INFO:tasks.workunit.client.1.smithi122.stdout:3/498: stat d2/d9/dc/df/d38/f3b 0 2024-03-22T02:28:57.275 INFO:tasks.workunit.client.1.smithi122.stdout:5/508: write d8/da/d28/d35/f5d [549257,127395] 0 2024-03-22T02:28:57.275 INFO:tasks.workunit.client.1.smithi122.stdout:4/513: dwrite d0/d12/d15/d42/d57/f58 [0,4194304] 0 2024-03-22T02:28:57.278 INFO:tasks.workunit.client.1.smithi122.stdout:4/514: chown d0/d12/d15/d3d 6941 1 2024-03-22T02:28:57.281 INFO:tasks.workunit.client.1.smithi122.stdout:4/515: write d0/d12/d15/d42/d57/f8e [1599991,46767] 0 2024-03-22T02:28:57.283 INFO:tasks.workunit.client.1.smithi122.stdout:7/556: creat d0/fae x:0 0 0 2024-03-22T02:28:57.285 INFO:tasks.workunit.client.1.smithi122.stdout:7/557: read - d0/d4/d20/d21/d28/fad zero size 2024-03-22T02:28:57.287 INFO:tasks.workunit.client.1.smithi122.stdout:0/477: symlink d11/la1 0 2024-03-22T02:28:57.289 INFO:tasks.workunit.client.1.smithi122.stdout:3/499: rename d2/d9/dc/df/d3c/d59/l61 to d2/d9/dc/df/d49/d85/l99 0 2024-03-22T02:28:57.294 INFO:tasks.workunit.client.1.smithi122.stdout:7/558: read d0/fd [633539,60028] 0 2024-03-22T02:28:57.295 INFO:tasks.workunit.client.1.smithi122.stdout:8/649: fdatasync d1/d40/d51/fb2 0 2024-03-22T02:28:57.295 INFO:tasks.workunit.client.1.smithi122.stdout:4/516: fsync d0/d12/d15/d42/d4e/f23 0 2024-03-22T02:28:57.296 INFO:tasks.workunit.client.1.smithi122.stdout:1/577: dwrite dc/f11 [0,4194304] 0 2024-03-22T02:28:57.302 INFO:tasks.workunit.client.1.smithi122.stdout:0/478: dwrite d11/d16/d44/d89/f4b [4194304,4194304] 0 2024-03-22T02:28:57.303 INFO:tasks.workunit.client.1.smithi122.stdout:0/479: chown d11/d14/d1d/d27/d2f/l6f 100040740 1 2024-03-22T02:28:57.305 INFO:tasks.workunit.client.1.smithi122.stdout:0/480: write d11/d14/d1b/f81 [952118,55021] 0 2024-03-22T02:28:57.311 INFO:tasks.workunit.client.1.smithi122.stdout:5/509: truncate d8/da/d28/d42/d4e/f92 2334796 0 2024-03-22T02:28:57.315 INFO:tasks.workunit.client.1.smithi122.stdout:4/517: symlink d0/d12/d15/d3d/d90/la6 0 2024-03-22T02:28:57.315 INFO:tasks.workunit.client.1.smithi122.stdout:3/500: write d2/d9/dc/d17/d28/f35 [2551034,42341] 0 2024-03-22T02:28:57.316 INFO:tasks.workunit.client.1.smithi122.stdout:0/481: creat d11/d14/d1d/fa2 x:0 0 0 2024-03-22T02:28:57.317 INFO:tasks.workunit.client.1.smithi122.stdout:1/578: mkdir dc/d3b/d46/db5/dba 0 2024-03-22T02:28:57.319 INFO:tasks.workunit.client.1.smithi122.stdout:0/482: rmdir d11 39 2024-03-22T02:28:57.322 INFO:tasks.workunit.client.1.smithi122.stdout:1/579: mkdir dc/d2b/d86/dbb 0 2024-03-22T02:28:57.322 INFO:tasks.workunit.client.1.smithi122.stdout:8/650: dread d1/f53 [0,4194304] 0 2024-03-22T02:28:57.323 INFO:tasks.workunit.client.1.smithi122.stdout:5/510: creat d8/fb0 x:0 0 0 2024-03-22T02:28:57.325 INFO:tasks.workunit.client.1.smithi122.stdout:5/511: chown d8/da/f5b 3301 1 2024-03-22T02:28:57.327 INFO:tasks.workunit.client.1.smithi122.stdout:5/512: stat d8/d1e/d51/l97 0 2024-03-22T02:28:57.330 INFO:tasks.workunit.client.1.smithi122.stdout:3/501: dread d2/d9/dc/d17/d45/f46 [0,4194304] 0 2024-03-22T02:28:57.331 INFO:tasks.workunit.client.1.smithi122.stdout:1/580: rename dc/d2b/c2e to dc/d3b/d8d/d72/d9c/db0/d90/cbc 0 2024-03-22T02:28:57.335 INFO:tasks.workunit.client.1.smithi122.stdout:5/513: rename d8/d1e/d22/d3a/d46/d4f/f54 to d8/d1e/d51/fb1 0 2024-03-22T02:28:57.336 INFO:tasks.workunit.client.1.smithi122.stdout:3/502: symlink d2/d9/l9a 0 2024-03-22T02:28:57.338 INFO:tasks.workunit.client.1.smithi122.stdout:1/581: dread dc/d3b/d8d/d72/d9c/db0/d90/f93 [0,4194304] 0 2024-03-22T02:28:57.339 INFO:tasks.workunit.client.1.smithi122.stdout:3/503: creat d2/d9/dc/df/d49/d85/f9b x:0 0 0 2024-03-22T02:28:57.339 INFO:tasks.workunit.client.1.smithi122.stdout:1/582: chown dc/d3b/d8d/d72/d9c/db0 945133 1 2024-03-22T02:28:57.340 INFO:tasks.workunit.client.1.smithi122.stdout:8/651: dread d1/d9/f28 [0,4194304] 0 2024-03-22T02:28:57.341 INFO:tasks.workunit.client.1.smithi122.stdout:3/504: write d2/d9/dc/d17/f42 [1778092,117818] 0 2024-03-22T02:28:57.342 INFO:tasks.workunit.client.1.smithi122.stdout:0/483: dread d11/d14/d1b/f3c [0,4194304] 0 2024-03-22T02:28:57.342 INFO:tasks.workunit.client.1.smithi122.stdout:5/514: rename d8/d1e/d22/d3a/d46/d4f/fa2 to d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 0 2024-03-22T02:28:57.347 INFO:tasks.workunit.client.1.smithi122.stdout:0/484: mkdir d11/d16/d37/d84/da3 0 2024-03-22T02:28:57.347 INFO:tasks.workunit.client.1.smithi122.stdout:5/515: dread d8/da/d28/d42/d4e/f58 [0,4194304] 0 2024-03-22T02:28:57.348 INFO:tasks.workunit.client.1.smithi122.stdout:0/485: dread - d11/d14/d1b/f8f zero size 2024-03-22T02:28:57.348 INFO:tasks.workunit.client.1.smithi122.stdout:8/652: symlink d1/d8/dba/dc0/le5 0 2024-03-22T02:28:57.349 INFO:tasks.workunit.client.1.smithi122.stdout:5/516: stat d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c 0 2024-03-22T02:28:57.350 INFO:tasks.workunit.client.1.smithi122.stdout:8/653: fsync d1/fd3 0 2024-03-22T02:28:57.351 INFO:tasks.workunit.client.1.smithi122.stdout:8/654: stat d1/d9/l37 0 2024-03-22T02:28:57.353 INFO:tasks.workunit.client.1.smithi122.stdout:0/486: dread d11/d14/d1d/d27/d32/f6b [0,4194304] 0 2024-03-22T02:28:57.354 INFO:tasks.workunit.client.1.smithi122.stdout:5/517: symlink d8/lb3 0 2024-03-22T02:28:57.358 INFO:tasks.workunit.client.1.smithi122.stdout:3/505: dwrite d2/d9/dc/df/d47/f76 [0,4194304] 0 2024-03-22T02:28:57.358 INFO:tasks.workunit.client.1.smithi122.stdout:8/655: creat d1/d40/d71/fe6 x:0 0 0 2024-03-22T02:28:57.363 INFO:tasks.workunit.client.1.smithi122.stdout:5/518: unlink d8/d1e/d22/d3a/d46/d4f/d55/d6a/c9e 0 2024-03-22T02:28:57.364 INFO:tasks.workunit.client.1.smithi122.stdout:3/506: chown d2/d9/d7e/l8e 79610 1 2024-03-22T02:28:57.364 INFO:tasks.workunit.client.1.smithi122.stdout:8/656: creat d1/d60/d73/d90/fe7 x:0 0 0 2024-03-22T02:28:57.366 INFO:tasks.workunit.client.1.smithi122.stdout:5/519: fdatasync d8/d1e/d22/d3a/d39/f96 0 2024-03-22T02:28:57.368 INFO:tasks.workunit.client.1.smithi122.stdout:0/487: dwrite d11/d14/d1d/d27/d32/f5f [0,4194304] 0 2024-03-22T02:28:57.371 INFO:tasks.workunit.client.1.smithi122.stdout:3/507: creat d2/d9/dc/df/d3c/d59/f9c x:0 0 0 2024-03-22T02:28:57.372 INFO:tasks.workunit.client.1.smithi122.stdout:3/508: fsync d2/f62 0 2024-03-22T02:28:57.373 INFO:tasks.workunit.client.1.smithi122.stdout:8/657: truncate d1/d8/f36 632312 0 2024-03-22T02:28:57.376 INFO:tasks.workunit.client.1.smithi122.stdout:3/509: fdatasync d2/d9/dc/d17/d28/d4c/f70 0 2024-03-22T02:28:57.377 INFO:tasks.workunit.client.1.smithi122.stdout:4/518: dwrite d0/d12/d15/d42/d4e/f34 [0,4194304] 0 2024-03-22T02:28:57.378 INFO:tasks.workunit.client.1.smithi122.stdout:0/488: fdatasync d11/d14/d23/d24/d33/d67/f7c 0 2024-03-22T02:28:57.382 INFO:tasks.workunit.client.1.smithi122.stdout:5/520: dwrite d8/da/d28/d35/f4c [0,4194304] 0 2024-03-22T02:28:57.384 INFO:tasks.workunit.client.1.smithi122.stdout:4/519: mkdir d0/d12/d15/da7 0 2024-03-22T02:28:57.385 INFO:tasks.workunit.client.1.smithi122.stdout:3/510: stat d2/d9/dc/df/f53 0 2024-03-22T02:28:57.388 INFO:tasks.workunit.client.1.smithi122.stdout:4/520: write d0/d25/f27 [816324,121868] 0 2024-03-22T02:28:57.392 INFO:tasks.workunit.client.1.smithi122.stdout:3/511: creat d2/d9/dc/df/d47/f9d x:0 0 0 2024-03-22T02:28:57.393 INFO:tasks.workunit.client.1.smithi122.stdout:4/521: symlink d0/d12/d15/d3d/la8 0 2024-03-22T02:28:57.395 INFO:tasks.workunit.client.1.smithi122.stdout:5/521: symlink d8/da/lb4 0 2024-03-22T02:28:57.397 INFO:tasks.workunit.client.1.smithi122.stdout:3/512: mknod d2/d9/dc/c9e 0 2024-03-22T02:28:57.398 INFO:tasks.workunit.client.1.smithi122.stdout:4/522: dread d0/d12/d15/d42/d57/f58 [0,4194304] 0 2024-03-22T02:28:57.400 INFO:tasks.workunit.client.1.smithi122.stdout:3/513: symlink d2/d9/dc/df/d47/l9f 0 2024-03-22T02:28:57.401 INFO:tasks.workunit.client.1.smithi122.stdout:3/514: read - d2/d9/dc/d17/d2b/d6b/f8c zero size 2024-03-22T02:28:57.403 INFO:tasks.workunit.client.1.smithi122.stdout:3/515: write d2/d9/dc/df/d38/f3b [3404180,85584] 0 2024-03-22T02:28:57.403 INFO:tasks.workunit.client.1.smithi122.stdout:4/523: creat d0/d12/d15/d42/d4e/de/d91/fa9 x:0 0 0 2024-03-22T02:28:57.406 INFO:tasks.workunit.client.1.smithi122.stdout:4/524: write d0/d12/d15/d3d/d90/f98 [225330,121448] 0 2024-03-22T02:28:57.408 INFO:tasks.workunit.client.1.smithi122.stdout:3/516: rmdir d2/d9/dc/d17/d45 39 2024-03-22T02:28:57.408 INFO:tasks.workunit.client.1.smithi122.stdout:4/525: truncate d0/d12/d15/d42/d57/f9a 269758 0 2024-03-22T02:28:57.411 INFO:tasks.workunit.client.1.smithi122.stdout:4/526: fsync d0/d12/d15/d2d/f72 0 2024-03-22T02:28:57.415 INFO:tasks.workunit.client.1.smithi122.stdout:4/527: creat d0/d12/faa x:0 0 0 2024-03-22T02:28:57.416 INFO:tasks.workunit.client.1.smithi122.stdout:3/517: creat d2/d9/dc/df/d3c/fa0 x:0 0 0 2024-03-22T02:28:57.418 INFO:tasks.workunit.client.1.smithi122.stdout:0/489: dwrite d11/d14/d1d/d27/d32/f6b [0,4194304] 0 2024-03-22T02:28:57.419 INFO:tasks.workunit.client.1.smithi122.stdout:8/658: dread d1/d9/f1f [0,4194304] 0 2024-03-22T02:28:57.423 INFO:tasks.workunit.client.1.smithi122.stdout:3/518: creat d2/d9/dc/df/d38/fa1 x:0 0 0 2024-03-22T02:28:57.427 INFO:tasks.workunit.client.1.smithi122.stdout:0/490: creat d11/d16/d37/d9f/fa4 x:0 0 0 2024-03-22T02:28:57.427 INFO:tasks.workunit.client.1.smithi122.stdout:5/522: dwrite d8/da/f73 [0,4194304] 0 2024-03-22T02:28:57.428 INFO:tasks.workunit.client.1.smithi122.stdout:3/519: write d2/d9/dc/d17/d28/f2d [1368973,54275] 0 2024-03-22T02:28:57.429 INFO:tasks.workunit.client.1.smithi122.stdout:5/523: readlink d8/da/lb4 0 2024-03-22T02:28:57.433 INFO:tasks.workunit.client.1.smithi122.stdout:5/524: readlink d8/d1e/d22/d3a/d46/l5f 0 2024-03-22T02:28:57.435 INFO:tasks.workunit.client.1.smithi122.stdout:5/525: write d8/da/d28/d35/f4c [4476126,107883] 0 2024-03-22T02:28:57.435 INFO:tasks.workunit.client.1.smithi122.stdout:4/528: truncate d0/d25/f13 1482696 0 2024-03-22T02:28:57.437 INFO:tasks.workunit.client.1.smithi122.stdout:0/491: creat d11/fa5 x:0 0 0 2024-03-22T02:28:57.438 INFO:tasks.workunit.client.1.smithi122.stdout:5/526: unlink d8/da/d28/d35/f4c 0 2024-03-22T02:28:57.439 INFO:tasks.workunit.client.1.smithi122.stdout:5/527: read d8/d3b/f69 [501994,130251] 0 2024-03-22T02:28:57.439 INFO:tasks.workunit.client.1.smithi122.stdout:4/529: rmdir d0/d12/d15/d42/d4e/de/d40 39 2024-03-22T02:28:57.442 INFO:tasks.workunit.client.1.smithi122.stdout:0/492: mknod d11/d14/d23/d24/d33/ca6 0 2024-03-22T02:28:57.442 INFO:tasks.workunit.client.1.smithi122.stdout:8/659: write d1/d3f/f41 [1286770,50879] 0 2024-03-22T02:28:57.444 INFO:tasks.workunit.client.1.smithi122.stdout:5/528: creat d8/d1e/d22/da3/fb5 x:0 0 0 2024-03-22T02:28:57.445 INFO:tasks.workunit.client.1.smithi122.stdout:0/493: dread d11/d14/d1b/f2c [0,4194304] 0 2024-03-22T02:28:57.446 INFO:tasks.workunit.client.1.smithi122.stdout:4/530: creat d0/d12/d61/fab x:0 0 0 2024-03-22T02:28:57.447 INFO:tasks.workunit.client.1.smithi122.stdout:5/529: truncate d8/d1e/d22/d3a/d46/f4b 4898025 0 2024-03-22T02:28:57.453 INFO:tasks.workunit.client.1.smithi122.stdout:5/530: fdatasync d8/da/f10 0 2024-03-22T02:28:57.455 INFO:tasks.workunit.client.1.smithi122.stdout:4/531: dwrite d0/d12/d61/fab [0,4194304] 0 2024-03-22T02:28:57.461 INFO:tasks.workunit.client.1.smithi122.stdout:5/531: rename d8/da/d28/d42 to d8/d1e/d22/d3a/d46/d4f/d55/db6 0 2024-03-22T02:28:57.464 INFO:tasks.workunit.client.1.smithi122.stdout:4/532: getdents d0/d12/d73 0 2024-03-22T02:28:57.465 INFO:tasks.workunit.client.1.smithi122.stdout:4/533: write d0/d25/f84 [4779713,108365] 0 2024-03-22T02:28:57.468 INFO:tasks.workunit.client.1.smithi122.stdout:0/494: write d11/d14/d23/d24/d33/d67/f7c [171984,88079] 0 2024-03-22T02:28:57.470 INFO:tasks.workunit.client.1.smithi122.stdout:4/534: symlink d0/d12/d15/d42/d57/lac 0 2024-03-22T02:28:57.473 INFO:tasks.workunit.client.1.smithi122.stdout:0/495: rename d11/d14/d1b/d91 to d11/d14/da7 0 2024-03-22T02:28:57.473 INFO:tasks.workunit.client.1.smithi122.stdout:4/535: read d0/d12/d15/d3d/f68 [756112,25612] 0 2024-03-22T02:28:57.474 INFO:tasks.workunit.client.1.smithi122.stdout:0/496: write d11/d16/d44/d89/f3e [550646,45931] 0 2024-03-22T02:28:57.478 INFO:tasks.workunit.client.1.smithi122.stdout:0/497: getdents d11/d14/d1d/d27/d2f 0 2024-03-22T02:28:57.486 INFO:tasks.workunit.client.1.smithi122.stdout:0/498: dwrite f9 [0,4194304] 0 2024-03-22T02:28:57.493 INFO:tasks.workunit.client.1.smithi122.stdout:0/499: truncate d11/d4f/f82 3445790 0 2024-03-22T02:28:57.495 INFO:tasks.workunit.client.1.smithi122.stdout:0/500: creat d11/d14/d23/d24/fa8 x:0 0 0 2024-03-22T02:28:57.497 INFO:tasks.workunit.client.1.smithi122.stdout:0/501: mknod d11/d14/d23/d24/d8b/ca9 0 2024-03-22T02:28:57.498 INFO:tasks.workunit.client.1.smithi122.stdout:0/502: mkdir d11/d14/d1d/d27/d9d/daa 0 2024-03-22T02:28:57.502 INFO:tasks.workunit.client.1.smithi122.stdout:0/503: dread d11/d14/d1b/f3c [0,4194304] 0 2024-03-22T02:28:57.503 INFO:tasks.workunit.client.1.smithi122.stdout:0/504: chown d11/d14/d1d/d27/d2f/d4e/f66 1 1 2024-03-22T02:28:57.506 INFO:tasks.workunit.client.1.smithi122.stdout:0/505: dread d11/d16/d44/d89/f4b [4194304,4194304] 0 2024-03-22T02:28:57.513 INFO:tasks.workunit.client.1.smithi122.stdout:0/506: dwrite d11/d14/d1d/d27/d2f/d4e/f66 [0,4194304] 0 2024-03-22T02:28:57.522 INFO:tasks.workunit.client.1.smithi122.stdout:0/507: dwrite d11/d14/d23/d24/d33/f8e [0,4194304] 0 2024-03-22T02:28:57.523 INFO:tasks.workunit.client.1.smithi122.stdout:0/508: chown d11/d14/d1d/d8a 2017608 1 2024-03-22T02:28:57.535 INFO:tasks.workunit.client.1.smithi122.stdout:0/509: read d11/d16/d44/d89/f3e [954253,83376] 0 2024-03-22T02:28:57.536 INFO:tasks.workunit.client.1.smithi122.stdout:0/510: chown d11/d14/d23/d24/d33 6 1 2024-03-22T02:28:57.540 INFO:tasks.workunit.client.1.smithi122.stdout:0/511: rename d11/d14/d1d/d27/d32/f5f to d11/d16/d37/fab 0 2024-03-22T02:28:57.543 INFO:tasks.workunit.client.1.smithi122.stdout:0/512: unlink d11/d14/d1d/f41 0 2024-03-22T02:28:57.545 INFO:tasks.workunit.client.1.smithi122.stdout:0/513: getdents d11/d14/d1d/d8a 0 2024-03-22T02:28:57.555 INFO:tasks.workunit.client.1.smithi122.stdout:0/514: write d11/d14/d1d/d27/f5e [783973,105052] 0 2024-03-22T02:28:57.558 INFO:tasks.workunit.client.1.smithi122.stdout:0/515: creat d11/d14/d1d/d27/d2f/fac x:0 0 0 2024-03-22T02:28:57.559 INFO:tasks.workunit.client.1.smithi122.stdout:0/516: write d11/d14/d1d/d27/d2f/f58 [3733916,13409] 0 2024-03-22T02:28:57.560 INFO:tasks.workunit.client.1.smithi122.stdout:0/517: stat d11/d14/d23/f6d 0 2024-03-22T02:28:57.561 INFO:tasks.workunit.client.1.smithi122.stdout:0/518: symlink d11/d14/d1b/lad 0 2024-03-22T02:28:57.668 INFO:tasks.workunit.client.1.smithi122.stdout:0/519: fdatasync d11/d14/d23/d24/fa8 0 2024-03-22T02:28:57.669 INFO:tasks.workunit.client.1.smithi122.stdout:0/520: unlink d11/d14/d1b/l94 0 2024-03-22T02:28:57.671 INFO:tasks.workunit.client.1.smithi122.stdout:0/521: symlink d11/d14/d1d/d27/lae 0 2024-03-22T02:28:57.672 INFO:tasks.workunit.client.1.smithi122.stdout:0/522: creat d11/d14/d1d/d27/d2f/d4e/faf x:0 0 0 2024-03-22T02:28:57.675 INFO:tasks.workunit.client.1.smithi122.stdout:0/523: dread d11/d14/d1d/d27/d32/f6c [0,4194304] 0 2024-03-22T02:28:57.678 INFO:tasks.workunit.client.1.smithi122.stdout:0/524: link d11/d14/d23/d24/d33/f95 d11/d14/d1d/d27/d59/fb0 0 2024-03-22T02:28:57.738 INFO:tasks.workunit.client.1.smithi122.stdout:2/625: sync 2024-03-22T02:28:57.739 INFO:tasks.workunit.client.1.smithi122.stdout:0/525: getdents d11/d14/d1b 0 2024-03-22T02:28:57.740 INFO:tasks.workunit.client.1.smithi122.stdout:2/626: write df/d1f/d33/f46 [1977645,127400] 0 2024-03-22T02:28:57.742 INFO:tasks.workunit.client.1.smithi122.stdout:2/627: chown df/d1f/d33/d4d/d94/dc2/d51/cb8 1364 1 2024-03-22T02:28:57.743 INFO:tasks.workunit.client.1.smithi122.stdout:0/526: creat d11/d14/d1d/d27/d59/fb1 x:0 0 0 2024-03-22T02:28:57.744 INFO:tasks.workunit.client.1.smithi122.stdout:0/527: write d11/d14/f8d [152841,77318] 0 2024-03-22T02:28:57.745 INFO:tasks.workunit.client.1.smithi122.stdout:2/628: getdents df/d1f/d33/d4d/d7a 0 2024-03-22T02:28:57.746 INFO:tasks.workunit.client.1.smithi122.stdout:0/528: dread d11/f3d [0,4194304] 0 2024-03-22T02:28:57.750 INFO:tasks.workunit.client.1.smithi122.stdout:0/529: rename d11/d14/d1d/d27/d2f/d4e/f66 to d11/d14/d1d/d27/d59/fb2 0 2024-03-22T02:28:57.752 INFO:tasks.workunit.client.1.smithi122.stdout:2/629: dwrite df/d1f/d33/d6a/f8d [0,4194304] 0 2024-03-22T02:28:57.758 INFO:tasks.workunit.client.1.smithi122.stdout:2/630: rename f4 to df/d1f/d33/fd7 0 2024-03-22T02:28:57.761 INFO:tasks.workunit.client.1.smithi122.stdout:2/631: truncate df/d1f/d23/dd2/f62 4054634 0 2024-03-22T02:28:57.797 INFO:tasks.workunit.client.1.smithi122.stdout:2/632: dread df/d1f/d33/d6a/f93 [0,4194304] 0 2024-03-22T02:28:57.799 INFO:tasks.workunit.client.1.smithi122.stdout:2/633: unlink df/d1f/d23/dd2/f53 0 2024-03-22T02:28:57.805 INFO:tasks.workunit.client.1.smithi122.stdout:2/634: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f5e [0,4194304] 0 2024-03-22T02:28:57.807 INFO:tasks.workunit.client.1.smithi122.stdout:2/635: write df/d1f/d33/d4d/f77 [952260,78333] 0 2024-03-22T02:28:57.809 INFO:tasks.workunit.client.1.smithi122.stdout:2/636: unlink df/d1f/d3d/c49 0 2024-03-22T02:28:57.809 INFO:tasks.workunit.client.1.smithi122.stdout:2/637: readlink df/d1f/lcb 0 2024-03-22T02:28:57.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/638: dread df/d1f/d33/d4d/d7a/f8a [0,4194304] 0 2024-03-22T02:28:57.818 INFO:tasks.workunit.client.1.smithi122.stdout:2/639: creat df/d1f/d33/d4d/fd8 x:0 0 0 2024-03-22T02:28:57.819 INFO:tasks.workunit.client.1.smithi122.stdout:2/640: rmdir df/d1f/d33/d6a/d7c/d8e/da0 39 2024-03-22T02:28:57.820 INFO:tasks.workunit.client.1.smithi122.stdout:2/641: dread - df/d1f/d33/d4d/d94/fc8 zero size 2024-03-22T02:28:57.822 INFO:tasks.workunit.client.1.smithi122.stdout:2/642: creat df/d1f/d33/d4d/d94/dc2/d51/fd9 x:0 0 0 2024-03-22T02:28:57.825 INFO:tasks.workunit.client.1.smithi122.stdout:2/643: getdents df/d1f/d33/d4d/d94 0 2024-03-22T02:28:57.830 INFO:tasks.workunit.client.1.smithi122.stdout:2/644: dread df/d1f/d33/d34/f5a [0,4194304] 0 2024-03-22T02:28:57.832 INFO:tasks.workunit.client.1.smithi122.stdout:2/645: symlink df/d1f/d3d/lda 0 2024-03-22T02:28:57.832 INFO:tasks.workunit.client.1.smithi122.stdout:2/646: chown df/d1f/d23/faf 341 1 2024-03-22T02:28:57.833 INFO:tasks.workunit.client.1.smithi122.stdout:2/647: rename df to df/d1f/d33/d4d/d94/dc2/d51/d67/db0/ddb 22 2024-03-22T02:28:57.837 INFO:tasks.workunit.client.1.smithi122.stdout:2/648: rename df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/d84 to df/d1f/d3d/d52/ddc 0 2024-03-22T02:28:57.838 INFO:tasks.workunit.client.1.smithi122.stdout:2/649: fsync df/d1f/d33/d4d/d94/dc2/f2f 0 2024-03-22T02:28:57.840 INFO:tasks.workunit.client.1.smithi122.stdout:2/650: mkdir df/d1f/d33/d34/ddd 0 2024-03-22T02:28:57.841 INFO:tasks.workunit.client.1.smithi122.stdout:2/651: write df/f2b [1175209,102686] 0 2024-03-22T02:28:57.843 INFO:tasks.workunit.client.1.smithi122.stdout:2/652: write df/d1f/d33/d4d/d94/dc2/d51/d67/f6e [6827471,34188] 0 2024-03-22T02:28:57.844 INFO:tasks.workunit.client.1.smithi122.stdout:2/653: chown df/d1f/d33/d4d/d94/dc2/d51/d67 24 1 2024-03-22T02:28:57.845 INFO:tasks.workunit.client.1.smithi122.stdout:2/654: write df/d1f/d9b/fa7 [316593,67856] 0 2024-03-22T02:28:57.847 INFO:tasks.workunit.client.1.smithi122.stdout:2/655: rmdir df/d1f/d33/d4d/d94/dc2/d51/d67 39 2024-03-22T02:28:57.848 INFO:tasks.workunit.client.1.smithi122.stdout:2/656: write df/d1f/d33/d6a/d7c/d8e/fa1 [121855,81708] 0 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: pgmap v22: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 115 MiB/s rd, 356 MiB/s wr, 652 op/s 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.852 INFO:tasks.workunit.client.1.smithi122.stdout:2/657: dwrite df/f10 [0,4194304] 0 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: pgmap v22: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 115 MiB/s rd, 356 MiB/s wr, 652 op/s 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:28:57.882 INFO:tasks.workunit.client.1.smithi122.stdout:6/745: sync 2024-03-22T02:28:57.883 INFO:tasks.workunit.client.1.smithi122.stdout:9/639: sync 2024-03-22T02:28:57.917 INFO:tasks.workunit.client.1.smithi122.stdout:6/746: unlink d7/lcb 0 2024-03-22T02:28:57.918 INFO:tasks.workunit.client.1.smithi122.stdout:6/747: chown d7/de/dd/d20/de0/d54 50740 1 2024-03-22T02:28:57.919 INFO:tasks.workunit.client.1.smithi122.stdout:2/658: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/fde x:0 0 0 2024-03-22T02:28:57.978 INFO:tasks.workunit.client.1.smithi122.stdout:6/748: truncate d7/de/d27/f36 646719 0 2024-03-22T02:28:57.979 INFO:tasks.workunit.client.1.smithi122.stdout:2/659: dread - df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/fde zero size 2024-03-22T02:28:57.980 INFO:tasks.workunit.client.1.smithi122.stdout:9/640: dwrite db/df/d3e/d81/d25/f55 [8388608,4194304] 0 2024-03-22T02:28:57.982 INFO:tasks.workunit.client.1.smithi122.stdout:9/641: write db/d23/f83 [4047123,77895] 0 2024-03-22T02:28:57.982 INFO:tasks.workunit.client.1.smithi122.stdout:2/660: unlink df/f36 0 2024-03-22T02:28:57.984 INFO:tasks.workunit.client.1.smithi122.stdout:6/749: mknod d7/de/dd/d20/de0/d54/cf1 0 2024-03-22T02:28:57.985 INFO:tasks.workunit.client.1.smithi122.stdout:6/750: fdatasync d7/de/dd/d20/de0/dae/fcd 0 2024-03-22T02:28:57.988 INFO:tasks.workunit.client.1.smithi122.stdout:9/642: rename db/df/d3e/f8b to db/df/d3e/d81/d25/d41/fc9 0 2024-03-22T02:28:57.990 INFO:tasks.workunit.client.1.smithi122.stdout:9/643: write db/d23/d1e/dad/fc2 [790631,50306] 0 2024-03-22T02:28:57.991 INFO:tasks.workunit.client.1.smithi122.stdout:9/644: write db/d1b/d28/dc1/f9f [650488,102119] 0 2024-03-22T02:28:57.992 INFO:tasks.workunit.client.1.smithi122.stdout:6/751: read d7/de/dd/d20/f45 [3540800,37849] 0 2024-03-22T02:28:57.993 INFO:tasks.workunit.client.1.smithi122.stdout:6/752: stat d7/de/d27/f69 0 2024-03-22T02:28:57.996 INFO:tasks.workunit.client.1.smithi122.stdout:6/753: mknod d7/de/dd/d39/cf2 0 2024-03-22T02:28:57.998 INFO:tasks.workunit.client.1.smithi122.stdout:6/754: rmdir d7/de/dd/d20/de0/d54/d76/d86 39 2024-03-22T02:28:58.000 INFO:tasks.workunit.client.1.smithi122.stdout:2/661: dread df/d1f/d23/dd2/f96 [0,4194304] 0 2024-03-22T02:28:58.000 INFO:tasks.workunit.client.1.smithi122.stdout:9/645: link db/df/d3e/d81/d25/ca2 db/d23/d1e/cca 0 2024-03-22T02:28:58.005 INFO:tasks.workunit.client.1.smithi122.stdout:2/662: mknod df/cdf 0 2024-03-22T02:28:58.007 INFO:tasks.workunit.client.1.smithi122.stdout:9/646: creat db/d23/d1f/d57/fcb x:0 0 0 2024-03-22T02:28:58.011 INFO:tasks.workunit.client.1.smithi122.stdout:2/663: mknod df/d1f/d3d/d52/ce0 0 2024-03-22T02:28:58.012 INFO:tasks.workunit.client.1.smithi122.stdout:6/755: write d7/de/d27/f80 [747513,111471] 0 2024-03-22T02:28:58.013 INFO:tasks.workunit.client.1.smithi122.stdout:2/664: read df/d1f/d33/d34/f5f [3683919,43683] 0 2024-03-22T02:28:58.014 INFO:tasks.workunit.client.1.smithi122.stdout:7/559: sync 2024-03-22T02:28:58.015 INFO:tasks.workunit.client.1.smithi122.stdout:1/583: sync 2024-03-22T02:28:58.016 INFO:tasks.workunit.client.1.smithi122.stdout:9/647: symlink db/d23/d1f/lcc 0 2024-03-22T02:28:58.023 INFO:tasks.workunit.client.1.smithi122.stdout:1/584: unlink dc/d4c/f64 0 2024-03-22T02:28:58.025 INFO:tasks.workunit.client.1.smithi122.stdout:6/756: write d7/de/dd/df/f6e [969189,115302] 0 2024-03-22T02:28:58.027 INFO:tasks.workunit.client.1.smithi122.stdout:2/665: mknod df/d1f/d33/d4d/d94/dc2/d51/ce1 0 2024-03-22T02:28:58.029 INFO:tasks.workunit.client.1.smithi122.stdout:1/585: creat dc/d3b/d8d/d72/d9c/db0/dab/fbd x:0 0 0 2024-03-22T02:28:58.032 INFO:tasks.workunit.client.1.smithi122.stdout:1/586: creat dc/d3b/d8d/d72/d9c/db0/d67/fbe x:0 0 0 2024-03-22T02:28:58.033 INFO:tasks.workunit.client.1.smithi122.stdout:9/648: creat db/d1b/da8/fcd x:0 0 0 2024-03-22T02:28:58.034 INFO:tasks.workunit.client.1.smithi122.stdout:9/649: dread - db/df/d3e/d81/d25/d41/fbc zero size 2024-03-22T02:28:58.038 INFO:tasks.workunit.client.1.smithi122.stdout:6/757: dread d7/de/dd/d20/f45 [0,4194304] 0 2024-03-22T02:28:58.038 INFO:tasks.workunit.client.1.smithi122.stdout:9/650: mknod db/df/d3e/d81/d3d/d86/cce 0 2024-03-22T02:28:58.039 INFO:tasks.workunit.client.1.smithi122.stdout:1/587: rmdir dc/d3b/d8d/d72 39 2024-03-22T02:28:58.043 INFO:tasks.workunit.client.1.smithi122.stdout:7/560: dwrite d0/d4/d94/fa9 [0,4194304] 0 2024-03-22T02:28:58.043 INFO:tasks.workunit.client.1.smithi122.stdout:9/651: mknod db/d23/ccf 0 2024-03-22T02:28:58.046 INFO:tasks.workunit.client.1.smithi122.stdout:7/561: mkdir d0/d4/d1d/daf 0 2024-03-22T02:28:58.047 INFO:tasks.workunit.client.1.smithi122.stdout:9/652: creat db/d23/d1f/d57/fd0 x:0 0 0 2024-03-22T02:28:58.047 INFO:tasks.workunit.client.1.smithi122.stdout:6/758: dwrite d7/de/d27/f80 [0,4194304] 0 2024-03-22T02:28:58.049 INFO:tasks.workunit.client.1.smithi122.stdout:7/562: truncate d0/d4/d94/fa5 847049 0 2024-03-22T02:28:58.049 INFO:tasks.workunit.client.1.smithi122.stdout:1/588: creat dc/d3b/d8d/d72/d9c/db0/dab/fbf x:0 0 0 2024-03-22T02:28:58.051 INFO:tasks.workunit.client.1.smithi122.stdout:7/563: symlink d0/d4e/d5d/lb0 0 2024-03-22T02:28:58.056 INFO:tasks.workunit.client.1.smithi122.stdout:7/564: dread d0/f16 [0,4194304] 0 2024-03-22T02:28:58.058 INFO:tasks.workunit.client.1.smithi122.stdout:7/565: mknod d0/d25/d46/d48/cb1 0 2024-03-22T02:28:58.060 INFO:tasks.workunit.client.1.smithi122.stdout:9/653: read db/d23/f1a [850348,41741] 0 2024-03-22T02:28:58.061 INFO:tasks.workunit.client.1.smithi122.stdout:7/566: symlink d0/d25/d46/d98/lb2 0 2024-03-22T02:28:58.061 INFO:tasks.workunit.client.1.smithi122.stdout:9/654: truncate db/d23/d1f/d57/fcb 457222 0 2024-03-22T02:28:58.062 INFO:tasks.workunit.client.1.smithi122.stdout:7/567: chown d0/d4/d20/d21/d28/f90 0 1 2024-03-22T02:28:58.063 INFO:tasks.workunit.client.1.smithi122.stdout:9/655: rmdir db/df/d3e/d59 39 2024-03-22T02:28:58.064 INFO:tasks.workunit.client.1.smithi122.stdout:7/568: stat d0/d4/d20/d21/d9b/l78 0 2024-03-22T02:28:58.064 INFO:tasks.workunit.client.1.smithi122.stdout:9/656: write db/fb3 [237615,82688] 0 2024-03-22T02:28:58.069 INFO:tasks.workunit.client.1.smithi122.stdout:7/569: rename d0/d4e/d5d/f6e to d0/d4/d22/fb3 0 2024-03-22T02:28:58.071 INFO:tasks.workunit.client.1.smithi122.stdout:9/657: dwrite db/d23/fa9 [0,4194304] 0 2024-03-22T02:28:58.076 INFO:tasks.workunit.client.1.smithi122.stdout:7/570: getdents d0/d25/d5c 0 2024-03-22T02:28:58.078 INFO:tasks.workunit.client.1.smithi122.stdout:1/589: dwrite dc/d2b/f33 [0,4194304] 0 2024-03-22T02:28:58.080 INFO:tasks.workunit.client.1.smithi122.stdout:7/571: write d0/d25/d46/f85 [259566,112073] 0 2024-03-22T02:28:58.082 INFO:tasks.workunit.client.1.smithi122.stdout:9/658: dwrite db/d1b/d28/dc1/f5f [0,4194304] 0 2024-03-22T02:28:58.091 INFO:tasks.workunit.client.1.smithi122.stdout:1/590: rename dc/d3b/d8d/d72/d9c/db0/d67/f6c to dc/d55/fc0 0 2024-03-22T02:28:58.093 INFO:tasks.workunit.client.1.smithi122.stdout:9/659: read db/fb1 [4257322,67810] 0 2024-03-22T02:28:58.095 INFO:tasks.workunit.client.1.smithi122.stdout:1/591: rename dc/d55/c5e to dc/d3b/d46/cc1 0 2024-03-22T02:28:58.096 INFO:tasks.workunit.client.1.smithi122.stdout:7/572: dwrite d0/d25/d5c/f8e [0,4194304] 0 2024-03-22T02:28:58.097 INFO:tasks.workunit.client.1.smithi122.stdout:9/660: write db/fb1 [5819261,31082] 0 2024-03-22T02:28:58.098 INFO:tasks.workunit.client.1.smithi122.stdout:1/592: unlink dc/l14 0 2024-03-22T02:28:58.102 INFO:tasks.workunit.client.1.smithi122.stdout:1/593: creat dc/d55/fc2 x:0 0 0 2024-03-22T02:28:58.103 INFO:tasks.workunit.client.1.smithi122.stdout:7/573: creat d0/d4/d20/d21/fb4 x:0 0 0 2024-03-22T02:28:58.106 INFO:tasks.workunit.client.1.smithi122.stdout:7/574: mknod d0/d30/cb5 0 2024-03-22T02:28:58.107 INFO:tasks.workunit.client.1.smithi122.stdout:7/575: fdatasync d0/d30/f53 0 2024-03-22T02:28:58.108 INFO:tasks.workunit.client.1.smithi122.stdout:1/594: dwrite dc/d2b/f33 [0,4194304] 0 2024-03-22T02:28:58.111 INFO:tasks.workunit.client.1.smithi122.stdout:7/576: getdents d0/d25/d46/d98 0 2024-03-22T02:28:58.114 INFO:tasks.workunit.client.1.smithi122.stdout:7/577: rmdir d0/d4 39 2024-03-22T02:28:58.115 INFO:tasks.workunit.client.1.smithi122.stdout:7/578: write d0/f87 [627434,3131] 0 2024-03-22T02:28:58.116 INFO:tasks.workunit.client.1.smithi122.stdout:1/595: dread - dc/d3b/d8d/d72/d9c/db0/d71/fb1 zero size 2024-03-22T02:28:58.116 INFO:tasks.workunit.client.1.smithi122.stdout:7/579: chown d0/d4e/f73 20692 1 2024-03-22T02:28:58.117 INFO:tasks.workunit.client.1.smithi122.stdout:7/580: chown d0/d30/cb5 789406792 1 2024-03-22T02:28:58.118 INFO:tasks.workunit.client.1.smithi122.stdout:1/596: chown dc/d3b/f81 8552582 1 2024-03-22T02:28:58.119 INFO:tasks.workunit.client.1.smithi122.stdout:7/581: stat d0/d4/c65 0 2024-03-22T02:28:58.120 INFO:tasks.workunit.client.1.smithi122.stdout:1/597: creat dc/d55/fc3 x:0 0 0 2024-03-22T02:28:58.120 INFO:tasks.workunit.client.1.smithi122.stdout:9/661: truncate db/d1b/d28/dc1/f5f 1629016 0 2024-03-22T02:28:58.123 INFO:tasks.workunit.client.1.smithi122.stdout:1/598: write dc/fd [11865443,112401] 0 2024-03-22T02:28:58.124 INFO:tasks.workunit.client.1.smithi122.stdout:1/599: stat dc/d3b/d46/cac 0 2024-03-22T02:28:58.124 INFO:tasks.workunit.client.1.smithi122.stdout:7/582: creat d0/d4/d20/d21/fb6 x:0 0 0 2024-03-22T02:28:58.125 INFO:tasks.workunit.client.1.smithi122.stdout:1/600: write dc/d4c/f63 [2460033,29112] 0 2024-03-22T02:28:58.125 INFO:tasks.workunit.client.1.smithi122.stdout:9/662: dread db/d23/d1e/dad/fc2 [0,4194304] 0 2024-03-22T02:28:58.126 INFO:tasks.workunit.client.1.smithi122.stdout:7/583: dread - d0/d4/d20/d21/fb4 zero size 2024-03-22T02:28:58.128 INFO:tasks.workunit.client.1.smithi122.stdout:9/663: dread db/d23/d1f/d57/fcb [0,4194304] 0 2024-03-22T02:28:58.129 INFO:tasks.workunit.client.1.smithi122.stdout:9/664: dread - db/d1b/da8/fab zero size 2024-03-22T02:28:58.131 INFO:tasks.workunit.client.1.smithi122.stdout:9/665: unlink db/d23/d1f/d57/fcb 0 2024-03-22T02:28:58.140 INFO:tasks.workunit.client.1.smithi122.stdout:7/584: dread d0/d30/f5b [0,4194304] 0 2024-03-22T02:28:58.142 INFO:tasks.workunit.client.1.smithi122.stdout:9/666: dread db/d1b/d6c/d58/d9d/fa6 [0,4194304] 0 2024-03-22T02:28:58.143 INFO:tasks.workunit.client.1.smithi122.stdout:9/667: chown db/df/d3e/d81/d25/d41 1018298285 1 2024-03-22T02:28:58.143 INFO:tasks.workunit.client.1.smithi122.stdout:7/585: creat d0/d4/d20/d21/d9b/d7d/fb7 x:0 0 0 2024-03-22T02:28:58.145 INFO:tasks.workunit.client.1.smithi122.stdout:7/586: dread d0/f87 [0,4194304] 0 2024-03-22T02:28:58.146 INFO:tasks.workunit.client.1.smithi122.stdout:9/668: mkdir db/d1b/d6c/dd1 0 2024-03-22T02:28:58.147 INFO:tasks.workunit.client.1.smithi122.stdout:9/669: mknod db/d1b/d6c/d58/cd2 0 2024-03-22T02:28:58.148 INFO:tasks.workunit.client.1.smithi122.stdout:7/587: rename d0/d25/d46 to d0/d4/d1d/db8 0 2024-03-22T02:28:58.149 INFO:tasks.workunit.client.1.smithi122.stdout:9/670: fdatasync db/df/d3e/d59/db5/fc6 0 2024-03-22T02:28:58.149 INFO:tasks.workunit.client.1.smithi122.stdout:7/588: truncate d0/f8 4351665 0 2024-03-22T02:28:58.150 INFO:tasks.workunit.client.1.smithi122.stdout:7/589: rmdir d0/d25/d5c 39 2024-03-22T02:28:58.152 INFO:tasks.workunit.client.1.smithi122.stdout:9/671: dread db/d23/d1f/d57/d66/d73/fa4 [0,4194304] 0 2024-03-22T02:28:58.155 INFO:tasks.workunit.client.1.smithi122.stdout:7/590: link d0/d4e/d5d/lb0 d0/d25/lb9 0 2024-03-22T02:28:58.158 INFO:tasks.workunit.client.1.smithi122.stdout:8/660: sync 2024-03-22T02:28:58.158 INFO:tasks.workunit.client.1.smithi122.stdout:3/520: sync 2024-03-22T02:28:58.159 INFO:tasks.workunit.client.1.smithi122.stdout:8/661: write d1/d8/d8a/fad [346326,116173] 0 2024-03-22T02:28:58.167 INFO:tasks.workunit.client.1.smithi122.stdout:9/672: dwrite db/d1b/f6a [0,4194304] 0 2024-03-22T02:28:58.169 INFO:tasks.workunit.client.1.smithi122.stdout:8/662: creat d1/d3f/fe8 x:0 0 0 2024-03-22T02:28:58.171 INFO:tasks.workunit.client.1.smithi122.stdout:9/673: chown db/df/d3e/d81/d25/ca2 108630681 1 2024-03-22T02:28:58.171 INFO:tasks.workunit.client.1.smithi122.stdout:7/591: dread d0/d4/d94/fa4 [0,4194304] 0 2024-03-22T02:28:58.173 INFO:tasks.workunit.client.1.smithi122.stdout:8/663: creat d1/d60/d73/da3/fe9 x:0 0 0 2024-03-22T02:28:58.174 INFO:tasks.workunit.client.1.smithi122.stdout:3/521: truncate d2/d9/dc/df/d38/f5d 2626457 0 2024-03-22T02:28:58.175 INFO:tasks.workunit.client.1.smithi122.stdout:9/674: mkdir db/d23/d1f/d57/dd3 0 2024-03-22T02:28:58.177 INFO:tasks.workunit.client.1.smithi122.stdout:8/664: mknod d1/d40/d71/d97/db8/dde/cea 0 2024-03-22T02:28:58.179 INFO:tasks.workunit.client.1.smithi122.stdout:3/522: getdents d2/d9/dc/d57/d89 0 2024-03-22T02:28:58.180 INFO:tasks.workunit.client.1.smithi122.stdout:8/665: symlink d1/d8/leb 0 2024-03-22T02:28:58.180 INFO:tasks.workunit.client.1.smithi122.stdout:7/592: link d0/d4/cf d0/d4/d94/cba 0 2024-03-22T02:28:58.183 INFO:tasks.workunit.client.1.smithi122.stdout:8/666: creat d1/d60/d73/d90/fec x:0 0 0 2024-03-22T02:28:58.184 INFO:tasks.workunit.client.1.smithi122.stdout:8/667: write d1/d22/f32 [4074889,32025] 0 2024-03-22T02:28:58.185 INFO:tasks.workunit.client.1.smithi122.stdout:3/523: getdents d2/d9/dc/df/d49/d85 0 2024-03-22T02:28:58.186 INFO:tasks.workunit.client.1.smithi122.stdout:3/524: stat d2/d9/dc/df/d3c/l3e 0 2024-03-22T02:28:58.187 INFO:tasks.workunit.client.1.smithi122.stdout:8/668: creat d1/d8/d2f/fed x:0 0 0 2024-03-22T02:28:58.188 INFO:tasks.workunit.client.1.smithi122.stdout:3/525: mkdir d2/d8a/da2 0 2024-03-22T02:28:58.190 INFO:tasks.workunit.client.1.smithi122.stdout:3/526: read d2/d9/dc/df/d47/f56 [44814,55481] 0 2024-03-22T02:28:58.191 INFO:tasks.workunit.client.1.smithi122.stdout:3/527: fdatasync d2/d9/dc/df/d3c/fa0 0 2024-03-22T02:28:58.195 INFO:tasks.workunit.client.1.smithi122.stdout:3/528: rename d2/d9/dc/df/d47/d79/l83 to d2/d9/d7e/la3 0 2024-03-22T02:28:58.197 INFO:tasks.workunit.client.1.smithi122.stdout:3/529: mkdir d2/d9/dc/d57/da4 0 2024-03-22T02:28:58.202 INFO:tasks.workunit.client.1.smithi122.stdout:3/530: mkdir d2/d9/dc/d17/d28/d4c/da5 0 2024-03-22T02:28:58.202 INFO:tasks.workunit.client.1.smithi122.stdout:7/593: dwrite d0/d4/d22/fb3 [0,4194304] 0 2024-03-22T02:28:58.205 INFO:tasks.workunit.client.1.smithi122.stdout:8/669: dwrite d1/f53 [0,4194304] 0 2024-03-22T02:28:58.206 INFO:tasks.workunit.client.1.smithi122.stdout:8/670: dread - d1/d22/d6a/fa2 zero size 2024-03-22T02:28:58.207 INFO:tasks.workunit.client.1.smithi122.stdout:8/671: chown d1/d22/d6a/c7d 1 1 2024-03-22T02:28:58.208 INFO:tasks.workunit.client.1.smithi122.stdout:7/594: chown d0/d4/d20/d21/d9b/l4b 61070364 1 2024-03-22T02:28:58.212 INFO:tasks.workunit.client.1.smithi122.stdout:3/531: unlink d2/d9/dc/d17/d28/f69 0 2024-03-22T02:28:58.219 INFO:tasks.workunit.client.1.smithi122.stdout:7/595: dwrite d0/d4/d20/d57/f6a [0,4194304] 0 2024-03-22T02:28:58.219 INFO:tasks.workunit.client.1.smithi122.stdout:8/672: dwrite d1/d3f/d45/fbf [0,4194304] 0 2024-03-22T02:28:58.220 INFO:tasks.workunit.client.1.smithi122.stdout:8/673: fdatasync d1/d60/d73/d90/f9b 0 2024-03-22T02:28:58.221 INFO:tasks.workunit.client.1.smithi122.stdout:7/596: read d0/d4/d20/d21/f2b [156338,67157] 0 2024-03-22T02:28:58.221 INFO:tasks.workunit.client.1.smithi122.stdout:3/532: unlink d2/d9/dc/df/d49/d85/l99 0 2024-03-22T02:28:58.224 INFO:tasks.workunit.client.1.smithi122.stdout:8/674: readlink d1/d60/d73/la0 0 2024-03-22T02:28:58.226 INFO:tasks.workunit.client.1.smithi122.stdout:8/675: symlink d1/d8/d63/lee 0 2024-03-22T02:28:58.229 INFO:tasks.workunit.client.1.smithi122.stdout:7/597: rmdir d0/d25/da1 0 2024-03-22T02:28:58.233 INFO:tasks.workunit.client.1.smithi122.stdout:8/676: dwrite d1/d3f/d45/f68 [0,4194304] 0 2024-03-22T02:28:58.234 INFO:tasks.workunit.client.1.smithi122.stdout:8/677: fsync d1/d3f/d45/f4b 0 2024-03-22T02:28:58.235 INFO:tasks.workunit.client.1.smithi122.stdout:7/598: mkdir d0/d4/d20/d21/d9b/dbb 0 2024-03-22T02:28:58.239 INFO:tasks.workunit.client.1.smithi122.stdout:8/678: creat d1/d60/d73/fef x:0 0 0 2024-03-22T02:28:58.241 INFO:tasks.workunit.client.1.smithi122.stdout:7/599: creat d0/d4/d20/d21/d9b/d5e/daa/fbc x:0 0 0 2024-03-22T02:28:58.243 INFO:tasks.workunit.client.1.smithi122.stdout:8/679: rename d1/d22 to d1/d8/d63/df0 0 2024-03-22T02:28:58.245 INFO:tasks.workunit.client.1.smithi122.stdout:8/680: dread - d1/f5d zero size 2024-03-22T02:28:58.247 INFO:tasks.workunit.client.1.smithi122.stdout:7/600: creat d0/d4/d20/d21/d9b/fbd x:0 0 0 2024-03-22T02:28:58.248 INFO:tasks.workunit.client.1.smithi122.stdout:8/681: creat d1/d4d/ff1 x:0 0 0 2024-03-22T02:28:58.250 INFO:tasks.workunit.client.1.smithi122.stdout:8/682: dread d1/d40/d71/dc9/f65 [0,4194304] 0 2024-03-22T02:28:58.251 INFO:tasks.workunit.client.1.smithi122.stdout:8/683: write d1/d60/d73/fef [530501,22847] 0 2024-03-22T02:28:58.252 INFO:tasks.workunit.client.1.smithi122.stdout:8/684: write d1/d8/d2f/fed [844150,85275] 0 2024-03-22T02:28:58.254 INFO:tasks.workunit.client.1.smithi122.stdout:8/685: mknod d1/d40/cf2 0 2024-03-22T02:28:58.257 INFO:tasks.workunit.client.1.smithi122.stdout:8/686: rename d1/d9/f28 to d1/d8/dba/dc0/ff3 0 2024-03-22T02:28:58.267 INFO:tasks.workunit.client.1.smithi122.stdout:7/601: dread d0/fd [0,4194304] 0 2024-03-22T02:28:58.270 INFO:tasks.workunit.client.1.smithi122.stdout:7/602: rename d0/d4/d20/d21/c2f to d0/d4/d20/d21/d9b/dbb/cbe 0 2024-03-22T02:28:58.273 INFO:tasks.workunit.client.1.smithi122.stdout:7/603: link d0/d4e/f73 d0/d4/d20/d57/fbf 0 2024-03-22T02:28:58.274 INFO:tasks.workunit.client.1.smithi122.stdout:7/604: mknod d0/d30/cc0 0 2024-03-22T02:28:58.277 INFO:tasks.workunit.client.1.smithi122.stdout:7/605: rename d0/d4/d20/d21/fb6 to d0/fc1 0 2024-03-22T02:28:58.277 INFO:tasks.workunit.client.1.smithi122.stdout:7/606: chown d0/d25 25 1 2024-03-22T02:28:58.285 INFO:tasks.workunit.client.1.smithi122.stdout:7/607: truncate d0/d4/d22/f64 3410039 0 2024-03-22T02:28:58.288 INFO:tasks.workunit.client.1.smithi122.stdout:7/608: rename d0/d25/l86 to d0/d4/d20/d21/d9b/d5e/lc2 0 2024-03-22T02:28:58.370 INFO:tasks.workunit.client.1.smithi122.stdout:0/530: sync 2024-03-22T02:28:58.370 INFO:tasks.workunit.client.1.smithi122.stdout:5/532: sync 2024-03-22T02:28:58.370 INFO:tasks.workunit.client.1.smithi122.stdout:4/536: sync 2024-03-22T02:28:58.371 INFO:tasks.workunit.client.1.smithi122.stdout:0/531: chown d11/d14/da7 29947 1 2024-03-22T02:28:58.373 INFO:tasks.workunit.client.1.smithi122.stdout:5/533: symlink d8/d1e/d22/d3a/d46/d4f/d55/d6a/lb7 0 2024-03-22T02:28:58.375 INFO:tasks.workunit.client.1.smithi122.stdout:4/537: creat d0/d12/d15/d42/d4e/fad x:0 0 0 2024-03-22T02:28:58.378 INFO:tasks.workunit.client.1.smithi122.stdout:5/534: mknod d8/d1e/d22/d3a/d7a/cb8 0 2024-03-22T02:28:58.380 INFO:tasks.workunit.client.1.smithi122.stdout:5/535: creat d8/d1e/d22/d3a/d7a/fb9 x:0 0 0 2024-03-22T02:28:58.380 INFO:tasks.workunit.client.1.smithi122.stdout:5/536: dread - d8/d3b/faf zero size 2024-03-22T02:28:58.381 INFO:tasks.workunit.client.1.smithi122.stdout:4/538: mknod d0/d25/cae 0 2024-03-22T02:28:58.382 INFO:tasks.workunit.client.1.smithi122.stdout:4/539: readlink d0/d12/l46 0 2024-03-22T02:28:58.382 INFO:tasks.workunit.client.1.smithi122.stdout:5/537: read - d8/d1e/d22/d3a/d46/f6f zero size 2024-03-22T02:28:58.384 INFO:tasks.workunit.client.1.smithi122.stdout:4/540: getdents d0/d12/da2 0 2024-03-22T02:28:58.386 INFO:tasks.workunit.client.1.smithi122.stdout:0/532: write d11/d14/d1b/f2c [715360,60541] 0 2024-03-22T02:28:58.388 INFO:tasks.workunit.client.1.smithi122.stdout:0/533: rmdir d11/d16/d44 39 2024-03-22T02:28:58.389 INFO:tasks.workunit.client.1.smithi122.stdout:0/534: chown d11/d14/d1b/f8f 1 1 2024-03-22T02:28:58.391 INFO:tasks.workunit.client.1.smithi122.stdout:4/541: getdents d0/d12/d15/d3d/d90 0 2024-03-22T02:28:58.395 INFO:tasks.workunit.client.1.smithi122.stdout:4/542: rename d0/l39 to d0/d12/d15/d3d/d90/laf 0 2024-03-22T02:28:58.396 INFO:tasks.workunit.client.1.smithi122.stdout:5/538: truncate d8/d1e/d51/fb1 1131804 0 2024-03-22T02:28:58.399 INFO:tasks.workunit.client.1.smithi122.stdout:4/543: dread d0/f37 [0,4194304] 0 2024-03-22T02:28:58.400 INFO:tasks.workunit.client.1.smithi122.stdout:5/539: rmdir d8/d1e/d22/d3a 39 2024-03-22T02:28:58.403 INFO:tasks.workunit.client.1.smithi122.stdout:5/540: fdatasync d8/d1e/d51/f75 0 2024-03-22T02:28:58.405 INFO:tasks.workunit.client.1.smithi122.stdout:0/535: truncate d11/d14/d23/f2d 970329 0 2024-03-22T02:28:58.405 INFO:tasks.workunit.client.1.smithi122.stdout:5/541: read d8/d1a/f70 [490323,81868] 0 2024-03-22T02:28:58.405 INFO:tasks.workunit.client.1.smithi122.stdout:4/544: truncate d0/d12/d15/d42/d4e/f22 577202 0 2024-03-22T02:28:58.406 INFO:tasks.workunit.client.1.smithi122.stdout:0/536: write d11/d14/d23/d24/d33/f79 [5048733,58697] 0 2024-03-22T02:28:58.407 INFO:tasks.workunit.client.1.smithi122.stdout:4/545: chown d0/d25/l65 5 1 2024-03-22T02:28:58.414 INFO:tasks.workunit.client.1.smithi122.stdout:0/537: rename d11/d14/d1d/d27/d32/f74 to d11/d14/d1d/d27/d59/fb3 0 2024-03-22T02:28:58.415 INFO:tasks.workunit.client.1.smithi122.stdout:4/546: creat d0/d12/d15/da7/fb0 x:0 0 0 2024-03-22T02:28:58.418 INFO:tasks.workunit.client.1.smithi122.stdout:5/542: rename d8/d1a/la0 to d8/d1e/d22/d3a/d46/d4f/d55/d6a/lba 0 2024-03-22T02:28:58.419 INFO:tasks.workunit.client.1.smithi122.stdout:4/547: mknod d0/d12/d15/d42/d4e/de/d91/cb1 0 2024-03-22T02:28:58.421 INFO:tasks.workunit.client.1.smithi122.stdout:5/543: creat d8/d74/fbb x:0 0 0 2024-03-22T02:28:58.422 INFO:tasks.workunit.client.1.smithi122.stdout:0/538: link d11/d14/d1b/lad d11/d14/d1d/d27/d9d/lb4 0 2024-03-22T02:28:58.426 INFO:tasks.workunit.client.1.smithi122.stdout:0/539: symlink d11/d14/d1d/d27/d9d/lb5 0 2024-03-22T02:28:58.426 INFO:tasks.workunit.client.1.smithi122.stdout:0/540: dread - d11/d14/d1d/fa2 zero size 2024-03-22T02:28:58.429 INFO:tasks.workunit.client.1.smithi122.stdout:0/541: creat d11/d14/d23/d24/d8b/fb6 x:0 0 0 2024-03-22T02:28:58.432 INFO:tasks.workunit.client.1.smithi122.stdout:5/544: getdents d8/d1a 0 2024-03-22T02:28:58.434 INFO:tasks.workunit.client.1.smithi122.stdout:4/548: dread d0/d12/d15/f55 [0,4194304] 0 2024-03-22T02:28:58.438 INFO:tasks.workunit.client.1.smithi122.stdout:4/549: rename d0/d25/f1d to d0/d12/fb2 0 2024-03-22T02:28:58.439 INFO:tasks.workunit.client.1.smithi122.stdout:5/545: dwrite d8/d74/fbb [0,4194304] 0 2024-03-22T02:28:58.441 INFO:tasks.workunit.client.1.smithi122.stdout:4/550: creat d0/d12/d73/fb3 x:0 0 0 2024-03-22T02:28:58.442 INFO:tasks.workunit.client.1.smithi122.stdout:5/546: creat d8/d1e/d22/d3a/d46/d4f/fbc x:0 0 0 2024-03-22T02:28:58.445 INFO:tasks.workunit.client.1.smithi122.stdout:4/551: rmdir d0/d12/d15/d42/d4e 39 2024-03-22T02:28:58.447 INFO:tasks.workunit.client.1.smithi122.stdout:4/552: write d0/d25/f8 [2323967,111134] 0 2024-03-22T02:28:58.447 INFO:tasks.workunit.client.1.smithi122.stdout:0/542: dread d11/d4f/d5a/f61 [0,4194304] 0 2024-03-22T02:28:58.448 INFO:tasks.workunit.client.1.smithi122.stdout:0/543: chown d11/d14/d1b/f3c 0 1 2024-03-22T02:28:58.452 INFO:tasks.workunit.client.1.smithi122.stdout:4/553: mknod d0/d7a/cb4 0 2024-03-22T02:28:58.453 INFO:tasks.workunit.client.1.smithi122.stdout:0/544: mknod d11/d4f/cb7 0 2024-03-22T02:28:58.454 INFO:tasks.workunit.client.1.smithi122.stdout:5/547: getdents d8/d1a 0 2024-03-22T02:28:58.456 INFO:tasks.workunit.client.1.smithi122.stdout:4/554: rmdir d0/d12/d15/d42/d4e/de/d91 39 2024-03-22T02:28:58.458 INFO:tasks.workunit.client.1.smithi122.stdout:5/548: rename d8/d1e/c27 to d8/d1e/d22/da3/cbd 0 2024-03-22T02:28:58.459 INFO:tasks.workunit.client.1.smithi122.stdout:0/545: creat d11/fb8 x:0 0 0 2024-03-22T02:28:58.462 INFO:tasks.workunit.client.1.smithi122.stdout:5/549: rename d8/d1e/d22/f47 to d8/d1e/d22/d78/fbe 0 2024-03-22T02:28:58.467 INFO:tasks.workunit.client.1.smithi122.stdout:0/546: dwrite d11/d14/d23/d24/d8b/fb6 [0,4194304] 0 2024-03-22T02:28:58.470 INFO:tasks.workunit.client.1.smithi122.stdout:0/547: rmdir d11/d14/d1d/d27/d9d 39 2024-03-22T02:28:58.472 INFO:tasks.workunit.client.1.smithi122.stdout:4/555: dwrite d0/d25/f7c [0,4194304] 0 2024-03-22T02:28:58.476 INFO:tasks.workunit.client.1.smithi122.stdout:0/548: rename d11/d14/d23/d24/f86 to d11/d16/d37/d84/da3/fb9 0 2024-03-22T02:28:58.478 INFO:tasks.workunit.client.1.smithi122.stdout:0/549: chown d11/d14/d23/l7d 38 1 2024-03-22T02:28:58.479 INFO:tasks.workunit.client.1.smithi122.stdout:4/556: dwrite d0/d25/f8 [0,4194304] 0 2024-03-22T02:28:58.481 INFO:tasks.workunit.client.1.smithi122.stdout:4/557: fsync d0/d25/f27 0 2024-03-22T02:28:58.483 INFO:tasks.workunit.client.1.smithi122.stdout:4/558: truncate d0/d12/d15/d42/d57/f8e 2539648 0 2024-03-22T02:28:58.484 INFO:tasks.workunit.client.1.smithi122.stdout:0/550: mknod d11/d14/d1d/d27/d9d/daa/cba 0 2024-03-22T02:28:58.485 INFO:tasks.workunit.client.1.smithi122.stdout:4/559: mkdir d0/d12/d15/d2d/db5 0 2024-03-22T02:28:58.503 INFO:tasks.workunit.client.1.smithi122.stdout:6/759: sync 2024-03-22T02:28:58.504 INFO:tasks.workunit.client.1.smithi122.stdout:2/666: sync 2024-03-22T02:28:58.508 INFO:tasks.workunit.client.1.smithi122.stdout:0/551: dwrite d11/d14/d1b/f2c [0,4194304] 0 2024-03-22T02:28:58.512 INFO:tasks.workunit.client.1.smithi122.stdout:6/760: mkdir d7/de/dd/d20/de0/d54/d76/df3 0 2024-03-22T02:28:58.515 INFO:tasks.workunit.client.1.smithi122.stdout:2/667: mkdir df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2 0 2024-03-22T02:28:58.515 INFO:tasks.workunit.client.1.smithi122.stdout:0/552: mknod d11/d14/d23/d24/d33/d67/cbb 0 2024-03-22T02:28:58.517 INFO:tasks.workunit.client.1.smithi122.stdout:0/553: chown d11/d4f 248 1 2024-03-22T02:28:58.518 INFO:tasks.workunit.client.1.smithi122.stdout:5/550: dwrite d8/d1e/d22/d78/fbe [0,4194304] 0 2024-03-22T02:28:58.518 INFO:tasks.workunit.client.1.smithi122.stdout:2/668: read df/f37 [1451498,26954] 0 2024-03-22T02:28:58.519 INFO:tasks.workunit.client.1.smithi122.stdout:4/560: dwrite d0/d12/d15/d3d/d90/f78 [0,4194304] 0 2024-03-22T02:28:58.521 INFO:tasks.workunit.client.1.smithi122.stdout:5/551: truncate d8/d1e/d22/d78/f98 885332 0 2024-03-22T02:28:58.522 INFO:tasks.workunit.client.1.smithi122.stdout:0/554: read - d11/d14/da7/f75 zero size 2024-03-22T02:28:58.523 INFO:tasks.workunit.client.1.smithi122.stdout:0/555: read d11/d14/d1d/d27/d59/fb2 [454478,17076] 0 2024-03-22T02:28:58.524 INFO:tasks.workunit.client.1.smithi122.stdout:6/761: unlink d7/de/dd/d20/de0/dae/fe6 0 2024-03-22T02:28:58.526 INFO:tasks.workunit.client.1.smithi122.stdout:4/561: creat d0/d7a/fb6 x:0 0 0 2024-03-22T02:28:58.527 INFO:tasks.workunit.client.1.smithi122.stdout:5/552: dread d8/d3b/f5a [0,4194304] 0 2024-03-22T02:28:58.528 INFO:tasks.workunit.client.1.smithi122.stdout:2/669: unlink df/d1f/d23/faf 0 2024-03-22T02:28:58.530 INFO:tasks.workunit.client.1.smithi122.stdout:0/556: creat d11/d14/da7/fbc x:0 0 0 2024-03-22T02:28:58.532 INFO:tasks.workunit.client.1.smithi122.stdout:0/557: read - d11/d14/d1d/d27/d2f/f53 zero size 2024-03-22T02:28:58.533 INFO:tasks.workunit.client.1.smithi122.stdout:4/562: chown d0/d25/c2c 14878383 1 2024-03-22T02:28:58.536 INFO:tasks.workunit.client.1.smithi122.stdout:0/558: write d11/d4f/d5a/f5c [1745049,129413] 0 2024-03-22T02:28:58.540 INFO:tasks.workunit.client.1.smithi122.stdout:5/553: creat d8/da/d28/d35/d71/fbf x:0 0 0 2024-03-22T02:28:58.541 INFO:tasks.workunit.client.1.smithi122.stdout:2/670: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f68 [0,4194304] 0 2024-03-22T02:28:58.543 INFO:tasks.workunit.client.1.smithi122.stdout:0/559: creat d11/d16/d37/d9f/fbd x:0 0 0 2024-03-22T02:28:58.543 INFO:tasks.workunit.client.1.smithi122.stdout:5/554: dread - d8/d1e/d22/da3/fb5 zero size 2024-03-22T02:28:58.546 INFO:tasks.workunit.client.1.smithi122.stdout:4/563: rename d0/d12/d15/d3d/la8 to d0/d12/d15/d42/d4e/lb7 0 2024-03-22T02:28:58.546 INFO:tasks.workunit.client.1.smithi122.stdout:4/564: fsync d0/d12/d15/d42/f7b 0 2024-03-22T02:28:58.547 INFO:tasks.workunit.client.1.smithi122.stdout:5/555: symlink d8/d1e/d22/d3a/d46/d4f/d55/lc0 0 2024-03-22T02:28:58.549 INFO:tasks.workunit.client.1.smithi122.stdout:0/560: creat d11/d16/d44/fbe x:0 0 0 2024-03-22T02:28:58.549 INFO:tasks.workunit.client.1.smithi122.stdout:2/671: read - df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f82 zero size 2024-03-22T02:28:58.550 INFO:tasks.workunit.client.1.smithi122.stdout:5/556: truncate d8/d1e/d22/d3a/d7a/fad 250021 0 2024-03-22T02:28:58.550 INFO:tasks.workunit.client.1.smithi122.stdout:4/565: mknod d0/d12/d73/cb8 0 2024-03-22T02:28:58.553 INFO:tasks.workunit.client.1.smithi122.stdout:5/557: truncate d8/da/d28/d35/d71/fbf 946717 0 2024-03-22T02:28:58.555 INFO:tasks.workunit.client.1.smithi122.stdout:4/566: creat d0/d12/d15/d3d/d90/fb9 x:0 0 0 2024-03-22T02:28:58.556 INFO:tasks.workunit.client.1.smithi122.stdout:2/672: creat df/d1f/d23/dd2/dca/fe3 x:0 0 0 2024-03-22T02:28:58.558 INFO:tasks.workunit.client.1.smithi122.stdout:4/567: dread d0/d12/d15/d42/d57/f9a [0,4194304] 0 2024-03-22T02:28:58.558 INFO:tasks.workunit.client.1.smithi122.stdout:5/558: rename d8/d1e/d22/d3a/d46/l5f to d8/d1e/d22/d3a/d46/d4f/d9b/lc1 0 2024-03-22T02:28:58.560 INFO:tasks.workunit.client.1.smithi122.stdout:2/673: mknod df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ce4 0 2024-03-22T02:28:58.560 INFO:tasks.workunit.client.1.smithi122.stdout:4/568: fsync d0/d12/d15/d3d/d90/f98 0 2024-03-22T02:28:58.563 INFO:tasks.workunit.client.1.smithi122.stdout:5/559: unlink d8/d1e/d22/d3a/d46/d4f/d55/db6/f7f 0 2024-03-22T02:28:58.564 INFO:tasks.workunit.client.1.smithi122.stdout:4/569: mknod d0/d12/d15/d2d/cba 0 2024-03-22T02:28:58.567 INFO:tasks.workunit.client.1.smithi122.stdout:2/674: symlink df/d1f/d33/d4d/d94/dc2/d72/le5 0 2024-03-22T02:28:58.569 INFO:tasks.workunit.client.1.smithi122.stdout:4/570: truncate d0/d12/d15/d42/d57/f64 747589 0 2024-03-22T02:28:58.570 INFO:tasks.workunit.client.1.smithi122.stdout:4/571: chown d0/d12/d15 102 1 2024-03-22T02:28:58.573 INFO:tasks.workunit.client.1.smithi122.stdout:2/675: link df/d1f/d33/d4d/f77 df/d1f/d33/d4d/d94/dc2/fe6 0 2024-03-22T02:28:58.575 INFO:tasks.workunit.client.1.smithi122.stdout:2/676: truncate df/d1f/d33/d6a/d7c/fce 176025 0 2024-03-22T02:28:58.578 INFO:tasks.workunit.client.1.smithi122.stdout:5/560: dread d8/da/d28/d35/f68 [0,4194304] 0 2024-03-22T02:28:58.578 INFO:tasks.workunit.client.1.smithi122.stdout:2/677: truncate df/d1f/d33/d34/f5f 4976179 0 2024-03-22T02:28:58.585 INFO:tasks.workunit.client.1.smithi122.stdout:0/561: dwrite d11/d14/d1d/d27/d2f/f70 [0,4194304] 0 2024-03-22T02:28:58.588 INFO:tasks.workunit.client.1.smithi122.stdout:0/562: creat d11/d16/d37/fbf x:0 0 0 2024-03-22T02:28:58.592 INFO:tasks.workunit.client.1.smithi122.stdout:0/563: mkdir d11/d14/d1d/d27/dc0 0 2024-03-22T02:28:58.593 INFO:tasks.workunit.client.1.smithi122.stdout:4/572: truncate d0/d12/d15/d42/d4e/de/f44 3682976 0 2024-03-22T02:28:58.595 INFO:tasks.workunit.client.1.smithi122.stdout:0/564: unlink d11/d14/d1d/d27/d32/c9c 0 2024-03-22T02:28:58.596 INFO:tasks.workunit.client.1.smithi122.stdout:0/565: stat d11/f3d 0 2024-03-22T02:28:58.596 INFO:tasks.workunit.client.1.smithi122.stdout:2/678: dread df/d1f/f20 [0,4194304] 0 2024-03-22T02:28:58.596 INFO:tasks.workunit.client.1.smithi122.stdout:4/573: mknod d0/d7a/cbb 0 2024-03-22T02:28:58.599 INFO:tasks.workunit.client.1.smithi122.stdout:0/566: mknod d11/d14/cc1 0 2024-03-22T02:28:58.599 INFO:tasks.workunit.client.1.smithi122.stdout:5/561: dread d8/da/d28/d35/f94 [0,4194304] 0 2024-03-22T02:28:58.600 INFO:tasks.workunit.client.1.smithi122.stdout:2/679: creat df/d1f/d33/d4d/d94/fe7 x:0 0 0 2024-03-22T02:28:58.603 INFO:tasks.workunit.client.1.smithi122.stdout:2/680: symlink df/d1f/d3d/d52/le8 0 2024-03-22T02:28:58.603 INFO:tasks.workunit.client.1.smithi122.stdout:0/567: read fb [6035133,84524] 0 2024-03-22T02:28:58.604 INFO:tasks.workunit.client.1.smithi122.stdout:2/681: dread - df/d1f/d33/d6a/f90 zero size 2024-03-22T02:28:58.605 INFO:tasks.workunit.client.1.smithi122.stdout:5/562: dread d8/d3b/f62 [0,4194304] 0 2024-03-22T02:28:58.608 INFO:tasks.workunit.client.1.smithi122.stdout:2/682: write df/d1f/d33/d6a/d7c/fce [776685,78569] 0 2024-03-22T02:28:58.611 INFO:tasks.workunit.client.1.smithi122.stdout:5/563: rmdir d8/d1e/d22/d3a/d46/d4f/d55/d6a 39 2024-03-22T02:28:58.611 INFO:tasks.workunit.client.1.smithi122.stdout:2/683: truncate df/d1f/d33/d4d/d7a/fc0 911315 0 2024-03-22T02:28:58.612 INFO:tasks.workunit.client.1.smithi122.stdout:0/568: dwrite d11/d14/d23/d24/d8b/fb6 [0,4194304] 0 2024-03-22T02:28:58.615 INFO:tasks.workunit.client.1.smithi122.stdout:0/569: readlink d11/d14/d1d/d27/d9d/lb4 0 2024-03-22T02:28:58.615 INFO:tasks.workunit.client.1.smithi122.stdout:5/564: read - d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f9d zero size 2024-03-22T02:28:58.617 INFO:tasks.workunit.client.1.smithi122.stdout:2/684: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/fe9 x:0 0 0 2024-03-22T02:28:58.618 INFO:tasks.workunit.client.1.smithi122.stdout:0/570: write d11/d14/d1d/d27/d2f/f70 [590848,41094] 0 2024-03-22T02:28:58.620 INFO:tasks.workunit.client.1.smithi122.stdout:4/574: dwrite d0/d25/f13 [0,4194304] 0 2024-03-22T02:28:58.622 INFO:tasks.workunit.client.1.smithi122.stdout:5/565: symlink d8/d1e/d22/d3a/d39/lc2 0 2024-03-22T02:28:58.626 INFO:tasks.workunit.client.1.smithi122.stdout:2/685: truncate df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/fbd 878837 0 2024-03-22T02:28:58.627 INFO:tasks.workunit.client.1.smithi122.stdout:0/571: dread f10 [0,4194304] 0 2024-03-22T02:28:58.629 INFO:tasks.workunit.client.1.smithi122.stdout:2/686: dread - df/d1f/d33/d4d/d94/fc8 zero size 2024-03-22T02:28:58.629 INFO:tasks.workunit.client.1.smithi122.stdout:4/575: dread d0/d25/f7c [0,4194304] 0 2024-03-22T02:28:58.631 INFO:tasks.workunit.client.1.smithi122.stdout:4/576: chown d0/d12/d15/d42/d4e 226922 1 2024-03-22T02:28:58.631 INFO:tasks.workunit.client.1.smithi122.stdout:5/566: dwrite d8/d3b/faf [0,4194304] 0 2024-03-22T02:28:58.638 INFO:tasks.workunit.client.1.smithi122.stdout:5/567: truncate d8/d1a/f5c 266358 0 2024-03-22T02:28:58.692 INFO:tasks.workunit.client.1.smithi122.stdout:9/675: sync 2024-03-22T02:28:58.692 INFO:tasks.workunit.client.1.smithi122.stdout:1/601: sync 2024-03-22T02:28:58.709 INFO:tasks.workunit.client.1.smithi122.stdout:2/687: getdents df/d1f/d23/dd2 0 2024-03-22T02:28:58.710 INFO:tasks.workunit.client.1.smithi122.stdout:2/688: readlink df/d1f/d33/d6a/lbe 0 2024-03-22T02:28:58.711 INFO:tasks.workunit.client.1.smithi122.stdout:4/577: creat d0/d12/d15/d42/d4e/fbc x:0 0 0 2024-03-22T02:28:58.715 INFO:tasks.workunit.client.1.smithi122.stdout:9/676: creat db/d1b/d28/dc1/fd4 x:0 0 0 2024-03-22T02:28:58.720 INFO:tasks.workunit.client.1.smithi122.stdout:4/578: mknod d0/d12/d15/d42/d4e/de/d40/cbd 0 2024-03-22T02:28:58.721 INFO:tasks.workunit.client.1.smithi122.stdout:9/677: mknod db/d1b/d28/dc1/cd5 0 2024-03-22T02:28:58.723 INFO:tasks.workunit.client.1.smithi122.stdout:5/568: dwrite d8/d3b/f52 [0,4194304] 0 2024-03-22T02:28:58.725 INFO:tasks.workunit.client.1.smithi122.stdout:4/579: symlink d0/d7a/lbe 0 2024-03-22T02:28:58.726 INFO:tasks.workunit.client.1.smithi122.stdout:1/602: dwrite dc/d4c/f62 [0,4194304] 0 2024-03-22T02:28:58.729 INFO:tasks.workunit.client.1.smithi122.stdout:2/689: dwrite df/d1f/d33/d4d/d94/dc2/f2f [0,4194304] 0 2024-03-22T02:28:58.732 INFO:tasks.workunit.client.1.smithi122.stdout:4/580: mkdir d0/d12/d15/d42/d4e/de/da1/dbf 0 2024-03-22T02:28:58.735 INFO:tasks.workunit.client.1.smithi122.stdout:1/603: mknod dc/d55/cc4 0 2024-03-22T02:28:58.740 INFO:tasks.workunit.client.1.smithi122.stdout:2/690: symlink df/d1f/d33/d6a/d7c/d8e/da0/dd4/lea 0 2024-03-22T02:28:58.742 INFO:tasks.workunit.client.1.smithi122.stdout:4/581: creat d0/d12/d15/d42/fc0 x:0 0 0 2024-03-22T02:28:58.744 INFO:tasks.workunit.client.1.smithi122.stdout:9/678: dwrite db/d23/d1e/f70 [0,4194304] 0 2024-03-22T02:28:58.745 INFO:tasks.workunit.client.1.smithi122.stdout:2/691: chown df/d1f/d33/d4d/d94/dc2/d51/c89 45803585 1 2024-03-22T02:28:58.746 INFO:tasks.workunit.client.1.smithi122.stdout:9/679: chown db/df/d3e/d81/d25/c8e 174 1 2024-03-22T02:28:58.749 INFO:tasks.workunit.client.1.smithi122.stdout:2/692: mkdir df/d1f/d33/d4d/d94/deb 0 2024-03-22T02:28:58.750 INFO:tasks.workunit.client.1.smithi122.stdout:9/680: mkdir db/d23/dd6 0 2024-03-22T02:28:58.754 INFO:tasks.workunit.client.1.smithi122.stdout:2/693: write df/d1f/d33/d6a/d7c/fce [137940,41037] 0 2024-03-22T02:28:58.756 INFO:tasks.workunit.client.1.smithi122.stdout:4/582: dwrite d0/d25/f7c [0,4194304] 0 2024-03-22T02:28:58.758 INFO:tasks.workunit.client.1.smithi122.stdout:9/681: rmdir db/df/d3e/d81/d25/d41 39 2024-03-22T02:28:58.758 INFO:tasks.workunit.client.1.smithi122.stdout:5/569: dread d8/da/f16 [0,4194304] 0 2024-03-22T02:28:58.763 INFO:tasks.workunit.client.1.smithi122.stdout:9/682: rmdir db/d23/d1f/d57/d66/d73 39 2024-03-22T02:28:58.766 INFO:tasks.workunit.client.1.smithi122.stdout:5/570: mkdir d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3 0 2024-03-22T02:28:58.768 INFO:tasks.workunit.client.1.smithi122.stdout:2/694: rename df/d1f/d33/d6a/d7c/lc6 to df/d1f/d33/d4d/d94/deb/lec 0 2024-03-22T02:28:58.769 INFO:tasks.workunit.client.1.smithi122.stdout:2/695: chown df/d1f/d33/d4d/d94/dc2/d72 30 1 2024-03-22T02:28:58.776 INFO:tasks.workunit.client.1.smithi122.stdout:2/696: mknod df/d1f/d33/d4d/d94/dc2/d51/d67/db0/ced 0 2024-03-22T02:28:58.777 INFO:tasks.workunit.client.1.smithi122.stdout:5/571: creat d8/fc4 x:0 0 0 2024-03-22T02:28:58.778 INFO:tasks.workunit.client.1.smithi122.stdout:2/697: chown df/d1f/d33/d4d/d94/dc2/d51/cb8 120874873 1 2024-03-22T02:28:58.780 INFO:tasks.workunit.client.1.smithi122.stdout:9/683: truncate db/df/d3e/d59/db5/f8a 4336224 0 2024-03-22T02:28:58.782 INFO:tasks.workunit.client.1.smithi122.stdout:5/572: dread d8/d1e/d22/d3a/d7a/fad [0,4194304] 0 2024-03-22T02:28:58.783 INFO:tasks.workunit.client.1.smithi122.stdout:2/698: read df/d1f/d33/d4d/d94/dc2/fe6 [807935,20362] 0 2024-03-22T02:28:58.784 INFO:tasks.workunit.client.1.smithi122.stdout:9/684: write db/d23/f1a [3324977,18561] 0 2024-03-22T02:28:58.785 INFO:tasks.workunit.client.1.smithi122.stdout:9/685: dread - db/d1b/da8/fb7 zero size 2024-03-22T02:28:58.787 INFO:tasks.workunit.client.1.smithi122.stdout:4/583: dread d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:58.789 INFO:tasks.workunit.client.1.smithi122.stdout:5/573: getdents d8/d1e/d22/d3a/d46/d4f/d9b/da9 0 2024-03-22T02:28:58.789 INFO:tasks.workunit.client.1.smithi122.stdout:2/699: dread df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/fbd [0,4194304] 0 2024-03-22T02:28:58.797 INFO:tasks.workunit.client.1.smithi122.stdout:2/700: rmdir df 39 2024-03-22T02:28:58.802 INFO:tasks.workunit.client.1.smithi122.stdout:4/584: rename d0/d7a/l8f to d0/d12/d15/d42/d57/lc1 0 2024-03-22T02:28:58.805 INFO:tasks.workunit.client.1.smithi122.stdout:5/574: rename d8/d1a/c56 to d8/d1a/cc5 0 2024-03-22T02:28:58.806 INFO:tasks.workunit.client.1.smithi122.stdout:9/686: creat db/df/d3e/d81/d25/fd7 x:0 0 0 2024-03-22T02:28:58.807 INFO:tasks.workunit.client.1.smithi122.stdout:4/585: unlink d0/d12/la5 0 2024-03-22T02:28:58.809 INFO:tasks.workunit.client.1.smithi122.stdout:5/575: symlink d8/d3b/lc6 0 2024-03-22T02:28:58.812 INFO:tasks.workunit.client.1.smithi122.stdout:4/586: truncate d0/d12/d15/d42/d4e/de/d40/f9d 752520 0 2024-03-22T02:28:58.813 INFO:tasks.workunit.client.1.smithi122.stdout:9/687: dwrite db/d1b/da8/fcd [0,4194304] 0 2024-03-22T02:28:58.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/688: write db/df/d3e/d81/d25/fd7 [166265,74050] 0 2024-03-22T02:28:58.815 INFO:tasks.workunit.client.1.smithi122.stdout:2/701: write df/d1f/d33/d4d/d94/f95 [224588,12044] 0 2024-03-22T02:28:58.816 INFO:tasks.workunit.client.1.smithi122.stdout:2/702: fsync df/f10 0 2024-03-22T02:28:58.822 INFO:tasks.workunit.client.1.smithi122.stdout:5/576: rename d8/da/d28 to d8/d1e/d22/d3a/d46/d4f/d55/dc7 0 2024-03-22T02:28:58.823 INFO:tasks.workunit.client.1.smithi122.stdout:4/587: truncate d0/d12/d61/fa3 598124 0 2024-03-22T02:28:58.826 INFO:tasks.workunit.client.1.smithi122.stdout:9/689: dwrite db/df/d3e/d59/db5/f93 [0,4194304] 0 2024-03-22T02:28:58.828 INFO:tasks.workunit.client.1.smithi122.stdout:9/690: write db/d23/f83 [1612544,30529] 0 2024-03-22T02:28:58.833 INFO:tasks.workunit.client.1.smithi122.stdout:9/691: write db/df/f80 [4313529,33371] 0 2024-03-22T02:28:58.834 INFO:tasks.workunit.client.1.smithi122.stdout:4/588: dwrite d0/d12/d73/fb3 [0,4194304] 0 2024-03-22T02:28:58.847 INFO:tasks.workunit.client.1.smithi122.stdout:5/577: truncate d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f68 1425357 0 2024-03-22T02:28:58.847 INFO:tasks.workunit.client.1.smithi122.stdout:4/589: truncate d0/d12/d15/d42/d4e/f4a 3288115 0 2024-03-22T02:28:58.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:58 smithi122 ceph-mon[36403]: Reconfiguring node-exporter.smithi103 (dependencies changed)... 2024-03-22T02:28:58.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:58 smithi122 ceph-mon[36403]: Reconfiguring daemon node-exporter.smithi103 on smithi103 2024-03-22T02:28:58.850 INFO:tasks.workunit.client.1.smithi122.stdout:4/590: creat d0/d25/fc2 x:0 0 0 2024-03-22T02:28:58.851 INFO:tasks.workunit.client.1.smithi122.stdout:4/591: read d0/d12/f79 [93333,57704] 0 2024-03-22T02:28:58.852 INFO:tasks.workunit.client.1.smithi122.stdout:4/592: chown d0/d12/d15/c59 71 1 2024-03-22T02:28:58.854 INFO:tasks.workunit.client.1.smithi122.stdout:4/593: fdatasync d0/d12/d15/d42/d57/f64 0 2024-03-22T02:28:58.858 INFO:tasks.workunit.client.1.smithi122.stdout:4/594: stat d0/d12/d15/d42/d4e/de/d40/c81 0 2024-03-22T02:28:58.862 INFO:tasks.workunit.client.1.smithi122.stdout:5/578: dread d8/d1e/d22/d3a/f32 [0,4194304] 0 2024-03-22T02:28:58.864 INFO:tasks.workunit.client.1.smithi122.stdout:4/595: rename d0/d12/d15/d42/d4e/fb to d0/d12/d15/d42/d57/fc3 0 2024-03-22T02:28:58.866 INFO:tasks.workunit.client.1.smithi122.stdout:4/596: dread - d0/d12/d15/d42/d4e/de/f53 zero size 2024-03-22T02:28:58.867 INFO:tasks.workunit.client.1.smithi122.stdout:5/579: link c2 d8/d1a/cc8 0 2024-03-22T02:28:58.868 INFO:tasks.workunit.client.1.smithi122.stdout:4/597: creat d0/d12/fc4 x:0 0 0 2024-03-22T02:28:58.869 INFO:tasks.workunit.client.1.smithi122.stdout:5/580: creat d8/d74/fc9 x:0 0 0 2024-03-22T02:28:58.870 INFO:tasks.workunit.client.1.smithi122.stdout:4/598: symlink d0/d12/da2/lc5 0 2024-03-22T02:28:58.870 INFO:tasks.workunit.client.1.smithi122.stdout:5/581: read d8/da/f73 [2370317,53673] 0 2024-03-22T02:28:58.873 INFO:tasks.workunit.client.1.smithi122.stdout:5/582: creat d8/d1e/d22/d3a/d46/d4f/fca x:0 0 0 2024-03-22T02:28:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:58 smithi103 ceph-mon[29258]: Reconfiguring node-exporter.smithi103 (dependencies changed)... 2024-03-22T02:28:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:58 smithi103 ceph-mon[29258]: Reconfiguring daemon node-exporter.smithi103 on smithi103 2024-03-22T02:28:58.878 INFO:tasks.workunit.client.1.smithi122.stdout:4/599: dwrite d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:28:58.880 INFO:tasks.workunit.client.1.smithi122.stdout:4/600: fdatasync d0/d12/fc4 0 2024-03-22T02:28:58.882 INFO:tasks.workunit.client.1.smithi122.stdout:4/601: fdatasync d0/d25/f8 0 2024-03-22T02:28:58.882 INFO:tasks.workunit.client.1.smithi122.stdout:5/583: dwrite d8/d1e/d22/d3a/d46/d4f/fbc [0,4194304] 0 2024-03-22T02:28:58.885 INFO:tasks.workunit.client.1.smithi122.stdout:5/584: mkdir d8/d1a/d83/dcb 0 2024-03-22T02:28:58.887 INFO:tasks.workunit.client.1.smithi122.stdout:5/585: write d8/d1e/d22/d78/f98 [857873,41582] 0 2024-03-22T02:28:58.903 INFO:tasks.workunit.client.1.smithi122.stdout:4/602: truncate d0/d12/d15/d42/d4e/de/f8b 3034595 0 2024-03-22T02:28:58.909 INFO:tasks.workunit.client.1.smithi122.stdout:4/603: write d0/d12/d61/fa3 [260988,69577] 0 2024-03-22T02:28:58.912 INFO:tasks.workunit.client.1.smithi122.stdout:4/604: getdents d0/d12/d15 0 2024-03-22T02:28:58.914 INFO:tasks.workunit.client.1.smithi122.stdout:4/605: fsync d0/d12/d15/d3d/f54 0 2024-03-22T02:28:58.915 INFO:tasks.workunit.client.1.smithi122.stdout:4/606: chown d0/d12/d15/d42/c50 214766259 1 2024-03-22T02:28:58.916 INFO:tasks.workunit.client.1.smithi122.stdout:4/607: write d0/d12/d15/d42/d4e/fad [42005,106222] 0 2024-03-22T02:28:58.917 INFO:tasks.workunit.client.1.smithi122.stdout:4/608: readlink d0/d12/da2/lc5 0 2024-03-22T02:28:58.919 INFO:tasks.workunit.client.1.smithi122.stdout:4/609: unlink d0/d12/d15/da7/fb0 0 2024-03-22T02:28:58.921 INFO:tasks.workunit.client.1.smithi122.stdout:4/610: unlink d0/d12/d15/d42/d4e/de/d40/c93 0 2024-03-22T02:28:58.927 INFO:tasks.workunit.client.1.smithi122.stdout:4/611: write d0/d12/d15/d42/d57/fc3 [1838379,44066] 0 2024-03-22T02:28:58.927 INFO:tasks.workunit.client.1.smithi122.stdout:4/612: fdatasync d0/d25/f13 0 2024-03-22T02:28:58.930 INFO:tasks.workunit.client.1.smithi122.stdout:4/613: mknod d0/d12/da2/cc6 0 2024-03-22T02:28:58.933 INFO:tasks.workunit.client.1.smithi122.stdout:4/614: mknod d0/d12/d15/d42/d4e/de/d91/cc7 0 2024-03-22T02:28:58.936 INFO:tasks.workunit.client.1.smithi122.stdout:4/615: dread d0/d12/d15/d42/d4e/de/f8b [0,4194304] 0 2024-03-22T02:28:58.937 INFO:tasks.workunit.client.1.smithi122.stdout:4/616: chown d0/d12/d15/d2d/db5 117226408 1 2024-03-22T02:28:58.941 INFO:tasks.workunit.client.1.smithi122.stdout:4/617: dwrite d0/d12/d15/d42/d4e/de/f66 [0,4194304] 0 2024-03-22T02:28:58.946 INFO:tasks.workunit.client.1.smithi122.stdout:4/618: getdents d0/d12/d15 0 2024-03-22T02:28:58.954 INFO:tasks.workunit.client.1.smithi122.stdout:4/619: dwrite d0/d12/d15/d42/d4e/fad [0,4194304] 0 2024-03-22T02:28:58.955 INFO:tasks.workunit.client.1.smithi122.stdout:4/620: chown d0/d12/d73/fb3 14794474 1 2024-03-22T02:28:58.955 INFO:tasks.workunit.client.1.smithi122.stdout:3/533: sync 2024-03-22T02:28:58.955 INFO:tasks.workunit.client.1.smithi122.stdout:6/762: sync 2024-03-22T02:28:58.955 INFO:tasks.workunit.client.1.smithi122.stdout:7/609: sync 2024-03-22T02:28:58.955 INFO:tasks.workunit.client.1.smithi122.stdout:8/687: sync 2024-03-22T02:28:58.957 INFO:tasks.workunit.client.1.smithi122.stdout:8/688: symlink d1/d8/d63/df0/lf4 0 2024-03-22T02:28:58.958 INFO:tasks.workunit.client.1.smithi122.stdout:4/621: creat d0/d12/d15/d2d/db5/fc8 x:0 0 0 2024-03-22T02:28:58.959 INFO:tasks.workunit.client.1.smithi122.stdout:7/610: symlink d0/d4/d1d/db8/d9a/lc3 0 2024-03-22T02:28:58.960 INFO:tasks.workunit.client.1.smithi122.stdout:3/534: symlink d2/d8a/da2/la6 0 2024-03-22T02:28:58.961 INFO:tasks.workunit.client.1.smithi122.stdout:3/535: write d2/d9/f8d [218510,90478] 0 2024-03-22T02:28:58.961 INFO:tasks.workunit.client.1.smithi122.stdout:8/689: unlink d1/d40/d71/c7f 0 2024-03-22T02:28:58.964 INFO:tasks.workunit.client.1.smithi122.stdout:6/763: rename d7/de/f68 to d7/de/dd/d20/de0/d54/d76/d86/ff4 0 2024-03-22T02:28:58.966 INFO:tasks.workunit.client.1.smithi122.stdout:3/536: creat d2/d9/dc/df/d47/d79/fa7 x:0 0 0 2024-03-22T02:28:58.967 INFO:tasks.workunit.client.1.smithi122.stdout:3/537: dread - d2/d9/dc/d57/f91 zero size 2024-03-22T02:28:58.968 INFO:tasks.workunit.client.1.smithi122.stdout:4/622: creat d0/d12/d15/d42/d4e/de/fc9 x:0 0 0 2024-03-22T02:28:58.969 INFO:tasks.workunit.client.1.smithi122.stdout:6/764: rename d7/de/dd/d20/de0/d54/l97 to d7/de/dd/d23/d7f/lf5 0 2024-03-22T02:28:58.972 INFO:tasks.workunit.client.1.smithi122.stdout:6/765: creat d7/de/dd/d23/ff6 x:0 0 0 2024-03-22T02:28:58.974 INFO:tasks.workunit.client.1.smithi122.stdout:8/690: getdents d1/d3f/d45 0 2024-03-22T02:28:58.978 INFO:tasks.workunit.client.1.smithi122.stdout:6/766: fsync d7/de/f34 0 2024-03-22T02:28:58.978 INFO:tasks.workunit.client.1.smithi122.stdout:6/767: chown d7/de/dd 569731009 1 2024-03-22T02:28:58.979 INFO:tasks.workunit.client.1.smithi122.stdout:6/768: fdatasync d7/de/dd/d20/fd6 0 2024-03-22T02:28:58.980 INFO:tasks.workunit.client.1.smithi122.stdout:3/538: link d2/d9/dc/d17/d51/c95 d2/d9/dc/d57/da4/ca8 0 2024-03-22T02:28:58.983 INFO:tasks.workunit.client.1.smithi122.stdout:4/623: rename d0/d12/d15/d42/d4e/f4a to d0/d25/fca 0 2024-03-22T02:28:58.983 INFO:tasks.workunit.client.1.smithi122.stdout:4/624: chown d0/d12/d15 1256426278 1 2024-03-22T02:28:58.984 INFO:tasks.workunit.client.1.smithi122.stdout:4/625: truncate d0/d12/d61/f87 741608 0 2024-03-22T02:28:58.987 INFO:tasks.workunit.client.1.smithi122.stdout:6/769: creat d7/de/dd/d39/dc2/ff7 x:0 0 0 2024-03-22T02:28:58.987 INFO:tasks.workunit.client.1.smithi122.stdout:8/691: dread d1/d8/d2f/f44 [0,4194304] 0 2024-03-22T02:28:58.988 INFO:tasks.workunit.client.1.smithi122.stdout:3/539: mknod d2/d9/dc/d17/d2b/d6b/ca9 0 2024-03-22T02:28:58.988 INFO:tasks.workunit.client.1.smithi122.stdout:6/770: dread - d7/de/dd/d20/de0/dae/fcd zero size 2024-03-22T02:28:58.994 INFO:tasks.workunit.client.1.smithi122.stdout:8/692: rename d1/d8/d63/d88 to d1/d40/d51/df5 0 2024-03-22T02:28:58.998 INFO:tasks.workunit.client.1.smithi122.stdout:6/771: rename d7/de/dd/d23/d7f/cd9 to d7/de/dd/d20/de0/d54/d76/d86/cf8 0 2024-03-22T02:28:59.001 INFO:tasks.workunit.client.1.smithi122.stdout:3/540: truncate d2/d9/dc/df/d47/f76 3480494 0 2024-03-22T02:28:59.002 INFO:tasks.workunit.client.1.smithi122.stdout:3/541: stat d2/d9/dc/df/d3c/d59/l96 0 2024-03-22T02:28:59.005 INFO:tasks.workunit.client.1.smithi122.stdout:3/542: write d2/d9/f8d [484261,74044] 0 2024-03-22T02:28:59.005 INFO:tasks.workunit.client.1.smithi122.stdout:8/693: dwrite d1/d8/d2f/f98 [0,4194304] 0 2024-03-22T02:28:59.008 INFO:tasks.workunit.client.1.smithi122.stdout:8/694: chown d1/d40/d48/f96 22652841 1 2024-03-22T02:28:59.011 INFO:tasks.workunit.client.1.smithi122.stdout:8/695: mkdir d1/d40/d71/d97/db8/df6 0 2024-03-22T02:28:59.012 INFO:tasks.workunit.client.1.smithi122.stdout:8/696: write d1/d4d/f56 [464837,1927] 0 2024-03-22T02:28:59.013 INFO:tasks.workunit.client.1.smithi122.stdout:6/772: dread d7/de/dd/d39/fa3 [0,4194304] 0 2024-03-22T02:28:59.020 INFO:tasks.workunit.client.1.smithi122.stdout:8/697: dread d1/d40/d51/fb2 [0,4194304] 0 2024-03-22T02:28:59.023 INFO:tasks.workunit.client.1.smithi122.stdout:3/543: write d2/d9/dc/df/d38/f5d [2613900,4842] 0 2024-03-22T02:28:59.023 INFO:tasks.workunit.client.1.smithi122.stdout:6/773: write d7/de/dd/d20/f7c [462602,27022] 0 2024-03-22T02:28:59.025 INFO:tasks.workunit.client.1.smithi122.stdout:8/698: truncate d1/d9/f77 3796479 0 2024-03-22T02:28:59.028 INFO:tasks.workunit.client.1.smithi122.stdout:8/699: dread d1/d40/f9e [0,4194304] 0 2024-03-22T02:28:59.030 INFO:tasks.workunit.client.1.smithi122.stdout:8/700: rename d1/d8/l2c to d1/d4d/daa/lf7 0 2024-03-22T02:28:59.031 INFO:tasks.workunit.client.1.smithi122.stdout:8/701: chown d1/d8/dba/dc0/db9/fc2 29262183 1 2024-03-22T02:28:59.032 INFO:tasks.workunit.client.1.smithi122.stdout:8/702: mknod d1/d40/d71/d97/cf8 0 2024-03-22T02:28:59.033 INFO:tasks.workunit.client.1.smithi122.stdout:8/703: readlink d1/d40/d71/ld9 0 2024-03-22T02:28:59.038 INFO:tasks.workunit.client.1.smithi122.stdout:3/544: dread d2/d9/dc/df/f53 [0,4194304] 0 2024-03-22T02:28:59.041 INFO:tasks.workunit.client.1.smithi122.stdout:3/545: mknod d2/d9/dc/df/d3c/d78/caa 0 2024-03-22T02:28:59.042 INFO:tasks.workunit.client.1.smithi122.stdout:3/546: dread - d2/d9/dc/df/d49/f4f zero size 2024-03-22T02:28:59.044 INFO:tasks.workunit.client.1.smithi122.stdout:3/547: mkdir d2/d9/dc/d57/dab 0 2024-03-22T02:28:59.045 INFO:tasks.workunit.client.1.smithi122.stdout:8/704: dwrite d1/d8/d2f/f38 [0,4194304] 0 2024-03-22T02:28:59.051 INFO:tasks.workunit.client.1.smithi122.stdout:3/548: dwrite d2/d9/dc/d17/d28/f35 [0,4194304] 0 2024-03-22T02:28:59.062 INFO:tasks.workunit.client.1.smithi122.stdout:3/549: write d2/d9/dc/df/f48 [810059,69988] 0 2024-03-22T02:28:59.065 INFO:tasks.workunit.client.1.smithi122.stdout:3/550: creat d2/d9/dc/df/d49/fac x:0 0 0 2024-03-22T02:28:59.066 INFO:tasks.workunit.client.1.smithi122.stdout:8/705: dwrite d1/d9/f77 [0,4194304] 0 2024-03-22T02:28:59.069 INFO:tasks.workunit.client.1.smithi122.stdout:8/706: truncate d1/d8/dba/dc0/db9/fc2 584935 0 2024-03-22T02:28:59.071 INFO:tasks.workunit.client.1.smithi122.stdout:8/707: mknod d1/d40/d71/d97/db8/df6/cf9 0 2024-03-22T02:28:59.072 INFO:tasks.workunit.client.1.smithi122.stdout:3/551: write d2/f8 [914361,14243] 0 2024-03-22T02:28:59.077 INFO:tasks.workunit.client.1.smithi122.stdout:8/708: getdents d1/d4d 0 2024-03-22T02:28:59.077 INFO:tasks.workunit.client.1.smithi122.stdout:8/709: write d1/d3f/f41 [1756007,83671] 0 2024-03-22T02:28:59.081 INFO:tasks.workunit.client.1.smithi122.stdout:3/552: dwrite d2/d9/dc/d17/f63 [0,4194304] 0 2024-03-22T02:28:59.087 INFO:tasks.workunit.client.1.smithi122.stdout:8/710: dwrite d1/d8/d8a/f86 [0,4194304] 0 2024-03-22T02:28:59.090 INFO:tasks.workunit.client.1.smithi122.stdout:8/711: write d1/d40/d71/db4/fd7 [754962,21257] 0 2024-03-22T02:28:59.092 INFO:tasks.workunit.client.1.smithi122.stdout:3/553: dwrite d2/d9/dc/df/d38/f6f [0,4194304] 0 2024-03-22T02:28:59.099 INFO:tasks.workunit.client.1.smithi122.stdout:8/712: mkdir d1/d40/d51/dfa 0 2024-03-22T02:28:59.100 INFO:tasks.workunit.client.1.smithi122.stdout:3/554: getdents d2/d9/dc/d17/d2b/d3f 0 2024-03-22T02:28:59.102 INFO:tasks.workunit.client.1.smithi122.stdout:3/555: mkdir d2/d8a/dad 0 2024-03-22T02:28:59.108 INFO:tasks.workunit.client.1.smithi122.stdout:3/556: dwrite d2/d9/dc/d17/d45/f81 [0,4194304] 0 2024-03-22T02:28:59.134 INFO:tasks.workunit.client.1.smithi122.stdout:3/557: dread d2/d9/dc/d17/d28/f31 [0,4194304] 0 2024-03-22T02:28:59.140 INFO:tasks.workunit.client.1.smithi122.stdout:3/558: dwrite d2/d9/dc/df/d38/f5d [0,4194304] 0 2024-03-22T02:28:59.146 INFO:tasks.workunit.client.1.smithi122.stdout:3/559: dwrite d2/d9/dc/d17/d28/f4d [0,4194304] 0 2024-03-22T02:28:59.152 INFO:tasks.workunit.client.1.smithi122.stdout:3/560: dread d2/d9/dc/df/d38/f5d [0,4194304] 0 2024-03-22T02:28:59.155 INFO:tasks.workunit.client.1.smithi122.stdout:3/561: unlink d2/d9/dc/df/d3c/d59/f9c 0 2024-03-22T02:28:59.162 INFO:tasks.workunit.client.1.smithi122.stdout:3/562: dwrite d2/d9/dc/d57/f93 [0,4194304] 0 2024-03-22T02:28:59.163 INFO:tasks.workunit.client.1.smithi122.stdout:3/563: creat d2/d9/dc/fae x:0 0 0 2024-03-22T02:28:59.165 INFO:tasks.workunit.client.1.smithi122.stdout:0/572: sync 2024-03-22T02:28:59.166 INFO:tasks.workunit.client.1.smithi122.stdout:3/564: chown d2/d9/dc/df/d47/c66 22043 1 2024-03-22T02:28:59.166 INFO:tasks.workunit.client.1.smithi122.stdout:0/573: stat d11/d14/d1d/d27/l2b 0 2024-03-22T02:28:59.167 INFO:tasks.workunit.client.1.smithi122.stdout:3/565: chown d2/d9/dc/d57/f91 21 1 2024-03-22T02:28:59.169 INFO:tasks.workunit.client.1.smithi122.stdout:3/566: truncate d2/d9/dc/df/d38/fa1 70605 0 2024-03-22T02:28:59.169 INFO:tasks.workunit.client.1.smithi122.stdout:0/574: unlink d11/d14/d1b/f60 0 2024-03-22T02:28:59.170 INFO:tasks.workunit.client.1.smithi122.stdout:3/567: chown d2/d9/dc/df/d3c/f7b 19971683 1 2024-03-22T02:28:59.174 INFO:tasks.workunit.client.1.smithi122.stdout:3/568: mknod d2/d9/dc/d17/d2b/caf 0 2024-03-22T02:28:59.177 INFO:tasks.workunit.client.1.smithi122.stdout:0/575: dwrite d11/d14/d23/d24/d33/d67/f7c [0,4194304] 0 2024-03-22T02:28:59.180 INFO:tasks.workunit.client.1.smithi122.stdout:3/569: dwrite d2/f39 [0,4194304] 0 2024-03-22T02:28:59.186 INFO:tasks.workunit.client.1.smithi122.stdout:3/570: dread d2/d9/dc/df/d47/f56 [0,4194304] 0 2024-03-22T02:28:59.187 INFO:tasks.workunit.client.1.smithi122.stdout:3/571: write d2/f8 [5383223,13796] 0 2024-03-22T02:28:59.188 INFO:tasks.workunit.client.1.smithi122.stdout:3/572: readlink d2/d9/dc/d17/d45/l58 0 2024-03-22T02:28:59.203 INFO:tasks.workunit.client.1.smithi122.stdout:3/573: truncate d2/d9/dc/d17/d2b/f82 676150 0 2024-03-22T02:28:59.204 INFO:tasks.workunit.client.1.smithi122.stdout:3/574: symlink d2/d9/dc/d57/lb0 0 2024-03-22T02:28:59.206 INFO:tasks.workunit.client.1.smithi122.stdout:3/575: mkdir d2/d9/dc/df/d47/db1 0 2024-03-22T02:28:59.208 INFO:tasks.workunit.client.1.smithi122.stdout:3/576: truncate d2/d9/dc/df/f53 1050056 0 2024-03-22T02:28:59.272 INFO:tasks.workunit.client.1.smithi122.stdout:3/577: dwrite d2/d9/dc/d17/d45/f46 [0,4194304] 0 2024-03-22T02:28:59.275 INFO:tasks.workunit.client.1.smithi122.stdout:3/578: creat d2/d9/dc/df/d47/db1/fb2 x:0 0 0 2024-03-22T02:28:59.278 INFO:tasks.workunit.client.1.smithi122.stdout:3/579: creat d2/d9/dc/d17/d2b/d6b/fb3 x:0 0 0 2024-03-22T02:28:59.279 INFO:tasks.workunit.client.1.smithi122.stdout:3/580: write d2/d9/f8d [644032,17775] 0 2024-03-22T02:28:59.312 INFO:tasks.workunit.client.1.smithi122.stdout:1/604: sync 2024-03-22T02:28:59.320 INFO:tasks.workunit.client.1.smithi122.stdout:3/581: write d2/d9/dc/df/d47/f76 [4888563,90564] 0 2024-03-22T02:28:59.321 INFO:tasks.workunit.client.1.smithi122.stdout:3/582: read d2/f39 [2359091,94114] 0 2024-03-22T02:28:59.321 INFO:tasks.workunit.client.1.smithi122.stdout:3/583: chown d2/d9 65950 1 2024-03-22T02:28:59.325 INFO:tasks.workunit.client.1.smithi122.stdout:3/584: dread d2/d9/dc/d17/d28/f2d [0,4194304] 0 2024-03-22T02:28:59.336 INFO:tasks.workunit.client.1.smithi122.stdout:1/605: dwrite dc/d4c/f92 [0,4194304] 0 2024-03-22T02:28:59.338 INFO:tasks.workunit.client.1.smithi122.stdout:3/585: rename d2/d9/dc/df/d47 to d2/d9/dc/d17/d28/d4c/da5/db4 0 2024-03-22T02:28:59.351 INFO:tasks.workunit.client.1.smithi122.stdout:1/606: dread dc/f25 [0,4194304] 0 2024-03-22T02:28:59.359 INFO:tasks.workunit.client.1.smithi122.stdout:1/607: dwrite dc/d3b/d8d/d72/d9c/fae [0,4194304] 0 2024-03-22T02:28:59.361 INFO:tasks.workunit.client.1.smithi122.stdout:1/608: mknod dc/d3b/d8d/d72/d9c/db0/d90/cc5 0 2024-03-22T02:28:59.361 INFO:tasks.workunit.client.1.smithi122.stdout:1/609: chown dc/d2b/f36 46 1 2024-03-22T02:28:59.362 INFO:tasks.workunit.client.1.smithi122.stdout:1/610: stat dc/d3b/d46/c97 0 2024-03-22T02:28:59.366 INFO:tasks.workunit.client.1.smithi122.stdout:1/611: write dc/d3b/d8d/d94/f98 [1635792,113935] 0 2024-03-22T02:28:59.368 INFO:tasks.workunit.client.1.smithi122.stdout:1/612: rename dc/d3b/d46/d47 to dc/d3b/d8d/d72/d9c/db0/d67/dc6 0 2024-03-22T02:28:59.375 INFO:tasks.workunit.client.1.smithi122.stdout:1/613: write dc/f1d [3760066,59735] 0 2024-03-22T02:28:59.411 INFO:tasks.workunit.client.1.smithi122.stdout:1/614: read - dc/d3b/d8d/d72/d9c/db0/d67/dc6/f48 zero size 2024-03-22T02:28:59.423 INFO:tasks.workunit.client.1.smithi122.stdout:1/615: dread dc/d3b/f81 [0,4194304] 0 2024-03-22T02:28:59.426 INFO:tasks.workunit.client.1.smithi122.stdout:1/616: mknod dc/d3b/d8d/d72/d9c/db0/d67/dc6/cc7 0 2024-03-22T02:28:59.429 INFO:tasks.workunit.client.1.smithi122.stdout:1/617: rename dc/f11 to dc/d3b/d8d/d72/fc8 0 2024-03-22T02:28:59.431 INFO:tasks.workunit.client.1.smithi122.stdout:1/618: rmdir dc/d3b/d8d/d94 39 2024-03-22T02:28:59.433 INFO:tasks.workunit.client.1.smithi122.stdout:2/703: sync 2024-03-22T02:28:59.433 INFO:tasks.workunit.client.1.smithi122.stdout:9/692: sync 2024-03-22T02:28:59.433 INFO:tasks.workunit.client.1.smithi122.stdout:1/619: mknod dc/d3b/d8d/d72/cc9 0 2024-03-22T02:28:59.436 INFO:tasks.workunit.client.1.smithi122.stdout:9/693: unlink db/df/d3e/d81/d25/fd7 0 2024-03-22T02:28:59.437 INFO:tasks.workunit.client.1.smithi122.stdout:2/704: creat df/d1f/d33/d6a/d7c/d8e/da0/fee x:0 0 0 2024-03-22T02:28:59.440 INFO:tasks.workunit.client.1.smithi122.stdout:2/705: truncate df/d1f/d33/d4d/d94/dc2/d51/d67/fbf 380982 0 2024-03-22T02:28:59.442 INFO:tasks.workunit.client.1.smithi122.stdout:1/620: creat dc/d3b/d46/db5/dba/fca x:0 0 0 2024-03-22T02:28:59.445 INFO:tasks.workunit.client.1.smithi122.stdout:9/694: dread db/d1b/d28/dc1/f5f [0,4194304] 0 2024-03-22T02:28:59.448 INFO:tasks.workunit.client.1.smithi122.stdout:9/695: mkdir db/df/d3e/d81/d3d/d86/dd8 0 2024-03-22T02:28:59.452 INFO:tasks.workunit.client.1.smithi122.stdout:9/696: truncate db/df/d3e/d81/d25/d41/f61 3971539 0 2024-03-22T02:28:59.460 INFO:tasks.workunit.client.1.smithi122.stdout:1/621: dread dc/d2b/f2d [0,4194304] 0 2024-03-22T02:28:59.463 INFO:tasks.workunit.client.1.smithi122.stdout:2/706: truncate f6 424681 0 2024-03-22T02:28:59.465 INFO:tasks.workunit.client.1.smithi122.stdout:1/622: creat dc/d3b/fcb x:0 0 0 2024-03-22T02:28:59.467 INFO:tasks.workunit.client.1.smithi122.stdout:2/707: rmdir df/d1f/d33/d4d/d94/dc2/d51/d67 39 2024-03-22T02:28:59.470 INFO:tasks.workunit.client.1.smithi122.stdout:2/708: mknod df/d1f/cef 0 2024-03-22T02:28:59.471 INFO:tasks.workunit.client.1.smithi122.stdout:1/623: getdents dc/d3b/d8d/d72/d9c/db0/d67 0 2024-03-22T02:28:59.472 INFO:tasks.workunit.client.1.smithi122.stdout:2/709: write df/d1f/d33/d6a/d7c/d8e/fa1 [238949,36041] 0 2024-03-22T02:28:59.473 INFO:tasks.workunit.client.1.smithi122.stdout:9/697: dread db/d23/f83 [0,4194304] 0 2024-03-22T02:28:59.474 INFO:tasks.workunit.client.1.smithi122.stdout:1/624: creat dc/d3b/d46/db5/dba/fcc x:0 0 0 2024-03-22T02:28:59.475 INFO:tasks.workunit.client.1.smithi122.stdout:1/625: write dc/d55/fc3 [490346,78135] 0 2024-03-22T02:28:59.476 INFO:tasks.workunit.client.1.smithi122.stdout:2/710: unlink df/d1f/d33/d4d/fd8 0 2024-03-22T02:28:59.479 INFO:tasks.workunit.client.1.smithi122.stdout:1/626: unlink dc/d3b/d46/db5/dba/fca 0 2024-03-22T02:28:59.480 INFO:tasks.workunit.client.1.smithi122.stdout:1/627: fsync dc/d4c/f62 0 2024-03-22T02:28:59.487 INFO:tasks.workunit.client.1.smithi122.stdout:2/711: dwrite df/d1f/d33/d4d/d7a/f8a [0,4194304] 0 2024-03-22T02:28:59.488 INFO:tasks.workunit.client.1.smithi122.stdout:1/628: dwrite dc/d2b/f36 [0,4194304] 0 2024-03-22T02:28:59.492 INFO:tasks.workunit.client.1.smithi122.stdout:2/712: mkdir df/d1f/d33/d4d/d94/df0 0 2024-03-22T02:28:59.493 INFO:tasks.workunit.client.1.smithi122.stdout:9/698: dread db/df/f7d [0,4194304] 0 2024-03-22T02:28:59.496 INFO:tasks.workunit.client.1.smithi122.stdout:9/699: chown db/d1b/d28/dc1/f5f 0 1 2024-03-22T02:28:59.501 INFO:tasks.workunit.client.1.smithi122.stdout:1/629: dwrite dc/f22 [4194304,4194304] 0 2024-03-22T02:28:59.505 INFO:tasks.workunit.client.1.smithi122.stdout:1/630: rmdir dc/d3b/d8d/d72/d9c/db0 39 2024-03-22T02:28:59.506 INFO:tasks.workunit.client.1.smithi122.stdout:2/713: link df/d1f/d33/d4d/d94/d9e/fa9 df/d1f/d3d/d52/ddc/ff1 0 2024-03-22T02:28:59.510 INFO:tasks.workunit.client.1.smithi122.stdout:1/631: symlink dc/d3b/lcd 0 2024-03-22T02:28:59.513 INFO:tasks.workunit.client.1.smithi122.stdout:2/714: read df/d1f/d33/d34/f5f [1939675,95144] 0 2024-03-22T02:28:59.516 INFO:tasks.workunit.client.1.smithi122.stdout:9/700: dwrite db/d1b/d6c/f4f [0,4194304] 0 2024-03-22T02:28:59.519 INFO:tasks.workunit.client.1.smithi122.stdout:2/715: getdents df/d1f/d33/d34 0 2024-03-22T02:28:59.521 INFO:tasks.workunit.client.1.smithi122.stdout:2/716: chown df/d1f/d33/d34/fb2 50957 1 2024-03-22T02:28:59.522 INFO:tasks.workunit.client.1.smithi122.stdout:9/701: rename db/d23/d1f/d57/d66/f7a to db/df/d3e/d81/d3d/d86/dd8/fd9 0 2024-03-22T02:28:59.524 INFO:tasks.workunit.client.1.smithi122.stdout:2/717: rmdir df/d1f/d23/dd2/dca 39 2024-03-22T02:28:59.525 INFO:tasks.workunit.client.1.smithi122.stdout:9/702: mkdir db/df/d3e/d59/dda 0 2024-03-22T02:28:59.527 INFO:tasks.workunit.client.1.smithi122.stdout:1/632: write dc/d3b/d8d/d72/d9c/db0/d71/fb1 [853671,106701] 0 2024-03-22T02:28:59.527 INFO:tasks.workunit.client.1.smithi122.stdout:7/611: sync 2024-03-22T02:28:59.527 INFO:tasks.workunit.client.1.smithi122.stdout:5/586: sync 2024-03-22T02:28:59.528 INFO:tasks.workunit.client.1.smithi122.stdout:9/703: creat db/d23/dd6/fdb x:0 0 0 2024-03-22T02:28:59.529 INFO:tasks.workunit.client.1.smithi122.stdout:2/718: mkdir df/d1f/d23/dd2/df2 0 2024-03-22T02:28:59.530 INFO:tasks.workunit.client.1.smithi122.stdout:5/587: chown d8/d1e/d22/d3a/d39/f96 6976 1 2024-03-22T02:28:59.531 INFO:tasks.workunit.client.1.smithi122.stdout:1/633: readlink dc/d3b/d8d/d72/d9c/db0/d71/la6 0 2024-03-22T02:28:59.535 INFO:tasks.workunit.client.1.smithi122.stdout:2/719: getdents df/d1f/d33/d4d/d94/dc2/d51 0 2024-03-22T02:28:59.535 INFO:tasks.workunit.client.1.smithi122.stdout:1/634: write dc/d3b/d8d/d72/d9c/db0/d71/fb1 [967426,71300] 0 2024-03-22T02:28:59.539 INFO:tasks.workunit.client.1.smithi122.stdout:9/704: dwrite db/df/d3e/d81/d3d/fa1 [0,4194304] 0 2024-03-22T02:28:59.540 INFO:tasks.workunit.client.1.smithi122.stdout:1/635: unlink dc/f1a 0 2024-03-22T02:28:59.542 INFO:tasks.workunit.client.1.smithi122.stdout:2/720: creat df/d1f/d33/d4d/d94/dc2/d72/ff3 x:0 0 0 2024-03-22T02:28:59.542 INFO:tasks.workunit.client.1.smithi122.stdout:1/636: read dc/d3b/d8d/d72/d9c/db0/d71/fb1 [584899,86962] 0 2024-03-22T02:28:59.547 INFO:tasks.workunit.client.1.smithi122.stdout:7/612: write d0/f84 [5015098,48035] 0 2024-03-22T02:28:59.549 INFO:tasks.workunit.client.1.smithi122.stdout:9/705: rename db/d23/d1f/d57/d66/d73/f91 to db/d1b/d28/fdc 0 2024-03-22T02:28:59.550 INFO:tasks.workunit.client.1.smithi122.stdout:5/588: write d8/da/f73 [1466856,115507] 0 2024-03-22T02:28:59.551 INFO:tasks.workunit.client.1.smithi122.stdout:9/706: read db/d23/d9a/fa3 [773052,8996] 0 2024-03-22T02:28:59.553 INFO:tasks.workunit.client.1.smithi122.stdout:9/707: chown db/d1b/d28/dc1/f5f 5405 1 2024-03-22T02:28:59.554 INFO:tasks.workunit.client.1.smithi122.stdout:2/721: truncate df/d1f/d33/d34/f43 3572689 0 2024-03-22T02:28:59.555 INFO:tasks.workunit.client.1.smithi122.stdout:1/637: creat dc/d2b/d86/dbb/fce x:0 0 0 2024-03-22T02:28:59.557 INFO:tasks.workunit.client.1.smithi122.stdout:7/613: write d0/d4/f12 [4543979,74734] 0 2024-03-22T02:28:59.557 INFO:tasks.workunit.client.1.smithi122.stdout:5/589: mkdir d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/dcc 0 2024-03-22T02:28:59.559 INFO:tasks.workunit.client.1.smithi122.stdout:5/590: chown d8/l36 81 1 2024-03-22T02:28:59.559 INFO:tasks.workunit.client.1.smithi122.stdout:9/708: mknod db/d1b/d28/dc1/cdd 0 2024-03-22T02:28:59.561 INFO:tasks.workunit.client.1.smithi122.stdout:7/614: mkdir d0/d4/d80/dc4 0 2024-03-22T02:28:59.563 INFO:tasks.workunit.client.1.smithi122.stdout:1/638: fsync dc/d3b/d8d/d72/d9c/db0/d67/dc6/f59 0 2024-03-22T02:28:59.563 INFO:tasks.workunit.client.1.smithi122.stdout:7/615: read - d0/d4/d80/f92 zero size 2024-03-22T02:28:59.571 INFO:tasks.workunit.client.1.smithi122.stdout:1/639: fdatasync dc/f4e 0 2024-03-22T02:28:59.572 INFO:tasks.workunit.client.1.smithi122.stdout:5/591: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf [0,4194304] 0 2024-03-22T02:28:59.574 INFO:tasks.workunit.client.1.smithi122.stdout:9/709: creat db/d1b/d6c/dd1/fde x:0 0 0 2024-03-22T02:28:59.574 INFO:tasks.workunit.client.1.smithi122.stdout:1/640: dread - dc/d3b/d46/f9e zero size 2024-03-22T02:28:59.576 INFO:tasks.workunit.client.1.smithi122.stdout:3/586: sync 2024-03-22T02:28:59.576 INFO:tasks.workunit.client.1.smithi122.stdout:8/713: sync 2024-03-22T02:28:59.576 INFO:tasks.workunit.client.1.smithi122.stdout:4/626: sync 2024-03-22T02:28:59.576 INFO:tasks.workunit.client.1.smithi122.stdout:0/576: sync 2024-03-22T02:28:59.577 INFO:tasks.workunit.client.1.smithi122.stdout:6/774: sync 2024-03-22T02:28:59.582 INFO:tasks.workunit.client.1.smithi122.stdout:2/722: getdents df/d1f/d3d 0 2024-03-22T02:28:59.584 INFO:tasks.workunit.client.1.smithi122.stdout:9/710: unlink db/d1b/f6a 0 2024-03-22T02:28:59.589 INFO:tasks.workunit.client.1.smithi122.stdout:7/616: rename d0/d25/d5c/f8e to d0/d4/d94/fc5 0 2024-03-22T02:28:59.589 INFO:tasks.workunit.client.1.smithi122.stdout:8/714: dread - d1/d40/d71/fe6 zero size 2024-03-22T02:28:59.595 INFO:tasks.workunit.client.1.smithi122.stdout:3/587: dwrite d2/d9/dc/df/d49/d85/f9b [0,4194304] 0 2024-03-22T02:28:59.598 INFO:tasks.workunit.client.1.smithi122.stdout:2/723: rmdir df 39 2024-03-22T02:28:59.601 INFO:tasks.workunit.client.1.smithi122.stdout:8/715: mkdir d1/d8/d8a/dfb 0 2024-03-22T02:28:59.602 INFO:tasks.workunit.client.1.smithi122.stdout:4/627: dread d0/d12/d61/fa3 [0,4194304] 0 2024-03-22T02:28:59.604 INFO:tasks.workunit.client.1.smithi122.stdout:0/577: mkdir d11/d14/d23/d24/d33/d67/dc2 0 2024-03-22T02:28:59.613 INFO:tasks.workunit.client.1.smithi122.stdout:4/628: symlink d0/d12/d61/lcb 0 2024-03-22T02:28:59.617 INFO:tasks.workunit.client.1.smithi122.stdout:3/588: mkdir d2/d9/dc/db5 0 2024-03-22T02:28:59.621 INFO:tasks.workunit.client.1.smithi122.stdout:8/716: rename d1/d4d/daa/lf7 to d1/d4d/lfc 0 2024-03-22T02:28:59.622 INFO:tasks.workunit.client.1.smithi122.stdout:2/724: dwrite df/d1f/d33/d6a/f8d [0,4194304] 0 2024-03-22T02:28:59.623 INFO:tasks.workunit.client.1.smithi122.stdout:2/725: fsync df/d1f/d23/f86 0 2024-03-22T02:28:59.625 INFO:tasks.workunit.client.1.smithi122.stdout:9/711: dwrite db/d23/d9a/fa3 [0,4194304] 0 2024-03-22T02:28:59.628 INFO:tasks.workunit.client.1.smithi122.stdout:2/726: read - df/d1f/d23/f86 zero size 2024-03-22T02:28:59.628 INFO:tasks.workunit.client.1.smithi122.stdout:0/578: mknod d11/d14/d23/d24/d33/d67/dc2/cc3 0 2024-03-22T02:28:59.632 INFO:tasks.workunit.client.1.smithi122.stdout:0/579: stat d11/d14/d1b/f2c 0 2024-03-22T02:28:59.632 INFO:tasks.workunit.client.1.smithi122.stdout:4/629: symlink d0/d12/d15/d42/d4e/de/da1/lcc 0 2024-03-22T02:28:59.634 INFO:tasks.workunit.client.1.smithi122.stdout:6/775: dread d7/de/dd/df/f6e [0,4194304] 0 2024-03-22T02:28:59.635 INFO:tasks.workunit.client.1.smithi122.stdout:9/712: stat db/d23/l4b 0 2024-03-22T02:28:59.641 INFO:tasks.workunit.client.1.smithi122.stdout:8/717: symlink d1/d40/d71/d97/db8/dde/lfd 0 2024-03-22T02:28:59.643 INFO:tasks.workunit.client.1.smithi122.stdout:1/641: dread dc/d3b/d46/f82 [0,4194304] 0 2024-03-22T02:28:59.643 INFO:tasks.workunit.client.1.smithi122.stdout:6/776: mkdir d7/de/dd/d20/de0/df9 0 2024-03-22T02:28:59.646 INFO:tasks.workunit.client.1.smithi122.stdout:9/713: mknod db/df/d3e/d59/cdf 0 2024-03-22T02:28:59.646 INFO:tasks.workunit.client.1.smithi122.stdout:3/589: rename d2/d9/dc/df/l44 to d2/d9/dc/d17/d28/d4c/da5/lb6 0 2024-03-22T02:28:59.650 INFO:tasks.workunit.client.1.smithi122.stdout:5/592: dread d8/d1e/d51/f75 [0,4194304] 0 2024-03-22T02:28:59.650 INFO:tasks.workunit.client.1.smithi122.stdout:3/590: fdatasync d2/d9/dc/fae 0 2024-03-22T02:28:59.651 INFO:tasks.workunit.client.1.smithi122.stdout:7/617: dread d0/d4/d20/f27 [0,4194304] 0 2024-03-22T02:28:59.652 INFO:tasks.workunit.client.1.smithi122.stdout:6/777: symlink d7/de/dd/d20/de0/d54/d76/d86/lfa 0 2024-03-22T02:28:59.653 INFO:tasks.workunit.client.1.smithi122.stdout:7/618: chown d0/d25/d5c 726 1 2024-03-22T02:28:59.654 INFO:tasks.workunit.client.1.smithi122.stdout:5/593: chown d8/l36 1 1 2024-03-22T02:28:59.654 INFO:tasks.workunit.client.1.smithi122.stdout:8/718: dread d1/d8/d2f/fed [0,4194304] 0 2024-03-22T02:28:59.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/580: link d11/d14/d23/l7d d11/d16/d44/d89/lc4 0 2024-03-22T02:28:59.655 INFO:tasks.workunit.client.1.smithi122.stdout:1/642: truncate dc/d3b/d8d/d72/d9c/db0/dab/fbd 437644 0 2024-03-22T02:28:59.656 INFO:tasks.workunit.client.1.smithi122.stdout:0/581: chown f9 311021 1 2024-03-22T02:28:59.658 INFO:tasks.workunit.client.1.smithi122.stdout:3/591: symlink d2/d9/dc/d17/d28/d4c/da5/lb7 0 2024-03-22T02:28:59.663 INFO:tasks.workunit.client.1.smithi122.stdout:6/778: unlink d7/de/dd/c84 0 2024-03-22T02:28:59.667 INFO:tasks.workunit.client.1.smithi122.stdout:8/719: rmdir d1/d40/d71 39 2024-03-22T02:28:59.668 INFO:tasks.workunit.client.1.smithi122.stdout:9/714: dread f8 [0,4194304] 0 2024-03-22T02:28:59.671 INFO:tasks.workunit.client.1.smithi122.stdout:2/727: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/fbd [0,4194304] 0 2024-03-22T02:28:59.674 INFO:tasks.workunit.client.1.smithi122.stdout:2/728: stat df/d1f/d33/d4d/d94/dc2/lae 0 2024-03-22T02:28:59.677 INFO:tasks.workunit.client.1.smithi122.stdout:3/592: truncate d2/f7 3505798 0 2024-03-22T02:28:59.678 INFO:tasks.workunit.client.1.smithi122.stdout:7/619: rename d0/fc1 to d0/d25/fc6 0 2024-03-22T02:28:59.679 INFO:tasks.workunit.client.1.smithi122.stdout:9/715: symlink db/d1b/d28/le0 0 2024-03-22T02:28:59.683 INFO:tasks.workunit.client.1.smithi122.stdout:5/594: dwrite d8/d1e/d51/f75 [0,4194304] 0 2024-03-22T02:28:59.684 INFO:tasks.workunit.client.1.smithi122.stdout:1/643: truncate dc/d2b/f36 2803735 0 2024-03-22T02:28:59.688 INFO:tasks.workunit.client.1.smithi122.stdout:9/716: dread - db/d1b/da8/dba/fbd zero size 2024-03-22T02:28:59.689 INFO:tasks.workunit.client.1.smithi122.stdout:5/595: stat d8/d1e/d22/d3a/d46/d4f/d55/c7c 0 2024-03-22T02:28:59.691 INFO:tasks.workunit.client.1.smithi122.stdout:0/582: dread fb [0,4194304] 0 2024-03-22T02:28:59.693 INFO:tasks.workunit.client.1.smithi122.stdout:7/620: getdents d0/d4/d1d/db8/d9a 0 2024-03-22T02:28:59.695 INFO:tasks.workunit.client.1.smithi122.stdout:3/593: rename d2/d9/dc/d17/d2b/d3f/c75 to d2/d9/dc/df/d38/d87/cb8 0 2024-03-22T02:28:59.698 INFO:tasks.workunit.client.1.smithi122.stdout:2/729: dwrite df/d1f/d33/d4d/d94/dc2/f2f [0,4194304] 0 2024-03-22T02:28:59.700 INFO:tasks.workunit.client.1.smithi122.stdout:3/594: write d2/d9/dc/d17/d2b/d6a/f7d [1393505,32554] 0 2024-03-22T02:28:59.702 INFO:tasks.workunit.client.1.smithi122.stdout:5/596: read d8/d3b/f69 [4322124,44378] 0 2024-03-22T02:28:59.703 INFO:tasks.workunit.client.1.smithi122.stdout:0/583: dread fb [0,4194304] 0 2024-03-22T02:28:59.704 INFO:tasks.workunit.client.1.smithi122.stdout:2/730: creat df/d1f/d33/d34/ff4 x:0 0 0 2024-03-22T02:28:59.705 INFO:tasks.workunit.client.1.smithi122.stdout:0/584: read - d11/d14/da7/f75 zero size 2024-03-22T02:28:59.708 INFO:tasks.workunit.client.1.smithi122.stdout:1/644: rename dc/l6a to dc/d3b/d8d/lcf 0 2024-03-22T02:28:59.710 INFO:tasks.workunit.client.1.smithi122.stdout:2/731: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ff5 x:0 0 0 2024-03-22T02:28:59.712 INFO:tasks.workunit.client.1.smithi122.stdout:5/597: rename d8/d3b/f5a to d8/d1e/d22/d3a/d46/d4f/d9b/da9/fcd 0 2024-03-22T02:28:59.712 INFO:tasks.workunit.client.1.smithi122.stdout:2/732: rmdir df 39 2024-03-22T02:28:59.717 INFO:tasks.workunit.client.1.smithi122.stdout:2/733: fsync df/d1f/d33/d4d/d94/fc8 0 2024-03-22T02:28:59.718 INFO:tasks.workunit.client.1.smithi122.stdout:0/585: rename d11/d14/da7/f75 to d11/d14/d1d/d27/dc0/fc5 0 2024-03-22T02:28:59.719 INFO:tasks.workunit.client.1.smithi122.stdout:2/734: write df/d1f/d33/d6a/f8d [1336382,82508] 0 2024-03-22T02:28:59.724 INFO:tasks.workunit.client.1.smithi122.stdout:0/586: mkdir d11/d14/d1d/d27/dc0/dc6 0 2024-03-22T02:28:59.728 INFO:tasks.workunit.client.1.smithi122.stdout:9/717: truncate db/df/d3e/d81/d3d/fa1 863615 0 2024-03-22T02:28:59.733 INFO:tasks.workunit.client.1.smithi122.stdout:0/587: symlink d11/d14/d1d/d27/dc0/lc7 0 2024-03-22T02:28:59.735 INFO:tasks.workunit.client.1.smithi122.stdout:0/588: chown d11/d16/d44/f76 3199 1 2024-03-22T02:28:59.737 INFO:tasks.workunit.client.1.smithi122.stdout:7/621: dwrite d0/d30/f5b [0,4194304] 0 2024-03-22T02:28:59.737 INFO:tasks.workunit.client.1.smithi122.stdout:9/718: truncate db/df/d3e/d81/f42 4765428 0 2024-03-22T02:28:59.737 INFO:tasks.workunit.client.1.smithi122.stdout:2/735: dwrite df/d1f/d33/d4d/d94/dc2/d51/d67/f6e [4194304,4194304] 0 2024-03-22T02:28:59.741 INFO:tasks.workunit.client.1.smithi122.stdout:7/622: fdatasync d0/fae 0 2024-03-22T02:28:59.747 INFO:tasks.workunit.client.1.smithi122.stdout:1/645: dread dc/d2b/f57 [0,4194304] 0 2024-03-22T02:28:59.748 INFO:tasks.workunit.client.1.smithi122.stdout:2/736: chown df/d1f/d33/d4d/d94/dc2/d51/ce1 22433 1 2024-03-22T02:28:59.749 INFO:tasks.workunit.client.1.smithi122.stdout:1/646: readlink dc/d3b/lad 0 2024-03-22T02:28:59.751 INFO:tasks.workunit.client.1.smithi122.stdout:7/623: dread d0/f16 [0,4194304] 0 2024-03-22T02:28:59.753 INFO:tasks.workunit.client.1.smithi122.stdout:1/647: fsync dc/d3b/d8d/d72/d9c/db0/dab/faf 0 2024-03-22T02:28:59.753 INFO:tasks.workunit.client.1.smithi122.stdout:9/719: dwrite db/d23/d1f/f2b [0,4194304] 0 2024-03-22T02:28:59.754 INFO:tasks.workunit.client.1.smithi122.stdout:2/737: read df/d1f/d33/f80 [233401,17638] 0 2024-03-22T02:28:59.755 INFO:tasks.workunit.client.1.smithi122.stdout:7/624: chown d0/d4/d22/f55 231 1 2024-03-22T02:28:59.759 INFO:tasks.workunit.client.1.smithi122.stdout:9/720: rmdir db/df/d3e 39 2024-03-22T02:28:59.761 INFO:tasks.workunit.client.1.smithi122.stdout:9/721: chown f8 1651666 1 2024-03-22T02:28:59.761 INFO:tasks.workunit.client.1.smithi122.stdout:1/648: read dc/d2b/f33 [1342809,6304] 0 2024-03-22T02:28:59.763 INFO:tasks.workunit.client.1.smithi122.stdout:7/625: creat d0/d4/d20/d21/d9b/dbb/fc7 x:0 0 0 2024-03-22T02:28:59.765 INFO:tasks.workunit.client.1.smithi122.stdout:2/738: getdents df/d1f/d33/d34/ddd 0 2024-03-22T02:28:59.775 INFO:tasks.workunit.client.1.smithi122.stdout:1/649: dread dc/d4c/f8b [0,4194304] 0 2024-03-22T02:28:59.775 INFO:tasks.workunit.client.1.smithi122.stdout:9/722: getdents db/d23/d1f 0 2024-03-22T02:28:59.776 INFO:tasks.workunit.client.1.smithi122.stdout:2/739: read df/d1f/f3e [2280070,108886] 0 2024-03-22T02:28:59.778 INFO:tasks.workunit.client.1.smithi122.stdout:9/723: dread - db/d23/d1f/d57/fd0 zero size 2024-03-22T02:28:59.779 INFO:tasks.workunit.client.1.smithi122.stdout:7/626: dwrite d0/d4/d94/fc5 [4194304,4194304] 0 2024-03-22T02:28:59.782 INFO:tasks.workunit.client.1.smithi122.stdout:9/724: write db/d1b/da8/dba/fbd [165875,13390] 0 2024-03-22T02:28:59.783 INFO:tasks.workunit.client.1.smithi122.stdout:7/627: truncate d0/f16 85673 0 2024-03-22T02:28:59.785 INFO:tasks.workunit.client.1.smithi122.stdout:9/725: chown db/d1b/d28/dc1/f38 429400182 1 2024-03-22T02:28:59.787 INFO:tasks.workunit.client.1.smithi122.stdout:2/740: getdents df 0 2024-03-22T02:28:59.792 INFO:tasks.workunit.client.1.smithi122.stdout:1/650: dread dc/d55/fc0 [0,4194304] 0 2024-03-22T02:28:59.794 INFO:tasks.workunit.client.1.smithi122.stdout:1/651: mknod dc/d4c/cd0 0 2024-03-22T02:28:59.795 INFO:tasks.workunit.client.1.smithi122.stdout:7/628: dwrite d0/d4e/f9e [0,4194304] 0 2024-03-22T02:28:59.795 INFO:tasks.workunit.client.1.smithi122.stdout:9/726: mknod db/df/d3e/ce1 0 2024-03-22T02:28:59.797 INFO:tasks.workunit.client.1.smithi122.stdout:2/741: mkdir df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/df6 0 2024-03-22T02:28:59.798 INFO:tasks.workunit.client.1.smithi122.stdout:7/629: creat d0/d4/d20/d21/d28/fc8 x:0 0 0 2024-03-22T02:28:59.799 INFO:tasks.workunit.client.1.smithi122.stdout:1/652: dread - dc/d2b/d86/dbb/fce zero size 2024-03-22T02:28:59.800 INFO:tasks.workunit.client.1.smithi122.stdout:7/630: fdatasync d0/d4/d20/d21/f95 0 2024-03-22T02:28:59.805 INFO:tasks.workunit.client.1.smithi122.stdout:9/727: creat db/fe2 x:0 0 0 2024-03-22T02:28:59.806 INFO:tasks.workunit.client.1.smithi122.stdout:1/653: dread - dc/d3b/d8d/d72/d9c/db0/dab/faf zero size 2024-03-22T02:28:59.807 INFO:tasks.workunit.client.1.smithi122.stdout:7/631: fdatasync d0/d4/d94/fc5 0 2024-03-22T02:28:59.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/728: rename db/df/d3e/d81/fb6 to db/d23/fe3 0 2024-03-22T02:28:59.813 INFO:tasks.workunit.client.1.smithi122.stdout:2/742: link df/d1f/d33/c7f df/d1f/d33/cf7 0 2024-03-22T02:28:59.815 INFO:tasks.workunit.client.1.smithi122.stdout:9/729: chown db/df/d3e/d81/d25/fae 76101484 1 2024-03-22T02:28:59.817 INFO:tasks.workunit.client.1.smithi122.stdout:2/743: stat df/d1f/d33/d4d/d7a/f8a 0 2024-03-22T02:28:59.817 INFO:tasks.workunit.client.1.smithi122.stdout:1/654: dwrite dc/d2b/d86/dbb/fce [0,4194304] 0 2024-03-22T02:28:59.819 INFO:tasks.workunit.client.1.smithi122.stdout:7/632: getdents d0/d4/d1d/db8/d98 0 2024-03-22T02:28:59.822 INFO:tasks.workunit.client.1.smithi122.stdout:2/744: mkdir df/d1f/d33/d4d/d94/dc2/d51/d67/db0/df8 0 2024-03-22T02:28:59.824 INFO:tasks.workunit.client.1.smithi122.stdout:7/633: truncate d0/d4/d20/d21/f2b 1153196 0 2024-03-22T02:28:59.825 INFO:tasks.workunit.client.1.smithi122.stdout:2/745: write fd [2500951,41595] 0 2024-03-22T02:28:59.830 INFO:tasks.workunit.client.1.smithi122.stdout:1/655: link dc/d3b/d8d/d72/cc9 dc/d2b/d86/dbb/cd1 0 2024-03-22T02:28:59.834 INFO:tasks.workunit.client.1.smithi122.stdout:2/746: dwrite df/d1f/d33/d34/fb2 [0,4194304] 0 2024-03-22T02:28:59.834 INFO:tasks.workunit.client.1.smithi122.stdout:1/656: read dc/d2b/f57 [2054056,42139] 0 2024-03-22T02:28:59.838 INFO:tasks.workunit.client.1.smithi122.stdout:2/747: symlink df/d1f/d33/d6a/d7c/lf9 0 2024-03-22T02:28:59.840 INFO:tasks.workunit.client.1.smithi122.stdout:2/748: write df/d1f/d33/d4d/d94/fe7 [675512,117169] 0 2024-03-22T02:28:59.842 INFO:tasks.workunit.client.1.smithi122.stdout:7/634: write d0/d25/d5c/f6d [811242,27834] 0 2024-03-22T02:28:59.844 INFO:tasks.workunit.client.1.smithi122.stdout:7/635: dread - d0/d4/d20/d21/d28/fc8 zero size 2024-03-22T02:28:59.844 INFO:tasks.workunit.client.1.smithi122.stdout:2/749: mknod df/d1f/d33/d4d/d94/dc2/d72/cfa 0 2024-03-22T02:28:59.845 INFO:tasks.workunit.client.1.smithi122.stdout:1/657: getdents dc/d2b/d86/dbb 0 2024-03-22T02:28:59.845 INFO:tasks.workunit.client.1.smithi122.stdout:7/636: write d0/d4e/d5d/f93 [1000917,57360] 0 2024-03-22T02:28:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:28:59 smithi122 ceph-mon[36403]: pgmap v23: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 87 MiB/s rd, 276 MiB/s wr, 495 op/s 2024-03-22T02:28:59.852 INFO:tasks.workunit.client.1.smithi122.stdout:7/637: getdents d0/d4/d20/d21/d28 0 2024-03-22T02:28:59.853 INFO:tasks.workunit.client.1.smithi122.stdout:1/658: dread dc/d3b/d46/f82 [0,4194304] 0 2024-03-22T02:28:59.854 INFO:tasks.workunit.client.1.smithi122.stdout:2/750: rename df/d1f/d33/d4d/d94/dc2/c9f to df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/cfb 0 2024-03-22T02:28:59.855 INFO:tasks.workunit.client.1.smithi122.stdout:7/638: symlink d0/d30/lc9 0 2024-03-22T02:28:59.857 INFO:tasks.workunit.client.1.smithi122.stdout:7/639: truncate d0/d4/d20/d21/f95 206222 0 2024-03-22T02:28:59.858 INFO:tasks.workunit.client.1.smithi122.stdout:7/640: fdatasync d0/d4/d20/d21/f95 0 2024-03-22T02:28:59.861 INFO:tasks.workunit.client.1.smithi122.stdout:7/641: link d0/f16 d0/d4/d80/dc4/fca 0 2024-03-22T02:28:59.863 INFO:tasks.workunit.client.1.smithi122.stdout:7/642: creat d0/fcb x:0 0 0 2024-03-22T02:28:59.865 INFO:tasks.workunit.client.1.smithi122.stdout:7/643: mknod d0/d4/d80/dc4/ccc 0 2024-03-22T02:28:59.866 INFO:tasks.workunit.client.1.smithi122.stdout:7/644: write d0/d4/d94/fc5 [7812888,126503] 0 2024-03-22T02:28:59.870 INFO:tasks.workunit.client.1.smithi122.stdout:2/751: dread df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f5e [0,4194304] 0 2024-03-22T02:28:59.871 INFO:tasks.workunit.client.1.smithi122.stdout:2/752: stat df/d1f/d3d/fa4 0 2024-03-22T02:28:59.871 INFO:tasks.workunit.client.1.smithi122.stdout:7/645: dwrite d0/d4/d94/fc5 [0,4194304] 0 2024-03-22T02:28:59.872 INFO:tasks.workunit.client.1.smithi122.stdout:2/753: dread - df/d1f/d33/d4d/d94/fc8 zero size 2024-03-22T02:28:59.872 INFO:tasks.workunit.client.1.smithi122.stdout:2/754: readlink df/d1f/d3d/lc4 0 2024-03-22T02:28:59.873 INFO:tasks.workunit.client.1.smithi122.stdout:2/755: stat df/d1f/d33/d4d/d94/dc2/d51/d67/lb4 0 2024-03-22T02:28:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:28:59 smithi103 ceph-mon[29258]: pgmap v23: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 87 MiB/s rd, 276 MiB/s wr, 495 op/s 2024-03-22T02:28:59.886 INFO:tasks.workunit.client.1.smithi122.stdout:2/756: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f47 [0,4194304] 0 2024-03-22T02:28:59.888 INFO:tasks.workunit.client.1.smithi122.stdout:2/757: read df/d1f/d33/d4d/d94/dc2/d51/d67/fbf [320430,2741] 0 2024-03-22T02:28:59.894 INFO:tasks.workunit.client.1.smithi122.stdout:2/758: write f6 [739917,120324] 0 2024-03-22T02:28:59.897 INFO:tasks.workunit.client.1.smithi122.stdout:2/759: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/ffc x:0 0 0 2024-03-22T02:28:59.899 INFO:tasks.workunit.client.1.smithi122.stdout:2/760: mkdir df/d1f/d33/dfd 0 2024-03-22T02:28:59.900 INFO:tasks.workunit.client.1.smithi122.stdout:2/761: symlink df/d1f/d33/d4d/d94/dc2/d51/d67/lfe 0 2024-03-22T02:28:59.902 INFO:tasks.workunit.client.1.smithi122.stdout:2/762: unlink df/d1f/d33/d4d/d94/dc2/l61 0 2024-03-22T02:28:59.903 INFO:tasks.workunit.client.1.smithi122.stdout:2/763: chown df/d1f/d33/d6a/f8d 1 1 2024-03-22T02:28:59.904 INFO:tasks.workunit.client.1.smithi122.stdout:2/764: symlink df/d1f/d33/d4d/d7a/lff 0 2024-03-22T02:28:59.906 INFO:tasks.workunit.client.1.smithi122.stdout:2/765: symlink df/d1f/d33/d6a/d7c/l100 0 2024-03-22T02:28:59.907 INFO:tasks.workunit.client.1.smithi122.stdout:2/766: fdatasync df/d1f/d33/f54 0 2024-03-22T02:28:59.912 INFO:tasks.workunit.client.1.smithi122.stdout:2/767: dwrite df/f13 [0,4194304] 0 2024-03-22T02:29:00.050 INFO:tasks.workunit.client.1.smithi122.stdout:4/630: sync 2024-03-22T02:29:00.052 INFO:tasks.workunit.client.1.smithi122.stdout:4/631: rename d0/d12/d15/d2d/l80 to d0/d12/d15/d2d/d75/lcd 0 2024-03-22T02:29:00.053 INFO:tasks.workunit.client.1.smithi122.stdout:4/632: mkdir d0/d12/d61/dce 0 2024-03-22T02:29:00.057 INFO:tasks.workunit.client.1.smithi122.stdout:4/633: getdents d0/d12/d15/d42/d57 0 2024-03-22T02:29:00.063 INFO:tasks.workunit.client.1.smithi122.stdout:4/634: dwrite d0/d25/fc2 [0,4194304] 0 2024-03-22T02:29:00.066 INFO:tasks.workunit.client.1.smithi122.stdout:4/635: symlink d0/d12/d15/d3d/d90/lcf 0 2024-03-22T02:29:00.068 INFO:tasks.workunit.client.1.smithi122.stdout:4/636: creat d0/d12/d61/fd0 x:0 0 0 2024-03-22T02:29:00.071 INFO:tasks.workunit.client.1.smithi122.stdout:4/637: getdents d0/d12/d15/d2d 0 2024-03-22T02:29:00.075 INFO:tasks.workunit.client.1.smithi122.stdout:4/638: getdents d0/d12/d15/d42 0 2024-03-22T02:29:00.080 INFO:tasks.workunit.client.1.smithi122.stdout:4/639: dwrite d0/d12/d15/d42/d4e/f23 [0,4194304] 0 2024-03-22T02:29:00.083 INFO:tasks.workunit.client.1.smithi122.stdout:4/640: fdatasync d0/d25/f4d 0 2024-03-22T02:29:00.085 INFO:tasks.workunit.client.1.smithi122.stdout:4/641: readlink d0/d12/d15/d42/d4e/lb7 0 2024-03-22T02:29:00.088 INFO:tasks.workunit.client.1.smithi122.stdout:4/642: mknod d0/d12/d15/d3d/d90/cd1 0 2024-03-22T02:29:00.089 INFO:tasks.workunit.client.1.smithi122.stdout:4/643: unlink d0/d12/d73/cb8 0 2024-03-22T02:29:00.091 INFO:tasks.workunit.client.1.smithi122.stdout:4/644: mkdir d0/d12/d15/d3d/dd2 0 2024-03-22T02:29:00.092 INFO:tasks.workunit.client.1.smithi122.stdout:4/645: symlink d0/d12/d61/dce/ld3 0 2024-03-22T02:29:00.093 INFO:tasks.workunit.client.1.smithi122.stdout:4/646: write d0/d12/d15/d42/d4e/de/f5d [5048881,56054] 0 2024-03-22T02:29:00.094 INFO:tasks.workunit.client.1.smithi122.stdout:4/647: chown d0/d12/d15/d42/d4e/de/c20 11 1 2024-03-22T02:29:00.097 INFO:tasks.workunit.client.1.smithi122.stdout:4/648: creat d0/d12/d61/dce/fd4 x:0 0 0 2024-03-22T02:29:00.097 INFO:tasks.workunit.client.1.smithi122.stdout:4/649: stat d0/d25/f8 0 2024-03-22T02:29:00.099 INFO:tasks.workunit.client.1.smithi122.stdout:4/650: rmdir d0/d12/d15/d42/d4e/de/da1 39 2024-03-22T02:29:00.105 INFO:tasks.workunit.client.1.smithi122.stdout:4/651: dwrite d0/d12/d73/f83 [0,4194304] 0 2024-03-22T02:29:00.108 INFO:tasks.workunit.client.1.smithi122.stdout:4/652: creat d0/d12/d73/fd5 x:0 0 0 2024-03-22T02:29:00.109 INFO:tasks.workunit.client.1.smithi122.stdout:4/653: chown d0/d12/d15/d42/d4e 0 1 2024-03-22T02:29:00.109 INFO:tasks.workunit.client.1.smithi122.stdout:4/654: readlink d0/d7a/lbe 0 2024-03-22T02:29:00.110 INFO:tasks.workunit.client.1.smithi122.stdout:4/655: truncate d0/f6a 1028359 0 2024-03-22T02:29:00.110 INFO:tasks.workunit.client.1.smithi122.stdout:4/656: chown d0/d25/f28 2213 1 2024-03-22T02:29:00.119 INFO:tasks.workunit.client.1.smithi122.stdout:4/657: dwrite d0/d25/f28 [0,4194304] 0 2024-03-22T02:29:00.120 INFO:tasks.workunit.client.1.smithi122.stdout:4/658: dread - d0/d12/d15/d42/d57/f96 zero size 2024-03-22T02:29:00.122 INFO:tasks.workunit.client.1.smithi122.stdout:4/659: unlink d0/d12/f82 0 2024-03-22T02:29:00.126 INFO:tasks.workunit.client.1.smithi122.stdout:4/660: dread d0/d12/d73/f83 [0,4194304] 0 2024-03-22T02:29:00.127 INFO:tasks.workunit.client.1.smithi122.stdout:4/661: chown d0/d12/d61/f97 209529 1 2024-03-22T02:29:00.133 INFO:tasks.workunit.client.1.smithi122.stdout:4/662: write d0/d12/d15/d42/d4e/de/f8b [2561554,28964] 0 2024-03-22T02:29:00.137 INFO:tasks.workunit.client.1.smithi122.stdout:4/663: write d0/d12/d15/d42/d4e/f5b [1043831,74987] 0 2024-03-22T02:29:00.139 INFO:tasks.workunit.client.1.smithi122.stdout:4/664: truncate d0/d12/f31 1433608 0 2024-03-22T02:29:00.140 INFO:tasks.workunit.client.1.smithi122.stdout:4/665: chown d0/d12/d15/d3d/d90/l5f 7 1 2024-03-22T02:29:00.141 INFO:tasks.workunit.client.1.smithi122.stdout:4/666: read - d0/d12/d15/d42/d4e/de/d91/fa9 zero size 2024-03-22T02:29:00.143 INFO:tasks.workunit.client.1.smithi122.stdout:4/667: truncate d0/d12/d15/d3d/d90/f48 3187414 0 2024-03-22T02:29:00.145 INFO:tasks.workunit.client.1.smithi122.stdout:4/668: mkdir d0/d12/d15/d42/d4e/dd6 0 2024-03-22T02:29:00.155 INFO:tasks.workunit.client.1.smithi122.stdout:4/669: dwrite d0/d12/d15/d42/d4e/de/f53 [0,4194304] 0 2024-03-22T02:29:00.159 INFO:tasks.workunit.client.1.smithi122.stdout:4/670: write d0/d12/d15/d42/d4e/f5a [263835,2800] 0 2024-03-22T02:29:00.161 INFO:tasks.workunit.client.1.smithi122.stdout:4/671: write d0/d12/d15/d42/d4e/de/d40/f60 [1574435,68762] 0 2024-03-22T02:29:00.162 INFO:tasks.workunit.client.1.smithi122.stdout:4/672: chown d0/d12/d15/d42/d4e/de/f53 28856400 1 2024-03-22T02:29:00.165 INFO:tasks.workunit.client.1.smithi122.stdout:4/673: creat d0/d12/d15/d42/fd7 x:0 0 0 2024-03-22T02:29:00.165 INFO:tasks.workunit.client.1.smithi122.stdout:4/674: chown d0/d25/f84 21 1 2024-03-22T02:29:00.166 INFO:tasks.workunit.client.1.smithi122.stdout:4/675: write d0/d12/fc4 [267407,47083] 0 2024-03-22T02:29:00.169 INFO:tasks.workunit.client.1.smithi122.stdout:4/676: rename d0/d12/d15/d2d/l2e to d0/d12/ld8 0 2024-03-22T02:29:00.171 INFO:tasks.workunit.client.1.smithi122.stdout:4/677: rename d0/d25/fc2 to d0/d7a/fd9 0 2024-03-22T02:29:00.173 INFO:tasks.workunit.client.1.smithi122.stdout:4/678: readlink d0/d12/d15/d3d/d90/laf 0 2024-03-22T02:29:00.179 INFO:tasks.workunit.client.1.smithi122.stdout:4/679: creat d0/d12/d15/d42/d4e/de/da1/fda x:0 0 0 2024-03-22T02:29:00.180 INFO:tasks.workunit.client.1.smithi122.stdout:4/680: read d0/d12/d15/d42/d4e/f23 [3070961,111049] 0 2024-03-22T02:29:00.187 INFO:tasks.workunit.client.1.smithi122.stdout:4/681: truncate d0/d12/d15/d42/d4e/de/f5d 2814383 0 2024-03-22T02:29:00.187 INFO:tasks.workunit.client.1.smithi122.stdout:4/682: fsync d0/d12/d61/fa3 0 2024-03-22T02:29:00.188 INFO:tasks.workunit.client.1.smithi122.stdout:4/683: fdatasync d0/d12/d15/d42/d4e/de/d91/fa9 0 2024-03-22T02:29:00.189 INFO:tasks.workunit.client.1.smithi122.stdout:4/684: fsync d0/d12/d15/d42/d57/f8e 0 2024-03-22T02:29:00.189 INFO:tasks.workunit.client.1.smithi122.stdout:4/685: stat d0/d7a 0 2024-03-22T02:29:00.191 INFO:tasks.workunit.client.1.smithi122.stdout:4/686: write d0/d12/d15/d42/fc0 [127922,75250] 0 2024-03-22T02:29:00.193 INFO:tasks.workunit.client.1.smithi122.stdout:4/687: fdatasync d0/d12/d61/f74 0 2024-03-22T02:29:00.195 INFO:tasks.workunit.client.1.smithi122.stdout:4/688: write d0/f9 [6850204,25168] 0 2024-03-22T02:29:00.197 INFO:tasks.workunit.client.1.smithi122.stdout:4/689: creat d0/d12/d15/d42/d57/fdb x:0 0 0 2024-03-22T02:29:00.200 INFO:tasks.workunit.client.1.smithi122.stdout:4/690: getdents d0/d12/d61 0 2024-03-22T02:29:00.211 INFO:tasks.workunit.client.1.smithi122.stdout:4/691: dread d0/d12/d15/d42/d4e/de/f5d [0,4194304] 0 2024-03-22T02:29:00.212 INFO:tasks.workunit.client.1.smithi122.stdout:4/692: readlink d0/d12/da2/lc5 0 2024-03-22T02:29:00.213 INFO:tasks.workunit.client.1.smithi122.stdout:4/693: write d0/d12/d15/d42/d4e/de/f53 [5137645,21482] 0 2024-03-22T02:29:00.213 INFO:tasks.workunit.client.1.smithi122.stdout:8/720: sync 2024-03-22T02:29:00.213 INFO:tasks.workunit.client.1.smithi122.stdout:6/779: sync 2024-03-22T02:29:00.214 INFO:tasks.workunit.client.1.smithi122.stdout:4/694: chown d0/d12/d15/d42/d4e/de/f53 3403443 1 2024-03-22T02:29:00.216 INFO:tasks.workunit.client.1.smithi122.stdout:6/780: mknod d7/de/dd/d23/cfb 0 2024-03-22T02:29:00.218 INFO:tasks.workunit.client.1.smithi122.stdout:6/781: symlink d7/de/dd/d23/lfc 0 2024-03-22T02:29:00.219 INFO:tasks.workunit.client.1.smithi122.stdout:4/695: symlink d0/d12/d15/d42/d4e/ldc 0 2024-03-22T02:29:00.219 INFO:tasks.workunit.client.1.smithi122.stdout:8/721: getdents d1/d60/d73 0 2024-03-22T02:29:00.220 INFO:tasks.workunit.client.1.smithi122.stdout:6/782: write d7/de/dd/d20/f21 [2950354,114303] 0 2024-03-22T02:29:00.223 INFO:tasks.workunit.client.1.smithi122.stdout:8/722: dread d1/d40/d71/db4/fbb [0,4194304] 0 2024-03-22T02:29:00.223 INFO:tasks.workunit.client.1.smithi122.stdout:6/783: mknod d7/de/dd/d20/de0/dae/cfd 0 2024-03-22T02:29:00.224 INFO:tasks.workunit.client.1.smithi122.stdout:8/723: write d1/d60/d73/d90/fec [538781,15039] 0 2024-03-22T02:29:00.225 INFO:tasks.workunit.client.1.smithi122.stdout:6/784: creat d7/de/dd/d23/ffe x:0 0 0 2024-03-22T02:29:00.225 INFO:tasks.workunit.client.1.smithi122.stdout:8/724: rmdir d1/d8/dba 39 2024-03-22T02:29:00.226 INFO:tasks.workunit.client.1.smithi122.stdout:8/725: write d1/d3f/d45/fe3 [1014468,127934] 0 2024-03-22T02:29:00.226 INFO:tasks.workunit.client.1.smithi122.stdout:8/726: stat d1/fd3 0 2024-03-22T02:29:00.236 INFO:tasks.workunit.client.1.smithi122.stdout:8/727: dwrite d1/d4d/f58 [0,4194304] 0 2024-03-22T02:29:00.236 INFO:tasks.workunit.client.1.smithi122.stdout:8/728: read - d1/d60/d73/da3/fe9 zero size 2024-03-22T02:29:00.237 INFO:tasks.workunit.client.1.smithi122.stdout:8/729: chown d1/d8/d2f/fed 1886863005 1 2024-03-22T02:29:00.239 INFO:tasks.workunit.client.1.smithi122.stdout:8/730: write d1/d4d/f56 [710447,85262] 0 2024-03-22T02:29:00.240 INFO:tasks.workunit.client.1.smithi122.stdout:8/731: write d1/d40/d48/faf [2459812,118307] 0 2024-03-22T02:29:00.254 INFO:tasks.workunit.client.1.smithi122.stdout:8/732: dwrite d1/d40/d71/dc9/f65 [0,4194304] 0 2024-03-22T02:29:00.254 INFO:tasks.workunit.client.1.smithi122.stdout:8/733: readlink d1/d8/d63/df0/l35 0 2024-03-22T02:29:00.258 INFO:tasks.workunit.client.1.smithi122.stdout:8/734: dread d1/d4d/f4e [0,4194304] 0 2024-03-22T02:29:00.259 INFO:tasks.workunit.client.1.smithi122.stdout:8/735: chown d1/d60/d73/d90/lbe 23304819 1 2024-03-22T02:29:00.264 INFO:tasks.workunit.client.1.smithi122.stdout:8/736: dwrite d1/fbd [0,4194304] 0 2024-03-22T02:29:00.268 INFO:tasks.workunit.client.1.smithi122.stdout:8/737: rename d1/d3f to d1/d40/d51/df5/dae/dfe 0 2024-03-22T02:29:00.270 INFO:tasks.workunit.client.1.smithi122.stdout:8/738: truncate d1/d9/f1f 3834181 0 2024-03-22T02:29:00.272 INFO:tasks.workunit.client.1.smithi122.stdout:8/739: symlink d1/d81/lff 0 2024-03-22T02:29:00.273 INFO:tasks.workunit.client.1.smithi122.stdout:8/740: truncate d1/d9/d50/da6/fd4 609679 0 2024-03-22T02:29:00.274 INFO:tasks.workunit.client.1.smithi122.stdout:8/741: chown d1/d8/d63/df0/d6a/c7d 0 1 2024-03-22T02:29:00.275 INFO:tasks.workunit.client.1.smithi122.stdout:8/742: fsync d1/d4d/f58 0 2024-03-22T02:29:00.278 INFO:tasks.workunit.client.1.smithi122.stdout:8/743: creat d1/d8/f100 x:0 0 0 2024-03-22T02:29:00.284 INFO:tasks.workunit.client.1.smithi122.stdout:8/744: rmdir d1/d8 39 2024-03-22T02:29:00.289 INFO:tasks.workunit.client.1.smithi122.stdout:8/745: dwrite d1/d9/f13 [0,4194304] 0 2024-03-22T02:29:00.297 INFO:tasks.workunit.client.1.smithi122.stdout:8/746: dwrite d1/d4d/fda [0,4194304] 0 2024-03-22T02:29:00.300 INFO:tasks.workunit.client.1.smithi122.stdout:8/747: rmdir d1/d40/d71/dc9 39 2024-03-22T02:29:00.304 INFO:tasks.workunit.client.1.smithi122.stdout:8/748: getdents d1/d8/d8a/dfb 0 2024-03-22T02:29:00.307 INFO:tasks.workunit.client.1.smithi122.stdout:8/749: dread d1/d4d/f58 [0,4194304] 0 2024-03-22T02:29:00.324 INFO:tasks.workunit.client.1.smithi122.stdout:8/750: dread d1/d8/f36 [0,4194304] 0 2024-03-22T02:29:00.385 INFO:tasks.workunit.client.1.smithi122.stdout:3/595: sync 2024-03-22T02:29:00.385 INFO:tasks.workunit.client.1.smithi122.stdout:3/596: chown d2/f39 31776 1 2024-03-22T02:29:00.392 INFO:tasks.workunit.client.1.smithi122.stdout:3/597: dwrite d2/d9/dc/d17/d2b/f67 [0,4194304] 0 2024-03-22T02:29:00.397 INFO:tasks.workunit.client.1.smithi122.stdout:3/598: link d2/d9/dc/d17/d28/f6c d2/d9/dc/d17/d28/d4c/da5/db4/d79/fb9 0 2024-03-22T02:29:00.399 INFO:tasks.workunit.client.1.smithi122.stdout:3/599: truncate d2/d9/dc/df/d49/d85/f94 1010651 0 2024-03-22T02:29:00.400 INFO:tasks.workunit.client.1.smithi122.stdout:3/600: write d2/d9/dc/d17/d45/f46 [76239,99438] 0 2024-03-22T02:29:00.402 INFO:tasks.workunit.client.1.smithi122.stdout:3/601: mkdir d2/d9/dc/df/dba 0 2024-03-22T02:29:00.404 INFO:tasks.workunit.client.1.smithi122.stdout:3/602: write d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 [838333,94453] 0 2024-03-22T02:29:00.407 INFO:tasks.workunit.client.1.smithi122.stdout:3/603: creat d2/d8a/da2/fbb x:0 0 0 2024-03-22T02:29:00.409 INFO:tasks.workunit.client.1.smithi122.stdout:3/604: creat d2/d9/dc/d17/d28/d4c/da5/db4/d79/fbc x:0 0 0 2024-03-22T02:29:00.415 INFO:tasks.workunit.client.1.smithi122.stdout:3/605: dwrite d2/d9/f8d [0,4194304] 0 2024-03-22T02:29:00.417 INFO:tasks.workunit.client.1.smithi122.stdout:3/606: fsync d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 0 2024-03-22T02:29:00.432 INFO:tasks.workunit.client.1.smithi122.stdout:3/607: write d2/d9/dc/d17/d28/f2d [2209945,77559] 0 2024-03-22T02:29:00.434 INFO:tasks.workunit.client.1.smithi122.stdout:3/608: mkdir d2/d9/d7e/dbd 0 2024-03-22T02:29:00.435 INFO:tasks.workunit.client.1.smithi122.stdout:3/609: write d2/d9/dc/d17/d51/f7c [531593,82581] 0 2024-03-22T02:29:00.447 INFO:tasks.workunit.client.1.smithi122.stdout:3/610: dread d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:29:00.450 INFO:tasks.workunit.client.1.smithi122.stdout:3/611: creat d2/d9/dc/d17/d2b/d3f/fbe x:0 0 0 2024-03-22T02:29:00.452 INFO:tasks.workunit.client.1.smithi122.stdout:3/612: creat d2/d9/dc/d57/d89/fbf x:0 0 0 2024-03-22T02:29:00.453 INFO:tasks.workunit.client.1.smithi122.stdout:3/613: mknod d2/cc0 0 2024-03-22T02:29:00.456 INFO:tasks.workunit.client.1.smithi122.stdout:3/614: rmdir d2/d9/dc/df/d3c/d78 39 2024-03-22T02:29:00.469 INFO:tasks.workunit.client.1.smithi122.stdout:3/615: dread d2/f62 [0,4194304] 0 2024-03-22T02:29:00.471 INFO:tasks.workunit.client.1.smithi122.stdout:3/616: creat d2/d9/dc/d57/fc1 x:0 0 0 2024-03-22T02:29:00.473 INFO:tasks.workunit.client.1.smithi122.stdout:3/617: symlink d2/d9/dc/d17/d28/d4c/da5/db4/lc2 0 2024-03-22T02:29:00.475 INFO:tasks.workunit.client.1.smithi122.stdout:3/618: rename d2/d8a/da2/la6 to d2/d9/dc/df/dba/lc3 0 2024-03-22T02:29:00.478 INFO:tasks.workunit.client.1.smithi122.stdout:3/619: rename d2/d9/dc/f16 to d2/d9/fc4 0 2024-03-22T02:29:00.481 INFO:tasks.workunit.client.1.smithi122.stdout:3/620: dread d2/d9/dc/d17/d28/d4c/da5/db4/f56 [0,4194304] 0 2024-03-22T02:29:00.494 INFO:tasks.workunit.client.1.smithi122.stdout:3/621: dread d2/d9/dc/d57/f93 [0,4194304] 0 2024-03-22T02:29:00.500 INFO:tasks.workunit.client.1.smithi122.stdout:3/622: dwrite d2/d9/dc/df/f48 [0,4194304] 0 2024-03-22T02:29:00.506 INFO:tasks.workunit.client.1.smithi122.stdout:3/623: dwrite d2/d9/dc/d17/d2b/d6b/f8c [0,4194304] 0 2024-03-22T02:29:00.509 INFO:tasks.workunit.client.1.smithi122.stdout:3/624: creat d2/d9/dc/d57/d89/fc5 x:0 0 0 2024-03-22T02:29:00.514 INFO:tasks.workunit.client.1.smithi122.stdout:3/625: rename d2/d9/dc/d17/d2b/d6b/ca9 to d2/d9/d7e/cc6 0 2024-03-22T02:29:00.514 INFO:tasks.workunit.client.1.smithi122.stdout:5/598: sync 2024-03-22T02:29:00.520 INFO:tasks.workunit.client.1.smithi122.stdout:3/626: dwrite d2/f5e [4194304,4194304] 0 2024-03-22T02:29:00.521 INFO:tasks.workunit.client.1.smithi122.stdout:3/627: readlink d2/d9/dc/df/d3c/d59/l96 0 2024-03-22T02:29:00.522 INFO:tasks.workunit.client.1.smithi122.stdout:3/628: dread - d2/d9/dc/d57/d89/fbf zero size 2024-03-22T02:29:00.526 INFO:tasks.workunit.client.1.smithi122.stdout:3/629: unlink d2/d8a/da2/fbb 0 2024-03-22T02:29:00.527 INFO:tasks.workunit.client.1.smithi122.stdout:3/630: chown d2/d9/dc/d17/d45/f81 74 1 2024-03-22T02:29:00.530 INFO:tasks.workunit.client.1.smithi122.stdout:3/631: mkdir d2/d9/dc/d57/da4/dc7 0 2024-03-22T02:29:00.530 INFO:tasks.workunit.client.1.smithi122.stdout:3/632: stat d2 0 2024-03-22T02:29:00.533 INFO:tasks.workunit.client.1.smithi122.stdout:5/599: dwrite d8/d1e/d22/d3a/f32 [0,4194304] 0 2024-03-22T02:29:00.533 INFO:tasks.workunit.client.1.smithi122.stdout:3/633: creat d2/d9/dc/d17/d2b/d6b/fc8 x:0 0 0 2024-03-22T02:29:00.542 INFO:tasks.workunit.client.1.smithi122.stdout:5/600: rename c5 to d8/d1e/d22/da3/cce 0 2024-03-22T02:29:00.544 INFO:tasks.workunit.client.1.smithi122.stdout:5/601: fdatasync d8/d1a/d83/fae 0 2024-03-22T02:29:00.546 INFO:tasks.workunit.client.1.smithi122.stdout:5/602: creat d8/d1e/d22/d3a/d39/fcf x:0 0 0 2024-03-22T02:29:00.550 INFO:tasks.workunit.client.1.smithi122.stdout:3/634: write d2/f7 [632566,73327] 0 2024-03-22T02:29:00.551 INFO:tasks.workunit.client.1.smithi122.stdout:4/696: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:2/768: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:1/659: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:9/730: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:7/646: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:0/589: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:8/751: sync 2024-03-22T02:29:00.552 INFO:tasks.workunit.client.1.smithi122.stdout:6/785: sync 2024-03-22T02:29:00.556 INFO:tasks.workunit.client.1.smithi122.stdout:9/731: chown db 0 1 2024-03-22T02:29:00.556 INFO:tasks.workunit.client.1.smithi122.stdout:2/769: fsync df/d1f/f2d 0 2024-03-22T02:29:00.558 INFO:tasks.workunit.client.1.smithi122.stdout:1/660: write dc/d2b/d86/dbb/fce [4200441,36443] 0 2024-03-22T02:29:00.558 INFO:tasks.workunit.client.1.smithi122.stdout:0/590: dread - d11/d14/d1d/d27/d59/fb1 zero size 2024-03-22T02:29:00.563 INFO:tasks.workunit.client.1.smithi122.stdout:4/697: write d0/d12/d15/d42/d4e/de/d40/f9d [2275,24892] 0 2024-03-22T02:29:00.564 INFO:tasks.workunit.client.1.smithi122.stdout:9/732: truncate db/d1b/da8/fab 64083 0 2024-03-22T02:29:00.564 INFO:tasks.workunit.client.1.smithi122.stdout:3/635: mkdir d2/d9/dc/d17/d2b/dc9 0 2024-03-22T02:29:00.566 INFO:tasks.workunit.client.1.smithi122.stdout:7/647: unlink d0/d4/d20/d21/d28/c89 0 2024-03-22T02:29:00.573 INFO:tasks.workunit.client.1.smithi122.stdout:6/786: dwrite d7/de/dd/d20/de0/dae/fc5 [0,4194304] 0 2024-03-22T02:29:00.575 INFO:tasks.workunit.client.1.smithi122.stdout:4/698: rename d0/d12/d15/f8c to d0/d12/fdd 0 2024-03-22T02:29:00.581 INFO:tasks.workunit.client.1.smithi122.stdout:7/648: rmdir d0/d4/d1d/db8/d48 39 2024-03-22T02:29:00.584 INFO:tasks.workunit.client.1.smithi122.stdout:8/752: dwrite d1/f3c [4194304,4194304] 0 2024-03-22T02:29:00.588 INFO:tasks.workunit.client.1.smithi122.stdout:9/733: symlink db/d1b/d6c/d58/d9d/le4 0 2024-03-22T02:29:00.588 INFO:tasks.workunit.client.1.smithi122.stdout:4/699: creat d0/fde x:0 0 0 2024-03-22T02:29:00.589 INFO:tasks.workunit.client.1.smithi122.stdout:3/636: link d2/l14 d2/d9/dc/d57/da4/dc7/lca 0 2024-03-22T02:29:00.590 INFO:tasks.workunit.client.1.smithi122.stdout:7/649: creat d0/d25/d5c/fcd x:0 0 0 2024-03-22T02:29:00.597 INFO:tasks.workunit.client.1.smithi122.stdout:8/753: rmdir d1/d8/d63/df0 39 2024-03-22T02:29:00.597 INFO:tasks.workunit.client.1.smithi122.stdout:6/787: dwrite d7/de/dd/d20/d8f/fd0 [0,4194304] 0 2024-03-22T02:29:00.597 INFO:tasks.workunit.client.1.smithi122.stdout:2/770: link df/d1f/d33/d4d/d94/dc2/d72/cfa df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/c101 0 2024-03-22T02:29:00.599 INFO:tasks.workunit.client.1.smithi122.stdout:2/771: write df/d1f/d33/f54 [4042463,908] 0 2024-03-22T02:29:00.600 INFO:tasks.workunit.client.1.smithi122.stdout:7/650: creat d0/d4e/d5d/dab/fce x:0 0 0 2024-03-22T02:29:00.602 INFO:tasks.workunit.client.1.smithi122.stdout:6/788: unlink d7/de/dd/d23/leb 0 2024-03-22T02:29:00.602 INFO:tasks.workunit.client.1.smithi122.stdout:7/651: chown d0/d4e/f9e 295940622 1 2024-03-22T02:29:00.609 INFO:tasks.workunit.client.1.smithi122.stdout:2/772: dread - df/d1f/d23/dd2/dca/fe3 zero size 2024-03-22T02:29:00.610 INFO:tasks.workunit.client.1.smithi122.stdout:6/789: read d7/de/dd/d20/f31 [402704,76014] 0 2024-03-22T02:29:00.610 INFO:tasks.workunit.client.1.smithi122.stdout:7/652: getdents d0/d4/d22 0 2024-03-22T02:29:00.613 INFO:tasks.workunit.client.1.smithi122.stdout:0/591: dread d11/d14/d23/f77 [0,4194304] 0 2024-03-22T02:29:00.613 INFO:tasks.workunit.client.1.smithi122.stdout:2/773: write df/d1f/d23/dd2/dca/fe3 [394521,114525] 0 2024-03-22T02:29:00.614 INFO:tasks.workunit.client.1.smithi122.stdout:1/661: dread dc/f51 [0,4194304] 0 2024-03-22T02:29:00.615 INFO:tasks.workunit.client.1.smithi122.stdout:7/653: readlink d0/d4/l7c 0 2024-03-22T02:29:00.618 INFO:tasks.workunit.client.1.smithi122.stdout:7/654: dread - d0/d25/d5c/fcd zero size 2024-03-22T02:29:00.618 INFO:tasks.workunit.client.1.smithi122.stdout:6/790: mknod d7/de/dd/d9b/cff 0 2024-03-22T02:29:00.625 INFO:tasks.workunit.client.1.smithi122.stdout:2/774: rename df/d1f/f9c to df/d1f/d33/d4d/d94/dc2/d51/f102 0 2024-03-22T02:29:00.627 INFO:tasks.workunit.client.1.smithi122.stdout:0/592: dwrite d11/d14/d1d/d27/d2f/d4e/faf [0,4194304] 0 2024-03-22T02:29:00.632 INFO:tasks.workunit.client.1.smithi122.stdout:0/593: write d11/d16/f55 [8417365,64190] 0 2024-03-22T02:29:00.633 INFO:tasks.workunit.client.1.smithi122.stdout:6/791: fdatasync d7/de/d27/d52/f7d 0 2024-03-22T02:29:00.638 INFO:tasks.workunit.client.1.smithi122.stdout:8/754: truncate d1/d40/d51/df5/dae/dfe/f41 2824686 0 2024-03-22T02:29:00.640 INFO:tasks.workunit.client.1.smithi122.stdout:0/594: write d11/d14/d1b/f8f [768862,96965] 0 2024-03-22T02:29:00.644 INFO:tasks.workunit.client.1.smithi122.stdout:7/655: dwrite d0/d4/d20/d21/d9b/fbd [0,4194304] 0 2024-03-22T02:29:00.644 INFO:tasks.workunit.client.1.smithi122.stdout:4/700: dread d0/f9 [0,4194304] 0 2024-03-22T02:29:00.646 INFO:tasks.workunit.client.1.smithi122.stdout:6/792: rmdir d7/de/dd/d23 39 2024-03-22T02:29:00.647 INFO:tasks.workunit.client.1.smithi122.stdout:7/656: write d0/d4/d20/d21/d9b/d5e/f77 [5241003,12854] 0 2024-03-22T02:29:00.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/595: mknod d11/d14/da7/cc8 0 2024-03-22T02:29:00.658 INFO:tasks.workunit.client.1.smithi122.stdout:7/657: symlink d0/d30/lcf 0 2024-03-22T02:29:00.666 INFO:tasks.workunit.client.1.smithi122.stdout:0/596: fdatasync d11/d14/d23/d24/f25 0 2024-03-22T02:29:00.666 INFO:tasks.workunit.client.1.smithi122.stdout:2/775: truncate df/d1f/f3c 4351706 0 2024-03-22T02:29:00.666 INFO:tasks.workunit.client.1.smithi122.stdout:1/662: dread dc/fd [0,4194304] 0 2024-03-22T02:29:00.667 INFO:tasks.workunit.client.1.smithi122.stdout:6/793: creat d7/de/dd/d20/de0/df9/f100 x:0 0 0 2024-03-22T02:29:00.668 INFO:tasks.workunit.client.1.smithi122.stdout:0/597: dread - d11/d16/d37/d9f/fa4 zero size 2024-03-22T02:29:00.669 INFO:tasks.workunit.client.1.smithi122.stdout:1/663: chown dc/d2b/f57 3 1 2024-03-22T02:29:00.670 INFO:tasks.workunit.client.1.smithi122.stdout:7/658: rename d0/d4/d20/l40 to d0/d4/d1d/daf/ld0 0 2024-03-22T02:29:00.671 INFO:tasks.workunit.client.1.smithi122.stdout:7/659: chown d0/d4/f2d 1107 1 2024-03-22T02:29:00.672 INFO:tasks.workunit.client.1.smithi122.stdout:7/660: stat d0/d4/d94/fa9 0 2024-03-22T02:29:00.672 INFO:tasks.workunit.client.1.smithi122.stdout:8/755: link d1/d9/f24 d1/d40/d48/d8d/f101 0 2024-03-22T02:29:00.676 INFO:tasks.workunit.client.1.smithi122.stdout:0/598: rmdir d11/d4f/d5a 39 2024-03-22T02:29:00.681 INFO:tasks.workunit.client.1.smithi122.stdout:1/664: truncate dc/d4c/f8b 1096857 0 2024-03-22T02:29:00.681 INFO:tasks.workunit.client.1.smithi122.stdout:8/756: dwrite d1/d40/d48/d8d/fca [0,4194304] 0 2024-03-22T02:29:00.685 INFO:tasks.workunit.client.1.smithi122.stdout:4/701: getdents d0/d12/d15/d3d/d90 0 2024-03-22T02:29:00.686 INFO:tasks.workunit.client.1.smithi122.stdout:7/661: creat d0/d4/d20/d21/d9b/d5e/daa/fd1 x:0 0 0 2024-03-22T02:29:00.688 INFO:tasks.workunit.client.1.smithi122.stdout:2/776: symlink df/d1f/d23/dd2/df2/l103 0 2024-03-22T02:29:00.692 INFO:tasks.workunit.client.1.smithi122.stdout:6/794: dwrite d7/de/dd/d20/de0/f30 [0,4194304] 0 2024-03-22T02:29:00.695 INFO:tasks.workunit.client.1.smithi122.stdout:4/702: getdents d0/d12/d15/d3d/d90 0 2024-03-22T02:29:00.724 INFO:tasks.workunit.client.1.smithi122.stdout:2/777: dwrite df/d1f/d33/f80 [0,4194304] 0 2024-03-22T02:29:00.725 INFO:tasks.workunit.client.1.smithi122.stdout:0/599: dwrite d11/d14/d1b/f38 [4194304,4194304] 0 2024-03-22T02:29:00.725 INFO:tasks.workunit.client.1.smithi122.stdout:1/665: truncate dc/d3b/d8d/d72/f96 632045 0 2024-03-22T02:29:00.725 INFO:tasks.workunit.client.1.smithi122.stdout:4/703: write d0/d12/d15/d42/d4e/de/d40/f9d [779133,113649] 0 2024-03-22T02:29:00.726 INFO:tasks.workunit.client.1.smithi122.stdout:8/757: truncate d1/d40/d51/df5/dae/dfe/d45/f68 3725468 0 2024-03-22T02:29:00.727 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:00 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:00.727 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:00 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:00.727 INFO:tasks.workunit.client.1.smithi122.stdout:7/662: write d0/d4/d20/d57/fbf [10917,41335] 0 2024-03-22T02:29:00.727 INFO:tasks.workunit.client.1.smithi122.stdout:1/666: chown dc/d3b/d46/f6b 363702 1 2024-03-22T02:29:00.730 INFO:tasks.workunit.client.1.smithi122.stdout:4/704: mknod d0/cdf 0 2024-03-22T02:29:00.737 INFO:tasks.workunit.client.1.smithi122.stdout:4/705: read d0/d25/f3b [1471446,109849] 0 2024-03-22T02:29:00.737 INFO:tasks.workunit.client.1.smithi122.stdout:0/600: creat d11/d14/d23/d24/d33/d67/dc2/fc9 x:0 0 0 2024-03-22T02:29:00.738 INFO:tasks.workunit.client.1.smithi122.stdout:7/663: symlink d0/d4e/ld2 0 2024-03-22T02:29:00.741 INFO:tasks.workunit.client.1.smithi122.stdout:7/664: write d0/d4/d20/d21/d9b/d5e/f77 [3043510,44617] 0 2024-03-22T02:29:00.744 INFO:tasks.workunit.client.1.smithi122.stdout:7/665: truncate d0/d4/d94/fa5 1706138 0 2024-03-22T02:29:00.748 INFO:tasks.workunit.client.1.smithi122.stdout:2/778: dwrite df/d1f/d33/d4d/d94/dc2/d51/fd9 [0,4194304] 0 2024-03-22T02:29:00.750 INFO:tasks.workunit.client.1.smithi122.stdout:7/666: symlink d0/d4/d20/d21/d9b/d5e/ld3 0 2024-03-22T02:29:00.750 INFO:tasks.workunit.client.1.smithi122.stdout:0/601: dread d11/d14/d1d/d27/d2f/f3a [0,4194304] 0 2024-03-22T02:29:00.750 INFO:tasks.workunit.client.1.smithi122.stdout:4/706: link d0/d12/d15/d42/d4e/f22 d0/d7a/fe0 0 2024-03-22T02:29:00.752 INFO:tasks.workunit.client.1.smithi122.stdout:0/602: stat d11/d14/l7e 0 2024-03-22T02:29:00.757 INFO:tasks.workunit.client.1.smithi122.stdout:7/667: creat d0/d4/d1d/db8/d9a/fd4 x:0 0 0 2024-03-22T02:29:00.759 INFO:tasks.workunit.client.1.smithi122.stdout:4/707: creat d0/d12/d15/d3d/dd2/fe1 x:0 0 0 2024-03-22T02:29:00.760 INFO:tasks.workunit.client.1.smithi122.stdout:2/779: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2/f104 x:0 0 0 2024-03-22T02:29:00.761 INFO:tasks.workunit.client.1.smithi122.stdout:7/668: readlink d0/d4/d1d/db8/d48/l7a 0 2024-03-22T02:29:00.762 INFO:tasks.workunit.client.1.smithi122.stdout:4/708: fsync d0/d25/f3b 0 2024-03-22T02:29:00.763 INFO:tasks.workunit.client.1.smithi122.stdout:0/603: creat d11/d14/d23/d24/fca x:0 0 0 2024-03-22T02:29:00.766 INFO:tasks.workunit.client.1.smithi122.stdout:1/667: dread dc/d2b/f33 [0,4194304] 0 2024-03-22T02:29:00.768 INFO:tasks.workunit.client.1.smithi122.stdout:0/604: creat d11/d14/d1d/d27/d9d/fcb x:0 0 0 2024-03-22T02:29:00.768 INFO:tasks.workunit.client.1.smithi122.stdout:7/669: symlink d0/d4/d20/d21/d9b/ld5 0 2024-03-22T02:29:00.774 INFO:tasks.workunit.client.1.smithi122.stdout:7/670: dread d0/d4/d94/fc5 [0,4194304] 0 2024-03-22T02:29:00.775 INFO:tasks.workunit.client.1.smithi122.stdout:7/671: dread d0/d4e/f73 [0,4194304] 0 2024-03-22T02:29:00.777 INFO:tasks.workunit.client.1.smithi122.stdout:4/709: dread d0/d12/d15/d42/d4e/f16 [0,4194304] 0 2024-03-22T02:29:00.777 INFO:tasks.workunit.client.1.smithi122.stdout:0/605: rename d11/d4f/d5a/f5c to d11/d14/d1d/d27/d9d/fcc 0 2024-03-22T02:29:00.781 INFO:tasks.workunit.client.1.smithi122.stdout:4/710: symlink d0/d12/d61/le2 0 2024-03-22T02:29:00.782 INFO:tasks.workunit.client.1.smithi122.stdout:0/606: mkdir d11/d14/d1d/d27/d9d/daa/dcd 0 2024-03-22T02:29:00.783 INFO:tasks.workunit.client.1.smithi122.stdout:2/780: truncate df/d1f/d33/d34/fb2 1303073 0 2024-03-22T02:29:00.785 INFO:tasks.workunit.client.1.smithi122.stdout:7/672: rename d0/d4/d20/d21/d9b/l3f to d0/d4/d20/ld6 0 2024-03-22T02:29:00.785 INFO:tasks.workunit.client.1.smithi122.stdout:5/603: sync 2024-03-22T02:29:00.786 INFO:tasks.workunit.client.1.smithi122.stdout:0/607: rmdir d11/d16/d37/d84 39 2024-03-22T02:29:00.788 INFO:tasks.workunit.client.1.smithi122.stdout:7/673: truncate d0/d4/d20/d21/d9b/dbb/fc7 886504 0 2024-03-22T02:29:00.789 INFO:tasks.workunit.client.1.smithi122.stdout:2/781: rmdir df/d1f/d33/d6a/d7c/d8e/da0 39 2024-03-22T02:29:00.789 INFO:tasks.workunit.client.1.smithi122.stdout:7/674: write d0/d4/d20/d21/d9b/d5e/f99 [2149801,130159] 0 2024-03-22T02:29:00.791 INFO:tasks.workunit.client.1.smithi122.stdout:7/675: readlink d0/d30/lc9 0 2024-03-22T02:29:00.794 INFO:tasks.workunit.client.1.smithi122.stdout:5/604: getdents d8/d1a/d83/dcb 0 2024-03-22T02:29:00.799 INFO:tasks.workunit.client.1.smithi122.stdout:2/782: unlink df/d1f/d9b/fa7 0 2024-03-22T02:29:00.799 INFO:tasks.workunit.client.1.smithi122.stdout:1/668: dwrite dc/f25 [0,4194304] 0 2024-03-22T02:29:00.799 INFO:tasks.workunit.client.1.smithi122.stdout:0/608: link d11/d16/d37/fbf d11/d14/d1b/fce 0 2024-03-22T02:29:00.799 INFO:tasks.workunit.client.1.smithi122.stdout:4/711: dread d0/d25/f4d [0,4194304] 0 2024-03-22T02:29:00.800 INFO:tasks.workunit.client.1.smithi122.stdout:0/609: readlink d11/d14/l7e 0 2024-03-22T02:29:00.801 INFO:tasks.workunit.client.1.smithi122.stdout:7/676: creat d0/d4/d20/d21/d9b/d5e/fd7 x:0 0 0 2024-03-22T02:29:00.804 INFO:tasks.workunit.client.1.smithi122.stdout:4/712: dread - d0/d12/d73/fd5 zero size 2024-03-22T02:29:00.805 INFO:tasks.workunit.client.1.smithi122.stdout:5/605: dwrite d8/d1e/f7e [0,4194304] 0 2024-03-22T02:29:00.813 INFO:tasks.workunit.client.1.smithi122.stdout:1/669: mkdir dc/d2b/dd2 0 2024-03-22T02:29:00.817 INFO:tasks.workunit.client.1.smithi122.stdout:4/713: dwrite d0/d12/d15/d42/d4e/de/f8b [0,4194304] 0 2024-03-22T02:29:00.817 INFO:tasks.workunit.client.1.smithi122.stdout:1/670: read - dc/d3b/fcb zero size 2024-03-22T02:29:00.820 INFO:tasks.workunit.client.1.smithi122.stdout:5/606: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf [0,4194304] 0 2024-03-22T02:29:00.822 INFO:tasks.workunit.client.1.smithi122.stdout:5/607: chown d8/d1e/d22/d3a/d46/c99 9146 1 2024-03-22T02:29:00.828 INFO:tasks.workunit.client.1.smithi122.stdout:5/608: symlink d8/da/ld0 0 2024-03-22T02:29:00.829 INFO:tasks.workunit.client.1.smithi122.stdout:2/783: write df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f82 [597222,88656] 0 2024-03-22T02:29:00.830 INFO:tasks.workunit.client.1.smithi122.stdout:4/714: creat d0/d12/d15/d42/d4e/de/fe3 x:0 0 0 2024-03-22T02:29:00.830 INFO:tasks.workunit.client.1.smithi122.stdout:7/677: write d0/d4/d22/f32 [545807,123191] 0 2024-03-22T02:29:00.831 INFO:tasks.workunit.client.1.smithi122.stdout:1/671: write dc/f2a [5170019,24714] 0 2024-03-22T02:29:00.840 INFO:tasks.workunit.client.1.smithi122.stdout:4/715: dwrite d0/d25/f5e [0,4194304] 0 2024-03-22T02:29:00.842 INFO:tasks.workunit.client.1.smithi122.stdout:7/678: link d0/d30/la3 d0/d4/d20/d21/d9b/d5e/daa/ld8 0 2024-03-22T02:29:00.846 INFO:tasks.workunit.client.1.smithi122.stdout:5/609: link d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/ca6 d8/d1e/d22/cd1 0 2024-03-22T02:29:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:00 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:00 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:00.849 INFO:tasks.workunit.client.1.smithi122.stdout:4/716: link d0/d12/d15/d2d/d75/f8a d0/d12/d15/d42/d4e/de/d40/fe4 0 2024-03-22T02:29:00.852 INFO:tasks.workunit.client.1.smithi122.stdout:5/610: truncate d8/d3b/f62 1539409 0 2024-03-22T02:29:00.852 INFO:tasks.workunit.client.1.smithi122.stdout:7/679: dwrite d0/d30/f8c [0,4194304] 0 2024-03-22T02:29:00.854 INFO:tasks.workunit.client.1.smithi122.stdout:4/717: creat d0/d12/d61/fe5 x:0 0 0 2024-03-22T02:29:00.857 INFO:tasks.workunit.client.1.smithi122.stdout:5/611: symlink d8/d1e/d22/d78/ld2 0 2024-03-22T02:29:00.858 INFO:tasks.workunit.client.1.smithi122.stdout:1/672: dwrite dc/d55/fc0 [0,4194304] 0 2024-03-22T02:29:00.859 INFO:tasks.workunit.client.1.smithi122.stdout:7/680: truncate d0/f87 716510 0 2024-03-22T02:29:00.860 INFO:tasks.workunit.client.1.smithi122.stdout:1/673: readlink dc/d2b/l5a 0 2024-03-22T02:29:00.862 INFO:tasks.workunit.client.1.smithi122.stdout:5/612: mknod d8/d3b/cd3 0 2024-03-22T02:29:00.863 INFO:tasks.workunit.client.1.smithi122.stdout:4/718: fdatasync d0/d12/f31 0 2024-03-22T02:29:00.865 INFO:tasks.workunit.client.1.smithi122.stdout:7/681: readlink d0/d4/d20/d21/d9b/l83 0 2024-03-22T02:29:00.868 INFO:tasks.workunit.client.1.smithi122.stdout:7/682: write d0/d30/f5b [778954,103261] 0 2024-03-22T02:29:00.868 INFO:tasks.workunit.client.1.smithi122.stdout:5/613: mknod d8/d1e/d22/d3a/d46/d4f/d9b/da9/cd4 0 2024-03-22T02:29:00.871 INFO:tasks.workunit.client.1.smithi122.stdout:1/674: dwrite dc/d3b/d8d/d72/d9c/db0/dab/faf [0,4194304] 0 2024-03-22T02:29:00.871 INFO:tasks.workunit.client.1.smithi122.stdout:4/719: mkdir d0/d25/de6 0 2024-03-22T02:29:00.873 INFO:tasks.workunit.client.1.smithi122.stdout:1/675: chown dc/d2b/c9a 13 1 2024-03-22T02:29:00.874 INFO:tasks.workunit.client.1.smithi122.stdout:7/683: getdents d0/d4/d1d/db8/d98 0 2024-03-22T02:29:00.876 INFO:tasks.workunit.client.1.smithi122.stdout:5/614: truncate d8/d1a/f5c 120050 0 2024-03-22T02:29:00.876 INFO:tasks.workunit.client.1.smithi122.stdout:4/720: mkdir d0/d12/d15/d42/de7 0 2024-03-22T02:29:00.879 INFO:tasks.workunit.client.1.smithi122.stdout:4/721: write d0/d12/d15/d42/fd7 [988418,66421] 0 2024-03-22T02:29:00.880 INFO:tasks.workunit.client.1.smithi122.stdout:5/615: read d8/d1e/d22/d3a/d46/d4f/d55/db6/f95 [1826509,111760] 0 2024-03-22T02:29:00.882 INFO:tasks.workunit.client.1.smithi122.stdout:1/676: truncate dc/d3b/d8d/d72/d9c/db0/d67/dc6/f50 980285 0 2024-03-22T02:29:00.883 INFO:tasks.workunit.client.1.smithi122.stdout:1/677: truncate dc/d3b/d46/fa7 5083625 0 2024-03-22T02:29:00.884 INFO:tasks.workunit.client.1.smithi122.stdout:7/684: rename d0/d4/d20/d21/fb4 to d0/d4/d20/d21/d9b/dbb/fd9 0 2024-03-22T02:29:00.893 INFO:tasks.workunit.client.1.smithi122.stdout:5/616: unlink d8/d1e/d22/da3/cbd 0 2024-03-22T02:29:00.896 INFO:tasks.workunit.client.1.smithi122.stdout:7/685: mkdir d0/d4/d20/d21/dda 0 2024-03-22T02:29:00.897 INFO:tasks.workunit.client.1.smithi122.stdout:4/722: creat d0/d12/d15/d42/d4e/de/da1/dbf/fe8 x:0 0 0 2024-03-22T02:29:00.899 INFO:tasks.workunit.client.1.smithi122.stdout:5/617: rename d8/d1e/d22/d78/l85 to d8/d1e/d22/d3a/d46/d4f/d9b/da9/ld5 0 2024-03-22T02:29:00.902 INFO:tasks.workunit.client.1.smithi122.stdout:7/686: unlink d0/c7b 0 2024-03-22T02:29:00.904 INFO:tasks.workunit.client.1.smithi122.stdout:4/723: rename d0/d12/d15/d3d/f68 to d0/d12/d15/d2d/db5/fe9 0 2024-03-22T02:29:00.906 INFO:tasks.workunit.client.1.smithi122.stdout:7/687: mknod d0/d4/d20/cdb 0 2024-03-22T02:29:00.909 INFO:tasks.workunit.client.1.smithi122.stdout:1/678: dwrite dc/d3b/d46/f6b [0,4194304] 0 2024-03-22T02:29:00.915 INFO:tasks.workunit.client.1.smithi122.stdout:5/618: getdents d8/d1e/d22/da3 0 2024-03-22T02:29:00.919 INFO:tasks.workunit.client.1.smithi122.stdout:5/619: rmdir d8/d1e/d22 39 2024-03-22T02:29:00.920 INFO:tasks.workunit.client.1.smithi122.stdout:7/688: rmdir d0/d4/d20 39 2024-03-22T02:29:00.920 INFO:tasks.workunit.client.1.smithi122.stdout:4/724: dread d0/d25/f27 [0,4194304] 0 2024-03-22T02:29:00.921 INFO:tasks.workunit.client.1.smithi122.stdout:5/620: read d8/d1a/f70 [75861,6293] 0 2024-03-22T02:29:00.921 INFO:tasks.workunit.client.1.smithi122.stdout:1/679: mkdir dc/d3b/d8d/d72/d9c/dd3 0 2024-03-22T02:29:00.922 INFO:tasks.workunit.client.1.smithi122.stdout:1/680: chown dc/d3b/d46/db5/dba/fcc 126 1 2024-03-22T02:29:00.923 INFO:tasks.workunit.client.1.smithi122.stdout:1/681: fdatasync dc/f25 0 2024-03-22T02:29:00.925 INFO:tasks.workunit.client.1.smithi122.stdout:4/725: mknod d0/d25/cea 0 2024-03-22T02:29:00.925 INFO:tasks.workunit.client.1.smithi122.stdout:1/682: dread - dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/fb7 zero size 2024-03-22T02:29:00.926 INFO:tasks.workunit.client.1.smithi122.stdout:7/689: fdatasync d0/d4/d20/d21/d9b/dbb/fd9 0 2024-03-22T02:29:00.928 INFO:tasks.workunit.client.1.smithi122.stdout:1/683: creat dc/d3b/d8d/d72/d9c/db0/d90/fd4 x:0 0 0 2024-03-22T02:29:00.930 INFO:tasks.workunit.client.1.smithi122.stdout:7/690: creat d0/d4/d20/d21/d28/fdc x:0 0 0 2024-03-22T02:29:00.931 INFO:tasks.workunit.client.1.smithi122.stdout:5/621: link d8/d3b/l59 d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/ld6 0 2024-03-22T02:29:00.933 INFO:tasks.workunit.client.1.smithi122.stdout:4/726: link d0/d12/d15/d42/d57/f64 d0/d12/d15/feb 0 2024-03-22T02:29:00.935 INFO:tasks.workunit.client.1.smithi122.stdout:1/684: dwrite dc/d2b/f57 [0,4194304] 0 2024-03-22T02:29:00.935 INFO:tasks.workunit.client.1.smithi122.stdout:7/691: truncate d0/d4/d20/d21/d9b/dbb/fc7 1579236 0 2024-03-22T02:29:00.941 INFO:tasks.workunit.client.1.smithi122.stdout:4/727: rename d0/d12/l47 to d0/d12/d15/d42/lec 0 2024-03-22T02:29:00.942 INFO:tasks.workunit.client.1.smithi122.stdout:1/685: dread dc/d2b/f33 [0,4194304] 0 2024-03-22T02:29:00.942 INFO:tasks.workunit.client.1.smithi122.stdout:1/686: readlink dc/d3b/l41 0 2024-03-22T02:29:00.943 INFO:tasks.workunit.client.1.smithi122.stdout:4/728: symlink d0/d12/d15/led 0 2024-03-22T02:29:00.943 INFO:tasks.workunit.client.1.smithi122.stdout:7/692: link d0/d4/d1d/db8/d9a/ca2 d0/d4e/d5d/cdd 0 2024-03-22T02:29:00.944 INFO:tasks.workunit.client.1.smithi122.stdout:5/622: read d8/d1e/d22/d3a/d46/d4f/d55/dc7/f2c [803781,56730] 0 2024-03-22T02:29:00.945 INFO:tasks.workunit.client.1.smithi122.stdout:1/687: unlink dc/d3b/c44 0 2024-03-22T02:29:00.946 INFO:tasks.workunit.client.1.smithi122.stdout:4/729: mkdir d0/d12/d61/dee 0 2024-03-22T02:29:00.947 INFO:tasks.workunit.client.1.smithi122.stdout:1/688: chown dc/d3b/d8d/d72/c79 5092 1 2024-03-22T02:29:00.948 INFO:tasks.workunit.client.1.smithi122.stdout:5/623: mkdir d8/d1e/d22/dd7 0 2024-03-22T02:29:00.950 INFO:tasks.workunit.client.1.smithi122.stdout:7/693: creat d0/d25/fde x:0 0 0 2024-03-22T02:29:00.952 INFO:tasks.workunit.client.1.smithi122.stdout:1/689: dread dc/f25 [0,4194304] 0 2024-03-22T02:29:00.952 INFO:tasks.workunit.client.1.smithi122.stdout:7/694: creat d0/d4/d80/fdf x:0 0 0 2024-03-22T02:29:00.954 INFO:tasks.workunit.client.1.smithi122.stdout:5/624: unlink d8/d1e/d22/d3a/d46/d4f/d9b/da9/fcd 0 2024-03-22T02:29:00.954 INFO:tasks.workunit.client.1.smithi122.stdout:4/730: getdents d0/d12/d15/d42/d4e/de/d91 0 2024-03-22T02:29:00.955 INFO:tasks.workunit.client.1.smithi122.stdout:1/690: rmdir dc/d4c 39 2024-03-22T02:29:00.958 INFO:tasks.workunit.client.1.smithi122.stdout:4/731: creat d0/d12/d15/fef x:0 0 0 2024-03-22T02:29:00.962 INFO:tasks.workunit.client.1.smithi122.stdout:1/691: symlink dc/d4c/ld5 0 2024-03-22T02:29:00.966 INFO:tasks.workunit.client.1.smithi122.stdout:7/695: dread d0/d4/d1d/db8/f85 [0,4194304] 0 2024-03-22T02:29:00.967 INFO:tasks.workunit.client.1.smithi122.stdout:7/696: chown d0/fcb 152 1 2024-03-22T02:29:00.969 INFO:tasks.workunit.client.1.smithi122.stdout:5/625: getdents d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35 0 2024-03-22T02:29:01.002 INFO:tasks.workunit.client.1.smithi122.stdout:9/734: sync 2024-03-22T02:29:01.002 INFO:tasks.workunit.client.1.smithi122.stdout:3/637: sync 2024-03-22T02:29:01.004 INFO:tasks.workunit.client.1.smithi122.stdout:7/697: mkdir d0/d4e/de0 0 2024-03-22T02:29:01.005 INFO:tasks.workunit.client.1.smithi122.stdout:5/626: truncate d8/d1e/d22/da3/fa5 1583174 0 2024-03-22T02:29:01.006 INFO:tasks.workunit.client.1.smithi122.stdout:7/698: write d0/d4/d20/d21/d28/f90 [516283,40460] 0 2024-03-22T02:29:01.013 INFO:tasks.workunit.client.1.smithi122.stdout:5/627: mknod d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/cd8 0 2024-03-22T02:29:01.014 INFO:tasks.workunit.client.1.smithi122.stdout:1/692: truncate dc/d3b/d8d/d72/d9c/db0/f7d 1649286 0 2024-03-22T02:29:01.016 INFO:tasks.workunit.client.1.smithi122.stdout:1/693: write dc/d3b/d46/f6b [1948121,18105] 0 2024-03-22T02:29:01.018 INFO:tasks.workunit.client.1.smithi122.stdout:9/735: write db/df/d3e/d81/d25/d41/f61 [1916268,42734] 0 2024-03-22T02:29:01.020 INFO:tasks.workunit.client.1.smithi122.stdout:4/732: dwrite d0/d12/d15/d2d/f4f [0,4194304] 0 2024-03-22T02:29:01.021 INFO:tasks.workunit.client.1.smithi122.stdout:7/699: dread d0/d30/f53 [0,4194304] 0 2024-03-22T02:29:01.021 INFO:tasks.workunit.client.1.smithi122.stdout:3/638: truncate d2/d9/dc/d17/d28/d4c/da5/db4/f76 2399058 0 2024-03-22T02:29:01.024 INFO:tasks.workunit.client.1.smithi122.stdout:1/694: fsync dc/f4e 0 2024-03-22T02:29:01.026 INFO:tasks.workunit.client.1.smithi122.stdout:9/736: creat db/d23/fe5 x:0 0 0 2024-03-22T02:29:01.029 INFO:tasks.workunit.client.1.smithi122.stdout:4/733: fdatasync d0/f37 0 2024-03-22T02:29:01.030 INFO:tasks.workunit.client.1.smithi122.stdout:7/700: readlink d0/d4/l7c 0 2024-03-22T02:29:01.031 INFO:tasks.workunit.client.1.smithi122.stdout:3/639: creat d2/d9/dc/d17/d2b/d6b/fcb x:0 0 0 2024-03-22T02:29:01.033 INFO:tasks.workunit.client.1.smithi122.stdout:5/628: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f68 [1432212,114085] 0 2024-03-22T02:29:01.033 INFO:tasks.workunit.client.1.smithi122.stdout:3/640: chown d2/d9/dc/d17/d28/d4c/da5/db4/l7f 25 1 2024-03-22T02:29:01.035 INFO:tasks.workunit.client.1.smithi122.stdout:9/737: mknod db/d23/d9a/ce6 0 2024-03-22T02:29:01.041 INFO:tasks.workunit.client.1.smithi122.stdout:4/734: mkdir d0/d12/d15/d42/df0 0 2024-03-22T02:29:01.043 INFO:tasks.workunit.client.1.smithi122.stdout:1/695: creat dc/d3b/d8d/d72/d9c/fd6 x:0 0 0 2024-03-22T02:29:01.044 INFO:tasks.workunit.client.1.smithi122.stdout:3/641: creat d2/d9/dc/df/d3c/d78/fcc x:0 0 0 2024-03-22T02:29:01.045 INFO:tasks.workunit.client.1.smithi122.stdout:1/696: chown dc/d3b/d8d/d72/d9c/db0/d67/dc6/la1 9 1 2024-03-22T02:29:01.046 INFO:tasks.workunit.client.1.smithi122.stdout:4/735: rename d0/d12/d61/dce/ld3 to d0/d12/d15/d42/d57/lf1 0 2024-03-22T02:29:01.050 INFO:tasks.workunit.client.1.smithi122.stdout:1/697: rename dc/d3b/d8d/d72/d9c/db0/dab/faf to dc/d2b/d86/fd7 0 2024-03-22T02:29:01.052 INFO:tasks.workunit.client.1.smithi122.stdout:3/642: creat d2/d8a/dad/fcd x:0 0 0 2024-03-22T02:29:01.055 INFO:tasks.workunit.client.1.smithi122.stdout:7/701: dread d0/d30/f52 [0,4194304] 0 2024-03-22T02:29:01.055 INFO:tasks.workunit.client.1.smithi122.stdout:1/698: truncate dc/f88 3013653 0 2024-03-22T02:29:01.056 INFO:tasks.workunit.client.1.smithi122.stdout:5/629: dread d8/d1e/d22/d3a/d46/d4f/d55/db6/f95 [0,4194304] 0 2024-03-22T02:29:01.058 INFO:tasks.workunit.client.1.smithi122.stdout:5/630: dread - d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d zero size 2024-03-22T02:29:01.060 INFO:tasks.workunit.client.1.smithi122.stdout:7/702: rmdir d0/d25 39 2024-03-22T02:29:01.061 INFO:tasks.workunit.client.1.smithi122.stdout:9/738: dwrite f8 [0,4194304] 0 2024-03-22T02:29:01.063 INFO:tasks.workunit.client.1.smithi122.stdout:4/736: rmdir d0/d12/d61/dee 0 2024-03-22T02:29:01.064 INFO:tasks.workunit.client.1.smithi122.stdout:9/739: dread db/d1b/da8/fab [0,4194304] 0 2024-03-22T02:29:01.065 INFO:tasks.workunit.client.1.smithi122.stdout:3/643: symlink d2/d9/dc/d57/da4/dc7/lce 0 2024-03-22T02:29:01.066 INFO:tasks.workunit.client.1.smithi122.stdout:3/644: readlink d2/d9/dc/d17/d2b/l64 0 2024-03-22T02:29:01.077 INFO:tasks.workunit.client.1.smithi122.stdout:7/703: creat d0/d4/d20/d21/dda/fe1 x:0 0 0 2024-03-22T02:29:01.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/704: stat d0/d4/d94/fa5 0 2024-03-22T02:29:01.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/705: fdatasync d0/d4/d80/f92 0 2024-03-22T02:29:01.079 INFO:tasks.workunit.client.1.smithi122.stdout:4/737: mkdir d0/d25/df2 0 2024-03-22T02:29:01.081 INFO:tasks.workunit.client.1.smithi122.stdout:5/631: truncate d8/d1e/d22/d3a/d46/faa 1453688 0 2024-03-22T02:29:01.081 INFO:tasks.workunit.client.1.smithi122.stdout:1/699: getdents dc/d3b/d46/db5 0 2024-03-22T02:29:01.083 INFO:tasks.workunit.client.1.smithi122.stdout:7/706: symlink d0/d4/d1d/daf/le2 0 2024-03-22T02:29:01.084 INFO:tasks.workunit.client.1.smithi122.stdout:4/738: mkdir d0/d12/d15/d3d/d90/df3 0 2024-03-22T02:29:01.084 INFO:tasks.workunit.client.1.smithi122.stdout:9/740: link db/d1b/l1c db/d1b/d28/dc1/le7 0 2024-03-22T02:29:01.085 INFO:tasks.workunit.client.1.smithi122.stdout:1/700: dread dc/d3b/d8d/d72/d9c/db0/dab/fbd [0,4194304] 0 2024-03-22T02:29:01.085 INFO:tasks.workunit.client.1.smithi122.stdout:9/741: chown db/d23/d1f/c6f 53 1 2024-03-22T02:29:01.087 INFO:tasks.workunit.client.1.smithi122.stdout:3/645: dread d2/d9/dc/d17/f42 [0,4194304] 0 2024-03-22T02:29:01.087 INFO:tasks.workunit.client.1.smithi122.stdout:5/632: creat d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/fd9 x:0 0 0 2024-03-22T02:29:01.089 INFO:tasks.workunit.client.1.smithi122.stdout:1/701: symlink dc/d55/ld8 0 2024-03-22T02:29:01.093 INFO:tasks.workunit.client.1.smithi122.stdout:9/742: mknod db/d1b/d6c/ce8 0 2024-03-22T02:29:01.096 INFO:tasks.workunit.client.1.smithi122.stdout:4/739: link d0/d12/d15/d42/d4e/de/fc9 d0/d12/d15/d42/d4e/de/da1/dbf/ff4 0 2024-03-22T02:29:01.098 INFO:tasks.workunit.client.1.smithi122.stdout:3/646: rename d2/d9/dc/d17/d28/d4c/da5/db4/l7f to d2/d9/dc/d17/d28/d4c/da5/db4/db1/lcf 0 2024-03-22T02:29:01.100 INFO:tasks.workunit.client.1.smithi122.stdout:7/707: write d0/d4/d22/f55 [1269571,123168] 0 2024-03-22T02:29:01.103 INFO:tasks.workunit.client.1.smithi122.stdout:7/708: dread - d0/d4e/fa6 zero size 2024-03-22T02:29:01.103 INFO:tasks.workunit.client.1.smithi122.stdout:4/740: dwrite d0/d25/f8 [0,4194304] 0 2024-03-22T02:29:01.105 INFO:tasks.workunit.client.1.smithi122.stdout:9/743: mkdir db/df/d3e/d59/db5/dc4/de9 0 2024-03-22T02:29:01.106 INFO:tasks.workunit.client.1.smithi122.stdout:9/744: stat db/fb1 0 2024-03-22T02:29:01.107 INFO:tasks.workunit.client.1.smithi122.stdout:7/709: creat d0/d4/d20/d21/d9b/d7d/fe3 x:0 0 0 2024-03-22T02:29:01.112 INFO:tasks.workunit.client.1.smithi122.stdout:4/741: rename d0/d12/d15/d42/d4e/de/d40/f9d to d0/d12/d15/ff5 0 2024-03-22T02:29:01.113 INFO:tasks.workunit.client.1.smithi122.stdout:1/702: write dc/d3b/f68 [112400,45560] 0 2024-03-22T02:29:01.116 INFO:tasks.workunit.client.1.smithi122.stdout:9/745: unlink db/d23/d1f/d57/c78 0 2024-03-22T02:29:01.117 INFO:tasks.workunit.client.1.smithi122.stdout:5/633: truncate d8/d1e/d22/d3a/d46/f4b 461501 0 2024-03-22T02:29:01.123 INFO:tasks.workunit.client.1.smithi122.stdout:4/742: mknod d0/d12/d15/d42/d4e/de/da1/cf6 0 2024-03-22T02:29:01.126 INFO:tasks.workunit.client.1.smithi122.stdout:3/647: link d2/d9/dc/d17/d2b/f82 d2/d9/dc/df/d49/d85/fd0 0 2024-03-22T02:29:01.126 INFO:tasks.workunit.client.1.smithi122.stdout:9/746: dwrite db/df/d3e/d81/d25/f55 [4194304,4194304] 0 2024-03-22T02:29:01.129 INFO:tasks.workunit.client.1.smithi122.stdout:3/648: write d2/f7 [496058,113882] 0 2024-03-22T02:29:01.131 INFO:tasks.workunit.client.1.smithi122.stdout:7/710: dread d0/d30/f5b [0,4194304] 0 2024-03-22T02:29:01.132 INFO:tasks.workunit.client.1.smithi122.stdout:3/649: chown d2/d9/dc/d17/d28/d4c/da5/db4/d79/fbc 27 1 2024-03-22T02:29:01.132 INFO:tasks.workunit.client.1.smithi122.stdout:9/747: unlink db/d23/d1e/cca 0 2024-03-22T02:29:01.134 INFO:tasks.workunit.client.1.smithi122.stdout:1/703: dread dc/f22 [0,4194304] 0 2024-03-22T02:29:01.137 INFO:tasks.workunit.client.1.smithi122.stdout:6/795: sync 2024-03-22T02:29:01.142 INFO:tasks.workunit.client.1.smithi122.stdout:1/704: fsync dc/d3b/d8d/d94/f98 0 2024-03-22T02:29:01.143 INFO:tasks.workunit.client.1.smithi122.stdout:9/748: truncate db/df/d3e/d81/f42 4875068 0 2024-03-22T02:29:01.144 INFO:tasks.workunit.client.1.smithi122.stdout:9/749: stat db/d23/c18 0 2024-03-22T02:29:01.145 INFO:tasks.workunit.client.1.smithi122.stdout:4/743: dread d0/d12/d15/d42/d4e/de/f66 [0,4194304] 0 2024-03-22T02:29:01.145 INFO:tasks.workunit.client.1.smithi122.stdout:9/750: read db/fb3 [130394,84640] 0 2024-03-22T02:29:01.145 INFO:tasks.workunit.client.1.smithi122.stdout:3/650: creat d2/d9/fd1 x:0 0 0 2024-03-22T02:29:01.148 INFO:tasks.workunit.client.1.smithi122.stdout:1/705: write dc/d2b/f33 [3109627,85021] 0 2024-03-22T02:29:01.151 INFO:tasks.workunit.client.1.smithi122.stdout:1/706: read dc/d3b/d46/f82 [955325,104937] 0 2024-03-22T02:29:01.152 INFO:tasks.workunit.client.1.smithi122.stdout:4/744: rename d0/d12/d15/d42/de7 to d0/d12/d15/df7 0 2024-03-22T02:29:01.154 INFO:tasks.workunit.client.1.smithi122.stdout:5/634: dwrite d8/da/f5b [0,4194304] 0 2024-03-22T02:29:01.154 INFO:tasks.workunit.client.1.smithi122.stdout:9/751: creat db/d23/d1f/d57/d66/d73/fea x:0 0 0 2024-03-22T02:29:01.162 INFO:tasks.workunit.client.1.smithi122.stdout:4/745: dwrite d0/d25/f28 [0,4194304] 0 2024-03-22T02:29:01.164 INFO:tasks.workunit.client.1.smithi122.stdout:7/711: dwrite d0/d4e/f73 [0,4194304] 0 2024-03-22T02:29:01.165 INFO:tasks.workunit.client.1.smithi122.stdout:7/712: fsync d0/d30/f53 0 2024-03-22T02:29:01.166 INFO:tasks.workunit.client.1.smithi122.stdout:3/651: write d2/d9/dc/d17/d28/f31 [6920270,55088] 0 2024-03-22T02:29:01.173 INFO:tasks.workunit.client.1.smithi122.stdout:4/746: creat d0/d7a/ff8 x:0 0 0 2024-03-22T02:29:01.176 INFO:tasks.workunit.client.1.smithi122.stdout:1/707: write dc/d3b/d8d/d72/d9c/db0/d67/dc6/f48 [65100,55488] 0 2024-03-22T02:29:01.177 INFO:tasks.workunit.client.1.smithi122.stdout:1/708: chown dc/d4c/c74 27282 1 2024-03-22T02:29:01.178 INFO:tasks.workunit.client.1.smithi122.stdout:9/752: link db/df/d3e/d59/c6d db/df/d3e/d59/ceb 0 2024-03-22T02:29:01.179 INFO:tasks.workunit.client.1.smithi122.stdout:9/753: stat db/d23/d1e/dad/fb2 0 2024-03-22T02:29:01.181 INFO:tasks.workunit.client.1.smithi122.stdout:1/709: mkdir dc/d3b/d46/db5/dd9 0 2024-03-22T02:29:01.182 INFO:tasks.workunit.client.1.smithi122.stdout:9/754: stat db/d23/d1f/d57/d66/d73/lb9 0 2024-03-22T02:29:01.183 INFO:tasks.workunit.client.1.smithi122.stdout:7/713: link d0/d4/d1d/db8/d48/l7a d0/le4 0 2024-03-22T02:29:01.184 INFO:tasks.workunit.client.1.smithi122.stdout:9/755: mknod db/d1b/d6c/dd1/cec 0 2024-03-22T02:29:01.188 INFO:tasks.workunit.client.1.smithi122.stdout:7/714: dread d0/f16 [0,4194304] 0 2024-03-22T02:29:01.189 INFO:tasks.workunit.client.1.smithi122.stdout:9/756: rename db/d23/f11 to db/df/d3e/d59/db5/dc3/fed 0 2024-03-22T02:29:01.190 INFO:tasks.workunit.client.1.smithi122.stdout:5/635: write d8/d1e/d22/da3/fa5 [2404287,35008] 0 2024-03-22T02:29:01.192 INFO:tasks.workunit.client.1.smithi122.stdout:9/757: write db/d23/d1e/f70 [4842014,68631] 0 2024-03-22T02:29:01.196 INFO:tasks.workunit.client.1.smithi122.stdout:3/652: truncate d2/d9/dc/d17/d28/f31 1760726 0 2024-03-22T02:29:01.196 INFO:tasks.workunit.client.1.smithi122.stdout:7/715: mknod d0/d4e/de0/ce5 0 2024-03-22T02:29:01.197 INFO:tasks.workunit.client.1.smithi122.stdout:9/758: creat db/d23/fee x:0 0 0 2024-03-22T02:29:01.203 INFO:tasks.workunit.client.1.smithi122.stdout:7/716: mkdir d0/d4/de6 0 2024-03-22T02:29:01.203 INFO:tasks.workunit.client.1.smithi122.stdout:5/636: dwrite d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d [0,4194304] 0 2024-03-22T02:29:01.205 INFO:tasks.workunit.client.1.smithi122.stdout:1/710: dread dc/d3b/d8d/d72/d9c/db0/d71/fb2 [0,4194304] 0 2024-03-22T02:29:01.206 INFO:tasks.workunit.client.1.smithi122.stdout:3/653: rmdir d2/d9/dc/d17/d28/d4c/da5/db4/d79 39 2024-03-22T02:29:01.206 INFO:tasks.workunit.client.1.smithi122.stdout:4/747: dread d0/d25/fca [0,4194304] 0 2024-03-22T02:29:01.207 INFO:tasks.workunit.client.1.smithi122.stdout:5/637: chown d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d 106824 1 2024-03-22T02:29:01.207 INFO:tasks.workunit.client.1.smithi122.stdout:1/711: read dc/d3b/d8d/d72/d9c/db0/d90/f93 [675464,14862] 0 2024-03-22T02:29:01.211 INFO:tasks.workunit.client.1.smithi122.stdout:9/759: mknod db/d23/d1f/d57/dd3/cef 0 2024-03-22T02:29:01.211 INFO:tasks.workunit.client.1.smithi122.stdout:3/654: chown d2/d9/dc/d17/d28/f2d 46350 1 2024-03-22T02:29:01.215 INFO:tasks.workunit.client.1.smithi122.stdout:7/717: truncate d0/d25/f56 825519 0 2024-03-22T02:29:01.215 INFO:tasks.workunit.client.1.smithi122.stdout:7/718: fsync d0/d4/d20/d21/d9b/dbb/fc7 0 2024-03-22T02:29:01.216 INFO:tasks.workunit.client.1.smithi122.stdout:7/719: chown d0/d4/d20/l96 246 1 2024-03-22T02:29:01.217 INFO:tasks.workunit.client.1.smithi122.stdout:1/712: creat dc/d3b/d8d/d72/d9c/db0/d71/fda x:0 0 0 2024-03-22T02:29:01.217 INFO:tasks.workunit.client.1.smithi122.stdout:7/720: dread - d0/d4/d80/f92 zero size 2024-03-22T02:29:01.218 INFO:tasks.workunit.client.1.smithi122.stdout:3/655: read d2/d9/dc/d17/d45/f81 [4111148,40203] 0 2024-03-22T02:29:01.221 INFO:tasks.workunit.client.1.smithi122.stdout:3/656: unlink d2/d9/fd1 0 2024-03-22T02:29:01.222 INFO:tasks.workunit.client.1.smithi122.stdout:9/760: creat db/d1b/da8/ff0 x:0 0 0 2024-03-22T02:29:01.223 INFO:tasks.workunit.client.1.smithi122.stdout:9/761: dread - db/d23/fee zero size 2024-03-22T02:29:01.223 INFO:tasks.workunit.client.1.smithi122.stdout:7/721: mkdir d0/d4/d80/de7 0 2024-03-22T02:29:01.225 INFO:tasks.workunit.client.1.smithi122.stdout:5/638: write d8/d1e/d22/d3a/d46/f4b [1110913,117224] 0 2024-03-22T02:29:01.225 INFO:tasks.workunit.client.1.smithi122.stdout:7/722: write d0/d4/d20/d21/f95 [253658,126476] 0 2024-03-22T02:29:01.228 INFO:tasks.workunit.client.1.smithi122.stdout:1/713: rename dc/d3b/d8d/d72/d9c/db0/d67/fbe to dc/d55/fdb 0 2024-03-22T02:29:01.229 INFO:tasks.workunit.client.1.smithi122.stdout:5/639: stat d8/d1e/d22/d3a/d46/d4f/d55/db6/f95 0 2024-03-22T02:29:01.229 INFO:tasks.workunit.client.1.smithi122.stdout:4/748: dread d0/d12/d15/d2d/db5/fe9 [0,4194304] 0 2024-03-22T02:29:01.230 INFO:tasks.workunit.client.1.smithi122.stdout:1/714: write dc/f38 [5701967,29918] 0 2024-03-22T02:29:01.234 INFO:tasks.workunit.client.1.smithi122.stdout:7/723: truncate d0/d4/d20/d21/f2b 1828498 0 2024-03-22T02:29:01.235 INFO:tasks.workunit.client.1.smithi122.stdout:7/724: fsync d0/d4/d80/fdf 0 2024-03-22T02:29:01.235 INFO:tasks.workunit.client.1.smithi122.stdout:1/715: creat dc/d2b/d86/dbb/fdc x:0 0 0 2024-03-22T02:29:01.236 INFO:tasks.workunit.client.1.smithi122.stdout:9/762: link db/df/d3e/d81/d3d/c63 db/df/d3e/cf1 0 2024-03-22T02:29:01.238 INFO:tasks.workunit.client.1.smithi122.stdout:9/763: truncate db/df/d3e/d59/db5/f93 4976042 0 2024-03-22T02:29:01.238 INFO:tasks.workunit.client.1.smithi122.stdout:5/640: creat d8/d1e/d22/fda x:0 0 0 2024-03-22T02:29:01.239 INFO:tasks.workunit.client.1.smithi122.stdout:1/716: fsync dc/f4e 0 2024-03-22T02:29:01.241 INFO:tasks.workunit.client.1.smithi122.stdout:7/725: mknod d0/d4e/d5d/ce8 0 2024-03-22T02:29:01.242 INFO:tasks.workunit.client.1.smithi122.stdout:3/657: dread d2/d9/dc/df/d38/f5d [0,4194304] 0 2024-03-22T02:29:01.243 INFO:tasks.workunit.client.1.smithi122.stdout:9/764: chown db/df/d3e/d81/f42 567143504 1 2024-03-22T02:29:01.243 INFO:tasks.workunit.client.1.smithi122.stdout:9/765: read - db/d23/fee zero size 2024-03-22T02:29:01.244 INFO:tasks.workunit.client.1.smithi122.stdout:4/749: write d0/d12/d15/d42/d4e/de/fc9 [266274,124838] 0 2024-03-22T02:29:01.247 INFO:tasks.workunit.client.1.smithi122.stdout:5/641: rename d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/l60 to d8/d1e/d22/d3a/d46/d4f/d55/db6/ldb 0 2024-03-22T02:29:01.248 INFO:tasks.workunit.client.1.smithi122.stdout:7/726: fdatasync d0/d4/d1d/db8/f85 0 2024-03-22T02:29:01.251 INFO:tasks.workunit.client.1.smithi122.stdout:9/766: creat db/d23/d1f/d57/dd3/ff2 x:0 0 0 2024-03-22T02:29:01.252 INFO:tasks.workunit.client.1.smithi122.stdout:4/750: dwrite d0/d12/d61/fe5 [0,4194304] 0 2024-03-22T02:29:01.253 INFO:tasks.workunit.client.1.smithi122.stdout:9/767: write db/d23/fa9 [2274966,83683] 0 2024-03-22T02:29:01.253 INFO:tasks.workunit.client.1.smithi122.stdout:3/658: unlink d2/d9/dc/d17/d51/c95 0 2024-03-22T02:29:01.256 INFO:tasks.workunit.client.1.smithi122.stdout:5/642: rename d8/da to d8/d1e/d22/da3/ddc 0 2024-03-22T02:29:01.258 INFO:tasks.workunit.client.1.smithi122.stdout:7/727: dread d0/f2 [0,4194304] 0 2024-03-22T02:29:01.258 INFO:tasks.workunit.client.1.smithi122.stdout:9/768: symlink db/d1b/d28/dc1/lf3 0 2024-03-22T02:29:01.259 INFO:tasks.workunit.client.1.smithi122.stdout:1/717: getdents dc/d3b/d8d 0 2024-03-22T02:29:01.260 INFO:tasks.workunit.client.1.smithi122.stdout:5/643: truncate d8/d1e/d22/d3a/d7a/fb9 64698 0 2024-03-22T02:29:01.263 INFO:tasks.workunit.client.1.smithi122.stdout:9/769: mknod db/d23/dd6/cf4 0 2024-03-22T02:29:01.266 INFO:tasks.workunit.client.1.smithi122.stdout:3/659: mknod d2/d9/dc/d17/d28/d4c/da5/db4/d79/cd2 0 2024-03-22T02:29:01.269 INFO:tasks.workunit.client.1.smithi122.stdout:5/644: creat d8/d1a/fdd x:0 0 0 2024-03-22T02:29:01.276 INFO:tasks.workunit.client.1.smithi122.stdout:4/751: dread d0/d7a/fd9 [0,4194304] 0 2024-03-22T02:29:01.276 INFO:tasks.workunit.client.1.smithi122.stdout:9/770: write db/d23/d1e/dad/fb2 [715688,29023] 0 2024-03-22T02:29:01.281 INFO:tasks.workunit.client.1.smithi122.stdout:7/728: getdents d0/d4/d20/d21 0 2024-03-22T02:29:01.282 INFO:tasks.workunit.client.1.smithi122.stdout:4/752: creat d0/d12/d15/d2d/db5/ff9 x:0 0 0 2024-03-22T02:29:01.283 INFO:tasks.workunit.client.1.smithi122.stdout:1/718: dread dc/d3b/d8d/d72/d9c/db0/d67/dc6/f59 [0,4194304] 0 2024-03-22T02:29:01.284 INFO:tasks.workunit.client.1.smithi122.stdout:3/660: rename d2/d9/dc/df/d38/f6f to d2/d9/dc/d17/d28/d4c/da5/db4/fd3 0 2024-03-22T02:29:01.284 INFO:tasks.workunit.client.1.smithi122.stdout:5/645: creat d8/d1e/d22/fde x:0 0 0 2024-03-22T02:29:01.290 INFO:tasks.workunit.client.1.smithi122.stdout:5/646: mkdir d8/d1e/d22/d3a/d46/d4f/d9b/ddf 0 2024-03-22T02:29:01.291 INFO:tasks.workunit.client.1.smithi122.stdout:4/753: creat d0/d12/d15/d42/ffa x:0 0 0 2024-03-22T02:29:01.291 INFO:tasks.workunit.client.1.smithi122.stdout:9/771: dread db/df/d3e/d81/d25/d41/f61 [0,4194304] 0 2024-03-22T02:29:01.292 INFO:tasks.workunit.client.1.smithi122.stdout:9/772: chown db/d23/dd6 1240 1 2024-03-22T02:29:01.293 INFO:tasks.workunit.client.1.smithi122.stdout:4/754: read d0/f37 [45142,43910] 0 2024-03-22T02:29:01.294 INFO:tasks.workunit.client.1.smithi122.stdout:9/773: chown db/df/d3e/d81/d3d/d86/cce 3466121 1 2024-03-22T02:29:01.296 INFO:tasks.workunit.client.1.smithi122.stdout:7/729: truncate d0/d4/fb 2848122 0 2024-03-22T02:29:01.298 INFO:tasks.workunit.client.1.smithi122.stdout:5/647: link d8/d1e/d22/da3/fb5 d8/d1e/d22/fe0 0 2024-03-22T02:29:01.300 INFO:tasks.workunit.client.1.smithi122.stdout:9/774: symlink db/d1b/d6c/lf5 0 2024-03-22T02:29:01.301 INFO:tasks.workunit.client.1.smithi122.stdout:9/775: fsync fa 0 2024-03-22T02:29:01.301 INFO:tasks.workunit.client.1.smithi122.stdout:3/661: dread d2/d9/dc/d17/d2b/d6b/f8c [0,4194304] 0 2024-03-22T02:29:01.301 INFO:tasks.workunit.client.1.smithi122.stdout:5/648: creat d8/d74/fe1 x:0 0 0 2024-03-22T02:29:01.301 INFO:tasks.workunit.client.1.smithi122.stdout:4/755: dwrite d0/d25/f13 [0,4194304] 0 2024-03-22T02:29:01.304 INFO:tasks.workunit.client.1.smithi122.stdout:7/730: rename d0/d25 to d0/d4/de9 0 2024-03-22T02:29:01.304 INFO:tasks.workunit.client.1.smithi122.stdout:4/756: fdatasync d0/d25/f28 0 2024-03-22T02:29:01.304 INFO:tasks.workunit.client.1.smithi122.stdout:3/662: write d2/f15 [1428470,38137] 0 2024-03-22T02:29:01.306 INFO:tasks.workunit.client.1.smithi122.stdout:0/610: sync 2024-03-22T02:29:01.306 INFO:tasks.workunit.client.1.smithi122.stdout:2/784: sync 2024-03-22T02:29:01.306 INFO:tasks.workunit.client.1.smithi122.stdout:8/758: sync 2024-03-22T02:29:01.306 INFO:tasks.workunit.client.1.smithi122.stdout:3/663: dread - d2/d9/dc/d57/f91 zero size 2024-03-22T02:29:01.307 INFO:tasks.workunit.client.1.smithi122.stdout:4/757: fsync d0/d25/f7c 0 2024-03-22T02:29:01.309 INFO:tasks.workunit.client.1.smithi122.stdout:0/611: chown d11/d14/d23/d24/fa8 3230026 1 2024-03-22T02:29:01.310 INFO:tasks.workunit.client.1.smithi122.stdout:7/731: write d0/d4/d20/d21/d9b/dbb/fd9 [863272,17012] 0 2024-03-22T02:29:01.311 INFO:tasks.workunit.client.1.smithi122.stdout:5/649: dread d8/d3b/f62 [0,4194304] 0 2024-03-22T02:29:01.333 INFO:tasks.workunit.client.1.smithi122.stdout:8/759: link d1/f53 d1/d60/da4/f102 0 2024-03-22T02:29:01.335 INFO:tasks.workunit.client.1.smithi122.stdout:4/758: rename d0/d12/d73/fb3 to d0/d25/df2/ffb 0 2024-03-22T02:29:01.336 INFO:tasks.workunit.client.1.smithi122.stdout:2/785: truncate df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f68 4834444 0 2024-03-22T02:29:01.337 INFO:tasks.workunit.client.1.smithi122.stdout:2/786: readlink df/d1f/d33/d4d/d94/dc2/d72/le5 0 2024-03-22T02:29:01.338 INFO:tasks.workunit.client.1.smithi122.stdout:2/787: write df/d1f/d33/d4d/d94/dc2/d51/fd9 [4612448,47269] 0 2024-03-22T02:29:01.343 INFO:tasks.workunit.client.1.smithi122.stdout:9/776: dread db/d23/d1f/d57/fb8 [0,4194304] 0 2024-03-22T02:29:01.344 INFO:tasks.workunit.client.1.smithi122.stdout:9/777: stat db/d1b/d28/f6b 0 2024-03-22T02:29:01.350 INFO:tasks.workunit.client.1.smithi122.stdout:3/664: mknod d2/d9/dc/d17/d2b/d3f/cd4 0 2024-03-22T02:29:01.351 INFO:tasks.workunit.client.1.smithi122.stdout:0/612: readlink d11/d14/d1d/d27/d2f/d4e/l51 0 2024-03-22T02:29:01.354 INFO:tasks.workunit.client.1.smithi122.stdout:0/613: truncate d11/d16/d37/d9f/fa4 915754 0 2024-03-22T02:29:01.355 INFO:tasks.workunit.client.1.smithi122.stdout:0/614: dread d11/d14/d1d/d27/d2f/f3a [0,4194304] 0 2024-03-22T02:29:01.358 INFO:tasks.workunit.client.1.smithi122.stdout:9/778: symlink db/d23/d1f/d57/dd3/lf6 0 2024-03-22T02:29:01.359 INFO:tasks.workunit.client.1.smithi122.stdout:7/732: mknod d0/d4/d80/de7/cea 0 2024-03-22T02:29:01.362 INFO:tasks.workunit.client.1.smithi122.stdout:3/665: rename d2/d8a/dad to d2/d9/dc/df/dba/dd5 0 2024-03-22T02:29:01.362 INFO:tasks.workunit.client.1.smithi122.stdout:0/615: dwrite d11/d16/d37/d9f/fbd [0,4194304] 0 2024-03-22T02:29:01.364 INFO:tasks.workunit.client.1.smithi122.stdout:4/759: symlink d0/d25/de6/lfc 0 2024-03-22T02:29:01.365 INFO:tasks.workunit.client.1.smithi122.stdout:0/616: write d11/d14/d1d/d27/d32/f6c [6054869,66747] 0 2024-03-22T02:29:01.366 INFO:tasks.workunit.client.1.smithi122.stdout:9/779: read - db/d23/d1f/d57/f87 zero size 2024-03-22T02:29:01.373 INFO:tasks.workunit.client.1.smithi122.stdout:3/666: mknod d2/d9/dc/d17/d28/cd6 0 2024-03-22T02:29:01.375 INFO:tasks.workunit.client.1.smithi122.stdout:4/760: creat d0/d12/ffd x:0 0 0 2024-03-22T02:29:01.376 INFO:tasks.workunit.client.1.smithi122.stdout:7/733: dwrite d0/d4/f9 [0,4194304] 0 2024-03-22T02:29:01.378 INFO:tasks.workunit.client.1.smithi122.stdout:5/650: link d8/l7b d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/le2 0 2024-03-22T02:29:01.381 INFO:tasks.workunit.client.1.smithi122.stdout:3/667: mknod d2/d9/dc/df/dba/dd5/cd7 0 2024-03-22T02:29:01.385 INFO:tasks.workunit.client.1.smithi122.stdout:4/761: unlink d0/d12/d15/d42/d57/f58 0 2024-03-22T02:29:01.387 INFO:tasks.workunit.client.1.smithi122.stdout:7/734: unlink d0/d4/d94/fa5 0 2024-03-22T02:29:01.387 INFO:tasks.workunit.client.1.smithi122.stdout:5/651: dwrite d8/d1e/d22/d3a/d46/d4f/fca [0,4194304] 0 2024-03-22T02:29:01.389 INFO:tasks.workunit.client.1.smithi122.stdout:3/668: rmdir d2/d9/dc/d17/d45 39 2024-03-22T02:29:01.391 INFO:tasks.workunit.client.1.smithi122.stdout:4/762: rmdir d0/d12/d15/d3d/d90 39 2024-03-22T02:29:01.393 INFO:tasks.workunit.client.1.smithi122.stdout:7/735: symlink d0/d4/d80/de7/leb 0 2024-03-22T02:29:01.395 INFO:tasks.workunit.client.1.smithi122.stdout:7/736: write d0/d4/d20/d21/d9b/d5e/daa/fd1 [491028,108229] 0 2024-03-22T02:29:01.395 INFO:tasks.workunit.client.1.smithi122.stdout:5/652: creat d8/d1e/d22/d3a/d46/d4f/fe3 x:0 0 0 2024-03-22T02:29:01.399 INFO:tasks.workunit.client.1.smithi122.stdout:4/763: dread - d0/d12/d15/d2d/d75/f8a zero size 2024-03-22T02:29:01.399 INFO:tasks.workunit.client.1.smithi122.stdout:5/653: write d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f9d [487160,102993] 0 2024-03-22T02:29:01.400 INFO:tasks.workunit.client.1.smithi122.stdout:4/764: chown d0/d7a/fb6 299767829 1 2024-03-22T02:29:01.405 INFO:tasks.workunit.client.1.smithi122.stdout:4/765: truncate d0/d12/d61/dce/fd4 301415 0 2024-03-22T02:29:01.405 INFO:tasks.workunit.client.1.smithi122.stdout:5/654: mknod d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/ce4 0 2024-03-22T02:29:01.405 INFO:tasks.workunit.client.1.smithi122.stdout:4/766: dread - d0/d12/d15/d42/d4e/de/fa0 zero size 2024-03-22T02:29:01.408 INFO:tasks.workunit.client.1.smithi122.stdout:8/760: dwrite d1/d40/f9e [0,4194304] 0 2024-03-22T02:29:01.412 INFO:tasks.workunit.client.1.smithi122.stdout:4/767: truncate d0/d12/d15/f29 3788027 0 2024-03-22T02:29:01.415 INFO:tasks.workunit.client.1.smithi122.stdout:8/761: unlink d1/d8/d63/df0/lf4 0 2024-03-22T02:29:01.415 INFO:tasks.workunit.client.1.smithi122.stdout:0/617: truncate d11/d14/d1d/f80 2339128 0 2024-03-22T02:29:01.417 INFO:tasks.workunit.client.1.smithi122.stdout:9/780: truncate db/d23/d1e/dad/fb2 555857 0 2024-03-22T02:29:01.419 INFO:tasks.workunit.client.1.smithi122.stdout:0/618: rmdir d11/d47 39 2024-03-22T02:29:01.420 INFO:tasks.workunit.client.1.smithi122.stdout:0/619: chown d11/d16/d44/d89/c36 264495 1 2024-03-22T02:29:01.425 INFO:tasks.workunit.client.1.smithi122.stdout:3/669: write d2/d9/dc/f36 [5168550,85427] 0 2024-03-22T02:29:01.426 INFO:tasks.workunit.client.1.smithi122.stdout:7/737: truncate d0/d4/f9 3529025 0 2024-03-22T02:29:01.430 INFO:tasks.workunit.client.1.smithi122.stdout:7/738: mkdir d0/d30/dec 0 2024-03-22T02:29:01.431 INFO:tasks.workunit.client.1.smithi122.stdout:9/781: dwrite db/df/d3e/d81/d25/d41/f61 [0,4194304] 0 2024-03-22T02:29:01.432 INFO:tasks.workunit.client.1.smithi122.stdout:3/670: symlink d2/d9/dc/df/dba/ld8 0 2024-03-22T02:29:01.438 INFO:tasks.workunit.client.1.smithi122.stdout:5/655: dwrite d8/d3b/f62 [0,4194304] 0 2024-03-22T02:29:01.439 INFO:tasks.workunit.client.1.smithi122.stdout:5/656: stat d8/d1a/f34 0 2024-03-22T02:29:01.442 INFO:tasks.workunit.client.1.smithi122.stdout:8/762: dread d1/f3d [0,4194304] 0 2024-03-22T02:29:01.442 INFO:tasks.workunit.client.1.smithi122.stdout:4/768: truncate d0/d12/d15/d42/d4e/de/da1/dbf/ff4 316522 0 2024-03-22T02:29:01.448 INFO:tasks.workunit.client.1.smithi122.stdout:5/657: symlink d8/d1e/d22/d3a/d46/d4f/le5 0 2024-03-22T02:29:01.449 INFO:tasks.workunit.client.1.smithi122.stdout:4/769: dwrite d0/d12/d15/d2d/f4f [0,4194304] 0 2024-03-22T02:29:01.454 INFO:tasks.workunit.client.1.smithi122.stdout:5/658: dread d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d [0,4194304] 0 2024-03-22T02:29:01.458 INFO:tasks.workunit.client.1.smithi122.stdout:0/620: truncate d11/d14/d1d/d27/d59/fb2 2169304 0 2024-03-22T02:29:01.462 INFO:tasks.workunit.client.1.smithi122.stdout:8/763: fdatasync d1/d40/d48/d8d/f101 0 2024-03-22T02:29:01.463 INFO:tasks.workunit.client.1.smithi122.stdout:8/764: chown d1/d9/d50/da6/fcf 2318208 1 2024-03-22T02:29:01.465 INFO:tasks.workunit.client.1.smithi122.stdout:5/659: mkdir d8/d1e/d22/d3a/d46/d4f/d55/db6/de6 0 2024-03-22T02:29:01.467 INFO:tasks.workunit.client.1.smithi122.stdout:3/671: rename d2/d9/dc/d17/d28/f31 to d2/d9/dc/df/fd9 0 2024-03-22T02:29:01.468 INFO:tasks.workunit.client.1.smithi122.stdout:0/621: creat d11/d14/d1d/d27/d9d/fcf x:0 0 0 2024-03-22T02:29:01.470 INFO:tasks.workunit.client.1.smithi122.stdout:3/672: chown d2/d9/dc/d17/d2b/d6b/fcb 76547 1 2024-03-22T02:29:01.471 INFO:tasks.workunit.client.1.smithi122.stdout:7/739: link d0/d4/l23 d0/d4/d22/led 0 2024-03-22T02:29:01.472 INFO:tasks.workunit.client.1.smithi122.stdout:5/660: dwrite d8/d1e/d22/d3a/d39/f8b [0,4194304] 0 2024-03-22T02:29:01.473 INFO:tasks.workunit.client.1.smithi122.stdout:7/740: fdatasync d0/d4/d20/d21/d9b/d5e/daa/fd1 0 2024-03-22T02:29:01.473 INFO:tasks.workunit.client.1.smithi122.stdout:8/765: creat d1/d40/d71/d97/db8/dde/f103 x:0 0 0 2024-03-22T02:29:01.477 INFO:tasks.workunit.client.1.smithi122.stdout:7/741: write d0/d4/d80/fdf [506095,94716] 0 2024-03-22T02:29:01.479 INFO:tasks.workunit.client.1.smithi122.stdout:9/782: truncate db/d23/d1e/f70 4341965 0 2024-03-22T02:29:01.482 INFO:tasks.workunit.client.1.smithi122.stdout:8/766: symlink d1/d8/d2f/l104 0 2024-03-22T02:29:01.482 INFO:tasks.workunit.client.1.smithi122.stdout:3/673: dwrite d2/d9/dc/d57/d89/fbf [0,4194304] 0 2024-03-22T02:29:01.484 INFO:tasks.workunit.client.1.smithi122.stdout:3/674: readlink d2/d9/dc/l37 0 2024-03-22T02:29:01.486 INFO:tasks.workunit.client.1.smithi122.stdout:0/622: mkdir d11/d14/d1d/d27/dc0/dc6/dd0 0 2024-03-22T02:29:01.489 INFO:tasks.workunit.client.1.smithi122.stdout:3/675: fsync d2/d9/dc/d17/d51/f7c 0 2024-03-22T02:29:01.492 INFO:tasks.workunit.client.1.smithi122.stdout:9/783: creat db/df/d3e/d59/db5/ff7 x:0 0 0 2024-03-22T02:29:01.493 INFO:tasks.workunit.client.1.smithi122.stdout:0/623: dread d11/d16/d37/d9f/fbd [0,4194304] 0 2024-03-22T02:29:01.493 INFO:tasks.workunit.client.1.smithi122.stdout:9/784: stat db/d23/d9a 0 2024-03-22T02:29:01.496 INFO:tasks.workunit.client.1.smithi122.stdout:5/661: mkdir d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7 0 2024-03-22T02:29:01.497 INFO:tasks.workunit.client.1.smithi122.stdout:5/662: fsync d8/d1e/d22/da3/fa5 0 2024-03-22T02:29:01.499 INFO:tasks.workunit.client.1.smithi122.stdout:3/676: rename d2/d9/dc/df/dba to d2/d9/dc/df/d49/d85/dda 0 2024-03-22T02:29:01.500 INFO:tasks.workunit.client.1.smithi122.stdout:0/624: unlink d11/d14/d1b/c68 0 2024-03-22T02:29:01.501 INFO:tasks.workunit.client.1.smithi122.stdout:9/785: symlink db/d23/lf8 0 2024-03-22T02:29:01.505 INFO:tasks.workunit.client.1.smithi122.stdout:3/677: write d2/d9/dc/df/fd9 [27405,52809] 0 2024-03-22T02:29:01.509 INFO:tasks.workunit.client.1.smithi122.stdout:9/786: mknod db/d23/d1f/d57/d66/d73/cf9 0 2024-03-22T02:29:01.510 INFO:tasks.workunit.client.1.smithi122.stdout:8/767: getdents d1/d8/d63 0 2024-03-22T02:29:01.510 INFO:tasks.workunit.client.1.smithi122.stdout:9/787: fdatasync db/fb1 0 2024-03-22T02:29:01.513 INFO:tasks.workunit.client.1.smithi122.stdout:3/678: dwrite d2/d9/dc/d17/f63 [0,4194304] 0 2024-03-22T02:29:01.516 INFO:tasks.workunit.client.1.smithi122.stdout:3/679: write d2/f7 [3656756,97430] 0 2024-03-22T02:29:01.516 INFO:tasks.workunit.client.1.smithi122.stdout:5/663: dread d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/f92 [0,4194304] 0 2024-03-22T02:29:01.517 INFO:tasks.workunit.client.1.smithi122.stdout:8/768: dread d1/d60/d73/fef [0,4194304] 0 2024-03-22T02:29:01.518 INFO:tasks.workunit.client.1.smithi122.stdout:8/769: chown d1/d9 1775 1 2024-03-22T02:29:01.518 INFO:tasks.workunit.client.1.smithi122.stdout:9/788: rename db/d23/d9a/ce6 to db/df/d3e/d59/dda/cfa 0 2024-03-22T02:29:01.522 INFO:tasks.workunit.client.1.smithi122.stdout:3/680: creat d2/d9/dc/df/d49/d85/dda/fdb x:0 0 0 2024-03-22T02:29:01.525 INFO:tasks.workunit.client.1.smithi122.stdout:8/770: creat d1/d40/d48/f105 x:0 0 0 2024-03-22T02:29:01.526 INFO:tasks.workunit.client.1.smithi122.stdout:9/789: symlink db/d23/lfb 0 2024-03-22T02:29:01.527 INFO:tasks.workunit.client.1.smithi122.stdout:9/790: stat db/df/lb4 0 2024-03-22T02:29:01.530 INFO:tasks.workunit.client.1.smithi122.stdout:3/681: dread d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 [0,4194304] 0 2024-03-22T02:29:01.531 INFO:tasks.workunit.client.1.smithi122.stdout:3/682: readlink d2/d9/dc/d57/l97 0 2024-03-22T02:29:01.533 INFO:tasks.workunit.client.1.smithi122.stdout:9/791: truncate db/d1b/d28/dc1/f38 1995316 0 2024-03-22T02:29:01.534 INFO:tasks.workunit.client.1.smithi122.stdout:5/664: getdents d8/d1e/d22/d3a/d39 0 2024-03-22T02:29:01.537 INFO:tasks.workunit.client.1.smithi122.stdout:5/665: creat d8/d3b/fe8 x:0 0 0 2024-03-22T02:29:01.538 INFO:tasks.workunit.client.1.smithi122.stdout:6/796: sync 2024-03-22T02:29:01.540 INFO:tasks.workunit.client.1.smithi122.stdout:5/666: unlink d8/d1e/d22/d3a/d46/l8a 0 2024-03-22T02:29:01.541 INFO:tasks.workunit.client.1.smithi122.stdout:5/667: readlink d8/d1e/d22/da3/ddc/lb4 0 2024-03-22T02:29:01.542 INFO:tasks.workunit.client.1.smithi122.stdout:5/668: write d8/fb0 [1010872,95223] 0 2024-03-22T02:29:01.542 INFO:tasks.workunit.client.1.smithi122.stdout:6/797: creat d7/de/dd/d9b/f101 x:0 0 0 2024-03-22T02:29:01.543 INFO:tasks.workunit.client.1.smithi122.stdout:8/771: truncate d1/f3c 2246781 0 2024-03-22T02:29:01.547 INFO:tasks.workunit.client.1.smithi122.stdout:3/683: write d2/d9/dc/d17/d28/d4c/da5/db4/f76 [441685,28356] 0 2024-03-22T02:29:01.547 INFO:tasks.workunit.client.1.smithi122.stdout:8/772: getdents d1/de1 0 2024-03-22T02:29:01.552 INFO:tasks.workunit.client.1.smithi122.stdout:3/684: symlink d2/d9/dc/d57/ldc 0 2024-03-22T02:29:01.554 INFO:tasks.workunit.client.1.smithi122.stdout:8/773: rmdir d1/d9/d50/da6 39 2024-03-22T02:29:01.555 INFO:tasks.workunit.client.1.smithi122.stdout:5/669: dread d8/f26 [0,4194304] 0 2024-03-22T02:29:01.556 INFO:tasks.workunit.client.1.smithi122.stdout:5/670: chown d8/d1a/f1d 3 1 2024-03-22T02:29:01.557 INFO:tasks.workunit.client.1.smithi122.stdout:9/792: dread db/d1b/d6c/d58/d9d/fa6 [0,4194304] 0 2024-03-22T02:29:01.558 INFO:tasks.workunit.client.1.smithi122.stdout:9/793: dread - db/d23/d1f/d57/fd0 zero size 2024-03-22T02:29:01.558 INFO:tasks.workunit.client.1.smithi122.stdout:5/671: read - d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fab zero size 2024-03-22T02:29:01.559 INFO:tasks.workunit.client.1.smithi122.stdout:5/672: readlink d8/d3b/l80 0 2024-03-22T02:29:01.559 INFO:tasks.workunit.client.1.smithi122.stdout:8/774: rename d1/d4d/ldf to d1/d8/d63/df0/l106 0 2024-03-22T02:29:01.561 INFO:tasks.workunit.client.1.smithi122.stdout:3/685: rename d2/d9/dc/d17/d2b/dc9 to d2/d9/dc/d17/d2b/ddd 0 2024-03-22T02:29:01.563 INFO:tasks.workunit.client.1.smithi122.stdout:5/673: mkdir d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/de9 0 2024-03-22T02:29:01.565 INFO:tasks.workunit.client.1.smithi122.stdout:5/674: read d8/d1a/f1c [4121859,110757] 0 2024-03-22T02:29:01.565 INFO:tasks.workunit.client.1.smithi122.stdout:3/686: write d2/d9/dc/d17/d45/f81 [4925602,32330] 0 2024-03-22T02:29:01.567 INFO:tasks.workunit.client.1.smithi122.stdout:8/775: link d1/d8/l29 d1/d40/d71/l107 0 2024-03-22T02:29:01.569 INFO:tasks.workunit.client.1.smithi122.stdout:3/687: creat d2/d9/dc/d17/d28/fde x:0 0 0 2024-03-22T02:29:01.569 INFO:tasks.workunit.client.1.smithi122.stdout:5/675: write d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 [484544,73336] 0 2024-03-22T02:29:01.571 INFO:tasks.workunit.client.1.smithi122.stdout:8/776: rename d1/d60/d73/da3/fe9 to d1/d81/f108 0 2024-03-22T02:29:01.572 INFO:tasks.workunit.client.1.smithi122.stdout:3/688: creat d2/d9/dc/d17/d28/fdf x:0 0 0 2024-03-22T02:29:01.572 INFO:tasks.workunit.client.1.smithi122.stdout:8/777: chown d1/d8/d63/lee 57 1 2024-03-22T02:29:01.578 INFO:tasks.workunit.client.1.smithi122.stdout:3/689: rename d2/d9/dc/df/d3c/d59 to d2/d9/dc/d17/d51/de0 0 2024-03-22T02:29:01.580 INFO:tasks.workunit.client.1.smithi122.stdout:3/690: stat d2/d9/dc/d17/d51/de0/l72 0 2024-03-22T02:29:01.582 INFO:tasks.workunit.client.1.smithi122.stdout:3/691: readlink d2/d9/dc/df/d49/d85/dda/lc3 0 2024-03-22T02:29:01.583 INFO:tasks.workunit.client.1.smithi122.stdout:3/692: readlink d2/d9/dc/d17/d2b/l64 0 2024-03-22T02:29:01.584 INFO:tasks.workunit.client.1.smithi122.stdout:5/676: dread d8/d1a/f1c [0,4194304] 0 2024-03-22T02:29:01.588 INFO:tasks.workunit.client.1.smithi122.stdout:5/677: creat d8/d1e/d22/d3a/fea x:0 0 0 2024-03-22T02:29:01.590 INFO:tasks.workunit.client.1.smithi122.stdout:3/693: link d2/d9/dc/df/d49/d85/dda/dd5/cd7 d2/d9/dc/d17/d2b/d6b/ce1 0 2024-03-22T02:29:01.591 INFO:tasks.workunit.client.1.smithi122.stdout:8/778: dwrite d1/d40/d71/dc9/f72 [0,4194304] 0 2024-03-22T02:29:01.598 INFO:tasks.workunit.client.1.smithi122.stdout:3/694: truncate d2/d9/dc/d17/d2b/d6b/fc8 869910 0 2024-03-22T02:29:01.598 INFO:tasks.workunit.client.1.smithi122.stdout:5/678: dwrite d8/d3b/f62 [0,4194304] 0 2024-03-22T02:29:01.603 INFO:tasks.workunit.client.1.smithi122.stdout:3/695: creat d2/d9/dc/df/d3c/fe2 x:0 0 0 2024-03-22T02:29:01.605 INFO:tasks.workunit.client.1.smithi122.stdout:5/679: link d8/d1a/f34 d8/d1e/d22/d3a/d46/d4f/d55/dc7/feb 0 2024-03-22T02:29:01.607 INFO:tasks.workunit.client.1.smithi122.stdout:3/696: symlink d2/d9/dc/d17/d28/d4c/da5/db4/le3 0 2024-03-22T02:29:01.608 INFO:tasks.workunit.client.1.smithi122.stdout:5/680: symlink d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/lec 0 2024-03-22T02:29:01.609 INFO:tasks.workunit.client.1.smithi122.stdout:3/697: read d2/d9/dc/d17/d28/f6c [158197,28194] 0 2024-03-22T02:29:01.610 INFO:tasks.workunit.client.1.smithi122.stdout:3/698: stat d2/d9/dc/df/d49 0 2024-03-22T02:29:01.611 INFO:tasks.workunit.client.1.smithi122.stdout:3/699: fsync d2/d9/dc/d17/d28/f35 0 2024-03-22T02:29:01.613 INFO:tasks.workunit.client.1.smithi122.stdout:8/779: dread d1/d9/f13 [0,4194304] 0 2024-03-22T02:29:01.614 INFO:tasks.workunit.client.1.smithi122.stdout:8/780: readlink d1/d40/d51/l8e 0 2024-03-22T02:29:01.617 INFO:tasks.workunit.client.1.smithi122.stdout:8/781: write d1/d81/f108 [771060,123971] 0 2024-03-22T02:29:01.617 INFO:tasks.workunit.client.1.smithi122.stdout:8/782: chown d1/d4d/fda 24 1 2024-03-22T02:29:01.619 INFO:tasks.workunit.client.1.smithi122.stdout:3/700: creat d2/d9/dc/d57/da4/fe4 x:0 0 0 2024-03-22T02:29:01.621 INFO:tasks.workunit.client.1.smithi122.stdout:3/701: symlink d2/d9/dc/d17/d28/d4c/da5/db4/db1/le5 0 2024-03-22T02:29:01.622 INFO:tasks.workunit.client.1.smithi122.stdout:8/783: link d1/d81/fa5 d1/d8/d8a/d78/f109 0 2024-03-22T02:29:01.624 INFO:tasks.workunit.client.1.smithi122.stdout:8/784: unlink d1/fd3 0 2024-03-22T02:29:01.625 INFO:tasks.workunit.client.1.smithi122.stdout:3/702: unlink d2/d9/dc/d57/fc1 0 2024-03-22T02:29:01.625 INFO:tasks.workunit.client.1.smithi122.stdout:5/681: dread d8/d1e/d51/fb1 [0,4194304] 0 2024-03-22T02:29:01.626 INFO:tasks.workunit.client.1.smithi122.stdout:8/785: symlink d1/d40/d51/df5/l10a 0 2024-03-22T02:29:01.630 INFO:tasks.workunit.client.1.smithi122.stdout:3/703: dwrite d2/d9/dc/f2f [0,4194304] 0 2024-03-22T02:29:01.670 INFO:tasks.workunit.client.1.smithi122.stdout:8/786: mknod d1/d4d/daa/c10b 0 2024-03-22T02:29:01.674 INFO:tasks.workunit.client.1.smithi122.stdout:8/787: dread d1/d8/f75 [0,4194304] 0 2024-03-22T02:29:01.679 INFO:tasks.workunit.client.1.smithi122.stdout:8/788: link d1/d8/d63/lee d1/d8/dba/l10c 0 2024-03-22T02:29:01.680 INFO:tasks.workunit.client.1.smithi122.stdout:3/704: truncate d2/f7 49111 0 2024-03-22T02:29:01.682 INFO:tasks.workunit.client.1.smithi122.stdout:8/789: mknod d1/d9/d50/c10d 0 2024-03-22T02:29:01.682 INFO:tasks.workunit.client.1.smithi122.stdout:3/705: write d2/d9/dc/d57/f91 [434488,92932] 0 2024-03-22T02:29:01.684 INFO:tasks.workunit.client.1.smithi122.stdout:5/682: dwrite d8/d1e/d22/d3a/d46/d4f/f63 [0,4194304] 0 2024-03-22T02:29:01.686 INFO:tasks.workunit.client.1.smithi122.stdout:8/790: mknod d1/d40/d71/db4/c10e 0 2024-03-22T02:29:01.686 INFO:tasks.workunit.client.1.smithi122.stdout:5/683: stat d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c 0 2024-03-22T02:29:01.688 INFO:tasks.workunit.client.1.smithi122.stdout:8/791: mknod d1/d60/c10f 0 2024-03-22T02:29:01.693 INFO:tasks.workunit.client.1.smithi122.stdout:8/792: link d1/d8/d8a/l93 d1/d40/d51/dfa/l110 0 2024-03-22T02:29:01.695 INFO:tasks.workunit.client.1.smithi122.stdout:8/793: mkdir d1/d8/dba/dc0/d111 0 2024-03-22T02:29:01.702 INFO:tasks.workunit.client.1.smithi122.stdout:8/794: dread d1/d40/d51/fb2 [0,4194304] 0 2024-03-22T02:29:01.705 INFO:tasks.workunit.client.1.smithi122.stdout:8/795: dread d1/d8/dba/dc0/db9/fc2 [0,4194304] 0 2024-03-22T02:29:01.711 INFO:tasks.workunit.client.1.smithi122.stdout:5/684: dread d8/d1e/d22/d78/fbe [0,4194304] 0 2024-03-22T02:29:01.714 INFO:tasks.workunit.client.1.smithi122.stdout:3/706: dread d2/d9/dc/d57/f93 [0,4194304] 0 2024-03-22T02:29:01.715 INFO:tasks.workunit.client.1.smithi122.stdout:3/707: chown d2/l14 9453 1 2024-03-22T02:29:01.717 INFO:tasks.workunit.client.1.smithi122.stdout:3/708: truncate d2/d9/dc/d57/d89/fc5 594202 0 2024-03-22T02:29:01.720 INFO:tasks.workunit.client.1.smithi122.stdout:3/709: creat d2/d9/dc/d17/d2b/fe6 x:0 0 0 2024-03-22T02:29:01.722 INFO:tasks.workunit.client.1.smithi122.stdout:8/796: dread d1/d9/f24 [0,4194304] 0 2024-03-22T02:29:01.723 INFO:tasks.workunit.client.1.smithi122.stdout:8/797: stat d1 0 2024-03-22T02:29:01.724 INFO:tasks.workunit.client.1.smithi122.stdout:8/798: dread d1/d8/f75 [0,4194304] 0 2024-03-22T02:29:01.725 INFO:tasks.workunit.client.1.smithi122.stdout:8/799: dread - d1/d9/d50/fe0 zero size 2024-03-22T02:29:01.727 INFO:tasks.workunit.client.1.smithi122.stdout:8/800: creat d1/d60/d73/da3/f112 x:0 0 0 2024-03-22T02:29:01.728 INFO:tasks.workunit.client.1.smithi122.stdout:5/685: dread d8/d1e/d22/d3a/d39/f96 [0,4194304] 0 2024-03-22T02:29:01.728 INFO:tasks.workunit.client.1.smithi122.stdout:5/686: fdatasync d8/f33 0 2024-03-22T02:29:01.729 INFO:tasks.workunit.client.1.smithi122.stdout:8/801: dread d1/d8/dba/dc0/db9/fc2 [0,4194304] 0 2024-03-22T02:29:01.731 INFO:tasks.workunit.client.1.smithi122.stdout:3/710: dread d2/d9/dc/d17/d28/d4c/da5/db4/f76 [0,4194304] 0 2024-03-22T02:29:01.731 INFO:tasks.workunit.client.1.smithi122.stdout:8/802: dread - d1/d40/d51/df5/dae/dfe/fe8 zero size 2024-03-22T02:29:01.731 INFO:tasks.workunit.client.1.smithi122.stdout:5/687: mknod d8/d1e/d22/d3a/d46/d4f/d9b/da9/ced 0 2024-03-22T02:29:01.732 INFO:tasks.workunit.client.1.smithi122.stdout:3/711: chown d2/d9/dc/df/d3c/d78/caa 415128791 1 2024-03-22T02:29:01.733 INFO:tasks.workunit.client.1.smithi122.stdout:8/803: chown d1/d9/d50/da6/fcf 23593 1 2024-03-22T02:29:01.737 INFO:tasks.workunit.client.1.smithi122.stdout:3/712: unlink d2/d9/dc/d17/d2b/f82 0 2024-03-22T02:29:01.739 INFO:tasks.workunit.client.1.smithi122.stdout:5/688: dwrite d8/d1e/d22/fde [0,4194304] 0 2024-03-22T02:29:01.743 INFO:tasks.workunit.client.1.smithi122.stdout:3/713: chown d2/d9/dc/df/d38/d87/cb8 1422311 1 2024-03-22T02:29:01.743 INFO:tasks.workunit.client.1.smithi122.stdout:8/804: link d1/d40/d51/dfa/l110 d1/d40/d51/df5/dae/dfe/l113 0 2024-03-22T02:29:01.746 INFO:tasks.workunit.client.1.smithi122.stdout:3/714: dread - d2/d9/dc/fae zero size 2024-03-22T02:29:01.748 INFO:tasks.workunit.client.1.smithi122.stdout:5/689: dwrite d8/fc4 [0,4194304] 0 2024-03-22T02:29:01.753 INFO:tasks.workunit.client.1.smithi122.stdout:5/690: unlink d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/lec 0 2024-03-22T02:29:01.754 INFO:tasks.workunit.client.1.smithi122.stdout:5/691: chown d8/d1a/f1d 781369 1 2024-03-22T02:29:01.754 INFO:tasks.workunit.client.1.smithi122.stdout:8/805: truncate d1/d40/d51/df5/dae/dfe/d45/f68 1827213 0 2024-03-22T02:29:01.759 INFO:tasks.workunit.client.1.smithi122.stdout:5/692: chown d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/c3e 449522 1 2024-03-22T02:29:01.760 INFO:tasks.workunit.client.1.smithi122.stdout:5/693: truncate d8/d3b/fe8 113593 0 2024-03-22T02:29:01.760 INFO:tasks.workunit.client.1.smithi122.stdout:5/694: stat d8/l36 0 2024-03-22T02:29:01.761 INFO:tasks.workunit.client.1.smithi122.stdout:8/806: rename d1/d40/d71/d97/db8/dde to d1/d60/d73/d114 0 2024-03-22T02:29:01.765 INFO:tasks.workunit.client.1.smithi122.stdout:3/715: link d2/d9/dc/d17/d28/d4c/da5/db4/c66 d2/d9/dc/df/d3c/d78/ce7 0 2024-03-22T02:29:01.766 INFO:tasks.workunit.client.1.smithi122.stdout:3/716: dread - d2/d9/dc/d17/d2b/d6b/fb3 zero size 2024-03-22T02:29:01.768 INFO:tasks.workunit.client.1.smithi122.stdout:3/717: write d2/d9/dc/d57/f91 [939682,31976] 0 2024-03-22T02:29:01.773 INFO:tasks.workunit.client.1.smithi122.stdout:8/807: creat d1/d8/f115 x:0 0 0 2024-03-22T02:29:01.775 INFO:tasks.workunit.client.1.smithi122.stdout:3/718: dwrite d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 [0,4194304] 0 2024-03-22T02:29:01.776 INFO:tasks.workunit.client.1.smithi122.stdout:8/808: mknod d1/d8/dba/dc0/d111/c116 0 2024-03-22T02:29:01.777 INFO:tasks.workunit.client.1.smithi122.stdout:8/809: fsync d1/d40/d71/fe6 0 2024-03-22T02:29:01.779 INFO:tasks.workunit.client.1.smithi122.stdout:3/719: creat d2/d9/dc/d17/d51/fe8 x:0 0 0 2024-03-22T02:29:01.784 INFO:tasks.workunit.client.1.smithi122.stdout:3/720: truncate d2/d9/dc/d17/d28/d4c/da5/db4/f56 723735 0 2024-03-22T02:29:01.789 INFO:tasks.workunit.client.1.smithi122.stdout:8/810: dwrite d1/d8/dba/dc0/db9/fc2 [0,4194304] 0 2024-03-22T02:29:01.790 INFO:tasks.workunit.client.1.smithi122.stdout:8/811: readlink d1/d60/d73/la0 0 2024-03-22T02:29:01.791 INFO:tasks.workunit.client.1.smithi122.stdout:3/721: dread d2/d9/dc/d17/d28/f6c [0,4194304] 0 2024-03-22T02:29:01.792 INFO:tasks.workunit.client.1.smithi122.stdout:8/812: symlink d1/d8/d8a/dfb/l117 0 2024-03-22T02:29:01.797 INFO:tasks.workunit.client.1.smithi122.stdout:8/813: link d1/d9/l16 d1/d8/d8a/d78/l118 0 2024-03-22T02:29:01.798 INFO:tasks.workunit.client.1.smithi122.stdout:3/722: link d2/d9/dc/d17/d2b/d3f/l68 d2/d9/d7e/le9 0 2024-03-22T02:29:01.799 INFO:tasks.workunit.client.1.smithi122.stdout:3/723: rename d2/d9/dc to d2/d9/dc/d17/d51/dea 22 2024-03-22T02:29:01.799 INFO:tasks.workunit.client.1.smithi122.stdout:8/814: chown d1/d40/d51/dfa/l110 51966298 1 2024-03-22T02:29:01.801 INFO:tasks.workunit.client.1.smithi122.stdout:8/815: dread d1/d60/d73/fef [0,4194304] 0 2024-03-22T02:29:01.801 INFO:tasks.workunit.client.1.smithi122.stdout:3/724: mkdir d2/d9/dc/d57/dab/deb 0 2024-03-22T02:29:01.801 INFO:tasks.workunit.client.1.smithi122.stdout:8/816: stat d1/d60/l6e 0 2024-03-22T02:29:01.803 INFO:tasks.workunit.client.1.smithi122.stdout:3/725: creat d2/d9/dc/d17/d51/fec x:0 0 0 2024-03-22T02:29:01.803 INFO:tasks.workunit.client.1.smithi122.stdout:8/817: mkdir d1/d40/d71/d97/db8/df6/d119 0 2024-03-22T02:29:01.805 INFO:tasks.workunit.client.1.smithi122.stdout:8/818: symlink d1/d8/d63/df0/l11a 0 2024-03-22T02:29:01.807 INFO:tasks.workunit.client.1.smithi122.stdout:8/819: creat d1/d8/d63/df0/d6a/f11b x:0 0 0 2024-03-22T02:29:01.808 INFO:tasks.workunit.client.1.smithi122.stdout:8/820: chown d1/d4d/f58 6946 1 2024-03-22T02:29:01.808 INFO:tasks.workunit.client.1.smithi122.stdout:3/726: dread d2/d9/dc/d17/d51/f7c [0,4194304] 0 2024-03-22T02:29:01.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/727: dwrite d2/d9/dc/df/d3c/d78/fcc [0,4194304] 0 2024-03-22T02:29:01.816 INFO:tasks.workunit.client.1.smithi122.stdout:3/728: dread - d2/d9/dc/d17/d51/fec zero size 2024-03-22T02:29:01.821 INFO:tasks.workunit.client.1.smithi122.stdout:3/729: write d2/d9/dc/d17/d28/f6c [1580954,113241] 0 2024-03-22T02:29:01.821 INFO:tasks.workunit.client.1.smithi122.stdout:8/821: dwrite d1/d40/d71/d97/f9f [0,4194304] 0 2024-03-22T02:29:01.822 INFO:tasks.workunit.client.1.smithi122.stdout:8/822: chown d1/d9/d50/lb0 1695802 1 2024-03-22T02:29:01.825 INFO:tasks.workunit.client.1.smithi122.stdout:3/730: mknod d2/d9/dc/d57/dab/ced 0 2024-03-22T02:29:01.829 INFO:tasks.workunit.client.1.smithi122.stdout:8/823: dwrite d1/d8/d8a/fad [0,4194304] 0 2024-03-22T02:29:01.833 INFO:tasks.workunit.client.1.smithi122.stdout:8/824: rmdir d1/d8 39 2024-03-22T02:29:01.838 INFO:tasks.workunit.client.1.smithi122.stdout:8/825: rename d1/d8/f36 to d1/d8/d2f/f11c 0 2024-03-22T02:29:01.839 INFO:tasks.workunit.client.1.smithi122.stdout:8/826: stat d1/d40/d71/d97/db8/df6/cf9 0 2024-03-22T02:29:01.842 INFO:tasks.workunit.client.1.smithi122.stdout:8/827: unlink d1/d40/f47 0 2024-03-22T02:29:01.844 INFO:tasks.workunit.client.1.smithi122.stdout:3/731: dwrite d2/d9/dc/fd [0,4194304] 0 2024-03-22T02:29:01.845 INFO:tasks.workunit.client.1.smithi122.stdout:8/828: dread d1/d9/d50/da6/fd4 [0,4194304] 0 2024-03-22T02:29:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:01 smithi122 ceph-mon[36403]: Reconfiguring alertmanager.smithi103 (dependencies changed)... 2024-03-22T02:29:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:01 smithi122 ceph-mon[36403]: Reconfiguring daemon alertmanager.smithi103 on smithi103 2024-03-22T02:29:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:01 smithi122 ceph-mon[36403]: pgmap v24: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 116 MiB/s rd, 343 MiB/s wr, 676 op/s 2024-03-22T02:29:01.849 INFO:tasks.workunit.client.1.smithi122.stdout:8/829: fdatasync d1/d40/d51/df5/dae/dfe/d45/f49 0 2024-03-22T02:29:01.849 INFO:tasks.workunit.client.1.smithi122.stdout:3/732: dread d2/d9/dc/df/fd9 [0,4194304] 0 2024-03-22T02:29:01.851 INFO:tasks.workunit.client.1.smithi122.stdout:3/733: fsync d2/d9/dc/d17/d51/fec 0 2024-03-22T02:29:01.852 INFO:tasks.workunit.client.1.smithi122.stdout:8/830: symlink d1/d40/d51/dfa/l11d 0 2024-03-22T02:29:01.853 INFO:tasks.workunit.client.1.smithi122.stdout:3/734: mknod d2/d9/dc/d17/d2b/ddd/cee 0 2024-03-22T02:29:01.855 INFO:tasks.workunit.client.1.smithi122.stdout:8/831: unlink d1/d9/d50/fe0 0 2024-03-22T02:29:01.856 INFO:tasks.workunit.client.1.smithi122.stdout:8/832: fdatasync d1/d8/d2f/f98 0 2024-03-22T02:29:01.857 INFO:tasks.workunit.client.1.smithi122.stdout:3/735: mknod d2/d9/dc/d17/d51/de0/cef 0 2024-03-22T02:29:01.859 INFO:tasks.workunit.client.1.smithi122.stdout:8/833: rmdir d1 39 2024-03-22T02:29:01.862 INFO:tasks.workunit.client.1.smithi122.stdout:3/736: dread d2/d9/dc/df/d3c/f7b [0,4194304] 0 2024-03-22T02:29:01.863 INFO:tasks.workunit.client.1.smithi122.stdout:3/737: chown d2/d9/dc/d17/d45/f46 96 1 2024-03-22T02:29:01.864 INFO:tasks.workunit.client.1.smithi122.stdout:3/738: read - d2/d9/dc/d17/d51/fec zero size 2024-03-22T02:29:01.866 INFO:tasks.workunit.client.1.smithi122.stdout:8/834: dread d1/d9/f13 [0,4194304] 0 2024-03-22T02:29:01.867 INFO:tasks.workunit.client.1.smithi122.stdout:3/739: unlink d2/d9/dc/d17/d51/de0/l84 0 2024-03-22T02:29:01.868 INFO:tasks.workunit.client.1.smithi122.stdout:8/835: getdents d1/d40/d51/df5/dae/dfe/d45 0 2024-03-22T02:29:01.869 INFO:tasks.workunit.client.1.smithi122.stdout:8/836: stat d1/d4d/f58 0 2024-03-22T02:29:01.869 INFO:tasks.workunit.client.1.smithi122.stdout:3/740: truncate d2/f55 2448538 0 2024-03-22T02:29:01.871 INFO:tasks.workunit.client.1.smithi122.stdout:8/837: write d1/d40/d51/df5/dae/dfe/fc6 [390319,30081] 0 2024-03-22T02:29:01.871 INFO:tasks.workunit.client.1.smithi122.stdout:8/838: chown d1/d40/f4a 56 1 2024-03-22T02:29:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:01 smithi103 ceph-mon[29258]: Reconfiguring alertmanager.smithi103 (dependencies changed)... 2024-03-22T02:29:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:01 smithi103 ceph-mon[29258]: Reconfiguring daemon alertmanager.smithi103 on smithi103 2024-03-22T02:29:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:01 smithi103 ceph-mon[29258]: pgmap v24: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 116 MiB/s rd, 343 MiB/s wr, 676 op/s 2024-03-22T02:29:01.877 INFO:tasks.workunit.client.1.smithi122.stdout:3/741: link d2/d9/dc/d17/d2b/d3f/cd4 d2/d9/dc/d57/da4/cf0 0 2024-03-22T02:29:01.878 INFO:tasks.workunit.client.1.smithi122.stdout:8/839: link d1/c1c d1/d40/d51/df5/dae/c11e 0 2024-03-22T02:29:01.881 INFO:tasks.workunit.client.1.smithi122.stdout:3/742: rename d2/d9/dc/d17/d28/d4c/da5/db4/l5c to d2/d9/dc/d17/d28/lf1 0 2024-03-22T02:29:01.883 INFO:tasks.workunit.client.1.smithi122.stdout:8/840: creat d1/d8/ddc/f11f x:0 0 0 2024-03-22T02:29:01.883 INFO:tasks.workunit.client.1.smithi122.stdout:3/743: chown d2/d9/dc/df/d38/d87/f90 180243785 1 2024-03-22T02:29:01.885 INFO:tasks.workunit.client.1.smithi122.stdout:3/744: symlink d2/d9/dc/d17/d51/de0/lf2 0 2024-03-22T02:29:01.888 INFO:tasks.workunit.client.1.smithi122.stdout:3/745: creat d2/d9/dc/d17/d28/d4c/da5/db4/d79/ff3 x:0 0 0 2024-03-22T02:29:01.889 INFO:tasks.workunit.client.1.smithi122.stdout:3/746: truncate d2/f62 5225541 0 2024-03-22T02:29:01.894 INFO:tasks.workunit.client.1.smithi122.stdout:3/747: link d2/d9/dc/d17/d28/d4c/da5/db4/d79/fbc d2/d9/dc/df/d38/d87/ff4 0 2024-03-22T02:29:01.894 INFO:tasks.workunit.client.1.smithi122.stdout:8/841: dread d1/d4d/fda [0,4194304] 0 2024-03-22T02:29:01.895 INFO:tasks.workunit.client.1.smithi122.stdout:8/842: chown d1/d8/d2f/f98 4091 1 2024-03-22T02:29:01.895 INFO:tasks.workunit.client.1.smithi122.stdout:8/843: chown d1/d60/d73/da3 553754940 1 2024-03-22T02:29:01.896 INFO:tasks.workunit.client.1.smithi122.stdout:3/748: creat d2/d9/dc/df/d49/ff5 x:0 0 0 2024-03-22T02:29:01.903 INFO:tasks.workunit.client.1.smithi122.stdout:3/749: dwrite d2/d9/dc/df/d3c/d78/fcc [4194304,4194304] 0 2024-03-22T02:29:01.903 INFO:tasks.workunit.client.1.smithi122.stdout:8/844: write d1/d40/d71/dc9/f65 [1749522,125763] 0 2024-03-22T02:29:01.908 INFO:tasks.workunit.client.1.smithi122.stdout:8/845: creat d1/d40/d71/d97/f120 x:0 0 0 2024-03-22T02:29:01.909 INFO:tasks.workunit.client.1.smithi122.stdout:8/846: chown d1/d40/d51/df5/dae/dfe 6838 1 2024-03-22T02:29:01.912 INFO:tasks.workunit.client.1.smithi122.stdout:1/719: sync 2024-03-22T02:29:01.915 INFO:tasks.workunit.client.1.smithi122.stdout:3/750: dwrite d2/f7 [0,4194304] 0 2024-03-22T02:29:01.915 INFO:tasks.workunit.client.1.smithi122.stdout:1/720: mkdir dc/d2b/d86/ddd 0 2024-03-22T02:29:01.917 INFO:tasks.workunit.client.1.smithi122.stdout:1/721: write dc/d3b/d8d/d72/d9c/fae [2244674,94894] 0 2024-03-22T02:29:01.918 INFO:tasks.workunit.client.1.smithi122.stdout:3/751: mkdir d2/d8a/df6 0 2024-03-22T02:29:01.921 INFO:tasks.workunit.client.1.smithi122.stdout:1/722: creat dc/d3b/d8d/d72/d9c/db0/dab/fde x:0 0 0 2024-03-22T02:29:01.922 INFO:tasks.workunit.client.1.smithi122.stdout:3/752: dread d2/d9/dc/d57/d89/fc5 [0,4194304] 0 2024-03-22T02:29:01.923 INFO:tasks.workunit.client.1.smithi122.stdout:8/847: write d1/d8/d8a/d78/f109 [1024226,65868] 0 2024-03-22T02:29:01.924 INFO:tasks.workunit.client.1.smithi122.stdout:8/848: write d1/d40/d48/d8d/fca [2190254,45101] 0 2024-03-22T02:29:01.928 INFO:tasks.workunit.client.1.smithi122.stdout:3/753: getdents d2/d9/dc/d17/d28 0 2024-03-22T02:29:01.932 INFO:tasks.workunit.client.1.smithi122.stdout:1/723: dwrite dc/d3b/d8d/d72/d9c/db0/f7a [0,4194304] 0 2024-03-22T02:29:01.936 INFO:tasks.workunit.client.1.smithi122.stdout:8/849: dwrite d1/d40/d51/df5/dae/dfe/d45/f4b [4194304,4194304] 0 2024-03-22T02:29:01.937 INFO:tasks.workunit.client.1.smithi122.stdout:8/850: chown d1/c3 1 1 2024-03-22T02:29:01.940 INFO:tasks.workunit.client.1.smithi122.stdout:1/724: rename dc/d2b/dd2 to dc/d3b/d8d/d72/d9c/db0/ddf 0 2024-03-22T02:29:01.947 INFO:tasks.workunit.client.1.smithi122.stdout:1/725: getdents dc/d3b/d8d/d72/d9c/db0/d67 0 2024-03-22T02:29:01.950 INFO:tasks.workunit.client.1.smithi122.stdout:1/726: dread dc/d3b/d8d/d72/f96 [0,4194304] 0 2024-03-22T02:29:01.951 INFO:tasks.workunit.client.1.smithi122.stdout:1/727: truncate dc/d55/fc3 1596948 0 2024-03-22T02:29:01.952 INFO:tasks.workunit.client.1.smithi122.stdout:1/728: chown dc/d3b/d8d/d72/d9c/db0/ddf 454 1 2024-03-22T02:29:01.954 INFO:tasks.workunit.client.1.smithi122.stdout:1/729: mkdir dc/d3b/d46/de0 0 2024-03-22T02:29:01.961 INFO:tasks.workunit.client.1.smithi122.stdout:3/754: dwrite d2/f55 [0,4194304] 0 2024-03-22T02:29:01.965 INFO:tasks.workunit.client.1.smithi122.stdout:3/755: truncate d2/d9/dc/df/f53 1398753 0 2024-03-22T02:29:01.965 INFO:tasks.workunit.client.1.smithi122.stdout:8/851: dread d1/f3c [0,4194304] 0 2024-03-22T02:29:02.014 INFO:tasks.workunit.client.1.smithi122.stdout:4/770: sync 2024-03-22T02:29:02.014 INFO:tasks.workunit.client.1.smithi122.stdout:2/788: sync 2024-03-22T02:29:02.015 INFO:tasks.workunit.client.1.smithi122.stdout:8/852: unlink d1/d8/d63/df0/d6a/fa2 0 2024-03-22T02:29:02.016 INFO:tasks.workunit.client.1.smithi122.stdout:8/853: stat d1/d40/d51/df5/dae/dfe/c4c 0 2024-03-22T02:29:02.017 INFO:tasks.workunit.client.1.smithi122.stdout:4/771: truncate d0/d12/d15/d2d/db5/fc8 190798 0 2024-03-22T02:29:02.018 INFO:tasks.workunit.client.1.smithi122.stdout:3/756: read d2/d9/dc/df/d49/d85/fd0 [564482,127351] 0 2024-03-22T02:29:02.019 INFO:tasks.workunit.client.1.smithi122.stdout:8/854: mknod d1/d40/d51/df5/dae/dfe/d45/c121 0 2024-03-22T02:29:02.020 INFO:tasks.workunit.client.1.smithi122.stdout:4/772: dread - d0/d12/d15/d42/d4e/de/f85 zero size 2024-03-22T02:29:02.025 INFO:tasks.workunit.client.1.smithi122.stdout:3/757: rename d2/d9/dc/df/d49/d85/dda/fdb to d2/d9/dc/d17/d28/d4c/da5/db4/d79/ff7 0 2024-03-22T02:29:02.033 INFO:tasks.workunit.client.1.smithi122.stdout:3/758: rename d2/d9/dc/d17/d28/d4c/da5/db4/f76 to d2/d9/dc/d57/ff8 0 2024-03-22T02:29:02.034 INFO:tasks.workunit.client.1.smithi122.stdout:8/855: dwrite d1/d8/f75 [0,4194304] 0 2024-03-22T02:29:02.035 INFO:tasks.workunit.client.1.smithi122.stdout:4/773: truncate d0/d12/d15/d2d/f4f 3403900 0 2024-03-22T02:29:02.036 INFO:tasks.workunit.client.1.smithi122.stdout:8/856: fsync d1/d60/d73/da3/f112 0 2024-03-22T02:29:02.038 INFO:tasks.workunit.client.1.smithi122.stdout:2/789: dread df/d1f/d33/d4d/d7a/f8a [0,4194304] 0 2024-03-22T02:29:02.038 INFO:tasks.workunit.client.1.smithi122.stdout:3/759: mkdir d2/d9/dc/df/d49/d85/dda/dd5/df9 0 2024-03-22T02:29:02.041 INFO:tasks.workunit.client.1.smithi122.stdout:4/774: link d0/d12/d15/d42/d4e/de/d40/cbd d0/d12/d15/d42/df0/cfe 0 2024-03-22T02:29:02.042 INFO:tasks.workunit.client.1.smithi122.stdout:3/760: write d2/d9/dc/d17/d2b/d6b/fcb [160558,103721] 0 2024-03-22T02:29:02.044 INFO:tasks.workunit.client.1.smithi122.stdout:4/775: rmdir d0/d25/de6 39 2024-03-22T02:29:02.045 INFO:tasks.workunit.client.1.smithi122.stdout:2/790: creat df/d1f/d33/d4d/d94/df0/f105 x:0 0 0 2024-03-22T02:29:02.045 INFO:tasks.workunit.client.1.smithi122.stdout:8/857: dwrite d1/d40/d71/d97/f120 [0,4194304] 0 2024-03-22T02:29:02.048 INFO:tasks.workunit.client.1.smithi122.stdout:3/761: getdents d2/d9/dc/d57 0 2024-03-22T02:29:02.051 INFO:tasks.workunit.client.1.smithi122.stdout:8/858: mkdir d1/d40/d71/d122 0 2024-03-22T02:29:02.057 INFO:tasks.workunit.client.1.smithi122.stdout:4/776: rename d0/d12/d15/d42/f7b to d0/d12/fff 0 2024-03-22T02:29:02.058 INFO:tasks.workunit.client.1.smithi122.stdout:8/859: mknod d1/d8/dba/c123 0 2024-03-22T02:29:02.061 INFO:tasks.workunit.client.1.smithi122.stdout:3/762: fdatasync d2/d9/fc4 0 2024-03-22T02:29:02.064 INFO:tasks.workunit.client.1.smithi122.stdout:8/860: rename d1/d8/d63/df0/l35 to d1/d40/d71/d97/l124 0 2024-03-22T02:29:02.069 INFO:tasks.workunit.client.1.smithi122.stdout:3/763: rename d2/f5e to d2/d8a/df6/ffa 0 2024-03-22T02:29:02.070 INFO:tasks.workunit.client.1.smithi122.stdout:8/861: readlink d1/d40/d51/l99 0 2024-03-22T02:29:02.072 INFO:tasks.workunit.client.1.smithi122.stdout:2/791: dwrite df/d1f/d33/d4d/d94/dc2/d51/d67/f71 [0,4194304] 0 2024-03-22T02:29:02.076 INFO:tasks.workunit.client.1.smithi122.stdout:3/764: dread d2/d9/dc/df/d3c/f7b [4194304,4194304] 0 2024-03-22T02:29:02.078 INFO:tasks.workunit.client.1.smithi122.stdout:4/777: dwrite d0/d25/f3b [4194304,4194304] 0 2024-03-22T02:29:02.080 INFO:tasks.workunit.client.1.smithi122.stdout:4/778: dread - d0/d12/d15/d2d/f7d zero size 2024-03-22T02:29:02.082 INFO:tasks.workunit.client.1.smithi122.stdout:4/779: truncate d0/f9 9005568 0 2024-03-22T02:29:02.083 INFO:tasks.workunit.client.1.smithi122.stdout:2/792: symlink df/d1f/d33/d4d/d94/d9e/l106 0 2024-03-22T02:29:02.086 INFO:tasks.workunit.client.1.smithi122.stdout:4/780: symlink d0/d12/d15/d42/d4e/de/da1/l100 0 2024-03-22T02:29:02.086 INFO:tasks.workunit.client.1.smithi122.stdout:8/862: link d1/d40/d71/db4/c10e d1/d8/dba/dc0/d111/c125 0 2024-03-22T02:29:02.086 INFO:tasks.workunit.client.1.smithi122.stdout:3/765: dwrite d2/d9/dc/df/d38/f3b [0,4194304] 0 2024-03-22T02:29:02.089 INFO:tasks.workunit.client.1.smithi122.stdout:0/625: sync 2024-03-22T02:29:02.089 INFO:tasks.workunit.client.1.smithi122.stdout:5/695: sync 2024-03-22T02:29:02.089 INFO:tasks.workunit.client.1.smithi122.stdout:9/794: sync 2024-03-22T02:29:02.089 INFO:tasks.workunit.client.1.smithi122.stdout:7/742: sync 2024-03-22T02:29:02.089 INFO:tasks.workunit.client.1.smithi122.stdout:6/798: sync 2024-03-22T02:29:02.093 INFO:tasks.workunit.client.1.smithi122.stdout:5/696: chown d8/f26 1517778 1 2024-03-22T02:29:02.095 INFO:tasks.workunit.client.1.smithi122.stdout:0/626: truncate d11/d16/d44/d89/f3e 1806685 0 2024-03-22T02:29:02.097 INFO:tasks.workunit.client.1.smithi122.stdout:4/781: fsync d0/d12/fdd 0 2024-03-22T02:29:02.099 INFO:tasks.workunit.client.1.smithi122.stdout:7/743: symlink d0/d4/d1d/lee 0 2024-03-22T02:29:02.101 INFO:tasks.workunit.client.1.smithi122.stdout:3/766: dwrite d2/d9/dc/d17/d51/f7c [0,4194304] 0 2024-03-22T02:29:02.101 INFO:tasks.workunit.client.1.smithi122.stdout:8/863: creat d1/de1/f126 x:0 0 0 2024-03-22T02:29:02.103 INFO:tasks.workunit.client.1.smithi122.stdout:9/795: mkdir db/df/d3e/dfc 0 2024-03-22T02:29:02.103 INFO:tasks.workunit.client.1.smithi122.stdout:5/697: dread d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 [0,4194304] 0 2024-03-22T02:29:02.107 INFO:tasks.workunit.client.1.smithi122.stdout:0/627: mkdir d11/d14/d1d/d27/d2f/dd1 0 2024-03-22T02:29:02.110 INFO:tasks.workunit.client.1.smithi122.stdout:6/799: link d7/de/lef d7/de/dd/d20/de0/l102 0 2024-03-22T02:29:02.115 INFO:tasks.workunit.client.1.smithi122.stdout:7/744: rename d0/d4/d22 to d0/d4/d20/d21/d9b/d5e/def 0 2024-03-22T02:29:02.118 INFO:tasks.workunit.client.1.smithi122.stdout:4/782: rmdir d0/d12/d15/d3d/d90 39 2024-03-22T02:29:02.122 INFO:tasks.workunit.client.1.smithi122.stdout:2/793: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f70 [0,4194304] 0 2024-03-22T02:29:02.122 INFO:tasks.workunit.client.1.smithi122.stdout:6/800: dwrite d7/de/dd/d20/de0/f30 [0,4194304] 0 2024-03-22T02:29:02.123 INFO:tasks.workunit.client.1.smithi122.stdout:5/698: creat d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/fee x:0 0 0 2024-03-22T02:29:02.124 INFO:tasks.workunit.client.1.smithi122.stdout:1/730: sync 2024-03-22T02:29:02.128 INFO:tasks.workunit.client.1.smithi122.stdout:9/796: link db/d1b/d6c/d58/d9d/fa6 db/df/d3e/d81/d25/d41/ffd 0 2024-03-22T02:29:02.130 INFO:tasks.workunit.client.1.smithi122.stdout:2/794: dread df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f82 [0,4194304] 0 2024-03-22T02:29:02.131 INFO:tasks.workunit.client.1.smithi122.stdout:3/767: write d2/d9/fc4 [40464,27833] 0 2024-03-22T02:29:02.134 INFO:tasks.workunit.client.1.smithi122.stdout:4/783: symlink d0/d12/d15/d2d/d75/l101 0 2024-03-22T02:29:02.134 INFO:tasks.workunit.client.1.smithi122.stdout:8/864: truncate d1/d9/f13 1895073 0 2024-03-22T02:29:02.136 INFO:tasks.workunit.client.1.smithi122.stdout:6/801: dwrite d7/de/dd/d20/de0/d54/f9d [0,4194304] 0 2024-03-22T02:29:02.137 INFO:tasks.workunit.client.1.smithi122.stdout:5/699: symlink d8/d1e/d22/d3a/d7a/lef 0 2024-03-22T02:29:02.138 INFO:tasks.workunit.client.1.smithi122.stdout:3/768: truncate d2/d9/dc/d57/f91 1146201 0 2024-03-22T02:29:02.139 INFO:tasks.workunit.client.1.smithi122.stdout:1/731: fdatasync dc/d2b/f91 0 2024-03-22T02:29:02.154 INFO:tasks.workunit.client.1.smithi122.stdout:3/769: stat d2/d9/dc/df/d49/d85/dda/dd5/cd7 0 2024-03-22T02:29:02.155 INFO:tasks.workunit.client.1.smithi122.stdout:8/865: dwrite d1/d8/dba/dc0/ff3 [0,4194304] 0 2024-03-22T02:29:02.156 INFO:tasks.workunit.client.1.smithi122.stdout:0/628: link c1 d11/d14/d23/d24/cd2 0 2024-03-22T02:29:02.159 INFO:tasks.workunit.client.1.smithi122.stdout:7/745: rename d0/d4/d1d/db8/d9a/l9f to d0/d4/de9/lf0 0 2024-03-22T02:29:02.163 INFO:tasks.workunit.client.1.smithi122.stdout:2/795: dread df/d1f/d33/d6a/f8d [0,4194304] 0 2024-03-22T02:29:02.164 INFO:tasks.workunit.client.1.smithi122.stdout:9/797: creat db/d1b/d6c/d58/d9d/ffe x:0 0 0 2024-03-22T02:29:02.165 INFO:tasks.workunit.client.1.smithi122.stdout:9/798: fsync db/df/f80 0 2024-03-22T02:29:02.165 INFO:tasks.workunit.client.1.smithi122.stdout:6/802: write f6 [8077371,77298] 0 2024-03-22T02:29:02.166 INFO:tasks.workunit.client.1.smithi122.stdout:2/796: write df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f70 [265972,95757] 0 2024-03-22T02:29:02.166 INFO:tasks.workunit.client.1.smithi122.stdout:3/770: readlink d2/d9/dc/l33 0 2024-03-22T02:29:02.170 INFO:tasks.workunit.client.1.smithi122.stdout:2/797: chown df/d1f/d23/l8b 1160514295 1 2024-03-22T02:29:02.170 INFO:tasks.workunit.client.1.smithi122.stdout:1/732: dread dc/d3b/d8d/d72/d9c/fae [0,4194304] 0 2024-03-22T02:29:02.172 INFO:tasks.workunit.client.1.smithi122.stdout:7/746: creat d0/d4/d20/d21/d9b/dbb/ff1 x:0 0 0 2024-03-22T02:29:02.178 INFO:tasks.workunit.client.1.smithi122.stdout:8/866: mkdir d1/d40/d51/d127 0 2024-03-22T02:29:02.179 INFO:tasks.workunit.client.1.smithi122.stdout:4/784: dwrite d0/d12/d15/d2d/f4f [0,4194304] 0 2024-03-22T02:29:02.181 INFO:tasks.workunit.client.1.smithi122.stdout:8/867: write d1/d8/f100 [312711,52475] 0 2024-03-22T02:29:02.182 INFO:tasks.workunit.client.1.smithi122.stdout:2/798: unlink df/d1f/d33/d34/c40 0 2024-03-22T02:29:02.184 INFO:tasks.workunit.client.1.smithi122.stdout:3/771: dread d2/d9/dc/d17/d28/d4c/da5/db4/f56 [0,4194304] 0 2024-03-22T02:29:02.186 INFO:tasks.workunit.client.1.smithi122.stdout:7/747: dwrite d0/d4/d1d/db8/f71 [0,4194304] 0 2024-03-22T02:29:02.187 INFO:tasks.workunit.client.1.smithi122.stdout:5/700: getdents d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e 0 2024-03-22T02:29:02.187 INFO:tasks.workunit.client.1.smithi122.stdout:9/799: symlink db/df/d3e/d81/d25/lff 0 2024-03-22T02:29:02.190 INFO:tasks.workunit.client.1.smithi122.stdout:1/733: mknod dc/d3b/d46/de0/ce1 0 2024-03-22T02:29:02.191 INFO:tasks.workunit.client.1.smithi122.stdout:0/629: creat d11/d16/d37/d84/d87/fd3 x:0 0 0 2024-03-22T02:29:02.192 INFO:tasks.workunit.client.1.smithi122.stdout:4/785: write d0/d12/d15/d42/d4e/de/f8b [1712507,44710] 0 2024-03-22T02:29:02.195 INFO:tasks.workunit.client.1.smithi122.stdout:7/748: fdatasync d0/d4/d20/d21/d9b/dbb/fd9 0 2024-03-22T02:29:02.197 INFO:tasks.workunit.client.1.smithi122.stdout:6/803: truncate d7/de/dd/d20/de0/d54/d76/f89 2244990 0 2024-03-22T02:29:02.197 INFO:tasks.workunit.client.1.smithi122.stdout:5/701: mknod d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/cf0 0 2024-03-22T02:29:02.198 INFO:tasks.workunit.client.1.smithi122.stdout:0/630: creat d11/d14/d23/d24/d33/d67/dc2/fd4 x:0 0 0 2024-03-22T02:29:02.200 INFO:tasks.workunit.client.1.smithi122.stdout:9/800: mkdir db/df/d100 0 2024-03-22T02:29:02.200 INFO:tasks.workunit.client.1.smithi122.stdout:4/786: mknod d0/d12/d15/d42/d4e/de/d40/c102 0 2024-03-22T02:29:02.201 INFO:tasks.workunit.client.1.smithi122.stdout:9/801: chown db/d23/f83 1539987 1 2024-03-22T02:29:02.202 INFO:tasks.workunit.client.1.smithi122.stdout:3/772: symlink d2/d9/dc/df/d49/d85/dda/dd5/df9/lfb 0 2024-03-22T02:29:02.204 INFO:tasks.workunit.client.1.smithi122.stdout:7/749: creat d0/d4/d80/dc4/ff2 x:0 0 0 2024-03-22T02:29:02.206 INFO:tasks.workunit.client.1.smithi122.stdout:1/734: truncate dc/d4c/f62 3987679 0 2024-03-22T02:29:02.208 INFO:tasks.workunit.client.1.smithi122.stdout:6/804: truncate d7/de/dd/f56 453271 0 2024-03-22T02:29:02.210 INFO:tasks.workunit.client.1.smithi122.stdout:0/631: creat d11/d14/d23/d24/d33/fd5 x:0 0 0 2024-03-22T02:29:02.212 INFO:tasks.workunit.client.1.smithi122.stdout:6/805: chown d7/de/dd/l78 7824 1 2024-03-22T02:29:02.212 INFO:tasks.workunit.client.1.smithi122.stdout:3/773: creat d2/d9/d7e/ffc x:0 0 0 2024-03-22T02:29:02.214 INFO:tasks.workunit.client.1.smithi122.stdout:9/802: dwrite db/d1b/da8/fb7 [0,4194304] 0 2024-03-22T02:29:02.216 INFO:tasks.workunit.client.1.smithi122.stdout:7/750: unlink d0/d4/d20/d21/d9b/d5e/def/c8b 0 2024-03-22T02:29:02.219 INFO:tasks.workunit.client.1.smithi122.stdout:8/868: dread d1/d60/da4/f102 [0,4194304] 0 2024-03-22T02:29:02.219 INFO:tasks.workunit.client.1.smithi122.stdout:2/799: dread df/d1f/d23/dd2/f96 [0,4194304] 0 2024-03-22T02:29:02.220 INFO:tasks.workunit.client.1.smithi122.stdout:7/751: write d0/d4/d20/d21/d9b/d7d/fb7 [663265,24949] 0 2024-03-22T02:29:02.221 INFO:tasks.workunit.client.1.smithi122.stdout:6/806: stat d7/de/dd/d23/f63 0 2024-03-22T02:29:02.225 INFO:tasks.workunit.client.1.smithi122.stdout:9/803: readlink db/d1b/d6c/l75 0 2024-03-22T02:29:02.225 INFO:tasks.workunit.client.1.smithi122.stdout:1/735: dwrite dc/fd [8388608,4194304] 0 2024-03-22T02:29:02.227 INFO:tasks.workunit.client.1.smithi122.stdout:9/804: chown db/d23/d1f/d57/d66 216689873 1 2024-03-22T02:29:02.229 INFO:tasks.workunit.client.1.smithi122.stdout:9/805: chown db/d1b/da8/fcd 2858 1 2024-03-22T02:29:02.230 INFO:tasks.workunit.client.1.smithi122.stdout:0/632: mknod d11/d14/d1d/d27/dc0/dc6/dd0/cd6 0 2024-03-22T02:29:02.234 INFO:tasks.workunit.client.1.smithi122.stdout:7/752: rmdir d0/d4e/de0 39 2024-03-22T02:29:02.234 INFO:tasks.workunit.client.1.smithi122.stdout:0/633: chown d11/f40 491 1 2024-03-22T02:29:02.234 INFO:tasks.workunit.client.1.smithi122.stdout:8/869: mknod d1/d60/d73/da3/c128 0 2024-03-22T02:29:02.234 INFO:tasks.workunit.client.1.smithi122.stdout:3/774: dwrite d2/d9/dc/df/d3c/f7b [4194304,4194304] 0 2024-03-22T02:29:02.236 INFO:tasks.workunit.client.1.smithi122.stdout:5/702: link d8/d1e/d22/d3a/d46/d4f/ca7 d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/cf1 0 2024-03-22T02:29:02.239 INFO:tasks.workunit.client.1.smithi122.stdout:1/736: symlink dc/d3b/d8d/d72/d9c/db0/le2 0 2024-03-22T02:29:02.242 INFO:tasks.workunit.client.1.smithi122.stdout:7/753: creat d0/d4e/d5d/ff3 x:0 0 0 2024-03-22T02:29:02.243 INFO:tasks.workunit.client.1.smithi122.stdout:0/634: symlink d11/d14/d23/d24/d33/d67/ld7 0 2024-03-22T02:29:02.244 INFO:tasks.workunit.client.1.smithi122.stdout:0/635: dread - d11/d14/d1d/d27/d2f/fac zero size 2024-03-22T02:29:02.245 INFO:tasks.workunit.client.1.smithi122.stdout:8/870: creat d1/d8/d63/f129 x:0 0 0 2024-03-22T02:29:02.248 INFO:tasks.workunit.client.1.smithi122.stdout:5/703: mknod d8/d1e/d51/cf2 0 2024-03-22T02:29:02.250 INFO:tasks.workunit.client.1.smithi122.stdout:2/800: creat df/d1f/d33/d4d/f107 x:0 0 0 2024-03-22T02:29:02.252 INFO:tasks.workunit.client.1.smithi122.stdout:2/801: dread - df/d1f/d33/d34/ff4 zero size 2024-03-22T02:29:02.252 INFO:tasks.workunit.client.1.smithi122.stdout:7/754: mkdir d0/d4/de9/df4 0 2024-03-22T02:29:02.256 INFO:tasks.workunit.client.1.smithi122.stdout:5/704: truncate d8/d1e/d22/da3/ddc/f10 2371357 0 2024-03-22T02:29:02.257 INFO:tasks.workunit.client.1.smithi122.stdout:6/807: dread d7/fdc [0,4194304] 0 2024-03-22T02:29:02.257 INFO:tasks.workunit.client.1.smithi122.stdout:5/705: chown d8/d1a/ca1 1943503 1 2024-03-22T02:29:02.258 INFO:tasks.workunit.client.1.smithi122.stdout:7/755: mkdir d0/d4e/d5d/df5 0 2024-03-22T02:29:02.259 INFO:tasks.workunit.client.1.smithi122.stdout:6/808: write d7/de/d27/d52/fac [1435165,173] 0 2024-03-22T02:29:02.262 INFO:tasks.workunit.client.1.smithi122.stdout:9/806: dread db/df/d3e/d81/d3d/d86/dd8/fd9 [0,4194304] 0 2024-03-22T02:29:02.266 INFO:tasks.workunit.client.1.smithi122.stdout:8/871: getdents d1/d40 0 2024-03-22T02:29:02.269 INFO:tasks.workunit.client.1.smithi122.stdout:6/809: dwrite d7/de/dd/d39/dc2/ff7 [0,4194304] 0 2024-03-22T02:29:02.271 INFO:tasks.workunit.client.1.smithi122.stdout:9/807: creat db/d23/d1e/f101 x:0 0 0 2024-03-22T02:29:02.271 INFO:tasks.workunit.client.1.smithi122.stdout:0/636: dread d11/d16/d44/d89/f4b [0,4194304] 0 2024-03-22T02:29:02.275 INFO:tasks.workunit.client.1.smithi122.stdout:6/810: dread - d7/de/dd/d20/d8f/fe7 zero size 2024-03-22T02:29:02.275 INFO:tasks.workunit.client.1.smithi122.stdout:5/706: creat d8/d1a/d83/dcb/ff3 x:0 0 0 2024-03-22T02:29:02.278 INFO:tasks.workunit.client.1.smithi122.stdout:2/802: creat df/d1f/d33/d4d/f108 x:0 0 0 2024-03-22T02:29:02.278 INFO:tasks.workunit.client.1.smithi122.stdout:0/637: symlink d11/d16/d44/ld8 0 2024-03-22T02:29:02.279 INFO:tasks.workunit.client.1.smithi122.stdout:7/756: dwrite d0/d4/d20/d57/fbf [0,4194304] 0 2024-03-22T02:29:02.280 INFO:tasks.workunit.client.1.smithi122.stdout:6/811: fdatasync d7/de/dd/d9b/f101 0 2024-03-22T02:29:02.282 INFO:tasks.workunit.client.1.smithi122.stdout:9/808: rename db/df/c24 to db/df/d3e/d59/db5/c102 0 2024-03-22T02:29:02.283 INFO:tasks.workunit.client.1.smithi122.stdout:0/638: write d11/d14/d1d/d27/d59/f9b [721794,62194] 0 2024-03-22T02:29:02.283 INFO:tasks.workunit.client.1.smithi122.stdout:7/757: chown d0/d4/d20/d21/dda/fe1 13154491 1 2024-03-22T02:29:02.284 INFO:tasks.workunit.client.1.smithi122.stdout:8/872: dread d1/d40/f5b [4194304,4194304] 0 2024-03-22T02:29:02.289 INFO:tasks.workunit.client.1.smithi122.stdout:8/873: chown d1/d40/d51/df5/dae/dfe/d45/c121 2 1 2024-03-22T02:29:02.291 INFO:tasks.workunit.client.1.smithi122.stdout:9/809: creat db/d1b/d6c/d58/f103 x:0 0 0 2024-03-22T02:29:02.292 INFO:tasks.workunit.client.1.smithi122.stdout:2/803: dread df/d1f/d33/d4d/d7a/f8a [0,4194304] 0 2024-03-22T02:29:02.293 INFO:tasks.workunit.client.1.smithi122.stdout:8/874: truncate d1/d40/d71/fe6 860970 0 2024-03-22T02:29:02.294 INFO:tasks.workunit.client.1.smithi122.stdout:9/810: write db/d23/d1f/d57/d66/d73/f76 [8136298,4978] 0 2024-03-22T02:29:02.294 INFO:tasks.workunit.client.1.smithi122.stdout:2/804: stat df/d1f/d33/d4d/d94/dc2/d72/le5 0 2024-03-22T02:29:02.295 INFO:tasks.workunit.client.1.smithi122.stdout:0/639: link d11/d16/d37/c64 d11/d14/d1d/d27/d32/cd9 0 2024-03-22T02:29:02.298 INFO:tasks.workunit.client.1.smithi122.stdout:8/875: unlink d1/d40/d51/df5/dae/dfe/d45/f49 0 2024-03-22T02:29:02.299 INFO:tasks.workunit.client.1.smithi122.stdout:7/758: dread d0/d4/d20/d21/f2b [0,4194304] 0 2024-03-22T02:29:02.300 INFO:tasks.workunit.client.1.smithi122.stdout:9/811: chown db/d23/d1e/dad/fb2 83 1 2024-03-22T02:29:02.301 INFO:tasks.workunit.client.1.smithi122.stdout:7/759: fsync d0/d4/d94/fa9 0 2024-03-22T02:29:02.305 INFO:tasks.workunit.client.1.smithi122.stdout:6/812: write d7/de/f43 [695561,99228] 0 2024-03-22T02:29:02.305 INFO:tasks.workunit.client.1.smithi122.stdout:0/640: dwrite d11/d14/d23/d24/fca [0,4194304] 0 2024-03-22T02:29:02.308 INFO:tasks.workunit.client.1.smithi122.stdout:0/641: write d11/d14/d23/f72 [1336773,129309] 0 2024-03-22T02:29:02.309 INFO:tasks.workunit.client.1.smithi122.stdout:8/876: readlink d1/d8/d63/df0/l11a 0 2024-03-22T02:29:02.343 INFO:tasks.workunit.client.1.smithi122.stdout:7/760: rename d0/d4e to d0/d4/d1d/df6 0 2024-03-22T02:29:02.345 INFO:tasks.workunit.client.1.smithi122.stdout:5/707: dwrite d8/d3b/f69 [0,4194304] 0 2024-03-22T02:29:02.362 INFO:tasks.workunit.client.1.smithi122.stdout:2/805: getdents df/d1f/d33/d4d/d94/d9e 0 2024-03-22T02:29:02.363 INFO:tasks.workunit.client.1.smithi122.stdout:2/806: chown df/d1f/d33/d4d/d94/dc2 293 1 2024-03-22T02:29:02.364 INFO:tasks.workunit.client.1.smithi122.stdout:9/812: creat db/d23/d1f/d57/d66/f104 x:0 0 0 2024-03-22T02:29:02.365 INFO:tasks.workunit.client.1.smithi122.stdout:0/642: mkdir d11/d16/d37/dda 0 2024-03-22T02:29:02.367 INFO:tasks.workunit.client.1.smithi122.stdout:8/877: mknod d1/d40/d51/c12a 0 2024-03-22T02:29:02.368 INFO:tasks.workunit.client.1.smithi122.stdout:8/878: chown d1/d40/d51/dfa/l110 0 1 2024-03-22T02:29:02.369 INFO:tasks.workunit.client.1.smithi122.stdout:9/813: mkdir db/d1b/d28/dc1/d105 0 2024-03-22T02:29:02.370 INFO:tasks.workunit.client.1.smithi122.stdout:0/643: truncate d11/d14/d1d/d27/d2f/f3a 659007 0 2024-03-22T02:29:02.371 INFO:tasks.workunit.client.1.smithi122.stdout:6/813: getdents d7/de/dd/d23/d7f 0 2024-03-22T02:29:02.371 INFO:tasks.workunit.client.1.smithi122.stdout:0/644: dread - d11/d14/d1d/d27/d59/fb0 zero size 2024-03-22T02:29:02.373 INFO:tasks.workunit.client.1.smithi122.stdout:6/814: readlink d7/de/dd/l26 0 2024-03-22T02:29:02.374 INFO:tasks.workunit.client.1.smithi122.stdout:7/761: link d0/d4/d94/fc5 d0/d4/d20/d21/d9b/ff7 0 2024-03-22T02:29:02.374 INFO:tasks.workunit.client.1.smithi122.stdout:0/645: fsync d11/d14/d1d/d27/d32/f6c 0 2024-03-22T02:29:02.377 INFO:tasks.workunit.client.1.smithi122.stdout:8/879: dwrite d1/d40/d51/df5/dae/dfe/fe8 [0,4194304] 0 2024-03-22T02:29:02.378 INFO:tasks.workunit.client.1.smithi122.stdout:6/815: fdatasync d7/de/dd/d20/de0/dae/fcd 0 2024-03-22T02:29:02.379 INFO:tasks.workunit.client.1.smithi122.stdout:8/880: fdatasync d1/d40/d71/dc9/f65 0 2024-03-22T02:29:02.379 INFO:tasks.workunit.client.1.smithi122.stdout:2/807: getdents df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2 0 2024-03-22T02:29:02.381 INFO:tasks.workunit.client.1.smithi122.stdout:2/808: write df/d1f/d33/d6a/d7c/fce [650715,65425] 0 2024-03-22T02:29:02.382 INFO:tasks.workunit.client.1.smithi122.stdout:7/762: symlink d0/d4/d80/dc4/lf8 0 2024-03-22T02:29:02.385 INFO:tasks.workunit.client.1.smithi122.stdout:0/646: creat d11/d14/d1d/d27/d9d/daa/fdb x:0 0 0 2024-03-22T02:29:02.387 INFO:tasks.workunit.client.1.smithi122.stdout:8/881: mknod d1/d8/ddc/c12b 0 2024-03-22T02:29:02.387 INFO:tasks.workunit.client.1.smithi122.stdout:0/647: write d11/d14/d23/d24/fca [2462262,81941] 0 2024-03-22T02:29:02.388 INFO:tasks.workunit.client.1.smithi122.stdout:8/882: chown d1/d40/cf2 3573410 1 2024-03-22T02:29:02.389 INFO:tasks.workunit.client.1.smithi122.stdout:2/809: fdatasync df/d1f/d33/d34/f5f 0 2024-03-22T02:29:02.391 INFO:tasks.workunit.client.1.smithi122.stdout:7/763: mknod d0/d4/d1d/cf9 0 2024-03-22T02:29:02.392 INFO:tasks.workunit.client.1.smithi122.stdout:6/816: getdents d7/de/dd/d20/de0/dc0 0 2024-03-22T02:29:02.396 INFO:tasks.workunit.client.1.smithi122.stdout:0/648: dwrite d11/d14/d23/f72 [0,4194304] 0 2024-03-22T02:29:02.396 INFO:tasks.workunit.client.1.smithi122.stdout:8/883: symlink d1/d40/l12c 0 2024-03-22T02:29:02.404 INFO:tasks.workunit.client.1.smithi122.stdout:8/884: unlink d1/d8/f33 0 2024-03-22T02:29:02.405 INFO:tasks.workunit.client.1.smithi122.stdout:0/649: creat d11/d14/d1d/d27/d9d/daa/fdc x:0 0 0 2024-03-22T02:29:02.406 INFO:tasks.workunit.client.1.smithi122.stdout:2/810: symlink df/d1f/d33/d6a/d7c/d8e/l109 0 2024-03-22T02:29:02.408 INFO:tasks.workunit.client.1.smithi122.stdout:7/764: mknod d0/d4/d20/d21/d9b/cfa 0 2024-03-22T02:29:02.410 INFO:tasks.workunit.client.1.smithi122.stdout:7/765: write d0/d4/d20/d21/d9b/d5e/def/fb3 [2057194,114863] 0 2024-03-22T02:29:02.410 INFO:tasks.workunit.client.1.smithi122.stdout:0/650: creat d11/d16/d37/d84/da3/fdd x:0 0 0 2024-03-22T02:29:02.414 INFO:tasks.workunit.client.1.smithi122.stdout:0/651: dread d11/d16/d37/d9f/fa4 [0,4194304] 0 2024-03-22T02:29:02.416 INFO:tasks.workunit.client.1.smithi122.stdout:2/811: unlink df/d1f/d33/d4d/d94/dc2/d72/c76 0 2024-03-22T02:29:02.419 INFO:tasks.workunit.client.1.smithi122.stdout:0/652: write d11/d14/d1b/f81 [1612144,56125] 0 2024-03-22T02:29:02.423 INFO:tasks.workunit.client.1.smithi122.stdout:2/812: rename df/d1f/d33/d4d/d94/dc2/d51/d67/db0/ced to df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2/c10a 0 2024-03-22T02:29:02.424 INFO:tasks.workunit.client.1.smithi122.stdout:8/885: dread d1/f1a [0,4194304] 0 2024-03-22T02:29:02.424 INFO:tasks.workunit.client.1.smithi122.stdout:7/766: getdents d0/d4/d94 0 2024-03-22T02:29:02.442 INFO:tasks.workunit.client.1.smithi122.stdout:9/814: dwrite db/df/f7d [4194304,4194304] 0 2024-03-22T02:29:02.443 INFO:tasks.workunit.client.1.smithi122.stdout:6/817: write d7/fe2 [43558,48875] 0 2024-03-22T02:29:02.445 INFO:tasks.workunit.client.1.smithi122.stdout:6/818: dread - d7/de/dd/d23/ff6 zero size 2024-03-22T02:29:02.445 INFO:tasks.workunit.client.1.smithi122.stdout:9/815: fdatasync db/d23/f1a 0 2024-03-22T02:29:02.449 INFO:tasks.workunit.client.1.smithi122.stdout:8/886: mkdir d1/d40/d71/d97/db8/df6/d12d 0 2024-03-22T02:29:02.451 INFO:tasks.workunit.client.1.smithi122.stdout:6/819: creat d7/de/dd/d23/d7f/f103 x:0 0 0 2024-03-22T02:29:02.453 INFO:tasks.workunit.client.1.smithi122.stdout:6/820: dread - d7/fab zero size 2024-03-22T02:29:02.459 INFO:tasks.workunit.client.1.smithi122.stdout:0/653: truncate d11/d14/d1d/f6a 7112863 0 2024-03-22T02:29:02.461 INFO:tasks.workunit.client.1.smithi122.stdout:2/813: write df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f82 [291643,69391] 0 2024-03-22T02:29:02.461 INFO:tasks.workunit.client.1.smithi122.stdout:0/654: chown d11/d14/d23/d24/d33/f79 2376053 1 2024-03-22T02:29:02.461 INFO:tasks.workunit.client.1.smithi122.stdout:0/655: stat d11/d16/d37/f9a 0 2024-03-22T02:29:02.463 INFO:tasks.workunit.client.1.smithi122.stdout:0/656: chown d11/d14/d23/d24/d33/d67 4 1 2024-03-22T02:29:02.468 INFO:tasks.workunit.client.1.smithi122.stdout:6/821: rename d7/de/d27/l11 to d7/de/l104 0 2024-03-22T02:29:02.468 INFO:tasks.workunit.client.1.smithi122.stdout:7/767: dwrite d0/d4/de9/f56 [0,4194304] 0 2024-03-22T02:29:02.469 INFO:tasks.workunit.client.1.smithi122.stdout:2/814: getdents df/d1f/d33/d4d/d94/dc2/d51/d67/db0/df8 0 2024-03-22T02:29:02.471 INFO:tasks.workunit.client.1.smithi122.stdout:8/887: truncate d1/d8/d63/df0/f32 563123 0 2024-03-22T02:29:02.496 INFO:tasks.workunit.client.1.smithi122.stdout:8/888: creat d1/d8/ddc/f12e x:0 0 0 2024-03-22T02:29:02.499 INFO:tasks.workunit.client.1.smithi122.stdout:9/816: dwrite db/d23/d1f/d57/f87 [0,4194304] 0 2024-03-22T02:29:02.499 INFO:tasks.workunit.client.1.smithi122.stdout:6/822: mknod d7/de/dd/c105 0 2024-03-22T02:29:02.500 INFO:tasks.workunit.client.1.smithi122.stdout:9/817: fsync db/d23/d1f/d57/fd0 0 2024-03-22T02:29:02.501 INFO:tasks.workunit.client.1.smithi122.stdout:6/823: chown d7/de/dd/d20/de0/dae 23 1 2024-03-22T02:29:02.501 INFO:tasks.workunit.client.1.smithi122.stdout:7/768: write d0/d4/f9 [1609969,4812] 0 2024-03-22T02:29:02.503 INFO:tasks.workunit.client.1.smithi122.stdout:8/889: read - d1/d9/d50/db1/fd8 zero size 2024-03-22T02:29:02.504 INFO:tasks.workunit.client.1.smithi122.stdout:9/818: write db/df/d3e/d81/d25/d41/f61 [2610501,40277] 0 2024-03-22T02:29:02.504 INFO:tasks.workunit.client.1.smithi122.stdout:8/890: readlink d1/d9/d50/db1/lbc 0 2024-03-22T02:29:02.506 INFO:tasks.workunit.client.1.smithi122.stdout:8/891: chown d1/d60/d73/da3/fc8 48563024 1 2024-03-22T02:29:02.507 INFO:tasks.workunit.client.1.smithi122.stdout:6/824: creat d7/de/dd/d39/f106 x:0 0 0 2024-03-22T02:29:02.509 INFO:tasks.workunit.client.1.smithi122.stdout:9/819: creat db/d1b/da8/dba/f106 x:0 0 0 2024-03-22T02:29:02.512 INFO:tasks.workunit.client.1.smithi122.stdout:7/769: getdents d0/d4/d20/d21/d28 0 2024-03-22T02:29:02.516 INFO:tasks.workunit.client.1.smithi122.stdout:8/892: dwrite d1/f53 [0,4194304] 0 2024-03-22T02:29:02.516 INFO:tasks.workunit.client.1.smithi122.stdout:7/770: unlink d0/d4/d20/d21/d9b/l8d 0 2024-03-22T02:29:02.518 INFO:tasks.workunit.client.1.smithi122.stdout:7/771: truncate d0/d4/f9 4345038 0 2024-03-22T02:29:02.518 INFO:tasks.workunit.client.1.smithi122.stdout:6/825: write d7/de/dd/d20/de0/f9a [4841652,55479] 0 2024-03-22T02:29:02.519 INFO:tasks.workunit.client.1.smithi122.stdout:9/820: truncate db/d23/d9a/fa3 3195081 0 2024-03-22T02:29:02.523 INFO:tasks.workunit.client.1.smithi122.stdout:8/893: symlink d1/d40/d51/dfa/l12f 0 2024-03-22T02:29:02.524 INFO:tasks.workunit.client.1.smithi122.stdout:8/894: readlink d1/d9/d50/lb0 0 2024-03-22T02:29:02.527 INFO:tasks.workunit.client.1.smithi122.stdout:6/826: write d7/de/f3e [787816,114070] 0 2024-03-22T02:29:02.528 INFO:tasks.workunit.client.1.smithi122.stdout:8/895: mkdir d1/d4d/daa/d130 0 2024-03-22T02:29:02.529 INFO:tasks.workunit.client.1.smithi122.stdout:7/772: rename d0/d4/l7c to d0/d4/d1d/df6/lfb 0 2024-03-22T02:29:02.530 INFO:tasks.workunit.client.1.smithi122.stdout:9/821: truncate db/df/d3e/d59/db5/dc3/fed 2929936 0 2024-03-22T02:29:02.533 INFO:tasks.workunit.client.1.smithi122.stdout:6/827: creat d7/de/dd/d20/de0/d54/d76/df3/f107 x:0 0 0 2024-03-22T02:29:02.535 INFO:tasks.workunit.client.1.smithi122.stdout:6/828: chown d7/f1b 369519 1 2024-03-22T02:29:02.536 INFO:tasks.workunit.client.1.smithi122.stdout:9/822: creat db/df/d3e/d81/d3d/d86/dd8/f107 x:0 0 0 2024-03-22T02:29:02.537 INFO:tasks.workunit.client.1.smithi122.stdout:8/896: dwrite d1/d8/f75 [4194304,4194304] 0 2024-03-22T02:29:02.538 INFO:tasks.workunit.client.1.smithi122.stdout:6/829: stat d7/de/dd/d20/de0/d54/d76/d86/d96/db2/lda 0 2024-03-22T02:29:02.538 INFO:tasks.workunit.client.1.smithi122.stdout:7/773: creat d0/d4/d20/d21/d9b/ffc x:0 0 0 2024-03-22T02:29:02.541 INFO:tasks.workunit.client.1.smithi122.stdout:7/774: write d0/d4/d20/d21/f95 [280997,57646] 0 2024-03-22T02:29:02.546 INFO:tasks.workunit.client.1.smithi122.stdout:8/897: mknod d1/d8/dba/dc0/d111/c131 0 2024-03-22T02:29:02.548 INFO:tasks.workunit.client.1.smithi122.stdout:9/823: mknod db/d1b/d6c/d58/d9d/c108 0 2024-03-22T02:29:02.548 INFO:tasks.workunit.client.1.smithi122.stdout:7/775: dwrite d0/d4/de9/f56 [0,4194304] 0 2024-03-22T02:29:02.549 INFO:tasks.workunit.client.1.smithi122.stdout:6/830: symlink d7/de/dd/l108 0 2024-03-22T02:29:02.554 INFO:tasks.workunit.client.1.smithi122.stdout:8/898: link d1/d40/d51/df5/dae/dfe/c4c d1/d8/d8a/c132 0 2024-03-22T02:29:02.558 INFO:tasks.workunit.client.1.smithi122.stdout:4/787: sync 2024-03-22T02:29:02.559 INFO:tasks.workunit.client.1.smithi122.stdout:7/776: link d0/d4/d94/cba d0/d30/cfd 0 2024-03-22T02:29:02.560 INFO:tasks.workunit.client.1.smithi122.stdout:9/824: getdents db/d23/d1f/d57/d66 0 2024-03-22T02:29:02.561 INFO:tasks.workunit.client.1.smithi122.stdout:4/788: read d0/d12/fb2 [1903406,110970] 0 2024-03-22T02:29:02.564 INFO:tasks.workunit.client.1.smithi122.stdout:9/825: mkdir db/d1b/da8/d109 0 2024-03-22T02:29:02.566 INFO:tasks.workunit.client.1.smithi122.stdout:7/777: getdents d0/d4/de9 0 2024-03-22T02:29:02.566 INFO:tasks.workunit.client.1.smithi122.stdout:9/826: dread - db/d1b/d6c/d58/f103 zero size 2024-03-22T02:29:02.567 INFO:tasks.workunit.client.1.smithi122.stdout:7/778: readlink d0/d4/de9/l5a 0 2024-03-22T02:29:02.567 INFO:tasks.workunit.client.1.smithi122.stdout:9/827: write db/fe2 [200148,73385] 0 2024-03-22T02:29:02.569 INFO:tasks.workunit.client.1.smithi122.stdout:4/789: rename d0/d12/d15/d42/d4e/de/d40 to d0/d12/d15/d42/d4e/de/da1/dbf/d103 0 2024-03-22T02:29:02.570 INFO:tasks.workunit.client.1.smithi122.stdout:9/828: write db/d1b/da8/dba/f106 [272989,78442] 0 2024-03-22T02:29:02.570 INFO:tasks.workunit.client.1.smithi122.stdout:7/779: write d0/d4/d1d/df6/f73 [678136,82499] 0 2024-03-22T02:29:02.575 INFO:tasks.workunit.client.1.smithi122.stdout:9/829: truncate db/fb3 1419594 0 2024-03-22T02:29:02.578 INFO:tasks.workunit.client.1.smithi122.stdout:4/790: symlink d0/d12/d15/d42/d4e/de/da1/dbf/l104 0 2024-03-22T02:29:02.579 INFO:tasks.workunit.client.1.smithi122.stdout:4/791: dread - d0/d12/d15/d2d/d75/f8a zero size 2024-03-22T02:29:02.580 INFO:tasks.workunit.client.1.smithi122.stdout:9/830: write db/d23/d1f/d57/fb8 [4708972,54152] 0 2024-03-22T02:29:02.580 INFO:tasks.workunit.client.1.smithi122.stdout:7/780: getdents d0/d4/d1d/df6/d5d/df5 0 2024-03-22T02:29:02.588 INFO:tasks.workunit.client.1.smithi122.stdout:6/831: dwrite d7/de/f34 [0,4194304] 0 2024-03-22T02:29:02.593 INFO:tasks.workunit.client.1.smithi122.stdout:6/832: readlink d7/de/dd/d20/de0/dc0/le9 0 2024-03-22T02:29:02.593 INFO:tasks.workunit.client.1.smithi122.stdout:7/781: creat d0/d4/d1d/df6/ffe x:0 0 0 2024-03-22T02:29:02.596 INFO:tasks.workunit.client.1.smithi122.stdout:6/833: dread - d7/de/dd/d20/de0/d54/d76/fe5 zero size 2024-03-22T02:29:02.598 INFO:tasks.workunit.client.1.smithi122.stdout:9/831: read db/df/d3e/d81/d25/d41/fc9 [4181363,113593] 0 2024-03-22T02:29:02.600 INFO:tasks.workunit.client.1.smithi122.stdout:8/899: dwrite d1/d9/d50/da6/fd4 [0,4194304] 0 2024-03-22T02:29:02.607 INFO:tasks.workunit.client.1.smithi122.stdout:7/782: dwrite d0/d4/d1d/df6/ffe [0,4194304] 0 2024-03-22T02:29:02.611 INFO:tasks.workunit.client.1.smithi122.stdout:4/792: dread d0/d12/d15/d42/d4e/fad [0,4194304] 0 2024-03-22T02:29:02.612 INFO:tasks.workunit.client.1.smithi122.stdout:7/783: read d0/d4/d20/d21/d9b/dbb/fd9 [338060,120653] 0 2024-03-22T02:29:02.614 INFO:tasks.workunit.client.1.smithi122.stdout:8/900: creat d1/d8/d8a/f133 x:0 0 0 2024-03-22T02:29:02.614 INFO:tasks.workunit.client.1.smithi122.stdout:7/784: read - d0/d4/d20/d21/d9b/dbb/ff1 zero size 2024-03-22T02:29:02.618 INFO:tasks.workunit.client.1.smithi122.stdout:6/834: write d7/de/d27/f18 [4808366,74778] 0 2024-03-22T02:29:02.620 INFO:tasks.workunit.client.1.smithi122.stdout:8/901: mkdir d1/d9/d134 0 2024-03-22T02:29:02.621 INFO:tasks.workunit.client.1.smithi122.stdout:9/832: link db/df/d3e/d59/db5/c102 db/df/d3e/d59/db5/dc3/c10a 0 2024-03-22T02:29:02.621 INFO:tasks.workunit.client.1.smithi122.stdout:8/902: chown d1/d8/l69 2 1 2024-03-22T02:29:02.625 INFO:tasks.workunit.client.1.smithi122.stdout:7/785: rename d0/d4/d1d/df6/d5d/df5 to d0/d4/d20/d21/d9b/dff 0 2024-03-22T02:29:02.626 INFO:tasks.workunit.client.1.smithi122.stdout:8/903: symlink d1/d8/l135 0 2024-03-22T02:29:02.627 INFO:tasks.workunit.client.1.smithi122.stdout:8/904: chown d1/d60/ld0 17176 1 2024-03-22T02:29:02.628 INFO:tasks.workunit.client.1.smithi122.stdout:9/833: rename db/d23/f1a to db/df/d3e/d59/dda/f10b 0 2024-03-22T02:29:02.633 INFO:tasks.workunit.client.1.smithi122.stdout:7/786: unlink d0/d4/d20/d21/d9b/l97 0 2024-03-22T02:29:02.636 INFO:tasks.workunit.client.1.smithi122.stdout:9/834: creat db/df/d100/f10c x:0 0 0 2024-03-22T02:29:02.639 INFO:tasks.workunit.client.1.smithi122.stdout:8/905: rename d1/d40/d71/d97/l124 to d1/d40/d48/l136 0 2024-03-22T02:29:02.640 INFO:tasks.workunit.client.1.smithi122.stdout:8/906: fsync d1/d40/d71/d97/f9f 0 2024-03-22T02:29:02.642 INFO:tasks.workunit.client.1.smithi122.stdout:8/907: truncate d1/d60/d73/da3/f112 862621 0 2024-03-22T02:29:02.643 INFO:tasks.workunit.client.1.smithi122.stdout:9/835: rename db/d1b/c5a to db/d1b/da8/c10d 0 2024-03-22T02:29:02.645 INFO:tasks.workunit.client.1.smithi122.stdout:8/908: read - d1/d8/d63/df0/f61 zero size 2024-03-22T02:29:02.647 INFO:tasks.workunit.client.1.smithi122.stdout:4/793: dread d0/d25/f84 [0,4194304] 0 2024-03-22T02:29:02.647 INFO:tasks.workunit.client.1.smithi122.stdout:9/836: unlink db/d23/fe5 0 2024-03-22T02:29:02.650 INFO:tasks.workunit.client.1.smithi122.stdout:9/837: dread - db/df/d3e/d81/d3d/d86/dd8/f107 zero size 2024-03-22T02:29:02.656 INFO:tasks.workunit.client.1.smithi122.stdout:4/794: getdents d0/d12/d15/d42/d4e 0 2024-03-22T02:29:02.657 INFO:tasks.workunit.client.1.smithi122.stdout:9/838: truncate db/d23/d1e/f70 359705 0 2024-03-22T02:29:02.659 INFO:tasks.workunit.client.1.smithi122.stdout:9/839: dread - db/df/d3e/d81/d25/d41/fbc zero size 2024-03-22T02:29:02.665 INFO:tasks.workunit.client.1.smithi122.stdout:6/835: dread d7/de/d27/f18 [0,4194304] 0 2024-03-22T02:29:02.668 INFO:tasks.workunit.client.1.smithi122.stdout:4/795: link d0/d12/c99 d0/d12/d15/d42/d4e/de/d91/c105 0 2024-03-22T02:29:02.674 INFO:tasks.workunit.client.1.smithi122.stdout:9/840: rename db/df/d3e/d81/d3d/c63 to db/df/d3e/d81/d25/c10e 0 2024-03-22T02:29:02.676 INFO:tasks.workunit.client.1.smithi122.stdout:6/836: dwrite d7/de/dd/d20/f31 [0,4194304] 0 2024-03-22T02:29:02.682 INFO:tasks.workunit.client.1.smithi122.stdout:6/837: creat d7/de/dd/d9b/f109 x:0 0 0 2024-03-22T02:29:02.684 INFO:tasks.workunit.client.1.smithi122.stdout:8/909: dwrite d1/d40/d48/fb5 [0,4194304] 0 2024-03-22T02:29:02.686 INFO:tasks.workunit.client.1.smithi122.stdout:6/838: read - d7/de/dd/d39/fb3 zero size 2024-03-22T02:29:02.688 INFO:tasks.workunit.client.1.smithi122.stdout:8/910: mknod d1/d60/d73/c137 0 2024-03-22T02:29:02.691 INFO:tasks.workunit.client.1.smithi122.stdout:6/839: creat d7/de/d27/d52/f10a x:0 0 0 2024-03-22T02:29:02.694 INFO:tasks.workunit.client.1.smithi122.stdout:6/840: mknod d7/de/dd/d20/de0/dae/c10b 0 2024-03-22T02:29:02.699 INFO:tasks.workunit.client.1.smithi122.stdout:8/911: dwrite d1/d60/d73/fef [0,4194304] 0 2024-03-22T02:29:02.701 INFO:tasks.workunit.client.1.smithi122.stdout:6/841: rmdir d7/de/dd/d20/de0/d54/d76/d86/d96 39 2024-03-22T02:29:02.702 INFO:tasks.workunit.client.1.smithi122.stdout:8/912: mkdir d1/d8/d63/df0/d138 0 2024-03-22T02:29:02.706 INFO:tasks.workunit.client.1.smithi122.stdout:6/842: creat d7/de/d27/f10c x:0 0 0 2024-03-22T02:29:02.710 INFO:tasks.workunit.client.1.smithi122.stdout:8/913: mkdir d1/d40/d71/d97/db8/df6/d119/d139 0 2024-03-22T02:29:02.719 INFO:tasks.workunit.client.1.smithi122.stdout:9/841: dread db/d1b/da8/fcd [0,4194304] 0 2024-03-22T02:29:02.722 INFO:tasks.workunit.client.1.smithi122.stdout:9/842: creat db/d1b/d6c/d58/f10f x:0 0 0 2024-03-22T02:29:02.725 INFO:tasks.workunit.client.1.smithi122.stdout:9/843: readlink db/d23/d1f/d57/l37 0 2024-03-22T02:29:02.728 INFO:tasks.workunit.client.1.smithi122.stdout:9/844: creat db/f110 x:0 0 0 2024-03-22T02:29:02.730 INFO:tasks.workunit.client.1.smithi122.stdout:9/845: mkdir db/df/d100/d111 0 2024-03-22T02:29:02.734 INFO:tasks.workunit.client.1.smithi122.stdout:6/843: dread d7/de/dd/d23/f63 [0,4194304] 0 2024-03-22T02:29:02.735 INFO:tasks.workunit.client.1.smithi122.stdout:9/846: dread db/d23/d1f/d57/f87 [0,4194304] 0 2024-03-22T02:29:02.736 INFO:tasks.workunit.client.1.smithi122.stdout:9/847: dread - db/f110 zero size 2024-03-22T02:29:02.736 INFO:tasks.workunit.client.1.smithi122.stdout:6/844: chown d7/de/dd/f16 1 1 2024-03-22T02:29:02.739 INFO:tasks.workunit.client.1.smithi122.stdout:9/848: rename db/d1b/d28/l45 to db/d23/d1f/d57/dd3/l112 0 2024-03-22T02:29:02.740 INFO:tasks.workunit.client.1.smithi122.stdout:9/849: fdatasync db/df/d3e/d59/db5/ff7 0 2024-03-22T02:29:02.743 INFO:tasks.workunit.client.1.smithi122.stdout:6/845: truncate d7/f15 1294208 0 2024-03-22T02:29:02.745 INFO:tasks.workunit.client.1.smithi122.stdout:6/846: chown d7/de/d27/d52/fe1 7229 1 2024-03-22T02:29:02.745 INFO:tasks.workunit.client.1.smithi122.stdout:9/850: link db/df/d3e/d59/ca7 db/d23/d1f/d57/d66/d73/c113 0 2024-03-22T02:29:02.746 INFO:tasks.workunit.client.1.smithi122.stdout:6/847: dread - d7/de/dd/d20/de0/d54/d76/fbd zero size 2024-03-22T02:29:02.748 INFO:tasks.workunit.client.1.smithi122.stdout:6/848: write d7/de/dd/d20/de0/df9/f100 [169084,73343] 0 2024-03-22T02:29:02.749 INFO:tasks.workunit.client.1.smithi122.stdout:6/849: fsync d7/de/d27/d52/f10a 0 2024-03-22T02:29:02.750 INFO:tasks.workunit.client.1.smithi122.stdout:6/850: chown d7/de/dd/d20/de0/dad/fd5 958205 1 2024-03-22T02:29:02.752 INFO:tasks.workunit.client.1.smithi122.stdout:9/851: truncate db/d23/f83 3506593 0 2024-03-22T02:29:02.753 INFO:tasks.workunit.client.1.smithi122.stdout:6/851: truncate d7/de/dd/f3a 1590514 0 2024-03-22T02:29:02.755 INFO:tasks.workunit.client.1.smithi122.stdout:9/852: creat db/d1b/f114 x:0 0 0 2024-03-22T02:29:02.757 INFO:tasks.workunit.client.1.smithi122.stdout:9/853: creat db/f115 x:0 0 0 2024-03-22T02:29:02.758 INFO:tasks.workunit.client.1.smithi122.stdout:9/854: truncate db/d23/f3a 252856 0 2024-03-22T02:29:02.759 INFO:tasks.workunit.client.1.smithi122.stdout:9/855: readlink db/d23/lf8 0 2024-03-22T02:29:02.765 INFO:tasks.workunit.client.1.smithi122.stdout:9/856: dwrite db/d1b/d6c/dd1/fde [0,4194304] 0 2024-03-22T02:29:02.772 INFO:tasks.workunit.client.1.smithi122.stdout:6/852: dread f6 [4194304,4194304] 0 2024-03-22T02:29:02.776 INFO:tasks.workunit.client.1.smithi122.stdout:9/857: dread db/d1b/d6c/d58/d9d/fa6 [0,4194304] 0 2024-03-22T02:29:02.776 INFO:tasks.workunit.client.1.smithi122.stdout:6/853: creat d7/de/dd/d39/f10d x:0 0 0 2024-03-22T02:29:02.778 INFO:tasks.workunit.client.1.smithi122.stdout:9/858: mkdir db/d23/d116 0 2024-03-22T02:29:02.785 INFO:tasks.workunit.client.1.smithi122.stdout:9/859: link db/df/d3e/d59/ca7 db/df/d3e/d81/d3d/c117 0 2024-03-22T02:29:02.787 INFO:tasks.workunit.client.1.smithi122.stdout:9/860: symlink db/df/d3e/d81/d3d/d86/l118 0 2024-03-22T02:29:02.787 INFO:tasks.workunit.client.1.smithi122.stdout:6/854: write d7/de/dd/f42 [2914452,59332] 0 2024-03-22T02:29:02.787 INFO:tasks.workunit.client.1.smithi122.stdout:6/855: chown d7 2123008 1 2024-03-22T02:29:02.790 INFO:tasks.workunit.client.1.smithi122.stdout:6/856: fsync d7/de/dd/d20/de0/dad/fd5 0 2024-03-22T02:29:02.791 INFO:tasks.workunit.client.1.smithi122.stdout:6/857: readlink d7/de/dd/d20/de0/d54/d76/d86/l8e 0 2024-03-22T02:29:02.796 INFO:tasks.workunit.client.1.smithi122.stdout:9/861: dwrite db/d1b/d28/dc1/f32 [0,4194304] 0 2024-03-22T02:29:02.798 INFO:tasks.workunit.client.1.smithi122.stdout:9/862: mknod db/d23/d9a/c119 0 2024-03-22T02:29:02.802 INFO:tasks.workunit.client.1.smithi122.stdout:9/863: rename db/d1b/d6c to db/df/d3e/d59/d11a 0 2024-03-22T02:29:02.811 INFO:tasks.workunit.client.1.smithi122.stdout:3/775: sync 2024-03-22T02:29:02.812 INFO:tasks.workunit.client.1.smithi122.stdout:3/776: dread - d2/d9/dc/d17/d2b/d6b/fb3 zero size 2024-03-22T02:29:02.812 INFO:tasks.workunit.client.1.smithi122.stdout:3/777: write d2/d9/dc/d17/d2b/fe6 [140431,78889] 0 2024-03-22T02:29:02.814 INFO:tasks.workunit.client.1.smithi122.stdout:3/778: mknod d2/d9/dc/d17/d28/d4c/cfd 0 2024-03-22T02:29:02.815 INFO:tasks.workunit.client.1.smithi122.stdout:3/779: readlink d2/d9/dc/d17/d2b/d3f/l68 0 2024-03-22T02:29:02.816 INFO:tasks.workunit.client.1.smithi122.stdout:3/780: rename d2/d9/dc/db5 to d2/d9/dc/db5/dfe 22 2024-03-22T02:29:02.817 INFO:tasks.workunit.client.1.smithi122.stdout:3/781: unlink d2/d9/dc/d17/c4b 0 2024-03-22T02:29:02.819 INFO:tasks.workunit.client.1.smithi122.stdout:3/782: creat d2/d9/fff x:0 0 0 2024-03-22T02:29:02.825 INFO:tasks.workunit.client.1.smithi122.stdout:3/783: getdents d2/d9/dc/d17/d51 0 2024-03-22T02:29:02.830 INFO:tasks.workunit.client.1.smithi122.stdout:3/784: truncate d2/d9/dc/f36 6198966 0 2024-03-22T02:29:02.831 INFO:tasks.workunit.client.1.smithi122.stdout:3/785: truncate d2/d9/dc/df/d49/f4f 86121 0 2024-03-22T02:29:02.833 INFO:tasks.workunit.client.1.smithi122.stdout:3/786: creat d2/d9/dc/d17/d2b/ddd/f100 x:0 0 0 2024-03-22T02:29:02.848 INFO:tasks.workunit.client.1.smithi122.stdout:3/787: dread d2/d9/dc/d57/f93 [0,4194304] 0 2024-03-22T02:29:02.850 INFO:tasks.workunit.client.1.smithi122.stdout:3/788: mknod d2/d9/dc/d17/d2b/d6b/c101 0 2024-03-22T02:29:02.850 INFO:tasks.workunit.client.1.smithi122.stdout:3/789: chown d2/d9/dc/d57/ff8 1 1 2024-03-22T02:29:02.857 INFO:tasks.workunit.client.1.smithi122.stdout:3/790: truncate d2/d9/dc/f2f 824436 0 2024-03-22T02:29:02.859 INFO:tasks.workunit.client.1.smithi122.stdout:3/791: creat d2/d9/dc/d57/d89/f102 x:0 0 0 2024-03-22T02:29:02.861 INFO:tasks.workunit.client.1.smithi122.stdout:3/792: symlink d2/d9/dc/df/d49/d85/dda/l103 0 2024-03-22T02:29:02.866 INFO:tasks.workunit.client.1.smithi122.stdout:3/793: dwrite d2/d9/dc/df/d3c/fe2 [0,4194304] 0 2024-03-22T02:29:02.868 INFO:tasks.workunit.client.1.smithi122.stdout:3/794: read d2/d9/dc/d57/ff8 [2028097,112844] 0 2024-03-22T02:29:02.877 INFO:tasks.workunit.client.1.smithi122.stdout:3/795: dwrite d2/d9/dc/df/d38/fa1 [0,4194304] 0 2024-03-22T02:29:02.878 INFO:tasks.workunit.client.1.smithi122.stdout:3/796: read - d2/d9/dc/df/d49/fac zero size 2024-03-22T02:29:02.882 INFO:tasks.workunit.client.1.smithi122.stdout:3/797: rename d2/d9/d7e/le9 to d2/d9/dc/d17/d28/l104 0 2024-03-22T02:29:02.885 INFO:tasks.workunit.client.1.smithi122.stdout:3/798: mkdir d2/d9/dc/df/d38/d87/d105 0 2024-03-22T02:29:02.885 INFO:tasks.workunit.client.1.smithi122.stdout:3/799: readlink d2/d9/d7e/l8e 0 2024-03-22T02:29:02.979 INFO:tasks.workunit.client.1.smithi122.stdout:1/737: sync 2024-03-22T02:29:02.982 INFO:tasks.workunit.client.1.smithi122.stdout:1/738: write dc/f10 [615231,86360] 0 2024-03-22T02:29:02.985 INFO:tasks.workunit.client.1.smithi122.stdout:1/739: unlink dc/d4c/la4 0 2024-03-22T02:29:02.988 INFO:tasks.workunit.client.1.smithi122.stdout:1/740: dread dc/fd [8388608,4194304] 0 2024-03-22T02:29:02.989 INFO:tasks.workunit.client.1.smithi122.stdout:1/741: mkdir dc/d3b/d46/de3 0 2024-03-22T02:29:02.994 INFO:tasks.workunit.client.1.smithi122.stdout:1/742: dwrite dc/d3b/d8d/d72/d9c/fae [0,4194304] 0 2024-03-22T02:29:02.998 INFO:tasks.workunit.client.1.smithi122.stdout:1/743: symlink dc/d3b/d8d/d72/d9c/le4 0 2024-03-22T02:29:02.999 INFO:tasks.workunit.client.1.smithi122.stdout:1/744: rmdir dc/d3b/d8d 39 2024-03-22T02:29:03.007 INFO:tasks.workunit.client.1.smithi122.stdout:1/745: dwrite dc/d3b/d8d/d72/d9c/db0/d67/dc6/f48 [0,4194304] 0 2024-03-22T02:29:03.011 INFO:tasks.workunit.client.1.smithi122.stdout:1/746: rename dc/l26 to dc/d3b/d8d/d72/d9c/db0/d71/le5 0 2024-03-22T02:29:03.021 INFO:tasks.workunit.client.1.smithi122.stdout:1/747: dwrite dc/ff [0,4194304] 0 2024-03-22T02:29:03.022 INFO:tasks.workunit.client.1.smithi122.stdout:1/748: readlink dc/d3b/d8d/d72/d9c/db0/le2 0 2024-03-22T02:29:03.030 INFO:tasks.workunit.client.1.smithi122.stdout:1/749: dwrite dc/d3b/d8d/d72/d9c/db0/d71/fda [0,4194304] 0 2024-03-22T02:29:03.052 INFO:tasks.workunit.client.1.smithi122.stdout:1/750: dread dc/f1d [0,4194304] 0 2024-03-22T02:29:03.064 INFO:tasks.workunit.client.1.smithi122.stdout:1/751: dread dc/f25 [0,4194304] 0 2024-03-22T02:29:03.065 INFO:tasks.workunit.client.1.smithi122.stdout:1/752: dread - dc/d3b/fb4 zero size 2024-03-22T02:29:03.104 INFO:tasks.workunit.client.1.smithi122.stdout:5/708: sync 2024-03-22T02:29:03.118 INFO:tasks.workunit.client.1.smithi122.stdout:5/709: dread d8/d74/fbb [0,4194304] 0 2024-03-22T02:29:03.120 INFO:tasks.workunit.client.1.smithi122.stdout:5/710: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/f44 [250854,22914] 0 2024-03-22T02:29:03.123 INFO:tasks.workunit.client.1.smithi122.stdout:5/711: rmdir d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/de9 0 2024-03-22T02:29:03.132 INFO:tasks.workunit.client.1.smithi122.stdout:5/712: dwrite d8/d1e/d22/d3a/d7a/fad [0,4194304] 0 2024-03-22T02:29:03.135 INFO:tasks.workunit.client.1.smithi122.stdout:5/713: creat d8/d1e/d22/d3a/d46/ff4 x:0 0 0 2024-03-22T02:29:03.138 INFO:tasks.workunit.client.1.smithi122.stdout:5/714: mknod d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/cf5 0 2024-03-22T02:29:03.139 INFO:tasks.workunit.client.1.smithi122.stdout:5/715: write d8/d1e/d22/fda [8130,130751] 0 2024-03-22T02:29:03.142 INFO:tasks.workunit.client.1.smithi122.stdout:5/716: symlink d8/d1e/d22/d3a/lf6 0 2024-03-22T02:29:03.148 INFO:tasks.workunit.client.1.smithi122.stdout:5/717: dwrite d8/d3b/faf [4194304,4194304] 0 2024-03-22T02:29:03.150 INFO:tasks.workunit.client.1.smithi122.stdout:5/718: fdatasync d8/d1e/f23 0 2024-03-22T02:29:03.154 INFO:tasks.workunit.client.1.smithi122.stdout:5/719: unlink d8/d1e/d22/d3a/d46/d4f/le5 0 2024-03-22T02:29:03.154 INFO:tasks.workunit.client.1.smithi122.stdout:5/720: chown d8/d74 29 1 2024-03-22T02:29:03.156 INFO:tasks.workunit.client.1.smithi122.stdout:5/721: rename d8/d1e/d22/d3a/d7a/lef to d8/d1e/d22/d3a/d46/d4f/lf7 0 2024-03-22T02:29:03.158 INFO:tasks.workunit.client.1.smithi122.stdout:5/722: stat d8/d1e/d22/d3a/d46/faa 0 2024-03-22T02:29:03.158 INFO:tasks.workunit.client.1.smithi122.stdout:5/723: truncate d8/f33 4623519 0 2024-03-22T02:29:03.160 INFO:tasks.workunit.client.1.smithi122.stdout:5/724: creat d8/d1e/d22/d3a/d46/d4f/d9b/ddf/ff8 x:0 0 0 2024-03-22T02:29:03.163 INFO:tasks.workunit.client.1.smithi122.stdout:5/725: link d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/f58 d8/d1e/d22/d3a/d46/d4f/ff9 0 2024-03-22T02:29:03.165 INFO:tasks.workunit.client.1.smithi122.stdout:5/726: chown d8/d1e/d22/cd1 485 1 2024-03-22T02:29:03.167 INFO:tasks.workunit.client.1.smithi122.stdout:5/727: creat d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7/ffa x:0 0 0 2024-03-22T02:29:03.169 INFO:tasks.workunit.client.1.smithi122.stdout:5/728: truncate d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 831550 0 2024-03-22T02:29:03.171 INFO:tasks.workunit.client.1.smithi122.stdout:5/729: mknod d8/d1e/d22/d3a/cfb 0 2024-03-22T02:29:03.174 INFO:tasks.workunit.client.1.smithi122.stdout:5/730: rename d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/f92 to d8/d1a/d83/dcb/ffc 0 2024-03-22T02:29:03.176 INFO:tasks.workunit.client.1.smithi122.stdout:5/731: creat d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/ffd x:0 0 0 2024-03-22T02:29:03.182 INFO:tasks.workunit.client.1.smithi122.stdout:5/732: chown d8/d1e/d22/d3a/d46/d4f/lf7 8 1 2024-03-22T02:29:03.185 INFO:tasks.workunit.client.1.smithi122.stdout:5/733: symlink d8/d1e/d22/dd7/lfe 0 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:4/796: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:8/914: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:7/787: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:2/815: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:9/864: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:0/657: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:3/800: sync 2024-03-22T02:29:03.189 INFO:tasks.workunit.client.1.smithi122.stdout:6/858: sync 2024-03-22T02:29:03.191 INFO:tasks.workunit.client.1.smithi122.stdout:9/865: fdatasync db/d1b/da8/dba/f106 0 2024-03-22T02:29:03.191 INFO:tasks.workunit.client.1.smithi122.stdout:5/734: truncate d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f68 1530529 0 2024-03-22T02:29:03.192 INFO:tasks.workunit.client.1.smithi122.stdout:8/915: readlink d1/d40/d51/dfa/l110 0 2024-03-22T02:29:03.193 INFO:tasks.workunit.client.1.smithi122.stdout:7/788: unlink d0/d4/d1d/lee 0 2024-03-22T02:29:03.193 INFO:tasks.workunit.client.1.smithi122.stdout:9/866: fsync db/df/d3e/d59/d11a/f4f 0 2024-03-22T02:29:03.201 INFO:tasks.workunit.client.1.smithi122.stdout:7/789: dread d0/d4/d20/d21/d9b/dbb/fc7 [0,4194304] 0 2024-03-22T02:29:03.203 INFO:tasks.workunit.client.1.smithi122.stdout:8/916: dwrite d1/d9/d50/da6/fd4 [4194304,4194304] 0 2024-03-22T02:29:03.204 INFO:tasks.workunit.client.1.smithi122.stdout:8/917: dread - d1/d8/ddc/f11f zero size 2024-03-22T02:29:03.208 INFO:tasks.workunit.client.1.smithi122.stdout:9/867: chown db/d1b/l1c 1 1 2024-03-22T02:29:03.210 INFO:tasks.workunit.client.1.smithi122.stdout:3/801: rename d2/d9/dc/d17/d2b/d3f/fbe to d2/d9/dc/df/d49/d85/dda/dd5/df9/f106 0 2024-03-22T02:29:03.213 INFO:tasks.workunit.client.1.smithi122.stdout:0/658: symlink d11/d14/d1d/d27/d9d/daa/dcd/lde 0 2024-03-22T02:29:03.216 INFO:tasks.workunit.client.1.smithi122.stdout:6/859: dwrite d7/de/dd/d23/d7f/fdb [0,4194304] 0 2024-03-22T02:29:03.217 INFO:tasks.workunit.client.1.smithi122.stdout:4/797: dread d0/d12/d15/d2d/f4f [0,4194304] 0 2024-03-22T02:29:03.218 INFO:tasks.workunit.client.1.smithi122.stdout:9/868: symlink db/df/d3e/d81/d3d/d86/dd8/l11b 0 2024-03-22T02:29:03.221 INFO:tasks.workunit.client.1.smithi122.stdout:0/659: mknod d11/d4f/d5a/cdf 0 2024-03-22T02:29:03.221 INFO:tasks.workunit.client.1.smithi122.stdout:4/798: dread - d0/d12/d15/d2d/f7d zero size 2024-03-22T02:29:03.222 INFO:tasks.workunit.client.1.smithi122.stdout:2/816: dwrite df/d1f/d23/dd2/f96 [0,4194304] 0 2024-03-22T02:29:03.227 INFO:tasks.workunit.client.1.smithi122.stdout:6/860: rmdir d7/de/dd/d39 39 2024-03-22T02:29:03.227 INFO:tasks.workunit.client.1.smithi122.stdout:7/790: getdents d0/d4/d1d/daf 0 2024-03-22T02:29:03.228 INFO:tasks.workunit.client.1.smithi122.stdout:5/735: dwrite d8/d1a/f70 [0,4194304] 0 2024-03-22T02:29:03.230 INFO:tasks.workunit.client.1.smithi122.stdout:4/799: dread - d0/d12/d61/f97 zero size 2024-03-22T02:29:03.231 INFO:tasks.workunit.client.1.smithi122.stdout:1/753: sync 2024-03-22T02:29:03.231 INFO:tasks.workunit.client.1.smithi122.stdout:5/736: chown d8/fc4 13733086 1 2024-03-22T02:29:03.234 INFO:tasks.workunit.client.1.smithi122.stdout:3/802: dwrite d2/d9/dc/d17/d2b/f67 [0,4194304] 0 2024-03-22T02:29:03.240 INFO:tasks.workunit.client.1.smithi122.stdout:2/817: dread df/d1f/d33/d4d/d7a/f8a [0,4194304] 0 2024-03-22T02:29:03.240 INFO:tasks.workunit.client.1.smithi122.stdout:4/800: chown d0/d12/d15/d42/c7f 1573958802 1 2024-03-22T02:29:03.241 INFO:tasks.workunit.client.1.smithi122.stdout:5/737: creat d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7/fff x:0 0 0 2024-03-22T02:29:03.242 INFO:tasks.workunit.client.1.smithi122.stdout:0/660: link d11/d14/d23/f77 d11/d14/d1d/d27/d9d/daa/dcd/fe0 0 2024-03-22T02:29:03.245 INFO:tasks.workunit.client.1.smithi122.stdout:3/803: unlink d2/d9/dc/d17/d2b/fe6 0 2024-03-22T02:29:03.248 INFO:tasks.workunit.client.1.smithi122.stdout:6/861: creat d7/de/dd/d20/de0/d54/d76/d86/d96/db2/f10e x:0 0 0 2024-03-22T02:29:03.250 INFO:tasks.workunit.client.1.smithi122.stdout:7/791: write d0/fac [574915,112192] 0 2024-03-22T02:29:03.250 INFO:tasks.workunit.client.1.smithi122.stdout:1/754: dwrite dc/d3b/d8d/d72/d9c/db0/d67/f7b [4194304,4194304] 0 2024-03-22T02:29:03.250 INFO:tasks.workunit.client.1.smithi122.stdout:2/818: fdatasync df/d1f/d33/d34/f5a 0 2024-03-22T02:29:03.252 INFO:tasks.workunit.client.1.smithi122.stdout:4/801: fdatasync d0/d12/d15/ff5 0 2024-03-22T02:29:03.254 INFO:tasks.workunit.client.1.smithi122.stdout:3/804: symlink d2/d9/dc/d57/d89/l107 0 2024-03-22T02:29:03.255 INFO:tasks.workunit.client.1.smithi122.stdout:0/661: mkdir d11/de1 0 2024-03-22T02:29:03.257 INFO:tasks.workunit.client.1.smithi122.stdout:4/802: truncate d0/d12/d61/fa3 1066538 0 2024-03-22T02:29:03.257 INFO:tasks.workunit.client.1.smithi122.stdout:1/755: fsync dc/d3b/d46/f9e 0 2024-03-22T02:29:03.259 INFO:tasks.workunit.client.1.smithi122.stdout:2/819: mknod df/d1f/d33/d6a/c10b 0 2024-03-22T02:29:03.262 INFO:tasks.workunit.client.1.smithi122.stdout:9/869: dwrite db/d23/d1f/d57/d66/d73/fa4 [0,4194304] 0 2024-03-22T02:29:03.262 INFO:tasks.workunit.client.1.smithi122.stdout:3/805: read d2/d9/dc/d17/d28/d4c/da5/db4/d79/fb9 [164646,83270] 0 2024-03-22T02:29:03.268 INFO:tasks.workunit.client.1.smithi122.stdout:1/756: creat dc/d3b/d46/fe6 x:0 0 0 2024-03-22T02:29:03.269 INFO:tasks.workunit.client.1.smithi122.stdout:8/918: dread d1/d8/d63/df0/f8b [0,4194304] 0 2024-03-22T02:29:03.271 INFO:tasks.workunit.client.1.smithi122.stdout:8/919: truncate d1/d8/f115 136045 0 2024-03-22T02:29:03.272 INFO:tasks.workunit.client.1.smithi122.stdout:9/870: symlink db/df/d100/l11c 0 2024-03-22T02:29:03.274 INFO:tasks.workunit.client.1.smithi122.stdout:3/806: creat d2/d9/dc/d17/d28/f108 x:0 0 0 2024-03-22T02:29:03.276 INFO:tasks.workunit.client.1.smithi122.stdout:5/738: dread d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf [0,4194304] 0 2024-03-22T02:29:03.276 INFO:tasks.workunit.client.1.smithi122.stdout:4/803: symlink d0/d25/de6/l106 0 2024-03-22T02:29:03.280 INFO:tasks.workunit.client.1.smithi122.stdout:2/820: rename df/d1f/d33/d34/ccf to df/c10c 0 2024-03-22T02:29:03.282 INFO:tasks.workunit.client.1.smithi122.stdout:6/862: creat d7/de/dd/d39/f10f x:0 0 0 2024-03-22T02:29:03.283 INFO:tasks.workunit.client.1.smithi122.stdout:9/871: creat db/d1b/d28/f11d x:0 0 0 2024-03-22T02:29:03.286 INFO:tasks.workunit.client.1.smithi122.stdout:7/792: getdents d0/d4/d20/d21/d9b/d5e 0 2024-03-22T02:29:03.286 INFO:tasks.workunit.client.1.smithi122.stdout:2/821: dread df/d1f/d33/d4d/d94/fe7 [0,4194304] 0 2024-03-22T02:29:03.290 INFO:tasks.workunit.client.1.smithi122.stdout:6/863: fsync d7/de/dd/d20/d8f/fa7 0 2024-03-22T02:29:03.292 INFO:tasks.workunit.client.1.smithi122.stdout:3/807: symlink d2/d9/dc/d17/l109 0 2024-03-22T02:29:03.292 INFO:tasks.workunit.client.1.smithi122.stdout:6/864: stat d7/de/dd/d23/d7f/f103 0 2024-03-22T02:29:03.294 INFO:tasks.workunit.client.1.smithi122.stdout:0/662: dwrite d11/d14/d23/f77 [0,4194304] 0 2024-03-22T02:29:03.294 INFO:tasks.workunit.client.1.smithi122.stdout:4/804: mknod d0/d12/d15/d42/df0/c107 0 2024-03-22T02:29:03.296 INFO:tasks.workunit.client.1.smithi122.stdout:4/805: read - d0/d12/d15/d2d/f72 zero size 2024-03-22T02:29:03.296 INFO:tasks.workunit.client.1.smithi122.stdout:7/793: creat d0/d4/d1d/f100 x:0 0 0 2024-03-22T02:29:03.297 INFO:tasks.workunit.client.1.smithi122.stdout:2/822: stat df/d1f/d33/d4d/d7a/c9a 0 2024-03-22T02:29:03.300 INFO:tasks.workunit.client.1.smithi122.stdout:6/865: mkdir d7/de/dd/d23/d7f/d110 0 2024-03-22T02:29:03.301 INFO:tasks.workunit.client.1.smithi122.stdout:3/808: creat d2/d9/dc/df/d49/d85/dda/f10a x:0 0 0 2024-03-22T02:29:03.302 INFO:tasks.workunit.client.1.smithi122.stdout:0/663: creat d11/d14/d23/d24/d33/d67/dc2/fe2 x:0 0 0 2024-03-22T02:29:03.304 INFO:tasks.workunit.client.1.smithi122.stdout:4/806: unlink d0/d25/l36 0 2024-03-22T02:29:03.305 INFO:tasks.workunit.client.1.smithi122.stdout:1/757: dread dc/d55/f75 [0,4194304] 0 2024-03-22T02:29:03.306 INFO:tasks.workunit.client.1.smithi122.stdout:9/872: getdents db/d23/d1e 0 2024-03-22T02:29:03.308 INFO:tasks.workunit.client.1.smithi122.stdout:5/739: dwrite d8/d1e/d22/d3a/d46/f6f [0,4194304] 0 2024-03-22T02:29:03.308 INFO:tasks.workunit.client.1.smithi122.stdout:6/866: mknod d7/de/dd/d20/de0/dad/c111 0 2024-03-22T02:29:03.311 INFO:tasks.workunit.client.1.smithi122.stdout:5/740: fdatasync d8/d1e/d22/fde 0 2024-03-22T02:29:03.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/664: mknod d11/d16/ce3 0 2024-03-22T02:29:03.313 INFO:tasks.workunit.client.1.smithi122.stdout:1/758: mkdir dc/d3b/d46/de7 0 2024-03-22T02:29:03.313 INFO:tasks.workunit.client.1.smithi122.stdout:6/867: chown d7/de/dd/d23/ff6 399 1 2024-03-22T02:29:03.314 INFO:tasks.workunit.client.1.smithi122.stdout:4/807: dread d0/d12/d15/d42/d4e/f5a [0,4194304] 0 2024-03-22T02:29:03.321 INFO:tasks.workunit.client.1.smithi122.stdout:4/808: dread d0/f37 [0,4194304] 0 2024-03-22T02:29:03.321 INFO:tasks.workunit.client.1.smithi122.stdout:4/809: dread - d0/d12/ffd zero size 2024-03-22T02:29:03.323 INFO:tasks.workunit.client.1.smithi122.stdout:0/665: unlink d11/d14/d1d/d27/dc0/dc6/dd0/cd6 0 2024-03-22T02:29:03.323 INFO:tasks.workunit.client.1.smithi122.stdout:7/794: write d0/d4/d1d/db8/f71 [4787208,58462] 0 2024-03-22T02:29:03.325 INFO:tasks.workunit.client.1.smithi122.stdout:2/823: truncate df/d1f/d33/fd7 178706 0 2024-03-22T02:29:03.328 INFO:tasks.workunit.client.1.smithi122.stdout:3/809: link d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 d2/f10b 0 2024-03-22T02:29:03.329 INFO:tasks.workunit.client.1.smithi122.stdout:1/759: mkdir dc/d3b/d8d/d72/d9c/de8 0 2024-03-22T02:29:03.331 INFO:tasks.workunit.client.1.smithi122.stdout:4/810: symlink d0/d12/d15/d2d/db5/l108 0 2024-03-22T02:29:03.332 INFO:tasks.workunit.client.1.smithi122.stdout:0/666: read d11/d16/d44/f76 [499154,35112] 0 2024-03-22T02:29:03.332 INFO:tasks.workunit.client.1.smithi122.stdout:4/811: fdatasync d0/d12/d15/d2d/db5/ff9 0 2024-03-22T02:29:03.333 INFO:tasks.workunit.client.1.smithi122.stdout:5/741: link d8/d74/fc9 d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/f100 0 2024-03-22T02:29:03.335 INFO:tasks.workunit.client.1.smithi122.stdout:4/812: write d0/f6a [1217733,12305] 0 2024-03-22T02:29:03.335 INFO:tasks.workunit.client.1.smithi122.stdout:9/873: dwrite db/df/d3e/d81/d25/d41/f54 [4194304,4194304] 0 2024-03-22T02:29:03.336 INFO:tasks.workunit.client.1.smithi122.stdout:7/795: rename d0/d4/d80/fdf to d0/d4/de9/f101 0 2024-03-22T02:29:03.339 INFO:tasks.workunit.client.1.smithi122.stdout:5/742: symlink d8/d3b/l101 0 2024-03-22T02:29:03.341 INFO:tasks.workunit.client.1.smithi122.stdout:3/810: write d2/d9/dc/df/d38/d87/f90 [294994,98541] 0 2024-03-22T02:29:03.342 INFO:tasks.workunit.client.1.smithi122.stdout:7/796: fdatasync d0/d30/f53 0 2024-03-22T02:29:03.349 INFO:tasks.workunit.client.1.smithi122.stdout:6/868: link d7/de/dd/fc9 d7/de/dd/d20/de0/d54/d76/f112 0 2024-03-22T02:29:03.350 INFO:tasks.workunit.client.1.smithi122.stdout:2/824: rename df/d1f/d3d/d52/lbc to df/d1f/d33/d4d/d94/deb/l10d 0 2024-03-22T02:29:03.352 INFO:tasks.workunit.client.1.smithi122.stdout:0/667: dwrite d11/d14/d1d/d27/d9d/daa/fdc [0,4194304] 0 2024-03-22T02:29:03.355 INFO:tasks.workunit.client.1.smithi122.stdout:2/825: write df/d1f/d33/d4d/d94/dc2/d72/ff3 [462733,110948] 0 2024-03-22T02:29:03.357 INFO:tasks.workunit.client.1.smithi122.stdout:2/826: stat df/d1f/d33/d4d/d94/fe7 0 2024-03-22T02:29:03.357 INFO:tasks.workunit.client.1.smithi122.stdout:9/874: dread db/df/d3e/d59/d11a/d58/d9d/fa6 [0,4194304] 0 2024-03-22T02:29:03.376 INFO:tasks.workunit.client.1.smithi122.stdout:4/813: getdents d0/d12/d15 0 2024-03-22T02:29:03.377 INFO:tasks.workunit.client.1.smithi122.stdout:1/760: rename dc/d2b/c9a to dc/d3b/d8d/d94/ce9 0 2024-03-22T02:29:03.378 INFO:tasks.workunit.client.1.smithi122.stdout:3/811: mknod d2/d9/dc/df/d49/c10c 0 2024-03-22T02:29:03.380 INFO:tasks.workunit.client.1.smithi122.stdout:0/668: truncate d11/d14/f15 762963 0 2024-03-22T02:29:03.380 INFO:tasks.workunit.client.1.smithi122.stdout:9/875: fsync db/d1b/d28/f6b 0 2024-03-22T02:29:03.382 INFO:tasks.workunit.client.1.smithi122.stdout:9/876: dread - db/df/d3e/d81/d3d/d86/dd8/f107 zero size 2024-03-22T02:29:03.383 INFO:tasks.workunit.client.1.smithi122.stdout:5/743: creat d8/d1e/d22/d3a/f102 x:0 0 0 2024-03-22T02:29:03.384 INFO:tasks.workunit.client.1.smithi122.stdout:4/814: mknod d0/d12/d15/c109 0 2024-03-22T02:29:03.390 INFO:tasks.workunit.client.1.smithi122.stdout:4/815: write d0/d12/d73/f83 [535769,9412] 0 2024-03-22T02:29:03.391 INFO:tasks.workunit.client.1.smithi122.stdout:4/816: read d0/d12/f69 [6328225,24739] 0 2024-03-22T02:29:03.393 INFO:tasks.workunit.client.1.smithi122.stdout:3/812: mknod d2/d9/dc/d57/da4/c10d 0 2024-03-22T02:29:03.394 INFO:tasks.workunit.client.1.smithi122.stdout:2/827: creat df/d1f/d33/d4d/f10e x:0 0 0 2024-03-22T02:29:03.397 INFO:tasks.workunit.client.1.smithi122.stdout:9/877: creat db/df/d3e/dfc/f11e x:0 0 0 2024-03-22T02:29:03.398 INFO:tasks.workunit.client.1.smithi122.stdout:0/669: dread d11/d14/d23/d24/d33/f79 [0,4194304] 0 2024-03-22T02:29:03.399 INFO:tasks.workunit.client.1.smithi122.stdout:9/878: write db/fb1 [4162408,20667] 0 2024-03-22T02:29:03.400 INFO:tasks.workunit.client.1.smithi122.stdout:3/813: dwrite d2/f7 [4194304,4194304] 0 2024-03-22T02:29:03.402 INFO:tasks.workunit.client.1.smithi122.stdout:7/797: getdents d0/d4/d20/d21/d9b/d5e/daa 0 2024-03-22T02:29:03.405 INFO:tasks.workunit.client.1.smithi122.stdout:5/744: link d8/d1a/ca1 d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/c103 0 2024-03-22T02:29:03.406 INFO:tasks.workunit.client.1.smithi122.stdout:3/814: truncate d2/d9/dc/d17/d28/d4c/da5/db4/db1/fb2 964737 0 2024-03-22T02:29:03.408 INFO:tasks.workunit.client.1.smithi122.stdout:1/761: rename dc/d3b/d8d/d72/d9c/db0/d67/dc6/f50 to dc/fea 0 2024-03-22T02:29:03.413 INFO:tasks.workunit.client.1.smithi122.stdout:7/798: symlink d0/d4/d20/d21/d9b/dff/l102 0 2024-03-22T02:29:03.414 INFO:tasks.workunit.client.1.smithi122.stdout:5/745: mkdir d8/d1a/d104 0 2024-03-22T02:29:03.417 INFO:tasks.workunit.client.1.smithi122.stdout:7/799: write d0/d4/d20/d21/d9b/d5e/f99 [4135365,124317] 0 2024-03-22T02:29:03.418 INFO:tasks.workunit.client.1.smithi122.stdout:2/828: rename df/d1f/d23/dd2/f96 to df/d1f/d33/dfd/f10f 0 2024-03-22T02:29:03.419 INFO:tasks.workunit.client.1.smithi122.stdout:6/869: dwrite d7/de/dd/df/f62 [0,4194304] 0 2024-03-22T02:29:03.424 INFO:tasks.workunit.client.1.smithi122.stdout:3/815: dread d2/d9/dc/df/fd9 [0,4194304] 0 2024-03-22T02:29:03.429 INFO:tasks.workunit.client.1.smithi122.stdout:7/800: creat d0/d4/d1d/db8/d9a/f103 x:0 0 0 2024-03-22T02:29:03.430 INFO:tasks.workunit.client.1.smithi122.stdout:8/920: sync 2024-03-22T02:29:03.431 INFO:tasks.workunit.client.1.smithi122.stdout:1/762: mkdir dc/d3b/d46/de7/deb 0 2024-03-22T02:29:03.431 INFO:tasks.workunit.client.1.smithi122.stdout:4/817: truncate d0/d12/d73/f83 1075716 0 2024-03-22T02:29:03.436 INFO:tasks.workunit.client.1.smithi122.stdout:9/879: creat db/df/d3e/d59/d11a/f11f x:0 0 0 2024-03-22T02:29:03.438 INFO:tasks.workunit.client.1.smithi122.stdout:3/816: rmdir d2/d9/dc/df/d3c 39 2024-03-22T02:29:03.443 INFO:tasks.workunit.client.1.smithi122.stdout:3/817: chown d2/d9/dc/d17/d28/d4c/da5/db4/d79/ff3 73 1 2024-03-22T02:29:03.444 INFO:tasks.workunit.client.1.smithi122.stdout:7/801: read d0/d4/f2d [668364,94401] 0 2024-03-22T02:29:03.445 INFO:tasks.workunit.client.1.smithi122.stdout:0/670: getdents d11/d16/d44 0 2024-03-22T02:29:03.445 INFO:tasks.workunit.client.1.smithi122.stdout:1/763: creat dc/d3b/d46/de0/fec x:0 0 0 2024-03-22T02:29:03.447 INFO:tasks.workunit.client.1.smithi122.stdout:9/880: dwrite db/d1b/da8/dba/fbd [0,4194304] 0 2024-03-22T02:29:03.451 INFO:tasks.workunit.client.1.smithi122.stdout:6/870: dread d7/de/dd/fc9 [0,4194304] 0 2024-03-22T02:29:03.455 INFO:tasks.workunit.client.1.smithi122.stdout:2/829: dread df/d1f/d33/d6a/f8c [0,4194304] 0 2024-03-22T02:29:03.456 INFO:tasks.workunit.client.1.smithi122.stdout:7/802: write d0/d30/f5b [4994151,108071] 0 2024-03-22T02:29:03.456 INFO:tasks.workunit.client.1.smithi122.stdout:3/818: mkdir d2/d8a/df6/d10e 0 2024-03-22T02:29:03.457 INFO:tasks.workunit.client.1.smithi122.stdout:6/871: dread - d7/de/dd/d20/de0/d54/d76/d86/d96/db2/f10e zero size 2024-03-22T02:29:03.461 INFO:tasks.workunit.client.1.smithi122.stdout:1/764: symlink dc/d55/led 0 2024-03-22T02:29:03.461 INFO:tasks.workunit.client.1.smithi122.stdout:3/819: write d2/d9/dc/fae [772635,26220] 0 2024-03-22T02:29:03.462 INFO:tasks.workunit.client.1.smithi122.stdout:9/881: dwrite db/df/f80 [4194304,4194304] 0 2024-03-22T02:29:03.463 INFO:tasks.workunit.client.1.smithi122.stdout:5/746: rename d8/d1a/fa4 to d8/d1e/d22/d3a/d46/d4f/d55/f105 0 2024-03-22T02:29:03.469 INFO:tasks.workunit.client.1.smithi122.stdout:1/765: creat dc/d3b/d8d/fee x:0 0 0 2024-03-22T02:29:03.471 INFO:tasks.workunit.client.1.smithi122.stdout:4/818: dwrite d0/d12/d61/f92 [0,4194304] 0 2024-03-22T02:29:03.473 INFO:tasks.workunit.client.1.smithi122.stdout:5/747: dread - d8/d1e/d22/d3a/d46/d4f/d9b/ddf/ff8 zero size 2024-03-22T02:29:03.477 INFO:tasks.workunit.client.1.smithi122.stdout:8/921: rename d1/d40/d71/db4 to d1/d8/d63/df0/d6a/d13a 0 2024-03-22T02:29:03.481 INFO:tasks.workunit.client.1.smithi122.stdout:6/872: dwrite d7/fdc [0,4194304] 0 2024-03-22T02:29:03.485 INFO:tasks.workunit.client.1.smithi122.stdout:9/882: truncate db/d1b/da8/fab 1045186 0 2024-03-22T02:29:03.488 INFO:tasks.workunit.client.1.smithi122.stdout:6/873: stat d7/de/dd/d20/de0/d54/d76/d86/d96/db2/lda 0 2024-03-22T02:29:03.488 INFO:tasks.workunit.client.1.smithi122.stdout:8/922: rename d1/d4d to d1/d9/d50/db1/d13b 0 2024-03-22T02:29:03.491 INFO:tasks.workunit.client.1.smithi122.stdout:6/874: chown d7/de/dd/d39/fb3 0 1 2024-03-22T02:29:03.491 INFO:tasks.workunit.client.1.smithi122.stdout:7/803: dread d0/d4/d1d/db8/f85 [0,4194304] 0 2024-03-22T02:29:03.491 INFO:tasks.workunit.client.1.smithi122.stdout:3/820: creat d2/d9/dc/d17/d28/d4c/da5/db4/f10f x:0 0 0 2024-03-22T02:29:03.493 INFO:tasks.workunit.client.1.smithi122.stdout:9/883: fsync db/d1b/da8/fcd 0 2024-03-22T02:29:03.493 INFO:tasks.workunit.client.1.smithi122.stdout:5/748: fsync d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf 0 2024-03-22T02:29:03.494 INFO:tasks.workunit.client.1.smithi122.stdout:0/671: write d11/d14/d23/f2d [646036,58461] 0 2024-03-22T02:29:03.494 INFO:tasks.workunit.client.1.smithi122.stdout:8/923: fdatasync d1/d40/d48/f6d 0 2024-03-22T02:29:03.496 INFO:tasks.workunit.client.1.smithi122.stdout:7/804: creat d0/d4/d1d/daf/f104 x:0 0 0 2024-03-22T02:29:03.498 INFO:tasks.workunit.client.1.smithi122.stdout:8/924: readlink d1/d60/d73/la9 0 2024-03-22T02:29:03.499 INFO:tasks.workunit.client.1.smithi122.stdout:2/830: write df/d1f/f5c [3342848,113078] 0 2024-03-22T02:29:03.500 INFO:tasks.workunit.client.1.smithi122.stdout:3/821: mkdir d2/d9/dc/d17/d28/d4c/d110 0 2024-03-22T02:29:03.502 INFO:tasks.workunit.client.1.smithi122.stdout:6/875: fdatasync d7/de/dd/d20/de0/d54/d76/d86/f87 0 2024-03-22T02:29:03.505 INFO:tasks.workunit.client.1.smithi122.stdout:9/884: dread db/d23/d1f/d57/d66/d73/fa4 [0,4194304] 0 2024-03-22T02:29:03.505 INFO:tasks.workunit.client.1.smithi122.stdout:4/819: truncate d0/d12/d73/f83 1801731 0 2024-03-22T02:29:03.507 INFO:tasks.workunit.client.1.smithi122.stdout:8/925: mkdir d1/d40/d71/d97/db8/df6/d13c 0 2024-03-22T02:29:03.509 INFO:tasks.workunit.client.1.smithi122.stdout:5/749: mkdir d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d106 0 2024-03-22T02:29:03.509 INFO:tasks.workunit.client.1.smithi122.stdout:8/926: write d1/d8/d2f/f38 [3881563,100677] 0 2024-03-22T02:29:03.512 INFO:tasks.workunit.client.1.smithi122.stdout:6/876: fdatasync d7/de/dd/d20/de0/d54/d76/fe5 0 2024-03-22T02:29:03.515 INFO:tasks.workunit.client.1.smithi122.stdout:1/766: dread dc/f2f [4194304,4194304] 0 2024-03-22T02:29:03.518 INFO:tasks.workunit.client.1.smithi122.stdout:8/927: creat d1/d40/d51/df5/f13d x:0 0 0 2024-03-22T02:29:03.521 INFO:tasks.workunit.client.1.smithi122.stdout:4/820: creat d0/d12/d15/d42/d4e/dd6/f10a x:0 0 0 2024-03-22T02:29:03.523 INFO:tasks.workunit.client.1.smithi122.stdout:6/877: creat d7/de/dd/d20/de0/f113 x:0 0 0 2024-03-22T02:29:03.523 INFO:tasks.workunit.client.1.smithi122.stdout:0/672: dread d11/d14/d23/d24/d33/f8e [0,4194304] 0 2024-03-22T02:29:03.524 INFO:tasks.workunit.client.1.smithi122.stdout:9/885: write db/d23/d9a/fa3 [1364424,125073] 0 2024-03-22T02:29:03.526 INFO:tasks.workunit.client.1.smithi122.stdout:8/928: creat d1/d8/d8a/dfb/f13e x:0 0 0 2024-03-22T02:29:03.527 INFO:tasks.workunit.client.1.smithi122.stdout:4/821: stat d0/d12/d15/d42/c95 0 2024-03-22T02:29:03.528 INFO:tasks.workunit.client.1.smithi122.stdout:8/929: write d1/d40/d71/dc9/f72 [2607770,38628] 0 2024-03-22T02:29:03.528 INFO:tasks.workunit.client.1.smithi122.stdout:1/767: creat dc/d3b/d46/de3/fef x:0 0 0 2024-03-22T02:29:03.531 INFO:tasks.workunit.client.1.smithi122.stdout:5/750: rename d8/f19 to d8/d1e/d22/f107 0 2024-03-22T02:29:03.532 INFO:tasks.workunit.client.1.smithi122.stdout:6/878: creat d7/de/dd/d20/de0/d54/d76/d86/d96/db2/f114 x:0 0 0 2024-03-22T02:29:03.533 INFO:tasks.workunit.client.1.smithi122.stdout:9/886: rmdir db/df/d3e/d59/d11a/dd1 39 2024-03-22T02:29:03.534 INFO:tasks.workunit.client.1.smithi122.stdout:5/751: read d8/fb0 [978341,85975] 0 2024-03-22T02:29:03.537 INFO:tasks.workunit.client.1.smithi122.stdout:5/752: dread - d8/d74/fe1 zero size 2024-03-22T02:29:03.538 INFO:tasks.workunit.client.1.smithi122.stdout:4/822: dwrite d0/d12/d15/d3d/dd2/fe1 [0,4194304] 0 2024-03-22T02:29:03.540 INFO:tasks.workunit.client.1.smithi122.stdout:3/822: link d2/d9/dc/d57/da4/dc7/lca d2/d9/dc/df/l111 0 2024-03-22T02:29:03.549 INFO:tasks.workunit.client.1.smithi122.stdout:5/753: mkdir d8/d1e/d22/dd7/d108 0 2024-03-22T02:29:03.550 INFO:tasks.workunit.client.1.smithi122.stdout:1/768: dread dc/d3b/d8d/d72/d9c/db0/d71/fda [0,4194304] 0 2024-03-22T02:29:03.552 INFO:tasks.workunit.client.1.smithi122.stdout:6/879: rename d7/de/dd/d20/de0/d54/d76/f112 to d7/de/dd/d9b/f115 0 2024-03-22T02:29:03.555 INFO:tasks.workunit.client.1.smithi122.stdout:3/823: creat d2/d9/dc/d17/d28/d4c/f112 x:0 0 0 2024-03-22T02:29:03.557 INFO:tasks.workunit.client.1.smithi122.stdout:3/824: read d2/f10b [295590,20024] 0 2024-03-22T02:29:03.559 INFO:tasks.workunit.client.1.smithi122.stdout:5/754: mknod d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/c109 0 2024-03-22T02:29:03.560 INFO:tasks.workunit.client.1.smithi122.stdout:6/880: mknod d7/de/dd/d39/dc2/c116 0 2024-03-22T02:29:03.564 INFO:tasks.workunit.client.1.smithi122.stdout:8/930: dwrite d1/d8/d2f/f11c [0,4194304] 0 2024-03-22T02:29:03.569 INFO:tasks.workunit.client.1.smithi122.stdout:4/823: write d0/d12/d15/d2d/f7d [337480,91587] 0 2024-03-22T02:29:03.569 INFO:tasks.workunit.client.1.smithi122.stdout:9/887: dwrite db/df/d3e/d81/d25/fae [4194304,4194304] 0 2024-03-22T02:29:03.572 INFO:tasks.workunit.client.1.smithi122.stdout:9/888: readlink db/d23/lf8 0 2024-03-22T02:29:03.577 INFO:tasks.workunit.client.1.smithi122.stdout:8/931: mkdir d1/d40/d51/dfa/d13f 0 2024-03-22T02:29:03.579 INFO:tasks.workunit.client.1.smithi122.stdout:6/881: fdatasync d7/de/d27/d52/f57 0 2024-03-22T02:29:03.583 INFO:tasks.workunit.client.1.smithi122.stdout:4/824: dread d0/d12/d15/d2d/f38 [0,4194304] 0 2024-03-22T02:29:03.585 INFO:tasks.workunit.client.1.smithi122.stdout:3/825: creat d2/d9/dc/df/f113 x:0 0 0 2024-03-22T02:29:03.586 INFO:tasks.workunit.client.1.smithi122.stdout:6/882: fdatasync d7/de/dd/d23/d7f/fe3 0 2024-03-22T02:29:03.589 INFO:tasks.workunit.client.1.smithi122.stdout:4/825: rmdir d0/d12/d15/d3d 39 2024-03-22T02:29:03.591 INFO:tasks.workunit.client.1.smithi122.stdout:4/826: read - d0/d12/d15/d42/d4e/de/da1/dbf/d103/fe4 zero size 2024-03-22T02:29:03.592 INFO:tasks.workunit.client.1.smithi122.stdout:8/932: rename d1/d9/d50/l6c to d1/d9/d50/l140 0 2024-03-22T02:29:03.594 INFO:tasks.workunit.client.1.smithi122.stdout:6/883: symlink d7/de/dd/d20/de0/d54/d76/df3/l117 0 2024-03-22T02:29:03.595 INFO:tasks.workunit.client.1.smithi122.stdout:4/827: stat d0/d12/d15/d42/d4e/de/da1/dbf/d103/fe4 0 2024-03-22T02:29:03.598 INFO:tasks.workunit.client.1.smithi122.stdout:1/769: dwrite dc/d4c/f8b [0,4194304] 0 2024-03-22T02:29:03.602 INFO:tasks.workunit.client.1.smithi122.stdout:4/828: symlink d0/d12/l10b 0 2024-03-22T02:29:03.602 INFO:tasks.workunit.client.1.smithi122.stdout:6/884: creat d7/de/dd/d20/de0/d54/f118 x:0 0 0 2024-03-22T02:29:03.605 INFO:tasks.workunit.client.1.smithi122.stdout:6/885: symlink d7/de/d27/d52/l119 0 2024-03-22T02:29:03.608 INFO:tasks.workunit.client.1.smithi122.stdout:4/829: read d0/d12/d15/d42/d4e/f34 [1656321,55169] 0 2024-03-22T02:29:03.608 INFO:tasks.workunit.client.1.smithi122.stdout:5/755: dread d8/d1e/d22/d3a/d46/d4f/d55/dc7/f44 [0,4194304] 0 2024-03-22T02:29:03.614 INFO:tasks.workunit.client.1.smithi122.stdout:6/886: link d7/de/dd/d39/f106 d7/de/dd/d20/de0/d54/d76/d86/f11a 0 2024-03-22T02:29:03.620 INFO:tasks.workunit.client.1.smithi122.stdout:4/830: getdents d0/d12/d15/d2d/d75 0 2024-03-22T02:29:03.625 INFO:tasks.workunit.client.1.smithi122.stdout:8/933: dwrite d1/f1a [0,4194304] 0 2024-03-22T02:29:03.630 INFO:tasks.workunit.client.1.smithi122.stdout:6/887: getdents d7/de 0 2024-03-22T02:29:03.634 INFO:tasks.workunit.client.1.smithi122.stdout:8/934: rmdir d1/d8/d2f 39 2024-03-22T02:29:03.634 INFO:tasks.workunit.client.1.smithi122.stdout:1/770: dwrite dc/d3b/d8d/d72/d9c/db0/d71/fda [4194304,4194304] 0 2024-03-22T02:29:03.647 INFO:tasks.workunit.client.1.smithi122.stdout:5/756: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f94 [4194304,4194304] 0 2024-03-22T02:29:03.650 INFO:tasks.workunit.client.1.smithi122.stdout:4/831: dwrite d0/d12/d61/fe5 [0,4194304] 0 2024-03-22T02:29:03.651 INFO:tasks.workunit.client.1.smithi122.stdout:3/826: dread d2/d9/dc/df/d38/f5d [0,4194304] 0 2024-03-22T02:29:03.655 INFO:tasks.workunit.client.1.smithi122.stdout:3/827: chown d2/d9/dc/d17/d51/de0/c92 7083 1 2024-03-22T02:29:03.657 INFO:tasks.workunit.client.1.smithi122.stdout:3/828: fsync d2/d9/dc/d17/d28/f4d 0 2024-03-22T02:29:03.666 INFO:tasks.workunit.client.1.smithi122.stdout:8/935: dread d1/d40/d51/df5/dae/dfe/d45/fe3 [0,4194304] 0 2024-03-22T02:29:03.672 INFO:tasks.workunit.client.1.smithi122.stdout:8/936: rename d1/d40/d51/df5/dae/dfe/cd1 to d1/d40/d51/d127/c141 0 2024-03-22T02:29:03.677 INFO:tasks.workunit.client.1.smithi122.stdout:8/937: symlink d1/d9/l142 0 2024-03-22T02:29:03.678 INFO:tasks.workunit.client.1.smithi122.stdout:4/832: write d0/d12/d15/d2d/db5/fe9 [2570070,82106] 0 2024-03-22T02:29:03.680 INFO:tasks.workunit.client.1.smithi122.stdout:8/938: readlink d1/d9/d50/l5f 0 2024-03-22T02:29:03.685 INFO:tasks.workunit.client.1.smithi122.stdout:4/833: fdatasync d0/d25/f5e 0 2024-03-22T02:29:03.689 INFO:tasks.workunit.client.1.smithi122.stdout:6/888: dwrite d7/de/dd/d39/f106 [0,4194304] 0 2024-03-22T02:29:03.699 INFO:tasks.workunit.client.1.smithi122.stdout:5/757: dwrite d8/fb0 [0,4194304] 0 2024-03-22T02:29:03.700 INFO:tasks.workunit.client.1.smithi122.stdout:3/829: dwrite d2/d9/dc/d17/d28/d4c/da5/db4/f56 [0,4194304] 0 2024-03-22T02:29:03.708 INFO:tasks.workunit.client.1.smithi122.stdout:4/834: dwrite d0/d12/d15/ff5 [0,4194304] 0 2024-03-22T02:29:03.710 INFO:tasks.workunit.client.1.smithi122.stdout:4/835: chown d0/d12/da2/lc5 339536359 1 2024-03-22T02:29:03.712 INFO:tasks.workunit.client.1.smithi122.stdout:3/830: truncate d2/d9/dc/d17/d28/f2d 4300976 0 2024-03-22T02:29:03.712 INFO:tasks.workunit.client.1.smithi122.stdout:1/771: dread dc/d3b/f68 [0,4194304] 0 2024-03-22T02:29:03.718 INFO:tasks.workunit.client.1.smithi122.stdout:3/831: creat d2/d9/dc/d57/f114 x:0 0 0 2024-03-22T02:29:03.718 INFO:tasks.workunit.client.1.smithi122.stdout:4/836: mkdir d0/d12/d15/d42/d4e/dd6/d10c 0 2024-03-22T02:29:03.720 INFO:tasks.workunit.client.1.smithi122.stdout:6/889: dread d7/de/dd/d20/de0/d54/fb4 [0,4194304] 0 2024-03-22T02:29:03.726 INFO:tasks.workunit.client.1.smithi122.stdout:3/832: creat d2/d9/dc/d17/d2b/f115 x:0 0 0 2024-03-22T02:29:03.727 INFO:tasks.workunit.client.1.smithi122.stdout:3/833: stat d2/d9/dc/d17/d28/l98 0 2024-03-22T02:29:03.729 INFO:tasks.workunit.client.1.smithi122.stdout:4/837: rename d0/d12/d15/d42/d4e/de/da1/dbf/l104 to d0/d12/d15/d2d/db5/l10d 0 2024-03-22T02:29:03.730 INFO:tasks.workunit.client.1.smithi122.stdout:4/838: fdatasync d0/d12/d15/d42/d57/fc3 0 2024-03-22T02:29:03.737 INFO:tasks.workunit.client.1.smithi122.stdout:6/890: write d7/de/dd/d39/f77 [105974,18302] 0 2024-03-22T02:29:03.739 INFO:tasks.workunit.client.1.smithi122.stdout:8/939: dread d1/d60/da4/f102 [0,4194304] 0 2024-03-22T02:29:03.746 INFO:tasks.workunit.client.1.smithi122.stdout:4/839: dwrite d0/f6a [0,4194304] 0 2024-03-22T02:29:03.749 INFO:tasks.workunit.client.1.smithi122.stdout:5/758: dwrite d8/d1e/d51/fb1 [0,4194304] 0 2024-03-22T02:29:03.751 INFO:tasks.workunit.client.1.smithi122.stdout:1/772: dwrite dc/d3b/d8d/d72/d9c/db0/dab/fbd [0,4194304] 0 2024-03-22T02:29:03.752 INFO:tasks.workunit.client.1.smithi122.stdout:8/940: getdents d1/d8/d63/df0/d138 0 2024-03-22T02:29:03.754 INFO:tasks.workunit.client.1.smithi122.stdout:4/840: chown d0/d12/d15/d42/d4e/l63 6 1 2024-03-22T02:29:03.759 INFO:tasks.workunit.client.1.smithi122.stdout:1/773: rmdir dc/d3b/d8d/d72/d9c/db0/d67 39 2024-03-22T02:29:03.760 INFO:tasks.workunit.client.1.smithi122.stdout:4/841: chown d0/d12/d15/d42/d4e/de/d91/c105 1020740643 1 2024-03-22T02:29:03.763 INFO:tasks.workunit.client.1.smithi122.stdout:8/941: mknod d1/d40/d71/d97/db8/df6/d12d/c143 0 2024-03-22T02:29:03.772 INFO:tasks.workunit.client.1.smithi122.stdout:5/759: dwrite d8/d1e/d22/da3/ddc/f10 [4194304,4194304] 0 2024-03-22T02:29:03.772 INFO:tasks.workunit.client.1.smithi122.stdout:1/774: getdents dc/d3b/d46/db5/dba 0 2024-03-22T02:29:03.776 INFO:tasks.workunit.client.1.smithi122.stdout:5/760: write d8/d1a/f70 [195548,70093] 0 2024-03-22T02:29:03.777 INFO:tasks.workunit.client.1.smithi122.stdout:4/842: dwrite d0/d12/d61/fe5 [0,4194304] 0 2024-03-22T02:29:03.778 INFO:tasks.workunit.client.1.smithi122.stdout:5/761: chown d8/d1e/d22/dd7 9374 1 2024-03-22T02:29:03.778 INFO:tasks.workunit.client.1.smithi122.stdout:3/834: dread d2/d9/dc/d17/d51/f7c [0,4194304] 0 2024-03-22T02:29:03.781 INFO:tasks.workunit.client.1.smithi122.stdout:4/843: chown d0/d12/d15/d42/d4e/ldc 215056050 1 2024-03-22T02:29:03.782 INFO:tasks.workunit.client.1.smithi122.stdout:1/775: dread dc/d4c/f8b [0,4194304] 0 2024-03-22T02:29:03.782 INFO:tasks.workunit.client.1.smithi122.stdout:3/835: write d2/d9/dc/d17/d28/f108 [798775,822] 0 2024-03-22T02:29:03.784 INFO:tasks.workunit.client.1.smithi122.stdout:4/844: read - d0/d12/d15/d42/d4e/de/fe3 zero size 2024-03-22T02:29:03.784 INFO:tasks.workunit.client.1.smithi122.stdout:6/891: dread d7/f3b [0,4194304] 0 2024-03-22T02:29:03.787 INFO:tasks.workunit.client.1.smithi122.stdout:1/776: symlink dc/d3b/d8d/d72/lf0 0 2024-03-22T02:29:03.788 INFO:tasks.workunit.client.1.smithi122.stdout:4/845: write d0/d12/d61/dce/fd4 [498444,20279] 0 2024-03-22T02:29:03.793 INFO:tasks.workunit.client.1.smithi122.stdout:1/777: creat dc/d3b/d8d/d72/d9c/db0/ff1 x:0 0 0 2024-03-22T02:29:03.793 INFO:tasks.workunit.client.1.smithi122.stdout:4/846: dread - d0/d12/d15/d42/d4e/de/f9b zero size 2024-03-22T02:29:03.796 INFO:tasks.workunit.client.1.smithi122.stdout:6/892: dwrite d7/de/dd/d23/ff6 [0,4194304] 0 2024-03-22T02:29:03.800 INFO:tasks.workunit.client.1.smithi122.stdout:4/847: creat d0/d12/d15/d42/d4e/de/f10e x:0 0 0 2024-03-22T02:29:03.804 INFO:tasks.workunit.client.1.smithi122.stdout:6/893: getdents d7/de/dd/d23/d7f 0 2024-03-22T02:29:03.804 INFO:tasks.workunit.client.1.smithi122.stdout:1/778: dwrite dc/d3b/d8d/d72/d9c/db0/d71/fda [0,4194304] 0 2024-03-22T02:29:03.808 INFO:tasks.workunit.client.1.smithi122.stdout:1/779: chown dc/d2b/l3a 76 1 2024-03-22T02:29:03.811 INFO:tasks.workunit.client.1.smithi122.stdout:8/942: dread d1/d8/dba/dc0/ff3 [0,4194304] 0 2024-03-22T02:29:03.811 INFO:tasks.workunit.client.1.smithi122.stdout:6/894: unlink d7/de/f92 0 2024-03-22T02:29:03.812 INFO:tasks.workunit.client.1.smithi122.stdout:5/762: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/f3d [1998131,67218] 0 2024-03-22T02:29:03.815 INFO:tasks.workunit.client.1.smithi122.stdout:4/848: symlink d0/d12/d15/d42/d4e/de/l10f 0 2024-03-22T02:29:03.816 INFO:tasks.workunit.client.1.smithi122.stdout:3/836: dread d2/d9/dc/d17/d28/d4c/da5/db4/fd3 [0,4194304] 0 2024-03-22T02:29:03.818 INFO:tasks.workunit.client.1.smithi122.stdout:1/780: mknod dc/d2b/d86/dbb/cf2 0 2024-03-22T02:29:03.819 INFO:tasks.workunit.client.1.smithi122.stdout:5/763: mknod d8/d74/c10a 0 2024-03-22T02:29:03.821 INFO:tasks.workunit.client.1.smithi122.stdout:8/943: creat d1/d40/d71/dc9/f144 x:0 0 0 2024-03-22T02:29:03.822 INFO:tasks.workunit.client.1.smithi122.stdout:1/781: write dc/fd [2849143,121559] 0 2024-03-22T02:29:03.823 INFO:tasks.workunit.client.1.smithi122.stdout:8/944: truncate d1/d40/d71/f80 448644 0 2024-03-22T02:29:03.824 INFO:tasks.workunit.client.1.smithi122.stdout:4/849: truncate d0/d12/d15/d42/d57/f64 44611 0 2024-03-22T02:29:03.825 INFO:tasks.workunit.client.1.smithi122.stdout:1/782: write dc/d2b/f33 [2304548,106261] 0 2024-03-22T02:29:03.826 INFO:tasks.workunit.client.1.smithi122.stdout:4/850: truncate d0/fde 357295 0 2024-03-22T02:29:03.826 INFO:tasks.workunit.client.1.smithi122.stdout:1/783: read - dc/d3b/d46/f9e zero size 2024-03-22T02:29:03.827 INFO:tasks.workunit.client.1.smithi122.stdout:3/837: unlink d2/d9/dc/f36 0 2024-03-22T02:29:03.830 INFO:tasks.workunit.client.1.smithi122.stdout:8/945: mkdir d1/d9/d50/da6/d145 0 2024-03-22T02:29:03.831 INFO:tasks.workunit.client.1.smithi122.stdout:8/946: chown d1/d81/lff 97125 1 2024-03-22T02:29:03.832 INFO:tasks.workunit.client.1.smithi122.stdout:4/851: symlink d0/d12/d61/l110 0 2024-03-22T02:29:03.832 INFO:tasks.workunit.client.1.smithi122.stdout:8/947: write d1/d40/d48/faf [1343656,19949] 0 2024-03-22T02:29:03.832 INFO:tasks.workunit.client.1.smithi122.stdout:7/805: sync 2024-03-22T02:29:03.834 INFO:tasks.workunit.client.1.smithi122.stdout:5/764: creat d8/d1e/d22/f10b x:0 0 0 2024-03-22T02:29:03.837 INFO:tasks.workunit.client.1.smithi122.stdout:4/852: creat d0/d12/d15/d42/d4e/f111 x:0 0 0 2024-03-22T02:29:03.837 INFO:tasks.workunit.client.1.smithi122.stdout:8/948: dread d1/d8/d63/df0/f8b [0,4194304] 0 2024-03-22T02:29:03.839 INFO:tasks.workunit.client.1.smithi122.stdout:7/806: unlink d0/d4/d20/l42 0 2024-03-22T02:29:03.840 INFO:tasks.workunit.client.1.smithi122.stdout:7/807: chown d0/d4/d20/d21/l31 283 1 2024-03-22T02:29:03.845 INFO:tasks.workunit.client.1.smithi122.stdout:3/838: creat d2/d9/dc/d17/f116 x:0 0 0 2024-03-22T02:29:03.846 INFO:tasks.workunit.client.1.smithi122.stdout:5/765: mknod d8/d1e/d22/d3a/d46/c10c 0 2024-03-22T02:29:03.847 INFO:tasks.workunit.client.1.smithi122.stdout:4/853: creat d0/d12/d15/da7/f112 x:0 0 0 2024-03-22T02:29:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:03 smithi122 ceph-mon[36403]: pgmap v25: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 80 MiB/s rd, 208 MiB/s wr, 442 op/s 2024-03-22T02:29:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:03 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:03.848 INFO:tasks.workunit.client.1.smithi122.stdout:8/949: mkdir d1/d40/d71/d97/d146 0 2024-03-22T02:29:03.850 INFO:tasks.workunit.client.1.smithi122.stdout:1/784: truncate dc/d2b/f33 2852809 0 2024-03-22T02:29:03.856 INFO:tasks.workunit.client.1.smithi122.stdout:8/950: creat d1/d8/ddc/f147 x:0 0 0 2024-03-22T02:29:03.857 INFO:tasks.workunit.client.1.smithi122.stdout:8/951: truncate d1/d40/d71/f80 911375 0 2024-03-22T02:29:03.858 INFO:tasks.workunit.client.1.smithi122.stdout:6/895: creat d7/de/dd/d20/de0/d54/d76/f11b x:0 0 0 2024-03-22T02:29:03.859 INFO:tasks.workunit.client.1.smithi122.stdout:1/785: symlink dc/d3b/d8d/d72/lf3 0 2024-03-22T02:29:03.864 INFO:tasks.workunit.client.1.smithi122.stdout:5/766: mknod d8/c10d 0 2024-03-22T02:29:03.865 INFO:tasks.workunit.client.1.smithi122.stdout:5/767: fsync d8/d1e/d22/d3a/d46/f6f 0 2024-03-22T02:29:03.867 INFO:tasks.workunit.client.1.smithi122.stdout:5/768: read d8/d3b/faf [7260301,65790] 0 2024-03-22T02:29:03.870 INFO:tasks.workunit.client.1.smithi122.stdout:3/839: creat d2/f117 x:0 0 0 2024-03-22T02:29:03.873 INFO:tasks.workunit.client.1.smithi122.stdout:4/854: fsync d0/d12/d15/d42/d4e/f23 0 2024-03-22T02:29:03.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:03 smithi103 ceph-mon[29258]: pgmap v25: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 80 MiB/s rd, 208 MiB/s wr, 442 op/s 2024-03-22T02:29:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:03 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:03.877 INFO:tasks.workunit.client.1.smithi122.stdout:6/896: symlink d7/de/dd/d20/de0/d54/d76/d86/l11c 0 2024-03-22T02:29:03.878 INFO:tasks.workunit.client.1.smithi122.stdout:1/786: dwrite dc/d3b/d8d/d72/f96 [0,4194304] 0 2024-03-22T02:29:03.880 INFO:tasks.workunit.client.1.smithi122.stdout:5/769: rmdir d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7 39 2024-03-22T02:29:03.880 INFO:tasks.workunit.client.1.smithi122.stdout:6/897: dread - d7/de/dd/d20/de0/d54/d76/df3/f107 zero size 2024-03-22T02:29:03.882 INFO:tasks.workunit.client.1.smithi122.stdout:1/787: chown dc/d3b/d8d/d72/d9c/db0/d90 510056226 1 2024-03-22T02:29:03.882 INFO:tasks.workunit.client.1.smithi122.stdout:7/808: link d0/d4/c6 d0/d4/de9/c105 0 2024-03-22T02:29:03.884 INFO:tasks.workunit.client.1.smithi122.stdout:8/952: creat d1/d8/d63/f148 x:0 0 0 2024-03-22T02:29:03.887 INFO:tasks.workunit.client.1.smithi122.stdout:0/673: sync 2024-03-22T02:29:03.888 INFO:tasks.workunit.client.1.smithi122.stdout:9/889: sync 2024-03-22T02:29:03.888 INFO:tasks.workunit.client.1.smithi122.stdout:2/831: sync 2024-03-22T02:29:03.888 INFO:tasks.workunit.client.1.smithi122.stdout:3/840: mkdir d2/d9/dc/d17/d28/d4c/da5/db4/d118 0 2024-03-22T02:29:03.895 INFO:tasks.workunit.client.1.smithi122.stdout:9/890: unlink db/df/d3e/d59/d11a/c97 0 2024-03-22T02:29:03.897 INFO:tasks.workunit.client.1.smithi122.stdout:5/770: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f5d [1636016,108599] 0 2024-03-22T02:29:03.899 INFO:tasks.workunit.client.1.smithi122.stdout:7/809: creat d0/d4/d1d/df6/de0/f106 x:0 0 0 2024-03-22T02:29:03.902 INFO:tasks.workunit.client.1.smithi122.stdout:8/953: symlink d1/d40/l149 0 2024-03-22T02:29:03.902 INFO:tasks.workunit.client.1.smithi122.stdout:7/810: write d0/d4/d1d/df6/d5d/dab/fce [793802,112674] 0 2024-03-22T02:29:03.903 INFO:tasks.workunit.client.1.smithi122.stdout:6/898: dwrite d7/de/d27/d52/fe1 [0,4194304] 0 2024-03-22T02:29:03.903 INFO:tasks.workunit.client.1.smithi122.stdout:4/855: mkdir d0/d12/d15/d3d/d90/df3/d113 0 2024-03-22T02:29:03.908 INFO:tasks.workunit.client.1.smithi122.stdout:1/788: truncate dc/d55/fc3 1415415 0 2024-03-22T02:29:03.908 INFO:tasks.workunit.client.1.smithi122.stdout:1/789: readlink dc/d2b/l66 0 2024-03-22T02:29:03.909 INFO:tasks.workunit.client.1.smithi122.stdout:5/771: creat d8/d1a/d83/f10e x:0 0 0 2024-03-22T02:29:03.912 INFO:tasks.workunit.client.1.smithi122.stdout:9/891: symlink db/df/d3e/d59/d11a/dd1/l120 0 2024-03-22T02:29:03.913 INFO:tasks.workunit.client.1.smithi122.stdout:4/856: mknod d0/d25/df2/c114 0 2024-03-22T02:29:03.915 INFO:tasks.workunit.client.1.smithi122.stdout:7/811: dwrite d0/d4/d20/d21/dda/fe1 [0,4194304] 0 2024-03-22T02:29:03.918 INFO:tasks.workunit.client.1.smithi122.stdout:0/674: link d11/d14/d1b/f38 d11/d14/d1d/d27/fe4 0 2024-03-22T02:29:03.921 INFO:tasks.workunit.client.1.smithi122.stdout:6/899: fsync d7/de/dd/d23/d7f/fbb 0 2024-03-22T02:29:03.923 INFO:tasks.workunit.client.1.smithi122.stdout:2/832: dwrite fd [0,4194304] 0 2024-03-22T02:29:03.927 INFO:tasks.workunit.client.1.smithi122.stdout:7/812: read d0/d4/d20/d21/d9b/dbb/fd9 [879780,14385] 0 2024-03-22T02:29:03.928 INFO:tasks.workunit.client.1.smithi122.stdout:1/790: dread - dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/fb7 zero size 2024-03-22T02:29:03.929 INFO:tasks.workunit.client.1.smithi122.stdout:7/813: chown d0/d4/d1d/db8/d98 19 1 2024-03-22T02:29:03.931 INFO:tasks.workunit.client.1.smithi122.stdout:1/791: fsync dc/d2b/f57 0 2024-03-22T02:29:03.932 INFO:tasks.workunit.client.1.smithi122.stdout:5/772: dwrite d8/d1e/d22/fde [0,4194304] 0 2024-03-22T02:29:03.933 INFO:tasks.workunit.client.1.smithi122.stdout:9/892: rename db/d23/fa9 to db/df/d3e/d59/f121 0 2024-03-22T02:29:03.936 INFO:tasks.workunit.client.1.smithi122.stdout:1/792: chown dc/d2b/d86/dbb/fdc 5907 1 2024-03-22T02:29:03.937 INFO:tasks.workunit.client.1.smithi122.stdout:7/814: write d0/d4/d20/d21/d9b/dbb/fc7 [1833095,82456] 0 2024-03-22T02:29:03.937 INFO:tasks.workunit.client.1.smithi122.stdout:2/833: dwrite df/d1f/d33/d4d/d94/dc2/f2f [0,4194304] 0 2024-03-22T02:29:03.940 INFO:tasks.workunit.client.1.smithi122.stdout:0/675: symlink d11/d14/d23/d24/d33/d67/dc2/le5 0 2024-03-22T02:29:03.941 INFO:tasks.workunit.client.1.smithi122.stdout:3/841: getdents d2/d9/dc/d17/d28/d4c/da5/db4 0 2024-03-22T02:29:03.943 INFO:tasks.workunit.client.1.smithi122.stdout:0/676: read - d11/d14/d1d/d27/d9d/fcb zero size 2024-03-22T02:29:03.944 INFO:tasks.workunit.client.1.smithi122.stdout:6/900: mkdir d7/de/dd/d20/de0/d54/d76/df3/d11d 0 2024-03-22T02:29:03.944 INFO:tasks.workunit.client.1.smithi122.stdout:3/842: fsync d2/d9/dc/d17/d2b/f67 0 2024-03-22T02:29:03.948 INFO:tasks.workunit.client.1.smithi122.stdout:9/893: symlink db/d23/d1f/l122 0 2024-03-22T02:29:03.948 INFO:tasks.workunit.client.1.smithi122.stdout:5/773: mknod d8/d1e/d22/dd7/c10f 0 2024-03-22T02:29:03.949 INFO:tasks.workunit.client.1.smithi122.stdout:8/954: dwrite d1/d9/f1f [0,4194304] 0 2024-03-22T02:29:03.950 INFO:tasks.workunit.client.1.smithi122.stdout:1/793: symlink dc/d2b/d86/lf4 0 2024-03-22T02:29:03.950 INFO:tasks.workunit.client.1.smithi122.stdout:9/894: chown db/df/d3e/d59/db5/dc4 133527092 1 2024-03-22T02:29:03.954 INFO:tasks.workunit.client.1.smithi122.stdout:9/895: write db/df/d3e/d81/d25/d41/fbc [371313,55758] 0 2024-03-22T02:29:03.960 INFO:tasks.workunit.client.1.smithi122.stdout:5/774: write d8/d1e/d22/f107 [3946188,130831] 0 2024-03-22T02:29:03.960 INFO:tasks.workunit.client.1.smithi122.stdout:3/843: rmdir d2/d9/dc/df/d49 39 2024-03-22T02:29:03.962 INFO:tasks.workunit.client.1.smithi122.stdout:3/844: chown d2/d9/dc/d17/d28/d4c/da5 78166 1 2024-03-22T02:29:03.963 INFO:tasks.workunit.client.1.smithi122.stdout:8/955: rename d1/d8/d63/df0/d6a/c9c to d1/d81/c14a 0 2024-03-22T02:29:03.963 INFO:tasks.workunit.client.1.smithi122.stdout:7/815: link d0/d4/d20/c34 d0/d4/d1d/db8/d98/c107 0 2024-03-22T02:29:03.964 INFO:tasks.workunit.client.1.smithi122.stdout:1/794: mknod dc/d3b/d8d/d72/d9c/de8/cf5 0 2024-03-22T02:29:03.965 INFO:tasks.workunit.client.1.smithi122.stdout:5/775: creat d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/f110 x:0 0 0 2024-03-22T02:29:03.968 INFO:tasks.workunit.client.1.smithi122.stdout:3/845: mkdir d2/d9/dc/d17/d2b/d6a/d119 0 2024-03-22T02:29:03.974 INFO:tasks.workunit.client.1.smithi122.stdout:8/956: dwrite d1/d9/d50/db1/d13b/fda [4194304,4194304] 0 2024-03-22T02:29:03.975 INFO:tasks.workunit.client.1.smithi122.stdout:7/816: rename d0/d4/d20/d21/d9b to d0/d4/d1d/d108 0 2024-03-22T02:29:03.979 INFO:tasks.workunit.client.1.smithi122.stdout:6/901: dread d7/de/dd/d20/fa5 [0,4194304] 0 2024-03-22T02:29:03.980 INFO:tasks.workunit.client.1.smithi122.stdout:9/896: dread db/df/d3e/d81/d3d/d86/dd8/fd9 [0,4194304] 0 2024-03-22T02:29:03.983 INFO:tasks.workunit.client.1.smithi122.stdout:8/957: creat d1/d8/d63/df0/d6a/f14b x:0 0 0 2024-03-22T02:29:03.984 INFO:tasks.workunit.client.1.smithi122.stdout:7/817: symlink d0/d4/d1d/db8/d9a/l109 0 2024-03-22T02:29:03.984 INFO:tasks.workunit.client.1.smithi122.stdout:4/857: write d0/d12/d15/d42/d4e/de/f44 [2785730,10414] 0 2024-03-22T02:29:03.986 INFO:tasks.workunit.client.1.smithi122.stdout:7/818: chown d0/d4/d1d/df6 13683336 1 2024-03-22T02:29:03.987 INFO:tasks.workunit.client.1.smithi122.stdout:2/834: truncate df/d1f/d33/f54 695773 0 2024-03-22T02:29:03.988 INFO:tasks.workunit.client.1.smithi122.stdout:7/819: fdatasync d0/f87 0 2024-03-22T02:29:03.989 INFO:tasks.workunit.client.1.smithi122.stdout:3/846: creat d2/d9/dc/f11a x:0 0 0 2024-03-22T02:29:03.993 INFO:tasks.workunit.client.1.smithi122.stdout:0/677: dwrite d11/d16/d37/f9a [0,4194304] 0 2024-03-22T02:29:03.993 INFO:tasks.workunit.client.1.smithi122.stdout:4/858: fdatasync d0/d12/d15/d3d/f5c 0 2024-03-22T02:29:03.995 INFO:tasks.workunit.client.1.smithi122.stdout:1/795: write dc/f80 [2614583,47891] 0 2024-03-22T02:29:03.996 INFO:tasks.workunit.client.1.smithi122.stdout:9/897: mknod db/df/d3e/c123 0 2024-03-22T02:29:04.001 INFO:tasks.workunit.client.1.smithi122.stdout:5/776: truncate d8/d1e/d22/d3a/d46/d4f/fca 2449268 0 2024-03-22T02:29:04.002 INFO:tasks.workunit.client.1.smithi122.stdout:2/835: dread - df/d1f/d33/d4d/d94/dc2/d51/d67/db0/fc9 zero size 2024-03-22T02:29:04.003 INFO:tasks.workunit.client.1.smithi122.stdout:8/958: symlink d1/d40/d51/dfa/l14c 0 2024-03-22T02:29:04.005 INFO:tasks.workunit.client.1.smithi122.stdout:8/959: write d1/d81/fa5 [1379395,10967] 0 2024-03-22T02:29:04.006 INFO:tasks.workunit.client.1.smithi122.stdout:4/859: mkdir d0/d12/d15/df7/d115 0 2024-03-22T02:29:04.009 INFO:tasks.workunit.client.1.smithi122.stdout:7/820: dread d0/d4/f2d [0,4194304] 0 2024-03-22T02:29:04.010 INFO:tasks.workunit.client.1.smithi122.stdout:0/678: mknod d11/d14/d1d/d27/d2f/ce6 0 2024-03-22T02:29:04.012 INFO:tasks.workunit.client.1.smithi122.stdout:9/898: mkdir db/d23/d9a/d124 0 2024-03-22T02:29:04.012 INFO:tasks.workunit.client.1.smithi122.stdout:6/902: dwrite d7/de/dd/d39/fa2 [4194304,4194304] 0 2024-03-22T02:29:04.013 INFO:tasks.workunit.client.1.smithi122.stdout:1/796: mkdir dc/d3b/d46/db5/dba/df6 0 2024-03-22T02:29:04.014 INFO:tasks.workunit.client.1.smithi122.stdout:9/899: write db/df/f7d [2081314,1019] 0 2024-03-22T02:29:04.020 INFO:tasks.workunit.client.1.smithi122.stdout:8/960: stat d1/d8/lc 0 2024-03-22T02:29:04.021 INFO:tasks.workunit.client.1.smithi122.stdout:7/821: readlink d0/d30/l8f 0 2024-03-22T02:29:04.029 INFO:tasks.workunit.client.1.smithi122.stdout:3/847: symlink d2/d9/dc/df/d49/d85/dda/dd5/l11b 0 2024-03-22T02:29:04.030 INFO:tasks.workunit.client.1.smithi122.stdout:8/961: dwrite d1/d9/f77 [4194304,4194304] 0 2024-03-22T02:29:04.033 INFO:tasks.workunit.client.1.smithi122.stdout:5/777: rename d8/d1e/d22/d3a/d39/c90 to d8/d1e/d22/d3a/d46/d4f/d55/db6/c111 0 2024-03-22T02:29:04.033 INFO:tasks.workunit.client.1.smithi122.stdout:8/962: fsync d1/d8/f100 0 2024-03-22T02:29:04.033 INFO:tasks.workunit.client.1.smithi122.stdout:7/822: dwrite d0/d4/d1d/df6/ffe [0,4194304] 0 2024-03-22T02:29:04.034 INFO:tasks.workunit.client.1.smithi122.stdout:0/679: mkdir d11/d14/d1d/d27/d2f/dd1/de7 0 2024-03-22T02:29:04.037 INFO:tasks.workunit.client.1.smithi122.stdout:7/823: read - d0/d4/d20/d21/d28/fc8 zero size 2024-03-22T02:29:04.037 INFO:tasks.workunit.client.1.smithi122.stdout:4/860: dread d0/d25/f84 [4194304,4194304] 0 2024-03-22T02:29:04.038 INFO:tasks.workunit.client.1.smithi122.stdout:9/900: readlink db/d23/d1f/d57/dd3/lf6 0 2024-03-22T02:29:04.042 INFO:tasks.workunit.client.1.smithi122.stdout:8/963: chown d1/d40/d71 679104923 1 2024-03-22T02:29:04.044 INFO:tasks.workunit.client.1.smithi122.stdout:0/680: creat d11/d14/fe8 x:0 0 0 2024-03-22T02:29:04.046 INFO:tasks.workunit.client.1.smithi122.stdout:0/681: chown d11/d16/d37/d84/da3/fb9 0 1 2024-03-22T02:29:04.048 INFO:tasks.workunit.client.1.smithi122.stdout:0/682: chown d11/d14/d1b/c57 1129014 1 2024-03-22T02:29:04.050 INFO:tasks.workunit.client.1.smithi122.stdout:6/903: dwrite d7/de/dd/d39/f4f [8388608,4194304] 0 2024-03-22T02:29:04.054 INFO:tasks.workunit.client.1.smithi122.stdout:2/836: getdents df/d1f/d33/d4d/d94/dc2/d72 0 2024-03-22T02:29:04.059 INFO:tasks.workunit.client.1.smithi122.stdout:6/904: stat d7/de/dd/d20/de0/d54/d76/d86/l11c 0 2024-03-22T02:29:04.059 INFO:tasks.workunit.client.1.smithi122.stdout:0/683: truncate d11/d14/d1d/d27/f5e 158645 0 2024-03-22T02:29:04.060 INFO:tasks.workunit.client.1.smithi122.stdout:3/848: creat d2/d9/dc/df/d49/d85/f11c x:0 0 0 2024-03-22T02:29:04.061 INFO:tasks.workunit.client.1.smithi122.stdout:2/837: read - df/d1f/d33/d4d/f108 zero size 2024-03-22T02:29:04.062 INFO:tasks.workunit.client.1.smithi122.stdout:1/797: getdents dc/d3b/d8d/d94 0 2024-03-22T02:29:04.064 INFO:tasks.workunit.client.1.smithi122.stdout:4/861: dwrite d0/d25/df2/ffb [4194304,4194304] 0 2024-03-22T02:29:04.064 INFO:tasks.workunit.client.1.smithi122.stdout:7/824: readlink d0/d4/de9/d5c/l7f 0 2024-03-22T02:29:04.068 INFO:tasks.workunit.client.1.smithi122.stdout:5/778: getdents d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71 0 2024-03-22T02:29:04.070 INFO:tasks.workunit.client.1.smithi122.stdout:7/825: chown d0/d4/cf 3568 1 2024-03-22T02:29:04.070 INFO:tasks.workunit.client.1.smithi122.stdout:6/905: creat d7/de/dd/d20/de0/dc0/f11e x:0 0 0 2024-03-22T02:29:04.071 INFO:tasks.workunit.client.1.smithi122.stdout:0/684: creat d11/d16/d37/dda/fe9 x:0 0 0 2024-03-22T02:29:04.073 INFO:tasks.workunit.client.1.smithi122.stdout:0/685: chown d11/d14/da7/cc8 51707870 1 2024-03-22T02:29:04.081 INFO:tasks.workunit.client.1.smithi122.stdout:4/862: truncate d0/d12/d15/d42/d4e/f5a 1017829 0 2024-03-22T02:29:04.082 INFO:tasks.workunit.client.1.smithi122.stdout:1/798: symlink dc/d3b/d8d/d72/d9c/db0/lf7 0 2024-03-22T02:29:04.084 INFO:tasks.workunit.client.1.smithi122.stdout:8/964: dread d1/d40/d71/d97/f120 [0,4194304] 0 2024-03-22T02:29:04.086 INFO:tasks.workunit.client.1.smithi122.stdout:1/799: readlink dc/d2b/d86/lf4 0 2024-03-22T02:29:04.089 INFO:tasks.workunit.client.1.smithi122.stdout:4/863: chown d0/d12/d15/d42/d4e/de/da1/fda 10 1 2024-03-22T02:29:04.090 INFO:tasks.workunit.client.1.smithi122.stdout:7/826: dwrite d0/d4/d20/d21/d28/f90 [0,4194304] 0 2024-03-22T02:29:04.090 INFO:tasks.workunit.client.1.smithi122.stdout:9/901: dread db/d23/f83 [0,4194304] 0 2024-03-22T02:29:04.090 INFO:tasks.workunit.client.1.smithi122.stdout:5/779: dread d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 [0,4194304] 0 2024-03-22T02:29:04.090 INFO:tasks.workunit.client.1.smithi122.stdout:2/838: truncate df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/f82 136177 0 2024-03-22T02:29:04.090 INFO:tasks.workunit.client.1.smithi122.stdout:6/906: dread d7/de/f43 [0,4194304] 0 2024-03-22T02:29:04.092 INFO:tasks.workunit.client.1.smithi122.stdout:9/902: read - db/f110 zero size 2024-03-22T02:29:04.092 INFO:tasks.workunit.client.1.smithi122.stdout:0/686: dread d11/d14/d1d/d27/d2f/f3a [0,4194304] 0 2024-03-22T02:29:04.093 INFO:tasks.workunit.client.1.smithi122.stdout:8/965: dread - d1/d8/d63/f129 zero size 2024-03-22T02:29:04.096 INFO:tasks.workunit.client.1.smithi122.stdout:0/687: chown d11/d14/d1d/d27/dc0 6791727 1 2024-03-22T02:29:04.097 INFO:tasks.workunit.client.1.smithi122.stdout:1/800: creat dc/d3b/d8d/d72/d9c/de8/ff8 x:0 0 0 2024-03-22T02:29:04.099 INFO:tasks.workunit.client.1.smithi122.stdout:7/827: truncate d0/d4/de9/fc6 371438 0 2024-03-22T02:29:04.099 INFO:tasks.workunit.client.1.smithi122.stdout:5/780: write d8/d1e/d22/d78/fbe [3520041,14283] 0 2024-03-22T02:29:04.100 INFO:tasks.workunit.client.1.smithi122.stdout:4/864: mkdir d0/d12/d61/dce/d116 0 2024-03-22T02:29:04.100 INFO:tasks.workunit.client.1.smithi122.stdout:7/828: chown d0/d4/de9/d5c/l7f 5 1 2024-03-22T02:29:04.103 INFO:tasks.workunit.client.1.smithi122.stdout:9/903: rename db/df/d3e/d81/d25/c8e to db/d23/c125 0 2024-03-22T02:29:04.104 INFO:tasks.workunit.client.1.smithi122.stdout:9/904: readlink db/df/d3e/d59/d11a/dd1/l120 0 2024-03-22T02:29:04.104 INFO:tasks.workunit.client.1.smithi122.stdout:0/688: write d11/d16/d37/fab [355096,23700] 0 2024-03-22T02:29:04.106 INFO:tasks.workunit.client.1.smithi122.stdout:1/801: unlink dc/d3b/d8d/d72/d9c/lb8 0 2024-03-22T02:29:04.110 INFO:tasks.workunit.client.1.smithi122.stdout:7/829: mkdir d0/d4/d20/d21/d10a 0 2024-03-22T02:29:04.111 INFO:tasks.workunit.client.1.smithi122.stdout:8/966: dwrite d1/d8/dba/dc0/ff3 [0,4194304] 0 2024-03-22T02:29:04.119 INFO:tasks.workunit.client.1.smithi122.stdout:0/689: mkdir d11/d14/d1d/d27/dc0/dc6/dd0/dea 0 2024-03-22T02:29:04.119 INFO:tasks.workunit.client.1.smithi122.stdout:7/830: dwrite d0/d4/d1d/d108/fbd [0,4194304] 0 2024-03-22T02:29:04.121 INFO:tasks.workunit.client.1.smithi122.stdout:9/905: dread db/df/d3e/d81/d3d/fa1 [0,4194304] 0 2024-03-22T02:29:04.122 INFO:tasks.workunit.client.1.smithi122.stdout:4/865: write d0/d12/d15/d42/d4e/de/f9b [258549,45813] 0 2024-03-22T02:29:04.122 INFO:tasks.workunit.client.1.smithi122.stdout:7/831: readlink d0/d4/d1d/daf/ld0 0 2024-03-22T02:29:04.126 INFO:tasks.workunit.client.1.smithi122.stdout:8/967: creat d1/d60/d73/f14d x:0 0 0 2024-03-22T02:29:04.133 INFO:tasks.workunit.client.1.smithi122.stdout:6/907: getdents d7/de/dd/d20/de0 0 2024-03-22T02:29:04.133 INFO:tasks.workunit.client.1.smithi122.stdout:0/690: fsync f10 0 2024-03-22T02:29:04.134 INFO:tasks.workunit.client.1.smithi122.stdout:8/968: write d1/d40/d71/d97/f9f [571727,42803] 0 2024-03-22T02:29:04.135 INFO:tasks.workunit.client.1.smithi122.stdout:5/781: dread d8/d1e/d22/d78/fbe [0,4194304] 0 2024-03-22T02:29:04.137 INFO:tasks.workunit.client.1.smithi122.stdout:4/866: creat d0/d12/d15/d3d/d90/df3/d113/f117 x:0 0 0 2024-03-22T02:29:04.140 INFO:tasks.workunit.client.1.smithi122.stdout:6/908: readlink d7/de/d27/d52/laa 0 2024-03-22T02:29:04.141 INFO:tasks.workunit.client.1.smithi122.stdout:2/839: dwrite df/d1f/d33/d4d/d94/d9e/fa9 [0,4194304] 0 2024-03-22T02:29:04.141 INFO:tasks.workunit.client.1.smithi122.stdout:9/906: link db/df/d3e/d59/c77 db/df/d3e/d59/db5/dc4/de9/c126 0 2024-03-22T02:29:04.143 INFO:tasks.workunit.client.1.smithi122.stdout:8/969: chown d1/d60/d73/d114/lfd 3972774 1 2024-03-22T02:29:04.147 INFO:tasks.workunit.client.1.smithi122.stdout:1/802: dwrite dc/f2f [0,4194304] 0 2024-03-22T02:29:04.148 INFO:tasks.workunit.client.1.smithi122.stdout:5/782: stat d8/d1e/d22/d3a/d46/d4f/d55/db6/c65 0 2024-03-22T02:29:04.148 INFO:tasks.workunit.client.1.smithi122.stdout:6/909: write f6 [1323304,96877] 0 2024-03-22T02:29:04.148 INFO:tasks.workunit.client.1.smithi122.stdout:0/691: truncate d11/d14/d1b/f81 2071870 0 2024-03-22T02:29:04.151 INFO:tasks.workunit.client.1.smithi122.stdout:9/907: stat db/df/d3e/d59/c6d 0 2024-03-22T02:29:04.151 INFO:tasks.workunit.client.1.smithi122.stdout:7/832: dwrite d0/d4/d1d/d108/d5e/def/f32 [4194304,4194304] 0 2024-03-22T02:29:04.153 INFO:tasks.workunit.client.1.smithi122.stdout:9/908: chown db/c36 5551841 1 2024-03-22T02:29:04.154 INFO:tasks.workunit.client.1.smithi122.stdout:1/803: write dc/d3b/d8d/d72/d9c/db0/d71/fb1 [4569530,8389] 0 2024-03-22T02:29:04.155 INFO:tasks.workunit.client.1.smithi122.stdout:8/970: symlink d1/d8/d8a/d78/l14e 0 2024-03-22T02:29:04.159 INFO:tasks.workunit.client.1.smithi122.stdout:8/971: readlink d1/d40/d51/df5/dae/dfe/d45/l70 0 2024-03-22T02:29:04.161 INFO:tasks.workunit.client.1.smithi122.stdout:4/867: write d0/d25/fca [610176,99485] 0 2024-03-22T02:29:04.162 INFO:tasks.workunit.client.1.smithi122.stdout:2/840: mkdir df/d1f/d33/d6a/d7c/d8e/da0/d110 0 2024-03-22T02:29:04.165 INFO:tasks.workunit.client.1.smithi122.stdout:7/833: dread d0/d4/d20/d21/dda/fe1 [0,4194304] 0 2024-03-22T02:29:04.166 INFO:tasks.workunit.client.1.smithi122.stdout:9/909: unlink db/d1b/f114 0 2024-03-22T02:29:04.168 INFO:tasks.workunit.client.1.smithi122.stdout:1/804: mknod dc/d3b/d8d/d72/d9c/db0/d67/cf9 0 2024-03-22T02:29:04.173 INFO:tasks.workunit.client.1.smithi122.stdout:2/841: truncate df/d1f/d9b/fa3 3043411 0 2024-03-22T02:29:04.174 INFO:tasks.workunit.client.1.smithi122.stdout:7/834: symlink d0/d4/d1d/daf/l10b 0 2024-03-22T02:29:04.175 INFO:tasks.workunit.client.1.smithi122.stdout:4/868: dread d0/d12/d15/d42/d4e/de/fc9 [0,4194304] 0 2024-03-22T02:29:04.176 INFO:tasks.workunit.client.1.smithi122.stdout:6/910: dwrite d7/f3b [4194304,4194304] 0 2024-03-22T02:29:04.176 INFO:tasks.workunit.client.1.smithi122.stdout:2/842: write df/f10 [626950,23567] 0 2024-03-22T02:29:04.177 INFO:tasks.workunit.client.1.smithi122.stdout:7/835: fdatasync d0/d4/d1d/db8/f71 0 2024-03-22T02:29:04.178 INFO:tasks.workunit.client.1.smithi122.stdout:9/910: mkdir db/d23/d1e/d127 0 2024-03-22T02:29:04.178 INFO:tasks.workunit.client.1.smithi122.stdout:1/805: mknod dc/d3b/d8d/d72/d9c/db0/ddf/cfa 0 2024-03-22T02:29:04.180 INFO:tasks.workunit.client.1.smithi122.stdout:8/972: mkdir d1/d40/d71/d97/db8/df6/d13c/d14f 0 2024-03-22T02:29:04.180 INFO:tasks.workunit.client.1.smithi122.stdout:9/911: chown db/d1b/da8/d109 1025 1 2024-03-22T02:29:04.181 INFO:tasks.workunit.client.1.smithi122.stdout:6/911: read d7/de/d27/d52/f7d [484954,121241] 0 2024-03-22T02:29:04.182 INFO:tasks.workunit.client.1.smithi122.stdout:5/783: creat d8/d1e/d22/d3a/d46/d4f/d9b/ddf/f112 x:0 0 0 2024-03-22T02:29:04.182 INFO:tasks.workunit.client.1.smithi122.stdout:4/869: rename d0/d25/de6 to d0/d12/d61/dce/d118 0 2024-03-22T02:29:04.185 INFO:tasks.workunit.client.1.smithi122.stdout:7/836: stat d0/d4/d1d/d108/dbb/cbe 0 2024-03-22T02:29:04.186 INFO:tasks.workunit.client.1.smithi122.stdout:0/692: write d11/d14/d1b/f38 [5702973,64049] 0 2024-03-22T02:29:04.188 INFO:tasks.workunit.client.1.smithi122.stdout:6/912: fsync d7/de/f34 0 2024-03-22T02:29:04.190 INFO:tasks.workunit.client.1.smithi122.stdout:5/784: read d8/d1e/d22/da3/fa5 [1967720,42363] 0 2024-03-22T02:29:04.191 INFO:tasks.workunit.client.1.smithi122.stdout:5/785: dread - d8/d1e/d22/d3a/fea zero size 2024-03-22T02:29:04.192 INFO:tasks.workunit.client.1.smithi122.stdout:4/870: fsync d0/d12/d15/d42/d4e/f16 0 2024-03-22T02:29:04.196 INFO:tasks.workunit.client.1.smithi122.stdout:5/786: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fab [273402,27855] 0 2024-03-22T02:29:04.199 INFO:tasks.workunit.client.1.smithi122.stdout:7/837: rename d0/d4/c62 to d0/d30/c10c 0 2024-03-22T02:29:04.201 INFO:tasks.workunit.client.1.smithi122.stdout:5/787: truncate d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/f58 1170189 0 2024-03-22T02:29:04.203 INFO:tasks.workunit.client.1.smithi122.stdout:4/871: creat d0/d25/df2/f119 x:0 0 0 2024-03-22T02:29:04.203 INFO:tasks.workunit.client.1.smithi122.stdout:1/806: dread dc/d3b/d46/f6b [0,4194304] 0 2024-03-22T02:29:04.205 INFO:tasks.workunit.client.1.smithi122.stdout:7/838: mknod d0/d4/d1d/d108/dbb/c10d 0 2024-03-22T02:29:04.205 INFO:tasks.workunit.client.1.smithi122.stdout:9/912: dread db/df/d3e/d81/d25/d41/fc9 [0,4194304] 0 2024-03-22T02:29:04.207 INFO:tasks.workunit.client.1.smithi122.stdout:2/843: getdents df/d1f/d33/d4d/d94/dc2/d51/d67/db0 0 2024-03-22T02:29:04.252 INFO:tasks.workunit.client.1.smithi122.stdout:7/839: write d0/d4/d80/f92 [549387,40999] 0 2024-03-22T02:29:04.261 INFO:tasks.workunit.client.1.smithi122.stdout:2/844: write df/d1f/f3e [3608645,117073] 0 2024-03-22T02:29:04.261 INFO:tasks.workunit.client.1.smithi122.stdout:5/788: dwrite d8/d1e/d22/d3a/d46/d4f/d9b/ddf/ff8 [0,4194304] 0 2024-03-22T02:29:04.262 INFO:tasks.workunit.client.1.smithi122.stdout:7/840: write d0/d4/d1d/df6/d5d/dab/fce [111028,69156] 0 2024-03-22T02:29:04.267 INFO:tasks.workunit.client.1.smithi122.stdout:6/913: dread d7/f1b [0,4194304] 0 2024-03-22T02:29:04.270 INFO:tasks.workunit.client.1.smithi122.stdout:0/693: dwrite d11/d14/d23/d24/d33/f8e [0,4194304] 0 2024-03-22T02:29:04.273 INFO:tasks.workunit.client.1.smithi122.stdout:1/807: dwrite dc/d3b/d8d/d72/d9c/db0/d67/dc6/f48 [4194304,4194304] 0 2024-03-22T02:29:04.275 INFO:tasks.workunit.client.1.smithi122.stdout:6/914: dread - d7/de/dd/d20/de0/dad/fd5 zero size 2024-03-22T02:29:04.276 INFO:tasks.workunit.client.1.smithi122.stdout:4/872: dread d0/d25/f3b [0,4194304] 0 2024-03-22T02:29:04.276 INFO:tasks.workunit.client.1.smithi122.stdout:9/913: dwrite db/d23/d1f/d57/f87 [0,4194304] 0 2024-03-22T02:29:04.282 INFO:tasks.workunit.client.1.smithi122.stdout:5/789: dwrite d8/d1e/d22/d3a/d46/f6f [4194304,4194304] 0 2024-03-22T02:29:04.284 INFO:tasks.workunit.client.1.smithi122.stdout:1/808: dwrite dc/fd [0,4194304] 0 2024-03-22T02:29:04.298 INFO:tasks.workunit.client.1.smithi122.stdout:5/790: read d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f5d [180830,119891] 0 2024-03-22T02:29:04.299 INFO:tasks.workunit.client.1.smithi122.stdout:5/791: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf [4331107,77517] 0 2024-03-22T02:29:04.311 INFO:tasks.workunit.client.1.smithi122.stdout:2/845: getdents df 0 2024-03-22T02:29:04.312 INFO:tasks.workunit.client.1.smithi122.stdout:7/841: mknod d0/d4/d1d/d108/dbb/c10e 0 2024-03-22T02:29:04.313 INFO:tasks.workunit.client.1.smithi122.stdout:7/842: chown d0/d4/de9/d5c 1548 1 2024-03-22T02:29:04.315 INFO:tasks.workunit.client.1.smithi122.stdout:0/694: rename d11/d14/da7/fbc to d11/d16/d37/d9f/feb 0 2024-03-22T02:29:04.323 INFO:tasks.workunit.client.1.smithi122.stdout:2/846: dwrite df/d1f/d33/d4d/d94/dc2/f2f [0,4194304] 0 2024-03-22T02:29:04.324 INFO:tasks.workunit.client.1.smithi122.stdout:7/843: dwrite d0/d4/de9/d5c/f6d [0,4194304] 0 2024-03-22T02:29:04.326 INFO:tasks.workunit.client.1.smithi122.stdout:7/844: write d0/d4/d1d/d108/d7d/fe3 [411158,74669] 0 2024-03-22T02:29:04.329 INFO:tasks.workunit.client.1.smithi122.stdout:4/873: mknod d0/d12/d15/d42/c11a 0 2024-03-22T02:29:04.329 INFO:tasks.workunit.client.1.smithi122.stdout:1/809: write dc/d3b/d8d/d72/d9c/db0/d67/dc6/f59 [3081323,114525] 0 2024-03-22T02:29:04.331 INFO:tasks.workunit.client.1.smithi122.stdout:5/792: truncate d8/d1a/d83/dcb/ffc 2490337 0 2024-03-22T02:29:04.332 INFO:tasks.workunit.client.1.smithi122.stdout:2/847: mknod df/d1f/d33/d6a/d7c/c111 0 2024-03-22T02:29:04.332 INFO:tasks.workunit.client.1.smithi122.stdout:5/793: dread - d8/d1e/d22/d3a/d46/d4f/fe3 zero size 2024-03-22T02:29:04.334 INFO:tasks.workunit.client.1.smithi122.stdout:7/845: mkdir d0/d4/d1d/daf/d10f 0 2024-03-22T02:29:04.335 INFO:tasks.workunit.client.1.smithi122.stdout:1/810: symlink dc/d55/lfb 0 2024-03-22T02:29:04.336 INFO:tasks.workunit.client.1.smithi122.stdout:4/874: creat d0/d12/d15/d3d/d90/df3/d113/f11b x:0 0 0 2024-03-22T02:29:04.337 INFO:tasks.workunit.client.1.smithi122.stdout:9/914: getdents db/df 0 2024-03-22T02:29:04.341 INFO:tasks.workunit.client.1.smithi122.stdout:6/915: rename d7/de/dd/d20/de0/d54 to d7/de/d11f 0 2024-03-22T02:29:04.342 INFO:tasks.workunit.client.1.smithi122.stdout:7/846: mknod d0/d30/c110 0 2024-03-22T02:29:04.343 INFO:tasks.workunit.client.1.smithi122.stdout:5/794: mknod d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/c113 0 2024-03-22T02:29:04.345 INFO:tasks.workunit.client.1.smithi122.stdout:1/811: symlink dc/d55/lfc 0 2024-03-22T02:29:04.349 INFO:tasks.workunit.client.1.smithi122.stdout:9/915: write db/d23/d1f/f2b [3655606,101874] 0 2024-03-22T02:29:04.356 INFO:tasks.workunit.client.1.smithi122.stdout:0/695: rename d11/d16/d44/ld8 to d11/d14/d1d/d27/d9d/daa/dcd/lec 0 2024-03-22T02:29:04.358 INFO:tasks.workunit.client.1.smithi122.stdout:4/875: mknod d0/d12/d15/c11c 0 2024-03-22T02:29:04.361 INFO:tasks.workunit.client.1.smithi122.stdout:4/876: chown d0/d12/d73 364862 1 2024-03-22T02:29:04.362 INFO:tasks.workunit.client.1.smithi122.stdout:6/916: truncate d7/de/dd/d20/de0/f9a 5372979 0 2024-03-22T02:29:04.362 INFO:tasks.workunit.client.1.smithi122.stdout:9/916: chown db/d23/d1f/d57/dd3/cef 1553957 1 2024-03-22T02:29:04.364 INFO:tasks.workunit.client.1.smithi122.stdout:7/847: dwrite d0/d4/d1d/d108/d5e/def/f32 [8388608,4194304] 0 2024-03-22T02:29:04.366 INFO:tasks.workunit.client.1.smithi122.stdout:4/877: readlink d0/d12/d61/l110 0 2024-03-22T02:29:04.366 INFO:tasks.workunit.client.1.smithi122.stdout:2/848: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f5e [4194304,4194304] 0 2024-03-22T02:29:04.368 INFO:tasks.workunit.client.1.smithi122.stdout:1/812: dwrite dc/d2b/f2d [0,4194304] 0 2024-03-22T02:29:04.373 INFO:tasks.workunit.client.1.smithi122.stdout:2/849: dread - df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ff5 zero size 2024-03-22T02:29:04.381 INFO:tasks.workunit.client.1.smithi122.stdout:5/795: creat d8/d1e/d22/dd7/d108/f114 x:0 0 0 2024-03-22T02:29:04.381 INFO:tasks.workunit.client.1.smithi122.stdout:9/917: dwrite db/d1b/da8/ff0 [0,4194304] 0 2024-03-22T02:29:04.395 INFO:tasks.workunit.client.1.smithi122.stdout:0/696: dread d11/d14/d1d/d27/d9d/fcc [0,4194304] 0 2024-03-22T02:29:04.441 INFO:tasks.workunit.client.1.smithi122.stdout:3/849: sync 2024-03-22T02:29:04.445 INFO:tasks.workunit.client.1.smithi122.stdout:9/918: mkdir db/df/d3e/d59/db5/dc4/d128 0 2024-03-22T02:29:04.445 INFO:tasks.workunit.client.1.smithi122.stdout:9/919: dread - db/d23/d1f/d57/dd3/ff2 zero size 2024-03-22T02:29:04.451 INFO:tasks.workunit.client.1.smithi122.stdout:7/848: unlink d0/d4/d1d/d108/d5e/daa/ld8 0 2024-03-22T02:29:04.451 INFO:tasks.workunit.client.1.smithi122.stdout:5/796: dwrite d8/d1e/d22/d3a/d46/d4f/fbc [0,4194304] 0 2024-03-22T02:29:04.452 INFO:tasks.workunit.client.1.smithi122.stdout:7/849: read d0/fac [16752,57970] 0 2024-03-22T02:29:04.454 INFO:tasks.workunit.client.1.smithi122.stdout:2/850: mknod df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/df6/c112 0 2024-03-22T02:29:04.455 INFO:tasks.workunit.client.1.smithi122.stdout:7/850: truncate d0/d4/d1d/d108/d5e/fd7 845495 0 2024-03-22T02:29:04.457 INFO:tasks.workunit.client.1.smithi122.stdout:3/850: mkdir d2/d9/dc/df/d49/d85/dda/dd5/df9/d11d 0 2024-03-22T02:29:04.460 INFO:tasks.workunit.client.1.smithi122.stdout:0/697: creat d11/d14/d1d/d27/dc0/dc6/dd0/dea/fed x:0 0 0 2024-03-22T02:29:04.466 INFO:tasks.workunit.client.1.smithi122.stdout:7/851: creat d0/d4/d1d/df6/de0/f111 x:0 0 0 2024-03-22T02:29:04.467 INFO:tasks.workunit.client.1.smithi122.stdout:7/852: dread - d0/d4/d1d/f100 zero size 2024-03-22T02:29:04.468 INFO:tasks.workunit.client.1.smithi122.stdout:7/853: chown d0/d4/d1d/d108/d5e/fd7 5180 1 2024-03-22T02:29:04.468 INFO:tasks.workunit.client.1.smithi122.stdout:3/851: truncate d2/d9/dc/d17/d28/f6c 875188 0 2024-03-22T02:29:04.472 INFO:tasks.workunit.client.1.smithi122.stdout:5/797: getdents d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71 0 2024-03-22T02:29:04.474 INFO:tasks.workunit.client.1.smithi122.stdout:5/798: fsync d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/f110 0 2024-03-22T02:29:04.477 INFO:tasks.workunit.client.1.smithi122.stdout:7/854: mknod d0/d4/d1d/d108/d5e/c112 0 2024-03-22T02:29:04.485 INFO:tasks.workunit.client.1.smithi122.stdout:7/855: dwrite d0/d4/d1d/d108/d5e/def/fb3 [0,4194304] 0 2024-03-22T02:29:04.487 INFO:tasks.workunit.client.1.smithi122.stdout:7/856: chown d0/d4/d1d/f100 65875 1 2024-03-22T02:29:04.488 INFO:tasks.workunit.client.1.smithi122.stdout:7/857: chown d0/d4/d20/d21/d28/c45 7726242 1 2024-03-22T02:29:04.492 INFO:tasks.workunit.client.1.smithi122.stdout:3/852: dread d2/f62 [0,4194304] 0 2024-03-22T02:29:04.493 INFO:tasks.workunit.client.1.smithi122.stdout:3/853: dread - d2/d9/d7e/ffc zero size 2024-03-22T02:29:04.497 INFO:tasks.workunit.client.1.smithi122.stdout:8/973: sync 2024-03-22T02:29:04.497 INFO:tasks.workunit.client.1.smithi122.stdout:3/854: write d2/f62 [2310392,10401] 0 2024-03-22T02:29:04.499 INFO:tasks.workunit.client.1.smithi122.stdout:7/858: link d0/d4/d1d/cf9 d0/d4/d80/c113 0 2024-03-22T02:29:04.500 INFO:tasks.workunit.client.1.smithi122.stdout:0/698: dread d11/d14/d1d/d27/d2f/f70 [0,4194304] 0 2024-03-22T02:29:04.502 INFO:tasks.workunit.client.1.smithi122.stdout:3/855: write d2/d9/dc/d57/f93 [4238141,36243] 0 2024-03-22T02:29:04.507 INFO:tasks.workunit.client.1.smithi122.stdout:7/859: mkdir d0/d4/d1d/db8/d114 0 2024-03-22T02:29:04.508 INFO:tasks.workunit.client.1.smithi122.stdout:7/860: fdatasync d0/d4/d1d/df6/f73 0 2024-03-22T02:29:04.508 INFO:tasks.workunit.client.1.smithi122.stdout:3/856: mknod d2/d9/dc/d17/d28/d4c/da5/db4/d79/c11e 0 2024-03-22T02:29:04.510 INFO:tasks.workunit.client.1.smithi122.stdout:8/974: rename d1/d40/d51/df5/dae/dfe/d45/fe3 to d1/d40/d51/f150 0 2024-03-22T02:29:04.519 INFO:tasks.workunit.client.1.smithi122.stdout:7/861: getdents d0/d4/d80/de7 0 2024-03-22T02:29:04.521 INFO:tasks.workunit.client.1.smithi122.stdout:8/975: dread d1/d40/d48/f96 [0,4194304] 0 2024-03-22T02:29:04.522 INFO:tasks.workunit.client.1.smithi122.stdout:8/976: fdatasync d1/d8/ddc/f11f 0 2024-03-22T02:29:04.522 INFO:tasks.workunit.client.1.smithi122.stdout:7/862: mkdir d0/d4/d1d/d108/dbb/d115 0 2024-03-22T02:29:04.526 INFO:tasks.workunit.client.1.smithi122.stdout:8/977: rename d1/d8/ddc/f11f to d1/d9/d134/f151 0 2024-03-22T02:29:04.527 INFO:tasks.workunit.client.1.smithi122.stdout:7/863: mknod d0/d4/d1d/df6/d5d/c116 0 2024-03-22T02:29:04.528 INFO:tasks.workunit.client.1.smithi122.stdout:8/978: readlink d1/d8/d63/df0/l3a 0 2024-03-22T02:29:04.528 INFO:tasks.workunit.client.1.smithi122.stdout:3/857: dread d2/f15 [4194304,4194304] 0 2024-03-22T02:29:04.533 INFO:tasks.workunit.client.1.smithi122.stdout:3/858: read d2/d9/dc/df/d49/d85/f9b [492071,39820] 0 2024-03-22T02:29:04.534 INFO:tasks.workunit.client.1.smithi122.stdout:7/864: dwrite d0/d4/d94/fa9 [0,4194304] 0 2024-03-22T02:29:04.536 INFO:tasks.workunit.client.1.smithi122.stdout:7/865: stat d0/d4/d1d/d108/dbb/fd9 0 2024-03-22T02:29:04.537 INFO:tasks.workunit.client.1.smithi122.stdout:7/866: write d0/d4/d1d/d108/fbd [1709358,70091] 0 2024-03-22T02:29:04.541 INFO:tasks.workunit.client.1.smithi122.stdout:1/813: mkdir dc/d3b/d46/dfd 0 2024-03-22T02:29:04.551 INFO:tasks.workunit.client.1.smithi122.stdout:3/859: dread d2/f10b [0,4194304] 0 2024-03-22T02:29:04.555 INFO:tasks.workunit.client.1.smithi122.stdout:4/878: write d0/d12/d15/d42/d4e/de/f85 [350231,114147] 0 2024-03-22T02:29:04.556 INFO:tasks.workunit.client.1.smithi122.stdout:6/917: dwrite d7/de/d11f/d76/d86/f87 [0,4194304] 0 2024-03-22T02:29:04.558 INFO:tasks.workunit.client.1.smithi122.stdout:7/867: dread d0/d4/d1d/d108/ff7 [0,4194304] 0 2024-03-22T02:29:04.570 INFO:tasks.workunit.client.1.smithi122.stdout:6/918: dwrite d7/de/dd/df/f6e [0,4194304] 0 2024-03-22T02:29:04.579 INFO:tasks.workunit.client.1.smithi122.stdout:3/860: dread d2/d9/dc/d17/f63 [0,4194304] 0 2024-03-22T02:29:04.580 INFO:tasks.workunit.client.1.smithi122.stdout:9/920: write db/d23/d1e/dad/fb2 [754057,7383] 0 2024-03-22T02:29:04.581 INFO:tasks.workunit.client.1.smithi122.stdout:3/861: readlink d2/d9/dc/d17/d2b/l86 0 2024-03-22T02:29:04.594 INFO:tasks.workunit.client.1.smithi122.stdout:2/851: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/fde [0,4194304] 0 2024-03-22T02:29:04.594 INFO:tasks.workunit.client.1.smithi122.stdout:9/921: creat db/df/d3e/d81/d25/f129 x:0 0 0 2024-03-22T02:29:04.600 INFO:tasks.workunit.client.1.smithi122.stdout:7/868: dread d0/d4/d1d/d108/d5e/f77 [0,4194304] 0 2024-03-22T02:29:04.602 INFO:tasks.workunit.client.1.smithi122.stdout:9/922: mknod db/df/d3e/d81/d3d/d86/dd8/c12a 0 2024-03-22T02:29:04.603 INFO:tasks.workunit.client.1.smithi122.stdout:9/923: chown db/d1b/d28/f11d 103 1 2024-03-22T02:29:04.606 INFO:tasks.workunit.client.1.smithi122.stdout:3/862: rename d2/d9/dc/d17/d2b/d3f/cd4 to d2/d9/dc/df/c11f 0 2024-03-22T02:29:04.611 INFO:tasks.workunit.client.1.smithi122.stdout:9/924: creat db/df/d3e/d59/d11a/d58/d9d/dc5/f12b x:0 0 0 2024-03-22T02:29:04.615 INFO:tasks.workunit.client.1.smithi122.stdout:7/869: getdents d0/d4/d1d/df6 0 2024-03-22T02:29:04.617 INFO:tasks.workunit.client.1.smithi122.stdout:9/925: mkdir db/df/d3e/d81/d25/d12c 0 2024-03-22T02:29:04.618 INFO:tasks.workunit.client.1.smithi122.stdout:9/926: chown db/df/d3e/d59/ceb 187 1 2024-03-22T02:29:04.620 INFO:tasks.workunit.client.1.smithi122.stdout:7/870: unlink d0/d4/de9/c105 0 2024-03-22T02:29:04.638 INFO:tasks.workunit.client.1.smithi122.stdout:0/699: dwrite d11/d16/f45 [0,4194304] 0 2024-03-22T02:29:04.650 INFO:tasks.workunit.client.1.smithi122.stdout:2/852: unlink df/d1f/d23/c64 0 2024-03-22T02:29:04.652 INFO:tasks.workunit.client.1.smithi122.stdout:3/863: symlink d2/d9/l120 0 2024-03-22T02:29:04.654 INFO:tasks.workunit.client.1.smithi122.stdout:7/871: dread d0/f16 [0,4194304] 0 2024-03-22T02:29:04.658 INFO:tasks.workunit.client.1.smithi122.stdout:8/979: dwrite d1/d8/d63/df0/d6a/d13a/fd7 [0,4194304] 0 2024-03-22T02:29:04.658 INFO:tasks.workunit.client.1.smithi122.stdout:2/853: creat df/d1f/d33/d34/ddd/f113 x:0 0 0 2024-03-22T02:29:04.660 INFO:tasks.workunit.client.1.smithi122.stdout:1/814: symlink dc/d3b/d8d/d72/d9c/lfe 0 2024-03-22T02:29:04.667 INFO:tasks.workunit.client.1.smithi122.stdout:3/864: rename d2/d9/dc/d57/da4/dc7/lca to d2/d9/dc/df/d49/d85/dda/dd5/l121 0 2024-03-22T02:29:04.668 INFO:tasks.workunit.client.1.smithi122.stdout:1/815: symlink dc/d3b/d8d/d94/lff 0 2024-03-22T02:29:04.670 INFO:tasks.workunit.client.1.smithi122.stdout:9/927: rmdir db/d23/d1e 39 2024-03-22T02:29:04.672 INFO:tasks.workunit.client.1.smithi122.stdout:7/872: rename d0/d4/d20/d57 to d0/d4/d1d/df6/de0/d117 0 2024-03-22T02:29:04.674 INFO:tasks.workunit.client.1.smithi122.stdout:1/816: mknod dc/d3b/d8d/d72/d9c/db0/ddf/c100 0 2024-03-22T02:29:04.675 INFO:tasks.workunit.client.1.smithi122.stdout:1/817: fdatasync dc/ff 0 2024-03-22T02:29:04.676 INFO:tasks.workunit.client.1.smithi122.stdout:4/879: write d0/d12/d15/d42/d57/f64 [161361,16973] 0 2024-03-22T02:29:04.679 INFO:tasks.workunit.client.1.smithi122.stdout:7/873: dread d0/d4/d94/fc5 [4194304,4194304] 0 2024-03-22T02:29:04.682 INFO:tasks.workunit.client.1.smithi122.stdout:8/980: rename d1/d9/d50/l5f to d1/d40/d71/l152 0 2024-03-22T02:29:04.687 INFO:tasks.workunit.client.1.smithi122.stdout:9/928: rename db/df/d3e/d59/d11a/d58/f103 to db/d1b/da8/dba/f12d 0 2024-03-22T02:29:04.689 INFO:tasks.workunit.client.1.smithi122.stdout:6/919: dwrite d7/fa [0,4194304] 0 2024-03-22T02:29:04.690 INFO:tasks.workunit.client.1.smithi122.stdout:0/700: mkdir d11/dee 0 2024-03-22T02:29:04.694 INFO:tasks.workunit.client.1.smithi122.stdout:0/701: truncate d11/d14/d1d/d27/d9d/fcb 751346 0 2024-03-22T02:29:04.696 INFO:tasks.workunit.client.1.smithi122.stdout:9/929: creat db/d23/d1e/dad/f12e x:0 0 0 2024-03-22T02:29:04.700 INFO:tasks.workunit.client.1.smithi122.stdout:0/702: chown c7 325951094 1 2024-03-22T02:29:04.703 INFO:tasks.workunit.client.1.smithi122.stdout:9/930: rename db/d23/d1f/d57/c60 to db/d1b/d28/c12f 0 2024-03-22T02:29:04.704 INFO:tasks.workunit.client.1.smithi122.stdout:5/799: creat d8/d1e/d22/f115 x:0 0 0 2024-03-22T02:29:04.704 INFO:tasks.workunit.client.1.smithi122.stdout:6/920: link d7/de/dd/df/fc7 d7/de/dd/d39/dc2/f120 0 2024-03-22T02:29:04.715 INFO:tasks.workunit.client.1.smithi122.stdout:3/865: mknod d2/c122 0 2024-03-22T02:29:04.717 INFO:tasks.workunit.client.1.smithi122.stdout:2/854: mkdir df/d1f/d33/d4d/d94/dc2/d114 0 2024-03-22T02:29:04.720 INFO:tasks.workunit.client.1.smithi122.stdout:5/800: dwrite d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 [0,4194304] 0 2024-03-22T02:29:04.723 INFO:tasks.workunit.client.1.smithi122.stdout:3/866: mkdir d2/d9/d123 0 2024-03-22T02:29:04.724 INFO:tasks.workunit.client.1.smithi122.stdout:2/855: fsync df/d1f/d33/d34/f5f 0 2024-03-22T02:29:04.727 INFO:tasks.workunit.client.1.smithi122.stdout:2/856: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ff5 206815255 1 2024-03-22T02:29:04.730 INFO:tasks.workunit.client.1.smithi122.stdout:2/857: write df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f5e [2123428,115401] 0 2024-03-22T02:29:04.730 INFO:tasks.workunit.client.1.smithi122.stdout:3/867: symlink d2/d8a/df6/d10e/l124 0 2024-03-22T02:29:04.731 INFO:tasks.workunit.client.1.smithi122.stdout:2/858: chown df/d1f/d33/d6a/d7c/fce 19065498 1 2024-03-22T02:29:04.733 INFO:tasks.workunit.client.1.smithi122.stdout:3/868: read - d2/d9/dc/d17/d2b/ddd/f100 zero size 2024-03-22T02:29:04.737 INFO:tasks.workunit.client.1.smithi122.stdout:3/869: rename d2/c122 to d2/d8a/df6/d10e/c125 0 2024-03-22T02:29:04.738 INFO:tasks.workunit.client.1.smithi122.stdout:3/870: fdatasync d2/d9/d7e/ffc 0 2024-03-22T02:29:04.738 INFO:tasks.workunit.client.1.smithi122.stdout:2/859: dwrite df/f10 [0,4194304] 0 2024-03-22T02:29:04.739 INFO:tasks.workunit.client.1.smithi122.stdout:4/880: write d0/d12/fdd [4092827,94503] 0 2024-03-22T02:29:04.744 INFO:tasks.workunit.client.1.smithi122.stdout:2/860: readlink df/d1f/d3d/lc4 0 2024-03-22T02:29:04.745 INFO:tasks.workunit.client.1.smithi122.stdout:3/871: read d2/d9/dc/d17/d28/f4d [2495498,12906] 0 2024-03-22T02:29:04.747 INFO:tasks.workunit.client.1.smithi122.stdout:4/881: creat d0/d12/d15/df7/d115/f11d x:0 0 0 2024-03-22T02:29:04.750 INFO:tasks.workunit.client.1.smithi122.stdout:0/703: write d11/d14/d1d/d27/d2f/d4e/f97 [45929,60005] 0 2024-03-22T02:29:04.751 INFO:tasks.workunit.client.1.smithi122.stdout:1/818: dwrite dc/f88 [0,4194304] 0 2024-03-22T02:29:04.756 INFO:tasks.workunit.client.1.smithi122.stdout:3/872: creat d2/d9/dc/d17/d51/de0/f126 x:0 0 0 2024-03-22T02:29:04.761 INFO:tasks.workunit.client.1.smithi122.stdout:5/801: write d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/f100 [651016,93836] 0 2024-03-22T02:29:04.763 INFO:tasks.workunit.client.1.smithi122.stdout:6/921: dwrite d7/de/dd/d39/fb3 [0,4194304] 0 2024-03-22T02:29:04.769 INFO:tasks.workunit.client.1.smithi122.stdout:4/882: mkdir d0/d12/da2/d11e 0 2024-03-22T02:29:04.771 INFO:tasks.workunit.client.1.smithi122.stdout:1/819: fdatasync dc/d3b/d8d/d72/d9c/db0/f7d 0 2024-03-22T02:29:04.774 INFO:tasks.workunit.client.1.smithi122.stdout:6/922: chown d7/de/dd/d23 13016108 1 2024-03-22T02:29:04.774 INFO:tasks.workunit.client.1.smithi122.stdout:9/931: dwrite db/df/d3e/d81/d25/f53 [0,4194304] 0 2024-03-22T02:29:04.774 INFO:tasks.workunit.client.1.smithi122.stdout:0/704: write d11/d16/d37/d9f/fa4 [193716,26156] 0 2024-03-22T02:29:04.774 INFO:tasks.workunit.client.1.smithi122.stdout:2/861: dwrite df/f13 [0,4194304] 0 2024-03-22T02:29:04.776 INFO:tasks.workunit.client.1.smithi122.stdout:3/873: rename d2/d9/dc/d17/d2b/d6a/d119 to d2/d9/dc/df/d38/d87/d105/d127 0 2024-03-22T02:29:04.783 INFO:tasks.workunit.client.1.smithi122.stdout:9/932: unlink db/df/d3e/d59/ceb 0 2024-03-22T02:29:04.785 INFO:tasks.workunit.client.1.smithi122.stdout:0/705: mkdir d11/d14/d1d/d27/dc0/def 0 2024-03-22T02:29:04.787 INFO:tasks.workunit.client.1.smithi122.stdout:3/874: dwrite d2/d9/dc/d57/d89/f102 [0,4194304] 0 2024-03-22T02:29:04.788 INFO:tasks.workunit.client.1.smithi122.stdout:9/933: truncate db/df/d3e/d81/d25/f129 492928 0 2024-03-22T02:29:04.789 INFO:tasks.workunit.client.1.smithi122.stdout:2/862: rename df/d1f/d23 to df/d1f/d33/d4d/d94/dc2/d114/d115 0 2024-03-22T02:29:04.792 INFO:tasks.workunit.client.1.smithi122.stdout:5/802: unlink d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7/ffa 0 2024-03-22T02:29:04.792 INFO:tasks.workunit.client.1.smithi122.stdout:6/923: mknod d7/de/d27/d52/c121 0 2024-03-22T02:29:04.796 INFO:tasks.workunit.client.1.smithi122.stdout:0/706: rename d11/d16/d37/d9f to d11/d14/d1d/d27/d2f/d4e/df0 0 2024-03-22T02:29:04.799 INFO:tasks.workunit.client.1.smithi122.stdout:9/934: mknod db/df/d3e/d59/db5/c130 0 2024-03-22T02:29:04.800 INFO:tasks.workunit.client.1.smithi122.stdout:3/875: dwrite d2/d9/dc/d17/d51/de0/f126 [0,4194304] 0 2024-03-22T02:29:04.801 INFO:tasks.workunit.client.1.smithi122.stdout:3/876: stat d2/d9/d7e 0 2024-03-22T02:29:04.803 INFO:tasks.workunit.client.1.smithi122.stdout:0/707: symlink d11/d16/d37/dda/lf1 0 2024-03-22T02:29:04.805 INFO:tasks.workunit.client.1.smithi122.stdout:2/863: mkdir df/d1f/d33/d4d/d7a/d116 0 2024-03-22T02:29:04.805 INFO:tasks.workunit.client.1.smithi122.stdout:0/708: write d11/d14/d1d/d27/d32/f6c [6822494,67578] 0 2024-03-22T02:29:04.807 INFO:tasks.workunit.client.1.smithi122.stdout:5/803: symlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/dcc/l116 0 2024-03-22T02:29:04.810 INFO:tasks.workunit.client.1.smithi122.stdout:9/935: symlink db/df/d3e/d59/d11a/d58/l131 0 2024-03-22T02:29:04.811 INFO:tasks.workunit.client.1.smithi122.stdout:3/877: unlink d2/d9/dc/d17/d28/f35 0 2024-03-22T02:29:04.816 INFO:tasks.workunit.client.1.smithi122.stdout:9/936: fdatasync db/d23/d1f/d57/d66/d73/f76 0 2024-03-22T02:29:04.821 INFO:tasks.workunit.client.1.smithi122.stdout:5/804: dwrite d8/d1e/d22/fde [0,4194304] 0 2024-03-22T02:29:04.821 INFO:tasks.workunit.client.1.smithi122.stdout:0/709: dwrite d11/d14/d23/f77 [0,4194304] 0 2024-03-22T02:29:04.822 INFO:tasks.workunit.client.1.smithi122.stdout:1/820: dwrite dc/d2b/f8c [0,4194304] 0 2024-03-22T02:29:04.823 INFO:tasks.workunit.client.1.smithi122.stdout:0/710: readlink l0 0 2024-03-22T02:29:04.830 INFO:tasks.workunit.client.1.smithi122.stdout:6/924: dread d7/fdc [0,4194304] 0 2024-03-22T02:29:04.833 INFO:tasks.workunit.client.1.smithi122.stdout:5/805: write d8/d74/fbb [3187068,120249] 0 2024-03-22T02:29:04.836 INFO:tasks.workunit.client.1.smithi122.stdout:2/864: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/f117 x:0 0 0 2024-03-22T02:29:04.837 INFO:tasks.workunit.client.1.smithi122.stdout:9/937: mkdir db/d23/d132 0 2024-03-22T02:29:04.838 INFO:tasks.workunit.client.1.smithi122.stdout:5/806: truncate d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/fd9 522310 0 2024-03-22T02:29:04.841 INFO:tasks.workunit.client.1.smithi122.stdout:6/925: rename d7/de/dd/d20/f3f to d7/de/dd/d39/dc2/f122 0 2024-03-22T02:29:04.842 INFO:tasks.workunit.client.1.smithi122.stdout:6/926: fsync d7/de/d27/d52/fe1 0 2024-03-22T02:29:04.843 INFO:tasks.workunit.client.1.smithi122.stdout:1/821: link dc/d3b/f68 dc/d3b/d46/de7/f101 0 2024-03-22T02:29:04.844 INFO:tasks.workunit.client.1.smithi122.stdout:2/865: rmdir df/d1f/d33/d4d/d94/dc2/d72 39 2024-03-22T02:29:04.844 INFO:tasks.workunit.client.1.smithi122.stdout:1/822: chown dc/d55/lfc 0 1 2024-03-22T02:29:04.845 INFO:tasks.workunit.client.1.smithi122.stdout:5/807: rmdir d8/d1e/d51 39 2024-03-22T02:29:04.846 INFO:tasks.workunit.client.1.smithi122.stdout:2/866: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ca5 438 1 2024-03-22T02:29:04.847 INFO:tasks.workunit.client.1.smithi122.stdout:0/711: link d11/d14/d1b/f8f d11/d14/d1d/d27/d2f/ff2 0 2024-03-22T02:29:04.850 INFO:tasks.workunit.client.1.smithi122.stdout:6/927: rename f5 to d7/de/dd/d23/d7f/d110/f123 0 2024-03-22T02:29:04.852 INFO:tasks.workunit.client.1.smithi122.stdout:5/808: readlink d8/d3b/l59 0 2024-03-22T02:29:04.854 INFO:tasks.workunit.client.1.smithi122.stdout:2/867: unlink df/d1f/d33/d6a/f93 0 2024-03-22T02:29:04.854 INFO:tasks.workunit.client.1.smithi122.stdout:3/878: dwrite d2/d9/dc/d17/d28/f6c [0,4194304] 0 2024-03-22T02:29:04.863 INFO:tasks.workunit.client.1.smithi122.stdout:6/928: dwrite d7/de/dd/d20/de0/dc0/f11e [0,4194304] 0 2024-03-22T02:29:04.866 INFO:tasks.workunit.client.1.smithi122.stdout:0/712: dwrite d11/d14/d23/d24/d33/f79 [0,4194304] 0 2024-03-22T02:29:04.866 INFO:tasks.workunit.client.1.smithi122.stdout:2/868: mknod df/d1f/d33/d6a/c118 0 2024-03-22T02:29:04.869 INFO:tasks.workunit.client.1.smithi122.stdout:5/809: creat d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/f117 x:0 0 0 2024-03-22T02:29:04.872 INFO:tasks.workunit.client.1.smithi122.stdout:0/713: write d11/d14/d23/d24/d33/f79 [2397085,39096] 0 2024-03-22T02:29:04.872 INFO:tasks.workunit.client.1.smithi122.stdout:1/823: rmdir dc/d3b/d46/dfd 0 2024-03-22T02:29:04.874 INFO:tasks.workunit.client.1.smithi122.stdout:9/938: link db/df/d3e/d59/d11a/d58/f67 db/d1b/da8/f133 0 2024-03-22T02:29:04.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:04 smithi103 ceph-mon[29258]: pgmap v26: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 83 MiB/s rd, 217 MiB/s wr, 457 op/s 2024-03-22T02:29:04.874 INFO:tasks.workunit.client.1.smithi122.stdout:5/810: stat d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/fee 0 2024-03-22T02:29:04.877 INFO:tasks.workunit.client.1.smithi122.stdout:6/929: truncate d7/de/d11f/d76/f6b 868671 0 2024-03-22T02:29:04.877 INFO:tasks.workunit.client.1.smithi122.stdout:3/879: dwrite d2/d9/dc/df/d49/fac [0,4194304] 0 2024-03-22T02:29:04.882 INFO:tasks.workunit.client.1.smithi122.stdout:5/811: fsync d8/d1e/d22/f10b 0 2024-03-22T02:29:04.887 INFO:tasks.workunit.client.1.smithi122.stdout:1/824: dwrite dc/d2b/d86/dbb/fdc [0,4194304] 0 2024-03-22T02:29:04.887 INFO:tasks.workunit.client.1.smithi122.stdout:3/880: write d2/d9/dc/d17/d51/fe8 [88984,69174] 0 2024-03-22T02:29:04.888 INFO:tasks.workunit.client.1.smithi122.stdout:9/939: truncate db/df/d3e/d81/d25/d41/fbc 1362651 0 2024-03-22T02:29:04.891 INFO:tasks.workunit.client.1.smithi122.stdout:5/812: creat d8/d1a/d104/f118 x:0 0 0 2024-03-22T02:29:04.895 INFO:tasks.workunit.client.1.smithi122.stdout:2/869: link df/f11 df/d1f/d33/d4d/d94/dc2/f119 0 2024-03-22T02:29:04.899 INFO:tasks.workunit.client.1.smithi122.stdout:3/881: rename d2/d9/dc/d17/d28/cd6 to d2/d9/dc/d57/da4/dc7/c128 0 2024-03-22T02:29:04.901 INFO:tasks.workunit.client.1.smithi122.stdout:1/825: creat dc/d3b/d8d/d72/d9c/dd3/f102 x:0 0 0 2024-03-22T02:29:04.902 INFO:tasks.workunit.client.1.smithi122.stdout:0/714: dwrite d11/d14/d1d/d27/d2f/f58 [0,4194304] 0 2024-03-22T02:29:04.903 INFO:tasks.workunit.client.1.smithi122.stdout:2/870: truncate df/d1f/d33/d34/f5f 3250694 0 2024-03-22T02:29:04.907 INFO:tasks.workunit.client.1.smithi122.stdout:5/813: rename d8/d1e/d22/d3a/d7a/fb9 to d8/d1e/d22/d3a/d46/d4f/d9b/da9/f119 0 2024-03-22T02:29:04.907 INFO:tasks.workunit.client.1.smithi122.stdout:1/826: creat dc/d2b/d86/f103 x:0 0 0 2024-03-22T02:29:04.911 INFO:tasks.workunit.client.1.smithi122.stdout:2/871: creat df/d1f/d3d/f11a x:0 0 0 2024-03-22T02:29:04.913 INFO:tasks.workunit.client.1.smithi122.stdout:0/715: creat d11/de1/ff3 x:0 0 0 2024-03-22T02:29:04.913 INFO:tasks.workunit.client.1.smithi122.stdout:2/872: chown df/d1f/d33/d4d/c6f 57245 1 2024-03-22T02:29:04.914 INFO:tasks.workunit.client.1.smithi122.stdout:6/930: dwrite d7/f15 [0,4194304] 0 2024-03-22T02:29:04.915 INFO:tasks.workunit.client.1.smithi122.stdout:5/814: symlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/dcc/l11a 0 2024-03-22T02:29:04.920 INFO:tasks.workunit.client.1.smithi122.stdout:6/931: mkdir d7/de/dd/d23/d7f/d110/d124 0 2024-03-22T02:29:04.921 INFO:tasks.workunit.client.1.smithi122.stdout:3/882: rename d2/d9/dc/d17/d45/f81 to d2/d9/dc/d17/d2b/f129 0 2024-03-22T02:29:04.922 INFO:tasks.workunit.client.1.smithi122.stdout:5/815: truncate d8/d1e/d22/d3a/d46/d4f/d9b/da9/f119 720043 0 2024-03-22T02:29:04.922 INFO:tasks.workunit.client.1.smithi122.stdout:6/932: chown d7/de/dd/f13 249979491 1 2024-03-22T02:29:04.923 INFO:tasks.workunit.client.1.smithi122.stdout:2/873: mkdir df/d1f/d33/d4d/d7a/d11b 0 2024-03-22T02:29:04.923 INFO:tasks.workunit.client.1.smithi122.stdout:5/816: fsync d8/d1e/d22/d3a/d46/d4f/f63 0 2024-03-22T02:29:04.924 INFO:tasks.workunit.client.1.smithi122.stdout:0/716: creat d11/d47/ff4 x:0 0 0 2024-03-22T02:29:04.925 INFO:tasks.workunit.client.1.smithi122.stdout:5/817: read - d8/d1a/fdd zero size 2024-03-22T02:29:04.927 INFO:tasks.workunit.client.1.smithi122.stdout:3/883: chown d2/d9/dc/df/d3c/d78 515031339 1 2024-03-22T02:29:04.929 INFO:tasks.workunit.client.1.smithi122.stdout:3/884: chown d2/d9/dc/d57/da4 586 1 2024-03-22T02:29:04.933 INFO:tasks.workunit.client.1.smithi122.stdout:0/717: unlink d11/d14/d23/d24/d33/d67/c7f 0 2024-03-22T02:29:04.934 INFO:tasks.workunit.client.1.smithi122.stdout:2/874: dread df/d1f/d33/d4d/d94/f95 [0,4194304] 0 2024-03-22T02:29:04.936 INFO:tasks.workunit.client.1.smithi122.stdout:5/818: unlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fab 0 2024-03-22T02:29:04.936 INFO:tasks.workunit.client.1.smithi122.stdout:0/718: truncate d11/d14/d23/d24/d33/d67/dc2/fc9 768381 0 2024-03-22T02:29:04.940 INFO:tasks.workunit.client.1.smithi122.stdout:2/875: unlink df/d1f/d3d/c6b 0 2024-03-22T02:29:04.941 INFO:tasks.workunit.client.1.smithi122.stdout:3/885: getdents d2/d9/d7e/dbd 0 2024-03-22T02:29:04.942 INFO:tasks.workunit.client.1.smithi122.stdout:5/819: fdatasync d8/d1e/f23 0 2024-03-22T02:29:04.943 INFO:tasks.workunit.client.1.smithi122.stdout:1/827: dwrite dc/d3b/f68 [0,4194304] 0 2024-03-22T02:29:04.945 INFO:tasks.workunit.client.1.smithi122.stdout:1/828: truncate dc/d3b/fcb 81004 0 2024-03-22T02:29:04.946 INFO:tasks.workunit.client.1.smithi122.stdout:0/719: rename d11/d14/da7/l96 to d11/d14/d1d/d27/d2f/d4e/lf5 0 2024-03-22T02:29:04.948 INFO:tasks.workunit.client.1.smithi122.stdout:3/886: truncate d2/d9/dc/d17/d2b/f67 4993215 0 2024-03-22T02:29:04.949 INFO:tasks.workunit.client.1.smithi122.stdout:6/933: dread d7/de/dd/d20/de0/f9a [0,4194304] 0 2024-03-22T02:29:04.950 INFO:tasks.workunit.client.1.smithi122.stdout:1/829: write dc/d3b/d8d/d72/d9c/dd3/f102 [440662,111443] 0 2024-03-22T02:29:04.951 INFO:tasks.workunit.client.1.smithi122.stdout:5/820: stat d8/d1e/d22/d3a/d46/d4f/d55/d6a/lba 0 2024-03-22T02:29:04.953 INFO:tasks.workunit.client.1.smithi122.stdout:2/876: symlink df/d1f/d33/l11c 0 2024-03-22T02:29:04.956 INFO:tasks.workunit.client.1.smithi122.stdout:0/720: unlink d11/d14/d23/d24/d8b/ca9 0 2024-03-22T02:29:04.957 INFO:tasks.workunit.client.1.smithi122.stdout:5/821: creat d8/d1e/d22/dd7/f11b x:0 0 0 2024-03-22T02:29:04.959 INFO:tasks.workunit.client.1.smithi122.stdout:3/887: fdatasync d2/d9/dc/d17/d28/d4c/da5/db4/d79/fa7 0 2024-03-22T02:29:04.962 INFO:tasks.workunit.client.1.smithi122.stdout:1/830: write dc/d3b/d8d/d72/d9c/db0/d90/f93 [5108297,29428] 0 2024-03-22T02:29:04.964 INFO:tasks.workunit.client.1.smithi122.stdout:2/877: mkdir df/d1f/d33/d4d/d94/deb/d11d 0 2024-03-22T02:29:04.965 INFO:tasks.workunit.client.1.smithi122.stdout:0/721: mkdir d11/d16/d37/d84/d87/df6 0 2024-03-22T02:29:04.966 INFO:tasks.workunit.client.1.smithi122.stdout:5/822: fdatasync d8/d3b/f52 0 2024-03-22T02:29:04.968 INFO:tasks.workunit.client.1.smithi122.stdout:3/888: fdatasync d2/d9/dc/df/d38/d87/ff4 0 2024-03-22T02:29:04.974 INFO:tasks.workunit.client.1.smithi122.stdout:5/823: mkdir d8/d1e/d22/d78/d11c 0 2024-03-22T02:29:04.975 INFO:tasks.workunit.client.1.smithi122.stdout:6/934: dwrite d7/de/f79 [0,4194304] 0 2024-03-22T02:29:04.976 INFO:tasks.workunit.client.1.smithi122.stdout:6/935: stat d7/de/dd/d23/f48 0 2024-03-22T02:29:04.976 INFO:tasks.workunit.client.1.smithi122.stdout:6/936: chown d7/l9c 4 1 2024-03-22T02:29:04.977 INFO:tasks.workunit.client.1.smithi122.stdout:5/824: dread d8/d1a/f1d [0,4194304] 0 2024-03-22T02:29:04.977 INFO:tasks.workunit.client.1.smithi122.stdout:1/831: symlink dc/d3b/l104 0 2024-03-22T02:29:04.978 INFO:tasks.workunit.client.1.smithi122.stdout:2/878: mknod df/d1f/d33/d4d/c11e 0 2024-03-22T02:29:04.981 INFO:tasks.workunit.client.1.smithi122.stdout:0/722: truncate d11/d14/d1d/d27/f5e 152269 0 2024-03-22T02:29:04.982 INFO:tasks.workunit.client.1.smithi122.stdout:1/832: symlink dc/d3b/d8d/d72/d9c/db0/ddf/l105 0 2024-03-22T02:29:04.984 INFO:tasks.workunit.client.1.smithi122.stdout:5/825: dread d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/f100 [0,4194304] 0 2024-03-22T02:29:04.985 INFO:tasks.workunit.client.1.smithi122.stdout:3/889: creat d2/d9/dc/d17/f12a x:0 0 0 2024-03-22T02:29:04.986 INFO:tasks.workunit.client.1.smithi122.stdout:6/937: dwrite d7/de/dd/d20/de0/dc0/f11e [0,4194304] 0 2024-03-22T02:29:04.987 INFO:tasks.workunit.client.1.smithi122.stdout:2/879: stat df/d1f/d3d/d52/ddc/cb3 0 2024-03-22T02:29:04.988 INFO:tasks.workunit.client.1.smithi122.stdout:1/833: read dc/d3b/d8d/d72/d9c/db0/d71/fb1 [647077,85003] 0 2024-03-22T02:29:04.995 INFO:tasks.workunit.client.1.smithi122.stdout:2/880: rmdir df/d1f/d33/d34 39 2024-03-22T02:29:04.998 INFO:tasks.workunit.client.1.smithi122.stdout:2/881: chown df/d1f/d33/d4d/d94/dc2/d51/fd9 64276 1 2024-03-22T02:29:04.999 INFO:tasks.workunit.client.1.smithi122.stdout:1/834: mkdir dc/d3b/d8d/d72/d9c/db0/d71/d106 0 2024-03-22T02:29:05.000 INFO:tasks.workunit.client.1.smithi122.stdout:5/826: dwrite d8/d1e/d22/d3a/d46/d4f/d55/db6/f95 [0,4194304] 0 2024-03-22T02:29:05.014 INFO:tasks.workunit.client.1.smithi122.stdout:1/835: dwrite dc/d2b/d86/dbb/fdc [4194304,4194304] 0 2024-03-22T02:29:05.015 INFO:tasks.workunit.client.1.smithi122.stdout:0/723: dwrite d11/d14/d1d/fa2 [0,4194304] 0 2024-03-22T02:29:05.016 INFO:tasks.workunit.client.1.smithi122.stdout:5/827: dread d8/d1a/f1c [0,4194304] 0 2024-03-22T02:29:05.019 INFO:tasks.workunit.client.1.smithi122.stdout:2/882: dread df/f2b [0,4194304] 0 2024-03-22T02:29:05.021 INFO:tasks.workunit.client.1.smithi122.stdout:0/724: creat d11/d14/ff7 x:0 0 0 2024-03-22T02:29:05.021 INFO:tasks.workunit.client.1.smithi122.stdout:6/938: dwrite d7/de/d11f/d76/f89 [0,4194304] 0 2024-03-22T02:29:05.022 INFO:tasks.workunit.client.1.smithi122.stdout:2/883: chown df/d1f/d33/l11c 5 1 2024-03-22T02:29:05.026 INFO:tasks.workunit.client.1.smithi122.stdout:0/725: write d11/d14/d1d/d27/d2f/d4e/df0/fa4 [734070,115807] 0 2024-03-22T02:29:05.027 INFO:tasks.workunit.client.1.smithi122.stdout:5/828: dread d8/d1e/d22/d3a/d46/d4f/d9b/da9/f119 [0,4194304] 0 2024-03-22T02:29:05.029 INFO:tasks.workunit.client.1.smithi122.stdout:6/939: symlink d7/de/dd/l125 0 2024-03-22T02:29:05.033 INFO:tasks.workunit.client.1.smithi122.stdout:1/836: dread dc/d55/f77 [0,4194304] 0 2024-03-22T02:29:05.034 INFO:tasks.workunit.client.1.smithi122.stdout:5/829: fdatasync d8/d1a/f5c 0 2024-03-22T02:29:05.038 INFO:tasks.workunit.client.1.smithi122.stdout:5/830: mkdir d8/d1e/d22/d78/d11d 0 2024-03-22T02:29:05.039 INFO:tasks.workunit.client.1.smithi122.stdout:0/726: dwrite d11/d14/d23/f72 [0,4194304] 0 2024-03-22T02:29:05.043 INFO:tasks.workunit.client.1.smithi122.stdout:0/727: creat d11/d4f/d5a/ff8 x:0 0 0 2024-03-22T02:29:05.048 INFO:tasks.workunit.client.1.smithi122.stdout:0/728: creat d11/d16/ff9 x:0 0 0 2024-03-22T02:29:05.056 INFO:tasks.workunit.client.1.smithi122.stdout:6/940: dwrite d7/de/d11f/d76/d86/d96/db2/fdd [0,4194304] 0 2024-03-22T02:29:05.057 INFO:tasks.workunit.client.1.smithi122.stdout:2/884: dread df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f70 [0,4194304] 0 2024-03-22T02:29:05.058 INFO:tasks.workunit.client.1.smithi122.stdout:0/729: dwrite d11/d47/ff4 [0,4194304] 0 2024-03-22T02:29:05.060 INFO:tasks.workunit.client.1.smithi122.stdout:6/941: mkdir d7/de/d27/d126 0 2024-03-22T02:29:05.063 INFO:tasks.workunit.client.1.smithi122.stdout:5/831: dread d8/d1a/f70 [0,4194304] 0 2024-03-22T02:29:05.066 INFO:tasks.workunit.client.1.smithi122.stdout:6/942: stat d7/de/d27/f36 0 2024-03-22T02:29:05.066 INFO:tasks.workunit.client.1.smithi122.stdout:2/885: getdents df/d1f/d33/d6a/d7c/d8e/da0/d110 0 2024-03-22T02:29:05.067 INFO:tasks.workunit.client.1.smithi122.stdout:2/886: readlink df/d1f/d33/d4d/d94/dc2/d51/d67/lfe 0 2024-03-22T02:29:05.071 INFO:tasks.workunit.client.1.smithi122.stdout:6/943: dread d7/de/dd/d20/de0/dc0/f11e [0,4194304] 0 2024-03-22T02:29:05.074 INFO:tasks.workunit.client.1.smithi122.stdout:0/730: dwrite d11/d14/d1d/f80 [0,4194304] 0 2024-03-22T02:29:05.077 INFO:tasks.workunit.client.1.smithi122.stdout:0/731: creat d11/d16/d37/dda/ffa x:0 0 0 2024-03-22T02:29:05.078 INFO:tasks.workunit.client.1.smithi122.stdout:6/944: rename d7/de/dd/d20/f31 to d7/de/dd/d20/d8f/f127 0 2024-03-22T02:29:05.089 INFO:tasks.workunit.client.1.smithi122.stdout:5/832: dread d8/d1e/d22/d3a/d46/d4f/f63 [0,4194304] 0 2024-03-22T02:29:05.090 INFO:tasks.workunit.client.1.smithi122.stdout:5/833: rmdir d8/d1e/d22/d3a/d46/d4f/d55/d6a 39 2024-03-22T02:29:05.093 INFO:tasks.workunit.client.1.smithi122.stdout:5/834: stat d8/d1a/d83/dcb/ffc 0 2024-03-22T02:29:05.094 INFO:tasks.workunit.client.1.smithi122.stdout:5/835: chown d8/d1e/d22/cd1 61245 1 2024-03-22T02:29:05.094 INFO:tasks.workunit.client.1.smithi122.stdout:0/732: dwrite d11/d14/d23/d24/f25 [0,4194304] 0 2024-03-22T02:29:05.095 INFO:tasks.workunit.client.1.smithi122.stdout:6/945: dwrite d7/de/f6c [0,4194304] 0 2024-03-22T02:29:05.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:04 smithi122 ceph-mon[36403]: pgmap v26: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 83 MiB/s rd, 217 MiB/s wr, 457 op/s 2024-03-22T02:29:05.105 INFO:tasks.workunit.client.1.smithi122.stdout:6/946: link d7/de/dd/d39/fa3 d7/de/dd/d23/f128 0 2024-03-22T02:29:05.108 INFO:tasks.workunit.client.1.smithi122.stdout:5/836: dread d8/d1e/d22/d78/f98 [0,4194304] 0 2024-03-22T02:29:05.110 INFO:tasks.workunit.client.1.smithi122.stdout:5/837: dread - d8/d1a/d83/f10e zero size 2024-03-22T02:29:05.110 INFO:tasks.workunit.client.1.smithi122.stdout:0/733: dwrite d11/fb8 [0,4194304] 0 2024-03-22T02:29:05.111 INFO:tasks.workunit.client.1.smithi122.stdout:0/734: fsync d11/d14/d1d/d27/d59/fb2 0 2024-03-22T02:29:05.112 INFO:tasks.workunit.client.1.smithi122.stdout:5/838: mkdir d8/d3b/d11e 0 2024-03-22T02:29:05.116 INFO:tasks.workunit.client.1.smithi122.stdout:5/839: read d8/d1e/d22/d3a/d46/d4f/d55/dc7/f2c [529709,85307] 0 2024-03-22T02:29:05.116 INFO:tasks.workunit.client.1.smithi122.stdout:6/947: read d7/de/dd/d39/f4f [465451,90132] 0 2024-03-22T02:29:05.117 INFO:tasks.workunit.client.1.smithi122.stdout:0/735: truncate d11/d14/d1d/d27/dc0/fc5 255278 0 2024-03-22T02:29:05.121 INFO:tasks.workunit.client.1.smithi122.stdout:6/948: creat d7/de/d11f/d76/d86/f129 x:0 0 0 2024-03-22T02:29:05.122 INFO:tasks.workunit.client.1.smithi122.stdout:6/949: fdatasync d7/de/dd/d23/d7f/f103 0 2024-03-22T02:29:05.123 INFO:tasks.workunit.client.1.smithi122.stdout:0/736: dwrite d11/d14/fe8 [0,4194304] 0 2024-03-22T02:29:05.124 INFO:tasks.workunit.client.1.smithi122.stdout:0/737: chown d11/d14/d23/d24/d33/f3f 637 1 2024-03-22T02:29:05.167 INFO:tasks.workunit.client.1.smithi122.stdout:7/874: sync 2024-03-22T02:29:05.167 INFO:tasks.workunit.client.1.smithi122.stdout:6/950: unlink d7/de/dd/d20/f5d 0 2024-03-22T02:29:05.168 INFO:tasks.workunit.client.1.smithi122.stdout:0/738: truncate d11/d14/d1d/d27/d59/f9b 1470166 0 2024-03-22T02:29:05.169 INFO:tasks.workunit.client.1.smithi122.stdout:5/840: write d8/d1a/f5c [687790,28767] 0 2024-03-22T02:29:05.170 INFO:tasks.workunit.client.1.smithi122.stdout:6/951: write d7/de/d27/d52/fac [398064,39352] 0 2024-03-22T02:29:05.172 INFO:tasks.workunit.client.1.smithi122.stdout:6/952: fsync d7/de/dd/d23/d7f/f103 0 2024-03-22T02:29:05.172 INFO:tasks.workunit.client.1.smithi122.stdout:0/739: rmdir d11/d14/d23/d24/d33/d67 39 2024-03-22T02:29:05.174 INFO:tasks.workunit.client.1.smithi122.stdout:0/740: stat d11/d14/d1d/d27/dc0/dc6/dd0/dea/fed 0 2024-03-22T02:29:05.174 INFO:tasks.workunit.client.1.smithi122.stdout:7/875: rename d0/d4/d1d/db8/d9a/l109 to d0/d4/d1d/df6/de0/l118 0 2024-03-22T02:29:05.175 INFO:tasks.workunit.client.1.smithi122.stdout:6/953: dread - d7/de/dd/d20/fd6 zero size 2024-03-22T02:29:05.176 INFO:tasks.workunit.client.1.smithi122.stdout:5/841: symlink d8/d1e/d22/d78/d11c/l11f 0 2024-03-22T02:29:05.178 INFO:tasks.workunit.client.1.smithi122.stdout:0/741: read d11/d14/d1d/d27/d32/f6b [1705197,107500] 0 2024-03-22T02:29:05.179 INFO:tasks.workunit.client.1.smithi122.stdout:5/842: creat d8/d1e/d22/da3/f120 x:0 0 0 2024-03-22T02:29:05.180 INFO:tasks.workunit.client.1.smithi122.stdout:6/954: link d7/de/dd/d39/fb3 d7/de/d27/d52/f12a 0 2024-03-22T02:29:05.181 INFO:tasks.workunit.client.1.smithi122.stdout:7/876: creat d0/d4/de9/f119 x:0 0 0 2024-03-22T02:29:05.182 INFO:tasks.workunit.client.1.smithi122.stdout:5/843: dread d8/d1e/d22/fda [0,4194304] 0 2024-03-22T02:29:05.183 INFO:tasks.workunit.client.1.smithi122.stdout:5/844: fsync d8/d3b/f62 0 2024-03-22T02:29:05.183 INFO:tasks.workunit.client.1.smithi122.stdout:7/877: chown d0/d4/d1d/df6/fa6 3088 1 2024-03-22T02:29:05.184 INFO:tasks.workunit.client.1.smithi122.stdout:6/955: mknod d7/de/dd/d39/c12b 0 2024-03-22T02:29:05.187 INFO:tasks.workunit.client.1.smithi122.stdout:7/878: mknod d0/d4/d1d/db8/d98/c11a 0 2024-03-22T02:29:05.188 INFO:tasks.workunit.client.1.smithi122.stdout:5/845: dwrite d8/d74/fbb [0,4194304] 0 2024-03-22T02:29:05.190 INFO:tasks.workunit.client.1.smithi122.stdout:6/956: creat d7/de/d27/d126/f12c x:0 0 0 2024-03-22T02:29:05.192 INFO:tasks.workunit.client.1.smithi122.stdout:7/879: symlink d0/d4/d1d/df6/l11b 0 2024-03-22T02:29:05.196 INFO:tasks.workunit.client.1.smithi122.stdout:6/957: chown d7/de/c7a 0 1 2024-03-22T02:29:05.197 INFO:tasks.workunit.client.1.smithi122.stdout:5/846: creat d8/d3b/d11e/f121 x:0 0 0 2024-03-22T02:29:05.198 INFO:tasks.workunit.client.1.smithi122.stdout:6/958: readlink d7/de/dd/d39/l70 0 2024-03-22T02:29:05.198 INFO:tasks.workunit.client.1.smithi122.stdout:0/742: truncate d11/d14/d1d/d27/d9d/daa/dcd/fe0 3753592 0 2024-03-22T02:29:05.199 INFO:tasks.workunit.client.1.smithi122.stdout:7/880: dwrite d0/d4/d80/dc4/fca [0,4194304] 0 2024-03-22T02:29:05.201 INFO:tasks.workunit.client.1.smithi122.stdout:6/959: truncate d7/de/dd/d20/de0/f113 1019084 0 2024-03-22T02:29:05.205 INFO:tasks.workunit.client.1.smithi122.stdout:0/743: mkdir d11/d16/d44/dfb 0 2024-03-22T02:29:05.207 INFO:tasks.workunit.client.1.smithi122.stdout:0/744: dread - d11/de1/ff3 zero size 2024-03-22T02:29:05.208 INFO:tasks.workunit.client.1.smithi122.stdout:5/847: dwrite d8/d1e/d22/da3/ddc/f10 [8388608,4194304] 0 2024-03-22T02:29:05.211 INFO:tasks.workunit.client.1.smithi122.stdout:0/745: symlink d11/d47/lfc 0 2024-03-22T02:29:05.213 INFO:tasks.workunit.client.1.smithi122.stdout:0/746: rmdir d11/d14/d23/d24 39 2024-03-22T02:29:05.220 INFO:tasks.workunit.client.1.smithi122.stdout:0/747: creat d11/d16/d37/d84/d87/df6/ffd x:0 0 0 2024-03-22T02:29:05.225 INFO:tasks.workunit.client.1.smithi122.stdout:7/881: dwrite d0/d4/d1d/df6/de0/d117/fbf [0,4194304] 0 2024-03-22T02:29:05.225 INFO:tasks.workunit.client.1.smithi122.stdout:0/748: creat d11/d14/d1d/d27/d2f/dd1/de7/ffe x:0 0 0 2024-03-22T02:29:05.226 INFO:tasks.workunit.client.1.smithi122.stdout:5/848: dread d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:29:05.229 INFO:tasks.workunit.client.1.smithi122.stdout:6/960: dwrite d7/de/dd/df/f94 [0,4194304] 0 2024-03-22T02:29:05.239 INFO:tasks.workunit.client.1.smithi122.stdout:7/882: dread d0/d4/d94/fc5 [0,4194304] 0 2024-03-22T02:29:05.240 INFO:tasks.workunit.client.1.smithi122.stdout:7/883: fdatasync d0/d4/d1d/d108/d7d/fb7 0 2024-03-22T02:29:05.242 INFO:tasks.workunit.client.1.smithi122.stdout:7/884: mkdir d0/d4/d20/d21/d28/d11c 0 2024-03-22T02:29:05.244 INFO:tasks.workunit.client.1.smithi122.stdout:7/885: symlink d0/d4/d80/dc4/l11d 0 2024-03-22T02:29:05.247 INFO:tasks.workunit.client.1.smithi122.stdout:7/886: link d0/d4/d1d/d108/d5e/f99 d0/d4/d1d/db8/d48/f11e 0 2024-03-22T02:29:05.253 INFO:tasks.workunit.client.1.smithi122.stdout:6/961: dread d7/de/dd/d39/fa2 [0,4194304] 0 2024-03-22T02:29:05.253 INFO:tasks.workunit.client.1.smithi122.stdout:7/887: dwrite d0/d4/d1d/db8/f71 [4194304,4194304] 0 2024-03-22T02:29:05.254 INFO:tasks.workunit.client.1.smithi122.stdout:7/888: fdatasync d0/d4/de9/d5c/fcd 0 2024-03-22T02:29:05.259 INFO:tasks.workunit.client.1.smithi122.stdout:7/889: symlink d0/d4/d20/d21/d10a/l11f 0 2024-03-22T02:29:05.261 INFO:tasks.workunit.client.1.smithi122.stdout:7/890: creat d0/d4/d80/de7/f120 x:0 0 0 2024-03-22T02:29:05.262 INFO:tasks.workunit.client.1.smithi122.stdout:7/891: readlink d0/le4 0 2024-03-22T02:29:05.263 INFO:tasks.workunit.client.1.smithi122.stdout:7/892: truncate d0/f84 5894708 0 2024-03-22T02:29:05.264 INFO:tasks.workunit.client.1.smithi122.stdout:7/893: chown d0/d4/d1d/db8/d48/cb1 184929 1 2024-03-22T02:29:05.266 INFO:tasks.workunit.client.1.smithi122.stdout:7/894: unlink d0/d4/d94/fa4 0 2024-03-22T02:29:05.270 INFO:tasks.workunit.client.1.smithi122.stdout:6/962: truncate d7/de/dd/d23/d7f/fdb 971096 0 2024-03-22T02:29:05.272 INFO:tasks.workunit.client.1.smithi122.stdout:7/895: dwrite d0/d4/d1d/d108/d7d/fb7 [0,4194304] 0 2024-03-22T02:29:05.273 INFO:tasks.workunit.client.1.smithi122.stdout:6/963: fsync d7/de/dd/d20/f7c 0 2024-03-22T02:29:05.278 INFO:tasks.workunit.client.1.smithi122.stdout:6/964: mkdir d7/de/d11f/d76/d86/d96/db2/d12d 0 2024-03-22T02:29:05.283 INFO:tasks.workunit.client.1.smithi122.stdout:7/896: link d0/d4/d1d/df6/l11b d0/d4/d1d/d108/l121 0 2024-03-22T02:29:05.284 INFO:tasks.workunit.client.1.smithi122.stdout:7/897: truncate d0/d4/de9/d5c/f6d 4544210 0 2024-03-22T02:29:05.287 INFO:tasks.workunit.client.1.smithi122.stdout:7/898: creat d0/d4/f122 x:0 0 0 2024-03-22T02:29:05.290 INFO:tasks.workunit.client.1.smithi122.stdout:7/899: fsync d0/d4/d1d/db8/f71 0 2024-03-22T02:29:05.291 INFO:tasks.workunit.client.1.smithi122.stdout:7/900: read - d0/d4/d20/d21/d28/fc8 zero size 2024-03-22T02:29:05.294 INFO:tasks.workunit.client.1.smithi122.stdout:7/901: creat d0/d4/d1d/db8/d114/f123 x:0 0 0 2024-03-22T02:29:05.295 INFO:tasks.workunit.client.1.smithi122.stdout:6/965: dwrite d7/f1b [0,4194304] 0 2024-03-22T02:29:05.299 INFO:tasks.workunit.client.1.smithi122.stdout:6/966: truncate d7/de/dd/d39/f73 897446 0 2024-03-22T02:29:05.300 INFO:tasks.workunit.client.1.smithi122.stdout:7/902: dread d0/d4/d1d/d108/d5e/daa/fd1 [0,4194304] 0 2024-03-22T02:29:05.305 INFO:tasks.workunit.client.1.smithi122.stdout:6/967: dwrite d7/de/dd/f16 [0,4194304] 0 2024-03-22T02:29:05.308 INFO:tasks.workunit.client.1.smithi122.stdout:6/968: write d7/de/d11f/d76/d86/ff4 [1656718,66111] 0 2024-03-22T02:29:05.314 INFO:tasks.workunit.client.1.smithi122.stdout:6/969: creat d7/de/f12e x:0 0 0 2024-03-22T02:29:05.314 INFO:tasks.workunit.client.1.smithi122.stdout:8/981: sync 2024-03-22T02:29:05.319 INFO:tasks.workunit.client.1.smithi122.stdout:7/903: dwrite d0/d30/f52 [0,4194304] 0 2024-03-22T02:29:05.321 INFO:tasks.workunit.client.1.smithi122.stdout:8/982: rename d1/d40/d71/d97/db8/df6/d12d to d1/d8/d63/df0/d6a/d153 0 2024-03-22T02:29:05.322 INFO:tasks.workunit.client.1.smithi122.stdout:7/904: fdatasync d0/d4/d1d/df6/f73 0 2024-03-22T02:29:05.323 INFO:tasks.workunit.client.1.smithi122.stdout:7/905: fdatasync d0/d4/d1d/db8/d9a/f103 0 2024-03-22T02:29:05.324 INFO:tasks.workunit.client.1.smithi122.stdout:6/970: dwrite d7/de/dd/d23/ff6 [0,4194304] 0 2024-03-22T02:29:05.326 INFO:tasks.workunit.client.1.smithi122.stdout:6/971: write d7/de/dd/d20/d8f/fe7 [462743,420] 0 2024-03-22T02:29:05.332 INFO:tasks.workunit.client.1.smithi122.stdout:6/972: dwrite d7/de/dd/d39/dc2/ff7 [0,4194304] 0 2024-03-22T02:29:05.332 INFO:tasks.workunit.client.1.smithi122.stdout:6/973: readlink d7/de/dd/le8 0 2024-03-22T02:29:05.339 INFO:tasks.workunit.client.1.smithi122.stdout:8/983: symlink d1/d9/d50/l154 0 2024-03-22T02:29:05.340 INFO:tasks.workunit.client.1.smithi122.stdout:6/974: dwrite d7/de/d11f/d76/df3/f107 [0,4194304] 0 2024-03-22T02:29:05.343 INFO:tasks.workunit.client.1.smithi122.stdout:7/906: creat d0/d4/d1d/db8/d114/f124 x:0 0 0 2024-03-22T02:29:05.344 INFO:tasks.workunit.client.1.smithi122.stdout:7/907: write d0/d4/f9 [2179447,109255] 0 2024-03-22T02:29:05.347 INFO:tasks.workunit.client.1.smithi122.stdout:8/984: creat d1/d8/d63/df0/d6a/f155 x:0 0 0 2024-03-22T02:29:05.401 INFO:tasks.workunit.client.1.smithi122.stdout:6/975: mknod d7/de/dd/d23/c12f 0 2024-03-22T02:29:05.402 INFO:tasks.workunit.client.1.smithi122.stdout:6/976: fdatasync d7/de/dd/d20/d8f/fa7 0 2024-03-22T02:29:05.404 INFO:tasks.workunit.client.1.smithi122.stdout:7/908: fsync d0/fac 0 2024-03-22T02:29:05.407 INFO:tasks.workunit.client.1.smithi122.stdout:8/985: creat d1/d8/d63/df0/d6a/d13a/f156 x:0 0 0 2024-03-22T02:29:05.424 INFO:tasks.workunit.client.1.smithi122.stdout:8/986: mkdir d1/d9/d134/d157 0 2024-03-22T02:29:05.425 INFO:tasks.workunit.client.1.smithi122.stdout:8/987: stat d1/d8/cc1 0 2024-03-22T02:29:05.426 INFO:tasks.workunit.client.1.smithi122.stdout:6/977: getdents d7/de/d11f/d76/df3 0 2024-03-22T02:29:05.428 INFO:tasks.workunit.client.1.smithi122.stdout:8/988: rename d1/d40/d71/dc9/ca7 to d1/d8/d8a/dfb/c158 0 2024-03-22T02:29:05.431 INFO:tasks.workunit.client.1.smithi122.stdout:8/989: read d1/d8/d63/df0/f32 [499399,37030] 0 2024-03-22T02:29:05.431 INFO:tasks.workunit.client.1.smithi122.stdout:6/978: getdents d7/de 0 2024-03-22T02:29:05.435 INFO:tasks.workunit.client.1.smithi122.stdout:6/979: dread d7/f1b [0,4194304] 0 2024-03-22T02:29:05.445 INFO:tasks.workunit.client.1.smithi122.stdout:8/990: write d1/f3d [2753362,33811] 0 2024-03-22T02:29:05.445 INFO:tasks.workunit.client.1.smithi122.stdout:6/980: truncate d7/de/dd/f42 160553 0 2024-03-22T02:29:05.449 INFO:tasks.workunit.client.1.smithi122.stdout:6/981: link d7/de/d11f/d76/d86/lfa d7/de/dd/d23/d7f/d110/d124/l130 0 2024-03-22T02:29:05.452 INFO:tasks.workunit.client.1.smithi122.stdout:6/982: rename d7/de/d11f/d76/d86/ff4 to d7/de/dd/d23/d7f/f131 0 2024-03-22T02:29:05.453 INFO:tasks.workunit.client.1.smithi122.stdout:6/983: write d7/f1b [4989875,67498] 0 2024-03-22T02:29:05.458 INFO:tasks.workunit.client.1.smithi122.stdout:6/984: dread d7/de/d27/f36 [0,4194304] 0 2024-03-22T02:29:05.460 INFO:tasks.workunit.client.1.smithi122.stdout:6/985: mkdir d7/d132 0 2024-03-22T02:29:05.462 INFO:tasks.workunit.client.1.smithi122.stdout:6/986: truncate d7/de/dd/d9b/f101 437663 0 2024-03-22T02:29:05.462 INFO:tasks.workunit.client.1.smithi122.stdout:6/987: chown d7/de/dd/d20/cb9 3804952 1 2024-03-22T02:29:05.473 INFO:tasks.workunit.client.1.smithi122.stdout:9/940: sync 2024-03-22T02:29:05.474 INFO:tasks.workunit.client.1.smithi122.stdout:4/883: sync 2024-03-22T02:29:05.476 INFO:tasks.workunit.client.1.smithi122.stdout:6/988: dwrite d7/de/dd/d39/dc2/f120 [0,4194304] 0 2024-03-22T02:29:05.478 INFO:tasks.workunit.client.1.smithi122.stdout:4/884: mknod d0/d12/d15/d42/df0/c11f 0 2024-03-22T02:29:05.478 INFO:tasks.workunit.client.1.smithi122.stdout:9/941: symlink db/df/d3e/d59/d11a/d58/d9d/dc5/l134 0 2024-03-22T02:29:05.480 INFO:tasks.workunit.client.1.smithi122.stdout:6/989: unlink d7/de/d11f/d76/fe5 0 2024-03-22T02:29:05.481 INFO:tasks.workunit.client.1.smithi122.stdout:6/990: readlink d7/de/dd/l125 0 2024-03-22T02:29:05.481 INFO:tasks.workunit.client.1.smithi122.stdout:9/942: symlink db/df/d3e/d81/d3d/d86/dd8/l135 0 2024-03-22T02:29:05.483 INFO:tasks.workunit.client.1.smithi122.stdout:6/991: unlink d7/de/dd/d39/f10d 0 2024-03-22T02:29:05.484 INFO:tasks.workunit.client.1.smithi122.stdout:9/943: chown db/df/d3e/dfc/f11e 0 1 2024-03-22T02:29:05.487 INFO:tasks.workunit.client.1.smithi122.stdout:4/885: dwrite d0/d12/fff [0,4194304] 0 2024-03-22T02:29:05.488 INFO:tasks.workunit.client.1.smithi122.stdout:6/992: mknod d7/de/dd/d20/de0/dae/c133 0 2024-03-22T02:29:05.490 INFO:tasks.workunit.client.1.smithi122.stdout:4/886: chown d0/d12/d15/d2d/f4f 222 1 2024-03-22T02:29:05.494 INFO:tasks.workunit.client.1.smithi122.stdout:4/887: write d0/d12/d15/d42/d4e/de/da1/dbf/fe8 [731291,69773] 0 2024-03-22T02:29:05.498 INFO:tasks.workunit.client.1.smithi122.stdout:9/944: dwrite db/d1b/da8/dba/f12d [0,4194304] 0 2024-03-22T02:29:05.498 INFO:tasks.workunit.client.1.smithi122.stdout:9/945: chown db/d23/c18 2 1 2024-03-22T02:29:05.499 INFO:tasks.workunit.client.1.smithi122.stdout:4/888: readlink d0/d12/ld8 0 2024-03-22T02:29:05.515 INFO:tasks.workunit.client.1.smithi122.stdout:9/946: write db/d23/d1e/f70 [1406062,51693] 0 2024-03-22T02:29:05.518 INFO:tasks.workunit.client.1.smithi122.stdout:9/947: mknod db/df/d3e/d59/db5/c136 0 2024-03-22T02:29:05.520 INFO:tasks.workunit.client.1.smithi122.stdout:4/889: dwrite d0/d12/fc4 [0,4194304] 0 2024-03-22T02:29:05.520 INFO:tasks.workunit.client.1.smithi122.stdout:6/993: dread d7/de/d11f/f9d [0,4194304] 0 2024-03-22T02:29:05.523 INFO:tasks.workunit.client.1.smithi122.stdout:6/994: write d7/de/dd/d23/d7f/f103 [48745,32117] 0 2024-03-22T02:29:05.523 INFO:tasks.workunit.client.1.smithi122.stdout:4/890: readlink d0/d12/d15/d42/d4e/ldc 0 2024-03-22T02:29:05.527 INFO:tasks.workunit.client.1.smithi122.stdout:9/948: dwrite db/d1b/da8/dba/fbd [0,4194304] 0 2024-03-22T02:29:05.528 INFO:tasks.workunit.client.1.smithi122.stdout:4/891: creat d0/d12/d15/d3d/d90/df3/d113/f120 x:0 0 0 2024-03-22T02:29:05.529 INFO:tasks.workunit.client.1.smithi122.stdout:6/995: creat d7/d132/f134 x:0 0 0 2024-03-22T02:29:05.536 INFO:tasks.workunit.client.1.smithi122.stdout:4/892: dread d0/d12/d15/d42/d4e/f5a [0,4194304] 0 2024-03-22T02:29:05.538 INFO:tasks.workunit.client.1.smithi122.stdout:4/893: chown d0/d12/d15/d42/d4e/ldc 6 1 2024-03-22T02:29:05.538 INFO:tasks.workunit.client.1.smithi122.stdout:6/996: dwrite d7/de/d11f/d76/df3/f107 [0,4194304] 0 2024-03-22T02:29:05.545 INFO:tasks.workunit.client.1.smithi122.stdout:4/894: link d0/d12/d61/f97 d0/d12/d61/dce/f121 0 2024-03-22T02:29:05.549 INFO:tasks.workunit.client.1.smithi122.stdout:9/949: truncate db/d23/d1e/f70 914902 0 2024-03-22T02:29:05.550 INFO:tasks.workunit.client.1.smithi122.stdout:4/895: dread d0/d12/d15/d42/d4e/de/da1/dbf/fe8 [0,4194304] 0 2024-03-22T02:29:05.586 INFO:tasks.workunit.client.1.smithi122.stdout:1/837: sync 2024-03-22T02:29:05.586 INFO:tasks.workunit.client.1.smithi122.stdout:3/890: sync 2024-03-22T02:29:05.589 INFO:tasks.workunit.client.1.smithi122.stdout:4/896: fdatasync d0/d12/d15/d3d/d90/f98 0 2024-03-22T02:29:05.591 INFO:tasks.workunit.client.1.smithi122.stdout:9/950: symlink db/d23/d1e/d127/l137 0 2024-03-22T02:29:05.594 INFO:tasks.workunit.client.1.smithi122.stdout:6/997: dwrite d7/de/dd/d23/d7f/fe3 [0,4194304] 0 2024-03-22T02:29:05.597 INFO:tasks.workunit.client.1.smithi122.stdout:9/951: chown db/df/d3e/d81/d25/fae 8851879 1 2024-03-22T02:29:05.597 INFO:tasks.workunit.client.1.smithi122.stdout:1/838: dwrite dc/d3b/fcb [0,4194304] 0 2024-03-22T02:29:05.601 INFO:tasks.workunit.client.1.smithi122.stdout:4/897: creat d0/d12/d61/dce/d116/f122 x:0 0 0 2024-03-22T02:29:05.603 INFO:tasks.workunit.client.1.smithi122.stdout:1/839: fsync dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/fb7 0 2024-03-22T02:29:05.613 INFO:tasks.workunit.client.1.smithi122.stdout:6/998: rename d7/de/dd/d39/f4f to d7/de/d11f/d76/d86/f135 0 2024-03-22T02:29:05.621 INFO:tasks.workunit.client.1.smithi122.stdout:1/840: dwrite dc/d2b/d86/f103 [0,4194304] 0 2024-03-22T02:29:05.621 INFO:tasks.workunit.client.1.smithi122.stdout:3/891: dwrite d2/d9/dc/df/d49/d85/fd0 [0,4194304] 0 2024-03-22T02:29:05.625 INFO:tasks.workunit.client.1.smithi122.stdout:9/952: dwrite db/d1b/d28/fdc [0,4194304] 0 2024-03-22T02:29:05.629 INFO:tasks.workunit.client.1.smithi122.stdout:4/898: dwrite d0/f9 [4194304,4194304] 0 2024-03-22T02:29:05.636 INFO:tasks.workunit.client.1.smithi122.stdout:6/999: dread d7/de/d11f/d76/d86/d96/db2/fdd [0,4194304] 0 2024-03-22T02:29:05.636 INFO:tasks.workunit.client.1.smithi122.stdout:1/841: rename dc/d3b/d46/de0 to dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/d107 0 2024-03-22T02:29:05.637 INFO:tasks.workunit.client.1.smithi122.stdout:9/953: symlink db/d1b/l138 0 2024-03-22T02:29:05.637 INFO:tasks.workunit.client.1.smithi122.stdout:1/842: readlink dc/d3b/d8d/d72/d9c/db0/lf7 0 2024-03-22T02:29:05.643 INFO:tasks.workunit.client.1.smithi122.stdout:1/843: mkdir dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/d107/d108 0 2024-03-22T02:29:05.645 INFO:tasks.workunit.client.1.smithi122.stdout:9/954: rename db/d23/dd6/cf4 to db/df/d3e/d59/dda/c139 0 2024-03-22T02:29:05.646 INFO:tasks.workunit.client.1.smithi122.stdout:1/844: fdatasync dc/d3b/d46/f6b 0 2024-03-22T02:29:05.649 INFO:tasks.workunit.client.1.smithi122.stdout:1/845: mknod dc/d3b/d8d/d94/c109 0 2024-03-22T02:29:05.653 INFO:tasks.workunit.client.1.smithi122.stdout:9/955: link db/d1b/da8/c10d db/df/d3e/d81/d25/d12c/c13a 0 2024-03-22T02:29:05.656 INFO:tasks.workunit.client.1.smithi122.stdout:9/956: creat db/df/d3e/d59/d11a/d58/d9d/f13b x:0 0 0 2024-03-22T02:29:05.657 INFO:tasks.workunit.client.1.smithi122.stdout:9/957: chown db/df/d3e/d59/d11a/dd1/l120 11432 1 2024-03-22T02:29:05.658 INFO:tasks.workunit.client.1.smithi122.stdout:9/958: write db/df/d3e/d81/d25/fae [8655891,464] 0 2024-03-22T02:29:05.659 INFO:tasks.workunit.client.1.smithi122.stdout:9/959: fsync db/d1b/d28/f11d 0 2024-03-22T02:29:05.690 INFO:tasks.workunit.client.1.smithi122.stdout:1/846: dread dc/f25 [0,4194304] 0 2024-03-22T02:29:05.691 INFO:tasks.workunit.client.1.smithi122.stdout:1/847: stat dc/d3b/d8d/d72/d9c/db0/d67/dc6/cc7 0 2024-03-22T02:29:05.693 INFO:tasks.workunit.client.1.smithi122.stdout:1/848: mknod dc/d3b/d8d/d94/c10a 0 2024-03-22T02:29:05.695 INFO:tasks.workunit.client.1.smithi122.stdout:1/849: chown dc/c3d 4777 1 2024-03-22T02:29:05.696 INFO:tasks.workunit.client.1.smithi122.stdout:9/960: dread db/df/d3e/d59/db5/f8a [4194304,4194304] 0 2024-03-22T02:29:05.704 INFO:tasks.workunit.client.1.smithi122.stdout:9/961: dwrite db/d23/d1e/f101 [0,4194304] 0 2024-03-22T02:29:05.708 INFO:tasks.workunit.client.1.smithi122.stdout:9/962: creat db/df/d100/d111/f13c x:0 0 0 2024-03-22T02:29:05.709 INFO:tasks.workunit.client.1.smithi122.stdout:9/963: fdatasync db/d23/d1f/d57/d66/d73/f76 0 2024-03-22T02:29:05.712 INFO:tasks.workunit.client.1.smithi122.stdout:9/964: truncate db/d23/d1f/d57/fd0 201023 0 2024-03-22T02:29:05.713 INFO:tasks.workunit.client.1.smithi122.stdout:9/965: write db/d23/d1f/d57/d66/d73/fea [431783,45049] 0 2024-03-22T02:29:05.715 INFO:tasks.workunit.client.1.smithi122.stdout:9/966: readlink db/df/d3e/d81/d3d/l52 0 2024-03-22T02:29:05.717 INFO:tasks.workunit.client.1.smithi122.stdout:9/967: write db/d1b/d28/dc1/f32 [4347675,57514] 0 2024-03-22T02:29:05.722 INFO:tasks.workunit.client.1.smithi122.stdout:9/968: dread db/d23/d1f/d57/fd0 [0,4194304] 0 2024-03-22T02:29:05.724 INFO:tasks.workunit.client.1.smithi122.stdout:9/969: truncate db/d23/d1f/d57/d66/d73/fea 677191 0 2024-03-22T02:29:05.732 INFO:tasks.workunit.client.1.smithi122.stdout:1/850: dread dc/d55/f75 [0,4194304] 0 2024-03-22T02:29:05.736 INFO:tasks.workunit.client.1.smithi122.stdout:2/887: sync 2024-03-22T02:29:05.737 INFO:tasks.workunit.client.1.smithi122.stdout:1/851: dread dc/d55/f75 [0,4194304] 0 2024-03-22T02:29:05.741 INFO:tasks.workunit.client.1.smithi122.stdout:9/970: dwrite db/d1b/da8/fab [0,4194304] 0 2024-03-22T02:29:05.743 INFO:tasks.workunit.client.1.smithi122.stdout:1/852: symlink dc/d3b/d8d/d72/d9c/db0/d67/l10b 0 2024-03-22T02:29:05.743 INFO:tasks.workunit.client.1.smithi122.stdout:2/888: mkdir df/d1f/d11f 0 2024-03-22T02:29:05.745 INFO:tasks.workunit.client.1.smithi122.stdout:9/971: creat db/d23/d1e/f13d x:0 0 0 2024-03-22T02:29:05.750 INFO:tasks.workunit.client.1.smithi122.stdout:1/853: creat dc/d3b/d8d/d72/d9c/de8/f10c x:0 0 0 2024-03-22T02:29:05.752 INFO:tasks.workunit.client.1.smithi122.stdout:9/972: symlink db/df/d3e/d59/d11a/d58/l13e 0 2024-03-22T02:29:05.755 INFO:tasks.workunit.client.1.smithi122.stdout:9/973: symlink db/d1b/d28/l13f 0 2024-03-22T02:29:05.755 INFO:tasks.workunit.client.1.smithi122.stdout:2/889: dwrite df/d1f/d33/d4d/d94/dc2/d51/d67/f71 [0,4194304] 0 2024-03-22T02:29:05.758 INFO:tasks.workunit.client.1.smithi122.stdout:1/854: rename dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/d107/d108 to dc/d2b/d86/d10d 0 2024-03-22T02:29:05.759 INFO:tasks.workunit.client.1.smithi122.stdout:1/855: readlink dc/d2b/l5a 0 2024-03-22T02:29:05.761 INFO:tasks.workunit.client.1.smithi122.stdout:2/890: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/df6 13444 1 2024-03-22T02:29:05.763 INFO:tasks.workunit.client.1.smithi122.stdout:2/891: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2/f104 344346912 1 2024-03-22T02:29:05.765 INFO:tasks.workunit.client.1.smithi122.stdout:9/974: read db/d1b/d28/dc1/f32 [2072780,5488] 0 2024-03-22T02:29:05.769 INFO:tasks.workunit.client.1.smithi122.stdout:9/975: write db/df/d3e/d81/d25/d41/f54 [140710,122607] 0 2024-03-22T02:29:05.771 INFO:tasks.workunit.client.1.smithi122.stdout:2/892: getdents df/d1f/d33/d6a/d7c/d8e/da0/dd4 0 2024-03-22T02:29:05.772 INFO:tasks.workunit.client.1.smithi122.stdout:9/976: mkdir db/d140 0 2024-03-22T02:29:05.773 INFO:tasks.workunit.client.1.smithi122.stdout:2/893: chown df/f11 499 1 2024-03-22T02:29:05.774 INFO:tasks.workunit.client.1.smithi122.stdout:9/977: dread - db/df/d3e/d59/d11a/d58/f10f zero size 2024-03-22T02:29:05.776 INFO:tasks.workunit.client.1.smithi122.stdout:2/894: creat df/d1f/d3d/f120 x:0 0 0 2024-03-22T02:29:05.779 INFO:tasks.workunit.client.1.smithi122.stdout:9/978: creat db/df/d3e/d59/dda/f141 x:0 0 0 2024-03-22T02:29:05.804 INFO:tasks.workunit.client.1.smithi122.stdout:9/979: dread db/df/d3e/d59/db5/f93 [0,4194304] 0 2024-03-22T02:29:05.806 INFO:tasks.workunit.client.1.smithi122.stdout:9/980: write db/df/d100/f10c [996826,75414] 0 2024-03-22T02:29:05.812 INFO:tasks.workunit.client.1.smithi122.stdout:9/981: dwrite db/d23/fe3 [0,4194304] 0 2024-03-22T02:29:05.816 INFO:tasks.workunit.client.1.smithi122.stdout:9/982: mkdir db/df/d3e/d81/d25/d12c/d142 0 2024-03-22T02:29:05.822 INFO:tasks.workunit.client.1.smithi122.stdout:9/983: rename db/df/d3e/d81/c5e to db/df/d3e/d81/c143 0 2024-03-22T02:29:05.823 INFO:tasks.workunit.client.1.smithi122.stdout:9/984: read db/d1b/da8/fab [1176542,122124] 0 2024-03-22T02:29:05.830 INFO:tasks.workunit.client.1.smithi122.stdout:9/985: dwrite db/df/d3e/d81/d3d/d86/dd8/f107 [0,4194304] 0 2024-03-22T02:29:05.882 INFO:tasks.workunit.client.1.smithi122.stdout:0/749: sync 2024-03-22T02:29:05.882 INFO:tasks.workunit.client.1.smithi122.stdout:5/849: sync 2024-03-22T02:29:05.885 INFO:tasks.workunit.client.1.smithi122.stdout:5/850: mknod d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/c122 0 2024-03-22T02:29:05.886 INFO:tasks.workunit.client.1.smithi122.stdout:5/851: dread - d8/d1e/d22/da3/f120 zero size 2024-03-22T02:29:05.887 INFO:tasks.workunit.client.1.smithi122.stdout:5/852: chown d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71 13693836 1 2024-03-22T02:29:05.890 INFO:tasks.workunit.client.1.smithi122.stdout:0/750: dread d11/d14/d1b/f81 [0,4194304] 0 2024-03-22T02:29:05.892 INFO:tasks.workunit.client.1.smithi122.stdout:0/751: symlink d11/dee/lff 0 2024-03-22T02:29:05.893 INFO:tasks.workunit.client.1.smithi122.stdout:0/752: readlink d11/d14/d23/l7d 0 2024-03-22T02:29:05.896 INFO:tasks.workunit.client.1.smithi122.stdout:0/753: rename d11/d16/d37/d84/d87/fd3 to d11/d16/d37/d84/d87/df6/f100 0 2024-03-22T02:29:05.898 INFO:tasks.workunit.client.1.smithi122.stdout:5/853: dread d8/d1e/d22/d3a/d46/d4f/d55/f105 [0,4194304] 0 2024-03-22T02:29:05.899 INFO:tasks.workunit.client.1.smithi122.stdout:0/754: creat d11/d14/d1d/d27/dc0/dc6/dd0/dea/f101 x:0 0 0 2024-03-22T02:29:05.907 INFO:tasks.workunit.client.1.smithi122.stdout:5/854: truncate d8/d1e/d22/d3a/f32 2268774 0 2024-03-22T02:29:05.907 INFO:tasks.workunit.client.1.smithi122.stdout:0/755: write d11/d14/d23/d24/fa8 [501353,35633] 0 2024-03-22T02:29:05.910 INFO:tasks.workunit.client.1.smithi122.stdout:5/855: unlink d8/d1e/d22/da3/fb5 0 2024-03-22T02:29:05.911 INFO:tasks.workunit.client.1.smithi122.stdout:0/756: mknod d11/d14/d1d/d27/d2f/d4e/df0/c102 0 2024-03-22T02:29:05.912 INFO:tasks.workunit.client.1.smithi122.stdout:0/757: readlink l0 0 2024-03-22T02:29:05.913 INFO:tasks.workunit.client.1.smithi122.stdout:5/856: rename d8/d1e/d22/d3a/d39/f8b to d8/d1a/f123 0 2024-03-22T02:29:05.914 INFO:tasks.workunit.client.1.smithi122.stdout:5/857: readlink d8/l36 0 2024-03-22T02:29:05.921 INFO:tasks.workunit.client.1.smithi122.stdout:5/858: dread d8/f26 [0,4194304] 0 2024-03-22T02:29:05.935 INFO:tasks.workunit.client.1.smithi122.stdout:5/859: dwrite d8/d1e/d22/d78/f98 [0,4194304] 0 2024-03-22T02:29:05.939 INFO:tasks.workunit.client.1.smithi122.stdout:5/860: mknod d8/d1e/d22/d3a/d7a/c124 0 2024-03-22T02:29:05.939 INFO:tasks.workunit.client.1.smithi122.stdout:5/861: write d8/d3b/f69 [12878215,121028] 0 2024-03-22T02:29:05.941 INFO:tasks.workunit.client.1.smithi122.stdout:5/862: chown d8/d1e/d22/d3a/d46/d4f/d55/f105 60 1 2024-03-22T02:29:05.952 INFO:tasks.workunit.client.1.smithi122.stdout:5/863: dwrite d8/d1e/d22/fe0 [0,4194304] 0 2024-03-22T02:29:05.963 INFO:tasks.workunit.client.1.smithi122.stdout:5/864: dwrite d8/d1e/d22/d3a/d46/d4f/ff9 [0,4194304] 0 2024-03-22T02:29:05.966 INFO:tasks.workunit.client.1.smithi122.stdout:5/865: symlink d8/d1e/d51/l125 0 2024-03-22T02:29:05.968 INFO:tasks.workunit.client.1.smithi122.stdout:5/866: fsync d8/d1a/f1c 0 2024-03-22T02:29:05.972 INFO:tasks.workunit.client.1.smithi122.stdout:5/867: mkdir d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3/d126 0 2024-03-22T02:29:06.032 INFO:tasks.workunit.client.1.smithi122.stdout:7/909: sync 2024-03-22T02:29:06.034 INFO:tasks.workunit.client.1.smithi122.stdout:7/910: creat d0/d4/de9/f125 x:0 0 0 2024-03-22T02:29:06.035 INFO:tasks.workunit.client.1.smithi122.stdout:7/911: mkdir d0/d4/d80/dc4/d126 0 2024-03-22T02:29:06.036 INFO:tasks.workunit.client.1.smithi122.stdout:7/912: readlink d0/d4/d1d/d108/d5e/def/led 0 2024-03-22T02:29:06.040 INFO:tasks.workunit.client.1.smithi122.stdout:7/913: write d0/d4/f2d [2556836,26557] 0 2024-03-22T02:29:06.045 INFO:tasks.workunit.client.1.smithi122.stdout:7/914: truncate d0/d30/f52 3599050 0 2024-03-22T02:29:06.047 INFO:tasks.workunit.client.1.smithi122.stdout:7/915: getdents d0/d4/d1d/d108/dbb/d115 0 2024-03-22T02:29:06.048 INFO:tasks.workunit.client.1.smithi122.stdout:7/916: write d0/d4/de9/f119 [28067,130114] 0 2024-03-22T02:29:06.051 INFO:tasks.workunit.client.1.smithi122.stdout:7/917: mknod d0/d4/d1d/df6/d5d/dab/c127 0 2024-03-22T02:29:06.056 INFO:tasks.workunit.client.1.smithi122.stdout:7/918: dwrite d0/d4/de9/d5c/fcd [0,4194304] 0 2024-03-22T02:29:06.063 INFO:tasks.workunit.client.1.smithi122.stdout:7/919: dwrite d0/d4/d80/dc4/fca [0,4194304] 0 2024-03-22T02:29:06.064 INFO:tasks.workunit.client.1.smithi122.stdout:7/920: readlink d0/d4/de9/d5c/l7f 0 2024-03-22T02:29:06.073 INFO:tasks.workunit.client.1.smithi122.stdout:7/921: write d0/d4/d20/d21/f2b [2400494,27640] 0 2024-03-22T02:29:06.074 INFO:tasks.workunit.client.1.smithi122.stdout:7/922: chown d0/d4/d94/fc5 20048175 1 2024-03-22T02:29:06.076 INFO:tasks.workunit.client.1.smithi122.stdout:7/923: symlink d0/d4/d1d/db8/d9a/l128 0 2024-03-22T02:29:06.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/924: stat d0/d4/de9/lf0 0 2024-03-22T02:29:06.078 INFO:tasks.workunit.client.1.smithi122.stdout:7/925: dread - d0/d4/d1d/f100 zero size 2024-03-22T02:29:06.086 INFO:tasks.workunit.client.1.smithi122.stdout:7/926: truncate d0/d4/de9/d5c/f6d 4181375 0 2024-03-22T02:29:06.088 INFO:tasks.workunit.client.1.smithi122.stdout:7/927: unlink d0/d4/d1d/d108/l4b 0 2024-03-22T02:29:06.089 INFO:tasks.workunit.client.1.smithi122.stdout:7/928: dread - d0/d4/d1d/d108/ffc zero size 2024-03-22T02:29:06.092 INFO:tasks.workunit.client.1.smithi122.stdout:7/929: getdents d0/d30/dec 0 2024-03-22T02:29:06.092 INFO:tasks.workunit.client.1.smithi122.stdout:7/930: readlink d0/d4/d1d/d108/l6b 0 2024-03-22T02:29:06.093 INFO:tasks.workunit.client.1.smithi122.stdout:7/931: chown d0/d30/f53 4064394 1 2024-03-22T02:29:06.094 INFO:tasks.workunit.client.1.smithi122.stdout:7/932: dread d0/d4/d1d/d108/d5e/daa/fd1 [0,4194304] 0 2024-03-22T02:29:06.097 INFO:tasks.workunit.client.1.smithi122.stdout:7/933: creat d0/d4/d1d/daf/d10f/f129 x:0 0 0 2024-03-22T02:29:06.102 INFO:tasks.workunit.client.1.smithi122.stdout:7/934: rename d0/d4/d1d/df6/de0/d117/l6f to d0/d4/d20/d21/l12a 0 2024-03-22T02:29:06.108 INFO:tasks.workunit.client.1.smithi122.stdout:7/935: dwrite d0/d4/d80/dc4/ff2 [0,4194304] 0 2024-03-22T02:29:06.111 INFO:tasks.workunit.client.1.smithi122.stdout:7/936: symlink d0/d4/d20/l12b 0 2024-03-22T02:29:06.114 INFO:tasks.workunit.client.1.smithi122.stdout:7/937: readlink d0/d4/d1d/d108/l83 0 2024-03-22T02:29:06.114 INFO:tasks.workunit.client.1.smithi122.stdout:7/938: chown d0/d4/d20/l38 18123 1 2024-03-22T02:29:06.115 INFO:tasks.workunit.client.1.smithi122.stdout:7/939: creat d0/d4/de9/d5c/f12c x:0 0 0 2024-03-22T02:29:06.117 INFO:tasks.workunit.client.1.smithi122.stdout:3/892: sync 2024-03-22T02:29:06.117 INFO:tasks.workunit.client.1.smithi122.stdout:8/991: sync 2024-03-22T02:29:06.119 INFO:tasks.workunit.client.1.smithi122.stdout:8/992: chown d1/d8/d63/df0/l11a 40980188 1 2024-03-22T02:29:06.120 INFO:tasks.workunit.client.1.smithi122.stdout:7/940: fdatasync d0/fcb 0 2024-03-22T02:29:06.125 INFO:tasks.workunit.client.1.smithi122.stdout:3/893: truncate d2/d9/dc/d17/d28/d4c/da5/db4/f9d 188810 0 2024-03-22T02:29:06.128 INFO:tasks.workunit.client.1.smithi122.stdout:3/894: symlink d2/d9/dc/d57/dab/l12b 0 2024-03-22T02:29:06.129 INFO:tasks.workunit.client.1.smithi122.stdout:7/941: getdents d0/d4/d1d/d108/d5e/daa 0 2024-03-22T02:29:06.137 INFO:tasks.workunit.client.1.smithi122.stdout:3/895: dwrite d2/d9/dc/d17/f63 [8388608,4194304] 0 2024-03-22T02:29:06.137 INFO:tasks.workunit.client.1.smithi122.stdout:7/942: write d0/d4/de9/d5c/f6d [3325767,117002] 0 2024-03-22T02:29:06.139 INFO:tasks.workunit.client.1.smithi122.stdout:7/943: chown d0/d4/d20/l4a 1869 1 2024-03-22T02:29:06.139 INFO:tasks.workunit.client.1.smithi122.stdout:7/944: stat d0/d4/d1d/db8 0 2024-03-22T02:29:06.140 INFO:tasks.workunit.client.1.smithi122.stdout:7/945: chown d0/d4/de9/d5c 381 1 2024-03-22T02:29:06.151 INFO:tasks.workunit.client.1.smithi122.stdout:7/946: dwrite d0/d4/d1d/d108/d5e/f99 [0,4194304] 0 2024-03-22T02:29:06.153 INFO:tasks.workunit.client.1.smithi122.stdout:7/947: creat d0/d4/d1d/df6/d5d/f12d x:0 0 0 2024-03-22T02:29:06.155 INFO:tasks.workunit.client.1.smithi122.stdout:7/948: fdatasync d0/d4/d1d/d108/d5e/daa/fbc 0 2024-03-22T02:29:06.161 INFO:tasks.workunit.client.1.smithi122.stdout:3/896: dread d2/d9/dc/d57/d89/f102 [0,4194304] 0 2024-03-22T02:29:06.163 INFO:tasks.workunit.client.1.smithi122.stdout:7/949: dwrite d0/d4/de9/f119 [0,4194304] 0 2024-03-22T02:29:06.165 INFO:tasks.workunit.client.1.smithi122.stdout:7/950: creat d0/d4/d80/f12e x:0 0 0 2024-03-22T02:29:06.167 INFO:tasks.workunit.client.1.smithi122.stdout:3/897: link d2/f39 d2/d9/dc/d17/d2b/d3f/f12c 0 2024-03-22T02:29:06.173 INFO:tasks.workunit.client.1.smithi122.stdout:7/951: dwrite d0/d4/d1d/d108/d7d/fb7 [4194304,4194304] 0 2024-03-22T02:29:06.176 INFO:tasks.workunit.client.1.smithi122.stdout:7/952: symlink d0/d4/d20/d21/d10a/l12f 0 2024-03-22T02:29:06.177 INFO:tasks.workunit.client.1.smithi122.stdout:7/953: dread - d0/d4/d20/d21/d28/fad zero size 2024-03-22T02:29:06.178 INFO:tasks.workunit.client.1.smithi122.stdout:7/954: chown d0/d4/d1d/l2a 104451 1 2024-03-22T02:29:06.182 INFO:tasks.workunit.client.1.smithi122.stdout:7/955: creat d0/d4/f130 x:0 0 0 2024-03-22T02:29:06.187 INFO:tasks.workunit.client.1.smithi122.stdout:3/898: dwrite d2/d9/dc/df/d38/d87/ff4 [0,4194304] 0 2024-03-22T02:29:06.195 INFO:tasks.workunit.client.1.smithi122.stdout:3/899: dwrite d2/d9/dc/d17/d28/f108 [0,4194304] 0 2024-03-22T02:29:06.196 INFO:tasks.workunit.client.1.smithi122.stdout:3/900: chown d2/d9/dc/d17/d28/lf1 1573 1 2024-03-22T02:29:06.197 INFO:tasks.workunit.client.1.smithi122.stdout:3/901: write d2/d9/dc/d57/f93 [3207338,51677] 0 2024-03-22T02:29:06.201 INFO:tasks.workunit.client.1.smithi122.stdout:3/902: creat d2/d9/dc/df/d38/d87/d105/f12d x:0 0 0 2024-03-22T02:29:06.204 INFO:tasks.workunit.client.1.smithi122.stdout:3/903: symlink d2/d9/dc/d57/da4/dc7/l12e 0 2024-03-22T02:29:06.222 INFO:tasks.workunit.client.1.smithi122.stdout:9/986: sync 2024-03-22T02:29:06.222 INFO:tasks.workunit.client.1.smithi122.stdout:4/899: sync 2024-03-22T02:29:06.222 INFO:tasks.workunit.client.1.smithi122.stdout:1/856: sync 2024-03-22T02:29:06.222 INFO:tasks.workunit.client.1.smithi122.stdout:2/895: sync 2024-03-22T02:29:06.222 INFO:tasks.workunit.client.1.smithi122.stdout:5/868: sync 2024-03-22T02:29:06.223 INFO:tasks.workunit.client.1.smithi122.stdout:0/758: sync 2024-03-22T02:29:06.224 INFO:tasks.workunit.client.1.smithi122.stdout:0/759: stat d11/d14/d1d/d27/d2f/ff2 0 2024-03-22T02:29:06.225 INFO:tasks.workunit.client.1.smithi122.stdout:4/900: creat d0/d7a/f123 x:0 0 0 2024-03-22T02:29:06.225 INFO:tasks.workunit.client.1.smithi122.stdout:5/869: mknod d8/d1a/d104/c127 0 2024-03-22T02:29:06.227 INFO:tasks.workunit.client.1.smithi122.stdout:3/904: dread d2/d9/dc/d57/d89/fbf [0,4194304] 0 2024-03-22T02:29:06.228 INFO:tasks.workunit.client.1.smithi122.stdout:2/896: unlink df/d1f/d33/d4d/d7a/f8a 0 2024-03-22T02:29:06.230 INFO:tasks.workunit.client.1.smithi122.stdout:9/987: mknod db/df/d3e/d81/d3d/d86/c144 0 2024-03-22T02:29:06.232 INFO:tasks.workunit.client.1.smithi122.stdout:4/901: dread d0/d12/d15/d42/d4e/de/f85 [0,4194304] 0 2024-03-22T02:29:06.233 INFO:tasks.workunit.client.1.smithi122.stdout:1/857: dwrite dc/d3b/d8d/d72/d9c/db0/d90/f93 [0,4194304] 0 2024-03-22T02:29:06.234 INFO:tasks.workunit.client.1.smithi122.stdout:0/760: rename d11/d14/d1d/d27/c42 to d11/d14/d1d/d27/dc0/dc6/dd0/dea/c103 0 2024-03-22T02:29:06.235 INFO:tasks.workunit.client.1.smithi122.stdout:5/870: mkdir d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/d128 0 2024-03-22T02:29:06.240 INFO:tasks.workunit.client.1.smithi122.stdout:0/761: rename d11/d47/l49 to d11/d4f/d5a/l104 0 2024-03-22T02:29:06.243 INFO:tasks.workunit.client.1.smithi122.stdout:3/905: dwrite d2/d9/dc/df/d38/f5d [0,4194304] 0 2024-03-22T02:29:06.243 INFO:tasks.workunit.client.1.smithi122.stdout:1/858: symlink dc/d3b/d8d/d72/d9c/db0/d67/l10e 0 2024-03-22T02:29:06.247 INFO:tasks.workunit.client.1.smithi122.stdout:5/871: symlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/l129 0 2024-03-22T02:29:06.255 INFO:tasks.workunit.client.1.smithi122.stdout:3/906: dwrite d2/d9/dc/d57/f114 [0,4194304] 0 2024-03-22T02:29:06.256 INFO:tasks.workunit.client.1.smithi122.stdout:1/859: dwrite dc/d2b/f57 [0,4194304] 0 2024-03-22T02:29:06.261 INFO:tasks.workunit.client.1.smithi122.stdout:3/907: chown d2/d9/dc/d17/d2b/d3f/l68 4 1 2024-03-22T02:29:06.263 INFO:tasks.workunit.client.1.smithi122.stdout:4/902: getdents d0/d12/d15/df7 0 2024-03-22T02:29:06.264 INFO:tasks.workunit.client.1.smithi122.stdout:5/872: unlink c2 0 2024-03-22T02:29:06.265 INFO:tasks.workunit.client.1.smithi122.stdout:3/908: write d2/d9/dc/d17/f63 [3883110,26091] 0 2024-03-22T02:29:06.268 INFO:tasks.workunit.client.1.smithi122.stdout:5/873: read d8/d74/fc9 [533736,107883] 0 2024-03-22T02:29:06.272 INFO:tasks.workunit.client.1.smithi122.stdout:2/897: dwrite df/f13 [0,4194304] 0 2024-03-22T02:29:06.274 INFO:tasks.workunit.client.1.smithi122.stdout:3/909: creat d2/d9/dc/df/d49/d85/dda/f12f x:0 0 0 2024-03-22T02:29:06.279 INFO:tasks.workunit.client.1.smithi122.stdout:3/910: read - d2/d9/dc/d57/da4/fe4 zero size 2024-03-22T02:29:06.279 INFO:tasks.workunit.client.1.smithi122.stdout:9/988: dwrite db/d23/d1f/fa0 [0,4194304] 0 2024-03-22T02:29:06.281 INFO:tasks.workunit.client.1.smithi122.stdout:3/911: dread - d2/d9/dc/d17/d2b/d6b/fb3 zero size 2024-03-22T02:29:06.283 INFO:tasks.workunit.client.1.smithi122.stdout:2/898: truncate df/d1f/d33/d4d/d94/dc2/d51/d67/f6e 4492198 0 2024-03-22T02:29:06.287 INFO:tasks.workunit.client.1.smithi122.stdout:9/989: rename db/df/d3e/d59/d11a/f11f to db/df/d3e/d59/db5/dc3/f145 0 2024-03-22T02:29:06.291 INFO:tasks.workunit.client.1.smithi122.stdout:4/903: truncate d0/d25/f84 4511730 0 2024-03-22T02:29:06.292 INFO:tasks.workunit.client.1.smithi122.stdout:0/762: dwrite d11/d14/f15 [0,4194304] 0 2024-03-22T02:29:06.294 INFO:tasks.workunit.client.1.smithi122.stdout:2/899: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/lad 0 1 2024-03-22T02:29:06.299 INFO:tasks.workunit.client.1.smithi122.stdout:4/904: creat d0/d7a/f124 x:0 0 0 2024-03-22T02:29:06.300 INFO:tasks.workunit.client.1.smithi122.stdout:5/874: dwrite d8/d3b/f52 [0,4194304] 0 2024-03-22T02:29:06.301 INFO:tasks.workunit.client.1.smithi122.stdout:9/990: truncate db/df/d3e/d81/d25/d41/ffd 626396 0 2024-03-22T02:29:06.303 INFO:tasks.workunit.client.1.smithi122.stdout:0/763: rename d11/d14/d23/d24/d8b/fb6 to d11/d47/f105 0 2024-03-22T02:29:06.309 INFO:tasks.workunit.client.1.smithi122.stdout:5/875: mknod d8/d1e/d22/d3a/d39/c12a 0 2024-03-22T02:29:06.310 INFO:tasks.workunit.client.1.smithi122.stdout:0/764: mknod d11/d14/d1d/c106 0 2024-03-22T02:29:06.310 INFO:tasks.workunit.client.1.smithi122.stdout:3/912: truncate d2/d9/dc/d17/d28/f108 3114013 0 2024-03-22T02:29:06.312 INFO:tasks.workunit.client.1.smithi122.stdout:3/913: truncate d2/d9/dc/d17/d28/fdf 344387 0 2024-03-22T02:29:06.314 INFO:tasks.workunit.client.1.smithi122.stdout:3/914: truncate d2/d9/dc/fd 4640091 0 2024-03-22T02:29:06.314 INFO:tasks.workunit.client.1.smithi122.stdout:4/905: mknod d0/d12/da2/d11e/c125 0 2024-03-22T02:29:06.318 INFO:tasks.workunit.client.1.smithi122.stdout:0/765: symlink d11/d14/d1d/d27/d2f/dd1/de7/l107 0 2024-03-22T02:29:06.322 INFO:tasks.workunit.client.1.smithi122.stdout:9/991: creat db/df/d3e/f146 x:0 0 0 2024-03-22T02:29:06.323 INFO:tasks.workunit.client.1.smithi122.stdout:0/766: creat d11/d16/d37/d84/d87/df6/f108 x:0 0 0 2024-03-22T02:29:06.325 INFO:tasks.workunit.client.1.smithi122.stdout:5/876: creat d8/d1e/d22/d78/d11d/f12b x:0 0 0 2024-03-22T02:29:06.328 INFO:tasks.workunit.client.1.smithi122.stdout:2/900: dread df/d1f/d33/d4d/d94/dc2/d72/fcc [0,4194304] 0 2024-03-22T02:29:06.329 INFO:tasks.workunit.client.1.smithi122.stdout:3/915: creat d2/d9/dc/df/d49/d85/dda/dd5/df9/d11d/f130 x:0 0 0 2024-03-22T02:29:06.333 INFO:tasks.workunit.client.1.smithi122.stdout:9/992: unlink db/df/c17 0 2024-03-22T02:29:06.333 INFO:tasks.workunit.client.1.smithi122.stdout:0/767: dwrite d11/d14/d23/d24/d33/f3f [4194304,4194304] 0 2024-03-22T02:29:06.339 INFO:tasks.workunit.client.1.smithi122.stdout:3/916: mkdir d2/d9/dc/d17/d28/d4c/da5/db4/db1/d131 0 2024-03-22T02:29:06.340 INFO:tasks.workunit.client.1.smithi122.stdout:2/901: truncate df/d1f/d33/d6a/d7c/d8e/da0/dd4/f58 2055118 0 2024-03-22T02:29:06.343 INFO:tasks.workunit.client.1.smithi122.stdout:9/993: creat db/d23/d1e/f147 x:0 0 0 2024-03-22T02:29:06.346 INFO:tasks.workunit.client.1.smithi122.stdout:3/917: mkdir d2/d9/dc/d17/d28/d4c/d132 0 2024-03-22T02:29:06.347 INFO:tasks.workunit.client.1.smithi122.stdout:0/768: truncate d11/d14/d1d/d27/dc0/fc5 496895 0 2024-03-22T02:29:06.349 INFO:tasks.workunit.client.1.smithi122.stdout:4/906: dwrite d0/d7a/fe0 [0,4194304] 0 2024-03-22T02:29:06.350 INFO:tasks.workunit.client.1.smithi122.stdout:5/877: getdents d8/d1e/d22/d3a/d46/d4f/d55/dc7 0 2024-03-22T02:29:06.355 INFO:tasks.workunit.client.1.smithi122.stdout:2/902: truncate df/d1f/d33/d4d/d94/dc2/d51/fcd 541951 0 2024-03-22T02:29:06.359 INFO:tasks.workunit.client.1.smithi122.stdout:0/769: dwrite d11/d14/d1d/d27/d9d/fcb [0,4194304] 0 2024-03-22T02:29:06.373 INFO:tasks.workunit.client.1.smithi122.stdout:0/770: dwrite d11/d14/f15 [4194304,4194304] 0 2024-03-22T02:29:06.385 INFO:tasks.workunit.client.1.smithi122.stdout:4/907: dwrite d0/d12/d15/d42/d4e/de/da1/dbf/d103/fe4 [0,4194304] 0 2024-03-22T02:29:06.387 INFO:tasks.workunit.client.1.smithi122.stdout:4/908: dread - d0/d12/d15/d2d/db5/ff9 zero size 2024-03-22T02:29:06.409 INFO:tasks.workunit.client.1.smithi122.stdout:8/993: sync 2024-03-22T02:29:06.410 INFO:tasks.workunit.client.1.smithi122.stdout:2/903: creat df/d1f/d11f/f121 x:0 0 0 2024-03-22T02:29:06.415 INFO:tasks.workunit.client.1.smithi122.stdout:3/918: link d2/d9/dc/d17/d28/d4c/c65 d2/d9/dc/d17/c133 0 2024-03-22T02:29:06.416 INFO:tasks.workunit.client.1.smithi122.stdout:5/878: getdents d8/d1e/d22/d3a/d46/d4f/d9b/da9 0 2024-03-22T02:29:06.417 INFO:tasks.workunit.client.1.smithi122.stdout:5/879: readlink d8/l1f 0 2024-03-22T02:29:06.417 INFO:tasks.workunit.client.1.smithi122.stdout:9/994: getdents db/df/d3e/d59/db5/dc4/d128 0 2024-03-22T02:29:06.418 INFO:tasks.workunit.client.1.smithi122.stdout:9/995: write db/d1b/da8/dba/f106 [130502,115702] 0 2024-03-22T02:29:06.422 INFO:tasks.workunit.client.1.smithi122.stdout:4/909: link d0/d12/d15/d42/d57/f64 d0/d12/d15/df7/d115/f126 0 2024-03-22T02:29:06.422 INFO:tasks.workunit.client.1.smithi122.stdout:0/771: dread d11/d16/f55 [4194304,4194304] 0 2024-03-22T02:29:06.425 INFO:tasks.workunit.client.1.smithi122.stdout:7/956: sync 2024-03-22T02:29:06.425 INFO:tasks.workunit.client.1.smithi122.stdout:3/919: truncate d2/d9/dc/df/d49/ff5 94758 0 2024-03-22T02:29:06.425 INFO:tasks.workunit.client.1.smithi122.stdout:5/880: write d8/d1e/d22/dd7/f11b [49824,112175] 0 2024-03-22T02:29:06.427 INFO:tasks.workunit.client.1.smithi122.stdout:8/994: symlink d1/d40/d71/d97/db8/df6/d13c/d14f/l159 0 2024-03-22T02:29:06.428 INFO:tasks.workunit.client.1.smithi122.stdout:8/995: truncate d1/d40/d48/f96 1363174 0 2024-03-22T02:29:06.430 INFO:tasks.workunit.client.1.smithi122.stdout:2/904: rename df/d1f/d33/d4d/d94/d9e to df/d1f/d33/d34/ddd/d122 0 2024-03-22T02:29:06.431 INFO:tasks.workunit.client.1.smithi122.stdout:0/772: rmdir d11/d14/d1d/d27/d9d 39 2024-03-22T02:29:06.431 INFO:tasks.workunit.client.1.smithi122.stdout:2/905: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/d41/ca5 14482494 1 2024-03-22T02:29:06.432 INFO:tasks.workunit.client.1.smithi122.stdout:7/957: creat d0/d4/d94/f131 x:0 0 0 2024-03-22T02:29:06.434 INFO:tasks.workunit.client.1.smithi122.stdout:5/881: write d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f9d [1173002,14618] 0 2024-03-22T02:29:06.439 INFO:tasks.workunit.client.1.smithi122.stdout:2/906: read df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/fbd [120099,99381] 0 2024-03-22T02:29:06.440 INFO:tasks.workunit.client.1.smithi122.stdout:3/920: symlink d2/d9/dc/d17/d28/d4c/d132/l134 0 2024-03-22T02:29:06.441 INFO:tasks.workunit.client.1.smithi122.stdout:7/958: unlink d0/d30/cfd 0 2024-03-22T02:29:06.443 INFO:tasks.workunit.client.1.smithi122.stdout:4/910: creat d0/d12/d15/d3d/f127 x:0 0 0 2024-03-22T02:29:06.444 INFO:tasks.workunit.client.1.smithi122.stdout:5/882: creat d8/d1e/d22/d3a/d7a/f12c x:0 0 0 2024-03-22T02:29:06.445 INFO:tasks.workunit.client.1.smithi122.stdout:5/883: readlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/l43 0 2024-03-22T02:29:06.446 INFO:tasks.workunit.client.1.smithi122.stdout:0/773: mknod d11/d14/d23/d24/d33/c109 0 2024-03-22T02:29:06.448 INFO:tasks.workunit.client.1.smithi122.stdout:2/907: mkdir df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/d123 0 2024-03-22T02:29:06.456 INFO:tasks.workunit.client.1.smithi122.stdout:0/774: chown d11/d14/d1d/d27/d9d/daa/dcd 127151 1 2024-03-22T02:29:06.457 INFO:tasks.workunit.client.1.smithi122.stdout:3/921: mkdir d2/d9/dc/d17/d135 0 2024-03-22T02:29:06.457 INFO:tasks.workunit.client.1.smithi122.stdout:5/884: dwrite d8/d1e/d22/da3/f120 [0,4194304] 0 2024-03-22T02:29:06.459 INFO:tasks.workunit.client.1.smithi122.stdout:8/996: link d1/d40/d51/df5/dae/dfe/c4c d1/d60/d73/d90/c15a 0 2024-03-22T02:29:06.461 INFO:tasks.workunit.client.1.smithi122.stdout:5/885: write d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7/fff [955611,88547] 0 2024-03-22T02:29:06.462 INFO:tasks.workunit.client.1.smithi122.stdout:8/997: truncate d1/d8/d63/f129 199580 0 2024-03-22T02:29:06.465 INFO:tasks.workunit.client.1.smithi122.stdout:4/911: link d0/d12/d15/d42/d4e/f22 d0/f128 0 2024-03-22T02:29:06.467 INFO:tasks.workunit.client.1.smithi122.stdout:2/908: dread df/d1f/f2d [0,4194304] 0 2024-03-22T02:29:06.471 INFO:tasks.workunit.client.1.smithi122.stdout:4/912: dread - d0/d12/d15/d3d/d90/fb9 zero size 2024-03-22T02:29:06.474 INFO:tasks.workunit.client.1.smithi122.stdout:5/886: read d8/d1a/d83/dcb/ffc [100400,44661] 0 2024-03-22T02:29:06.476 INFO:tasks.workunit.client.1.smithi122.stdout:8/998: mkdir d1/d60/da4/d15b 0 2024-03-22T02:29:06.477 INFO:tasks.workunit.client.1.smithi122.stdout:4/913: symlink d0/d12/d15/d42/df0/l129 0 2024-03-22T02:29:06.477 INFO:tasks.workunit.client.1.smithi122.stdout:8/999: chown d1/d8/dba/c123 114329067 1 2024-03-22T02:29:06.479 INFO:tasks.workunit.client.1.smithi122.stdout:3/922: creat d2/d9/dc/d57/da4/f136 x:0 0 0 2024-03-22T02:29:06.482 INFO:tasks.workunit.client.1.smithi122.stdout:3/923: stat d2/d9/dc/d57/f114 0 2024-03-22T02:29:06.482 INFO:tasks.workunit.client.1.smithi122.stdout:9/996: dwrite db/d23/d1e/f70 [0,4194304] 0 2024-03-22T02:29:06.482 INFO:tasks.workunit.client.1.smithi122.stdout:2/909: link df/d1f/d33/d4d/f108 df/d1f/d11f/f124 0 2024-03-22T02:29:06.485 INFO:tasks.workunit.client.1.smithi122.stdout:1/860: sync 2024-03-22T02:29:06.490 INFO:tasks.workunit.client.1.smithi122.stdout:5/887: fdatasync d8/d1a/f34 0 2024-03-22T02:29:06.493 INFO:tasks.workunit.client.1.smithi122.stdout:7/959: write d0/d4/d20/d21/dda/fe1 [4996563,111792] 0 2024-03-22T02:29:06.497 INFO:tasks.workunit.client.1.smithi122.stdout:4/914: dwrite d0/d12/d15/d3d/d90/f48 [0,4194304] 0 2024-03-22T02:29:06.499 INFO:tasks.workunit.client.1.smithi122.stdout:9/997: dwrite db/d23/d1f/d57/d66/d73/f76 [0,4194304] 0 2024-03-22T02:29:06.503 INFO:tasks.workunit.client.1.smithi122.stdout:0/775: dread d11/d14/d23/d24/d33/d67/f7c [0,4194304] 0 2024-03-22T02:29:06.507 INFO:tasks.workunit.client.1.smithi122.stdout:4/915: symlink d0/d12/d15/da7/l12a 0 2024-03-22T02:29:06.507 INFO:tasks.workunit.client.1.smithi122.stdout:0/776: chown d11/d14/d1d/c106 36 1 2024-03-22T02:29:06.510 INFO:tasks.workunit.client.1.smithi122.stdout:0/777: write d11/d14/d23/d24/d33/d67/dc2/fd4 [961295,21391] 0 2024-03-22T02:29:06.510 INFO:tasks.workunit.client.1.smithi122.stdout:9/998: fdatasync db/d1b/d28/dc1/f5f 0 2024-03-22T02:29:06.513 INFO:tasks.workunit.client.1.smithi122.stdout:1/861: dread dc/f1d [0,4194304] 0 2024-03-22T02:29:06.522 INFO:tasks.workunit.client.1.smithi122.stdout:7/960: link d0/d4/d1d/df6/d5d/l91 d0/d4/d80/dc4/d126/l132 0 2024-03-22T02:29:06.522 INFO:tasks.workunit.client.1.smithi122.stdout:1/862: dwrite dc/d3b/d8d/d72/f96 [0,4194304] 0 2024-03-22T02:29:06.522 INFO:tasks.workunit.client.1.smithi122.stdout:9/999: dread db/d1b/da8/fab [0,4194304] 0 2024-03-22T02:29:06.524 INFO:tasks.workunit.client.1.smithi122.stdout:3/924: truncate d2/d9/dc/fae 725918 0 2024-03-22T02:29:06.525 INFO:tasks.workunit.client.1.smithi122.stdout:2/910: link df/d1f/d3d/d52/ddc/fb7 df/d1f/d33/d4d/d94/dc2/f125 0 2024-03-22T02:29:06.526 INFO:tasks.workunit.client.1.smithi122.stdout:1/863: dread - dc/d55/faa zero size 2024-03-22T02:29:06.529 INFO:tasks.workunit.client.1.smithi122.stdout:4/916: write d0/d12/d15/d42/fd7 [532246,84477] 0 2024-03-22T02:29:06.530 INFO:tasks.workunit.client.1.smithi122.stdout:4/917: stat d0/d12/l4b 0 2024-03-22T02:29:06.531 INFO:tasks.workunit.client.1.smithi122.stdout:7/961: rmdir d0/d4/d80/dc4 39 2024-03-22T02:29:06.533 INFO:tasks.workunit.client.1.smithi122.stdout:4/918: dread - d0/d12/d15/da7/f112 zero size 2024-03-22T02:29:06.536 INFO:tasks.workunit.client.1.smithi122.stdout:1/864: dwrite dc/f88 [0,4194304] 0 2024-03-22T02:29:06.536 INFO:tasks.workunit.client.1.smithi122.stdout:0/778: link d11/d14/da7/c65 d11/c10a 0 2024-03-22T02:29:06.542 INFO:tasks.workunit.client.1.smithi122.stdout:5/888: dread d8/d1e/d22/fde [0,4194304] 0 2024-03-22T02:29:06.544 INFO:tasks.workunit.client.1.smithi122.stdout:7/962: fdatasync d0/d4/d1d/d108/d5e/daa/fbc 0 2024-03-22T02:29:06.545 INFO:tasks.workunit.client.1.smithi122.stdout:7/963: fsync d0/d4/de9/d5c/f6d 0 2024-03-22T02:29:06.546 INFO:tasks.workunit.client.1.smithi122.stdout:4/919: rename d0/d12/d15/d3d/c86 to d0/d12/d73/c12b 0 2024-03-22T02:29:06.549 INFO:tasks.workunit.client.1.smithi122.stdout:4/920: chown d0/d12/d15/d2d/db5/l108 35 1 2024-03-22T02:29:06.553 INFO:tasks.workunit.client.1.smithi122.stdout:7/964: truncate d0/fac 583666 0 2024-03-22T02:29:06.554 INFO:tasks.workunit.client.1.smithi122.stdout:3/925: truncate d2/d9/dc/df/d38/fa1 3825807 0 2024-03-22T02:29:06.554 INFO:tasks.workunit.client.1.smithi122.stdout:1/865: symlink dc/d3b/d46/de7/deb/l10f 0 2024-03-22T02:29:06.559 INFO:tasks.workunit.client.1.smithi122.stdout:4/921: dwrite d0/d12/d15/d3d/d90/df3/d113/f120 [0,4194304] 0 2024-03-22T02:29:06.561 INFO:tasks.workunit.client.1.smithi122.stdout:2/911: dwrite df/d1f/d33/d34/f43 [0,4194304] 0 2024-03-22T02:29:06.564 INFO:tasks.workunit.client.1.smithi122.stdout:1/866: mkdir dc/d2b/d86/d110 0 2024-03-22T02:29:06.565 INFO:tasks.workunit.client.1.smithi122.stdout:3/926: write d2/d9/dc/d17/d28/d4c/da5/db4/d79/fb9 [3748802,117864] 0 2024-03-22T02:29:06.567 INFO:tasks.workunit.client.1.smithi122.stdout:2/912: chown df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f70 256370053 1 2024-03-22T02:29:06.572 INFO:tasks.workunit.client.1.smithi122.stdout:2/913: read df/f10 [96703,25645] 0 2024-03-22T02:29:06.572 INFO:tasks.workunit.client.1.smithi122.stdout:4/922: mknod d0/d12/d61/c12c 0 2024-03-22T02:29:06.574 INFO:tasks.workunit.client.1.smithi122.stdout:5/889: truncate d8/d1e/d22/d3a/d46/d4f/d55/db6/f95 3148608 0 2024-03-22T02:29:06.575 INFO:tasks.workunit.client.1.smithi122.stdout:1/867: mknod dc/d2b/d86/c111 0 2024-03-22T02:29:06.577 INFO:tasks.workunit.client.1.smithi122.stdout:0/779: dread d11/d14/d1d/fa2 [0,4194304] 0 2024-03-22T02:29:06.578 INFO:tasks.workunit.client.1.smithi122.stdout:7/965: mknod d0/d4/de9/df4/c133 0 2024-03-22T02:29:06.580 INFO:tasks.workunit.client.1.smithi122.stdout:0/780: readlink d11/d14/d1b/lad 0 2024-03-22T02:29:06.580 INFO:tasks.workunit.client.1.smithi122.stdout:7/966: chown d0/d4/d1d/d108/dbb/fd9 0 1 2024-03-22T02:29:06.580 INFO:tasks.workunit.client.1.smithi122.stdout:4/923: symlink d0/d12/d15/d3d/d90/df3/d113/l12d 0 2024-03-22T02:29:06.581 INFO:tasks.workunit.client.1.smithi122.stdout:1/868: mknod dc/d3b/d8d/d72/d9c/db0/dab/c112 0 2024-03-22T02:29:06.581 INFO:tasks.workunit.client.1.smithi122.stdout:7/967: dread - d0/fcb zero size 2024-03-22T02:29:06.582 INFO:tasks.workunit.client.1.smithi122.stdout:3/927: symlink d2/d9/dc/d57/da4/l137 0 2024-03-22T02:29:06.584 INFO:tasks.workunit.client.1.smithi122.stdout:0/781: mknod d11/d16/d37/dda/c10b 0 2024-03-22T02:29:06.587 INFO:tasks.workunit.client.1.smithi122.stdout:7/968: mknod d0/d4/d1d/df6/de0/c134 0 2024-03-22T02:29:06.589 INFO:tasks.workunit.client.1.smithi122.stdout:4/924: dread d0/d12/d15/d2d/f4f [0,4194304] 0 2024-03-22T02:29:06.590 INFO:tasks.workunit.client.1.smithi122.stdout:1/869: rename dc/d3b/d46/de7/deb to dc/d3b/d8d/d72/d9c/db0/d90/d113 0 2024-03-22T02:29:06.595 INFO:tasks.workunit.client.1.smithi122.stdout:0/782: symlink d11/d16/d37/d84/d87/l10c 0 2024-03-22T02:29:06.598 INFO:tasks.workunit.client.1.smithi122.stdout:0/783: stat d11/d16/d37/fbf 0 2024-03-22T02:29:06.600 INFO:tasks.workunit.client.1.smithi122.stdout:4/925: dwrite d0/f9 [8388608,4194304] 0 2024-03-22T02:29:06.601 INFO:tasks.workunit.client.1.smithi122.stdout:2/914: write df/d1f/d11f/f124 [113892,28206] 0 2024-03-22T02:29:06.606 INFO:tasks.workunit.client.1.smithi122.stdout:1/870: link dc/d3b/d8d/d72/d9c/db0/d67/c6d dc/d3b/d8d/d72/d9c/db0/dab/c114 0 2024-03-22T02:29:06.608 INFO:tasks.workunit.client.1.smithi122.stdout:0/784: mknod d11/d14/d1d/c10d 0 2024-03-22T02:29:06.608 INFO:tasks.workunit.client.1.smithi122.stdout:1/871: fsync dc/d2b/f2d 0 2024-03-22T02:29:06.614 INFO:tasks.workunit.client.1.smithi122.stdout:4/926: rename d0/d12/d15/d42/f89 to d0/d12/d15/d2d/db5/f12e 0 2024-03-22T02:29:06.616 INFO:tasks.workunit.client.1.smithi122.stdout:0/785: rmdir d11/d14/d1d/d27/dc0/dc6/dd0/dea 39 2024-03-22T02:29:06.623 INFO:tasks.workunit.client.1.smithi122.stdout:4/927: dwrite d0/d12/d15/d2d/db5/ff9 [0,4194304] 0 2024-03-22T02:29:06.623 INFO:tasks.workunit.client.1.smithi122.stdout:3/928: truncate d2/f15 3986707 0 2024-03-22T02:29:06.624 INFO:tasks.workunit.client.1.smithi122.stdout:7/969: read d0/fac [291955,12169] 0 2024-03-22T02:29:06.627 INFO:tasks.workunit.client.1.smithi122.stdout:2/915: dread df/d1f/d33/d34/fb2 [0,4194304] 0 2024-03-22T02:29:06.628 INFO:tasks.workunit.client.1.smithi122.stdout:0/786: unlink d11/d16/d37/d84/d87/df6/f100 0 2024-03-22T02:29:06.634 INFO:tasks.workunit.client.1.smithi122.stdout:7/970: creat d0/d4/d1d/db8/d9a/f135 x:0 0 0 2024-03-22T02:29:06.634 INFO:tasks.workunit.client.1.smithi122.stdout:3/929: mknod d2/d9/dc/d57/dab/c138 0 2024-03-22T02:29:06.636 INFO:tasks.workunit.client.1.smithi122.stdout:4/928: mknod d0/d12/d15/d42/d4e/de/c12f 0 2024-03-22T02:29:06.637 INFO:tasks.workunit.client.1.smithi122.stdout:0/787: dwrite d11/d47/ff4 [0,4194304] 0 2024-03-22T02:29:06.637 INFO:tasks.workunit.client.1.smithi122.stdout:1/872: dread dc/d3b/f81 [0,4194304] 0 2024-03-22T02:29:06.642 INFO:tasks.workunit.client.1.smithi122.stdout:2/916: creat df/d1f/d33/d6a/d7c/d8e/da0/dd4/f126 x:0 0 0 2024-03-22T02:29:06.644 INFO:tasks.workunit.client.1.smithi122.stdout:0/788: fsync d11/d14/d1d/d27/d9d/fcb 0 2024-03-22T02:29:06.649 INFO:tasks.workunit.client.1.smithi122.stdout:1/873: mknod dc/d3b/d8d/d72/d9c/db0/d67/c115 0 2024-03-22T02:29:06.651 INFO:tasks.workunit.client.1.smithi122.stdout:4/929: mknod d0/d12/d15/d42/c130 0 2024-03-22T02:29:06.654 INFO:tasks.workunit.client.1.smithi122.stdout:0/789: creat d11/d14/d23/d24/d33/d67/dc2/f10e x:0 0 0 2024-03-22T02:29:06.666 INFO:tasks.workunit.client.1.smithi122.stdout:3/930: write d2/d9/dc/fae [43217,4821] 0 2024-03-22T02:29:06.671 INFO:tasks.workunit.client.1.smithi122.stdout:3/931: write d2/d9/dc/d17/d28/f6c [4721877,90222] 0 2024-03-22T02:29:06.672 INFO:tasks.workunit.client.1.smithi122.stdout:7/971: dwrite d0/d30/f8c [4194304,4194304] 0 2024-03-22T02:29:06.684 INFO:tasks.workunit.client.1.smithi122.stdout:3/932: symlink d2/d9/dc/d57/l139 0 2024-03-22T02:29:06.685 INFO:tasks.workunit.client.1.smithi122.stdout:4/930: dwrite d0/d12/d61/fd0 [0,4194304] 0 2024-03-22T02:29:06.685 INFO:tasks.workunit.client.1.smithi122.stdout:7/972: mknod d0/d4/d80/dc4/c136 0 2024-03-22T02:29:06.686 INFO:tasks.workunit.client.1.smithi122.stdout:2/917: dwrite df/d1f/d3d/f92 [0,4194304] 0 2024-03-22T02:29:06.688 INFO:tasks.workunit.client.1.smithi122.stdout:0/790: dwrite d11/d47/f105 [0,4194304] 0 2024-03-22T02:29:06.694 INFO:tasks.workunit.client.1.smithi122.stdout:0/791: fdatasync d11/d14/d1d/fa2 0 2024-03-22T02:29:06.695 INFO:tasks.workunit.client.1.smithi122.stdout:4/931: fdatasync d0/d12/d15/d42/d4e/f22 0 2024-03-22T02:29:06.696 INFO:tasks.workunit.client.1.smithi122.stdout:0/792: fsync d11/d16/ff9 0 2024-03-22T02:29:06.700 INFO:tasks.workunit.client.1.smithi122.stdout:2/918: truncate df/d1f/d33/d4d/d94/dc2/d72/fba 988906 0 2024-03-22T02:29:06.702 INFO:tasks.workunit.client.1.smithi122.stdout:7/973: creat d0/d4/d80/f137 x:0 0 0 2024-03-22T02:29:06.703 INFO:tasks.workunit.client.1.smithi122.stdout:0/793: fdatasync d11/d14/d1d/d27/d9d/fcb 0 2024-03-22T02:29:06.703 INFO:tasks.workunit.client.1.smithi122.stdout:3/933: dwrite d2/d9/dc/d17/d51/fec [0,4194304] 0 2024-03-22T02:29:06.704 INFO:tasks.workunit.client.1.smithi122.stdout:1/874: dread dc/f38 [0,4194304] 0 2024-03-22T02:29:06.710 INFO:tasks.workunit.client.1.smithi122.stdout:7/974: creat d0/d4/d20/d21/dda/f138 x:0 0 0 2024-03-22T02:29:06.712 INFO:tasks.workunit.client.1.smithi122.stdout:3/934: truncate d2/d9/dc/d57/d89/fc5 1201619 0 2024-03-22T02:29:06.712 INFO:tasks.workunit.client.1.smithi122.stdout:7/975: chown d0/d4/d1d/d108/dbb/fc7 564260624 1 2024-03-22T02:29:06.713 INFO:tasks.workunit.client.1.smithi122.stdout:7/976: dread - d0/d4/de9/f125 zero size 2024-03-22T02:29:06.717 INFO:tasks.workunit.client.1.smithi122.stdout:4/932: rename d0/d12/d15/d2d/db5/fe9 to d0/d12/d15/d42/d4e/de/f131 0 2024-03-22T02:29:06.719 INFO:tasks.workunit.client.1.smithi122.stdout:0/794: creat d11/d4f/f10f x:0 0 0 2024-03-22T02:29:06.719 INFO:tasks.workunit.client.1.smithi122.stdout:0/795: fdatasync f10 0 2024-03-22T02:29:06.721 INFO:tasks.workunit.client.1.smithi122.stdout:7/977: read d0/d4/d1d/df6/ffe [2517095,21251] 0 2024-03-22T02:29:06.723 INFO:tasks.workunit.client.1.smithi122.stdout:4/933: rename d0/d12/d15/d3d/d90/l70 to d0/d12/da2/l132 0 2024-03-22T02:29:06.724 INFO:tasks.workunit.client.1.smithi122.stdout:4/934: fsync d0/d12/d15/d42/d4e/de/f10e 0 2024-03-22T02:29:06.726 INFO:tasks.workunit.client.1.smithi122.stdout:2/919: write df/d1f/d33/d4d/d94/dc2/d51/d67/fbf [1084498,116651] 0 2024-03-22T02:29:06.727 INFO:tasks.workunit.client.1.smithi122.stdout:0/796: creat d11/d14/d1d/d27/d2f/d4e/df0/f110 x:0 0 0 2024-03-22T02:29:06.727 INFO:tasks.workunit.client.1.smithi122.stdout:2/920: fdatasync df/d1f/d3d/f11a 0 2024-03-22T02:29:06.729 INFO:tasks.workunit.client.1.smithi122.stdout:7/978: rename d0/d4/d20/d21/f95 to d0/d4/d1d/db8/d98/f139 0 2024-03-22T02:29:06.733 INFO:tasks.workunit.client.1.smithi122.stdout:1/875: dwrite dc/d3b/d8d/d72/d9c/fd6 [0,4194304] 0 2024-03-22T02:29:06.734 INFO:tasks.workunit.client.1.smithi122.stdout:3/935: truncate d2/d9/dc/d57/d89/f102 2065309 0 2024-03-22T02:29:06.734 INFO:tasks.workunit.client.1.smithi122.stdout:1/876: chown dc/d3b/d8d/d72/d9c/db0/d71 4504 1 2024-03-22T02:29:06.736 INFO:tasks.workunit.client.1.smithi122.stdout:2/921: readlink df/d1f/d33/d4d/d94/deb/l10d 0 2024-03-22T02:29:06.737 INFO:tasks.workunit.client.1.smithi122.stdout:1/877: truncate dc/d3b/d46/f9e 9114 0 2024-03-22T02:29:06.738 INFO:tasks.workunit.client.1.smithi122.stdout:4/935: symlink d0/d12/d15/d42/l133 0 2024-03-22T02:29:06.741 INFO:tasks.workunit.client.1.smithi122.stdout:2/922: unlink df/d1f/d33/d4d/d7a/lff 0 2024-03-22T02:29:06.744 INFO:tasks.workunit.client.1.smithi122.stdout:0/797: creat d11/d14/d1d/d27/dc0/dc6/dd0/f111 x:0 0 0 2024-03-22T02:29:06.744 INFO:tasks.workunit.client.1.smithi122.stdout:2/923: truncate df/d1f/d33/d4d/d94/dc2/d72/ff3 1041292 0 2024-03-22T02:29:06.744 INFO:tasks.workunit.client.1.smithi122.stdout:3/936: creat d2/d9/dc/d17/d135/f13a x:0 0 0 2024-03-22T02:29:06.748 INFO:tasks.workunit.client.1.smithi122.stdout:2/924: symlink df/d1f/d33/d34/ddd/l127 0 2024-03-22T02:29:06.751 INFO:tasks.workunit.client.1.smithi122.stdout:7/979: truncate d0/d4/d1d/df6/ffe 824033 0 2024-03-22T02:29:06.751 INFO:tasks.workunit.client.1.smithi122.stdout:1/878: dread dc/d55/fc3 [0,4194304] 0 2024-03-22T02:29:06.751 INFO:tasks.workunit.client.1.smithi122.stdout:3/937: truncate d2/d9/dc/df/d49/d85/f94 824922 0 2024-03-22T02:29:06.754 INFO:tasks.workunit.client.1.smithi122.stdout:0/798: dwrite d11/d14/d23/d24/d33/d67/dc2/fd4 [0,4194304] 0 2024-03-22T02:29:06.756 INFO:tasks.workunit.client.1.smithi122.stdout:7/980: mknod d0/d4/d80/dc4/c13a 0 2024-03-22T02:29:06.758 INFO:tasks.workunit.client.1.smithi122.stdout:7/981: truncate d0/d4/d1d/d108/d7d/fe3 1298934 0 2024-03-22T02:29:06.759 INFO:tasks.workunit.client.1.smithi122.stdout:1/879: mkdir dc/d3b/d8d/d72/d9c/de8/d116 0 2024-03-22T02:29:06.762 INFO:tasks.workunit.client.1.smithi122.stdout:3/938: write d2/d9/dc/d17/d28/d4c/da5/db4/fd3 [4187858,13804] 0 2024-03-22T02:29:06.763 INFO:tasks.workunit.client.1.smithi122.stdout:0/799: unlink d11/d14/d1d/d27/d2f/d4e/faf 0 2024-03-22T02:29:06.765 INFO:tasks.workunit.client.1.smithi122.stdout:7/982: readlink d0/d4/de9/lb9 0 2024-03-22T02:29:06.767 INFO:tasks.workunit.client.1.smithi122.stdout:2/925: fdatasync df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2/f104 0 2024-03-22T02:29:06.769 INFO:tasks.workunit.client.1.smithi122.stdout:3/939: mkdir d2/d9/dc/df/d49/d85/dda/dd5/d13b 0 2024-03-22T02:29:06.769 INFO:tasks.workunit.client.1.smithi122.stdout:4/936: dwrite d0/d25/f8 [0,4194304] 0 2024-03-22T02:29:06.772 INFO:tasks.workunit.client.1.smithi122.stdout:3/940: chown d2/d9/dc/c9e 499 1 2024-03-22T02:29:06.773 INFO:tasks.workunit.client.1.smithi122.stdout:3/941: fdatasync d2/d9/dc/d17/d28/fde 0 2024-03-22T02:29:06.776 INFO:tasks.workunit.client.1.smithi122.stdout:1/880: getdents dc/d3b/d46/db5/dd9 0 2024-03-22T02:29:06.776 INFO:tasks.workunit.client.1.smithi122.stdout:7/983: mknod d0/d4/d1d/d108/d7d/c13b 0 2024-03-22T02:29:06.778 INFO:tasks.workunit.client.1.smithi122.stdout:2/926: symlink df/d1f/d33/d6a/d7c/d8e/da0/l128 0 2024-03-22T02:29:06.779 INFO:tasks.workunit.client.1.smithi122.stdout:1/881: write dc/ff [37431,6049] 0 2024-03-22T02:29:06.780 INFO:tasks.workunit.client.1.smithi122.stdout:0/800: dread d11/d14/d1d/d27/fe4 [4194304,4194304] 0 2024-03-22T02:29:06.784 INFO:tasks.workunit.client.1.smithi122.stdout:4/937: dwrite d0/d12/d61/fe5 [4194304,4194304] 0 2024-03-22T02:29:06.785 INFO:tasks.workunit.client.1.smithi122.stdout:0/801: chown d11/d14/d1d/c10d 15946498 1 2024-03-22T02:29:06.785 INFO:tasks.workunit.client.1.smithi122.stdout:7/984: chown d0/le4 1 1 2024-03-22T02:29:06.786 INFO:tasks.workunit.client.1.smithi122.stdout:3/942: symlink d2/d9/dc/d57/d89/l13c 0 2024-03-22T02:29:06.788 INFO:tasks.workunit.client.1.smithi122.stdout:7/985: write d0/d4/d20/d21/dda/fe1 [5471203,13853] 0 2024-03-22T02:29:06.789 INFO:tasks.workunit.client.1.smithi122.stdout:2/927: creat df/d1f/d33/d4d/d94/df0/f129 x:0 0 0 2024-03-22T02:29:06.790 INFO:tasks.workunit.client.1.smithi122.stdout:1/882: mknod dc/d2b/d86/ddd/c117 0 2024-03-22T02:29:06.791 INFO:tasks.workunit.client.1.smithi122.stdout:0/802: creat d11/d14/d1d/d27/d9d/daa/f112 x:0 0 0 2024-03-22T02:29:06.792 INFO:tasks.workunit.client.1.smithi122.stdout:4/938: dread - d0/d12/d15/d42/d4e/dd6/f10a zero size 2024-03-22T02:29:06.794 INFO:tasks.workunit.client.1.smithi122.stdout:7/986: chown d0/d4/d80/dc4/d126/l132 50169539 1 2024-03-22T02:29:06.795 INFO:tasks.workunit.client.1.smithi122.stdout:4/939: truncate d0/d12/d61/fd0 4712034 0 2024-03-22T02:29:06.797 INFO:tasks.workunit.client.1.smithi122.stdout:1/883: truncate dc/d2b/d86/dbb/fce 3822898 0 2024-03-22T02:29:06.798 INFO:tasks.workunit.client.1.smithi122.stdout:2/928: fsync df/d1f/d33/d4d/d94/dc2/d51/f102 0 2024-03-22T02:29:06.800 INFO:tasks.workunit.client.1.smithi122.stdout:7/987: rmdir d0/d4/d1d/df6/de0 39 2024-03-22T02:29:06.800 INFO:tasks.workunit.client.1.smithi122.stdout:2/929: write df/d1f/f3e [4385047,110003] 0 2024-03-22T02:29:06.804 INFO:tasks.workunit.client.1.smithi122.stdout:4/940: rename d0/d12/l46 to d0/d12/d15/d42/d4e/dd6/l134 0 2024-03-22T02:29:06.810 INFO:tasks.workunit.client.1.smithi122.stdout:1/884: creat dc/d3b/d46/db5/dd9/f118 x:0 0 0 2024-03-22T02:29:06.811 INFO:tasks.workunit.client.1.smithi122.stdout:1/885: creat dc/d4c/f119 x:0 0 0 2024-03-22T02:29:06.816 INFO:tasks.workunit.client.1.smithi122.stdout:4/941: readlink d0/d12/d15/d42/d4e/dd6/l134 0 2024-03-22T02:29:06.817 INFO:tasks.workunit.client.1.smithi122.stdout:1/886: mknod dc/d3b/d8d/d72/d9c/dd3/c11a 0 2024-03-22T02:29:06.820 INFO:tasks.workunit.client.1.smithi122.stdout:4/942: dread - d0/d12/d61/dce/f121 zero size 2024-03-22T02:29:06.821 INFO:tasks.workunit.client.1.smithi122.stdout:7/988: dread d0/d4/d1d/db8/f71 [0,4194304] 0 2024-03-22T02:29:06.823 INFO:tasks.workunit.client.1.smithi122.stdout:0/803: dwrite d11/d14/d1b/f8f [0,4194304] 0 2024-03-22T02:29:06.825 INFO:tasks.workunit.client.1.smithi122.stdout:7/989: readlink d0/d4/d80/dc4/l11d 0 2024-03-22T02:29:06.826 INFO:tasks.workunit.client.1.smithi122.stdout:4/943: unlink d0/d25/f28 0 2024-03-22T02:29:06.827 INFO:tasks.workunit.client.1.smithi122.stdout:2/930: dwrite df/d1f/d33/d4d/d94/dc2/f119 [0,4194304] 0 2024-03-22T02:29:06.828 INFO:tasks.workunit.client.1.smithi122.stdout:1/887: dread dc/d55/f77 [0,4194304] 0 2024-03-22T02:29:06.830 INFO:tasks.workunit.client.1.smithi122.stdout:4/944: chown d0/d12/d15/d42/d4e/dd6/l134 109987551 1 2024-03-22T02:29:06.831 INFO:tasks.workunit.client.1.smithi122.stdout:7/990: chown d0/d4/d1d/df6/fa6 0 1 2024-03-22T02:29:06.832 INFO:tasks.workunit.client.1.smithi122.stdout:1/888: mknod dc/d3b/d8d/d72/d9c/dd3/c11b 0 2024-03-22T02:29:06.832 INFO:tasks.workunit.client.1.smithi122.stdout:7/991: dread - d0/d4/d20/d21/d28/fad zero size 2024-03-22T02:29:06.833 INFO:tasks.workunit.client.1.smithi122.stdout:2/931: mknod df/d1f/d33/d4d/d7a/d116/c12a 0 2024-03-22T02:29:06.836 INFO:tasks.workunit.client.1.smithi122.stdout:4/945: creat d0/d12/d15/d3d/d90/f135 x:0 0 0 2024-03-22T02:29:06.836 INFO:tasks.workunit.client.1.smithi122.stdout:7/992: fdatasync d0/d4/d20/d21/d28/f90 0 2024-03-22T02:29:06.836 INFO:tasks.workunit.client.1.smithi122.stdout:1/889: creat dc/d3b/d8d/d72/d9c/db0/d67/f11c x:0 0 0 2024-03-22T02:29:06.839 INFO:tasks.workunit.client.1.smithi122.stdout:4/946: readlink d0/d12/d15/d42/d4e/dd6/l134 0 2024-03-22T02:29:06.839 INFO:tasks.workunit.client.1.smithi122.stdout:1/890: chown dc/d3b/d8d/d94/lff 32298 1 2024-03-22T02:29:06.840 INFO:tasks.workunit.client.1.smithi122.stdout:2/932: mkdir df/d1f/d33/d4d/d94/dc2/d12b 0 2024-03-22T02:29:06.841 INFO:tasks.workunit.client.1.smithi122.stdout:1/891: stat dc/d3b/d8d/d72/d9c/db0/f7a 0 2024-03-22T02:29:06.841 INFO:tasks.workunit.client.1.smithi122.stdout:7/993: mknod d0/d30/c13c 0 2024-03-22T02:29:06.842 INFO:tasks.workunit.client.1.smithi122.stdout:1/892: chown dc/d2b/d86/lf4 0 1 2024-03-22T02:29:06.843 INFO:tasks.workunit.client.1.smithi122.stdout:4/947: rename d0/d12/ffd to d0/d12/d15/df7/f136 0 2024-03-22T02:29:06.844 INFO:tasks.workunit.client.1.smithi122.stdout:1/893: write dc/d3b/fcb [1941294,49914] 0 2024-03-22T02:29:06.845 INFO:tasks.workunit.client.1.smithi122.stdout:4/948: chown d0/d12/da2/lc5 1479186501 1 2024-03-22T02:29:06.845 INFO:tasks.workunit.client.1.smithi122.stdout:7/994: creat d0/d4/d1d/f13d x:0 0 0 2024-03-22T02:29:06.846 INFO:tasks.workunit.client.1.smithi122.stdout:2/933: dread - df/d1f/d33/d6a/d7c/d8e/da0/fee zero size 2024-03-22T02:29:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:06 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:06 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:06.849 INFO:tasks.workunit.client.1.smithi122.stdout:1/894: unlink dc/d2b/f8c 0 2024-03-22T02:29:06.851 INFO:tasks.workunit.client.1.smithi122.stdout:4/949: symlink d0/d12/l137 0 2024-03-22T02:29:06.853 INFO:tasks.workunit.client.1.smithi122.stdout:0/804: dread d11/d14/d23/f77 [0,4194304] 0 2024-03-22T02:29:06.854 INFO:tasks.workunit.client.1.smithi122.stdout:0/805: chown d11/d14/d1d/d27/d9d/daa/dcd 2790 1 2024-03-22T02:29:06.856 INFO:tasks.workunit.client.1.smithi122.stdout:4/950: read - d0/d12/d73/fd5 zero size 2024-03-22T02:29:06.856 INFO:tasks.workunit.client.1.smithi122.stdout:4/951: dread - d0/d7a/f123 zero size 2024-03-22T02:29:06.857 INFO:tasks.workunit.client.1.smithi122.stdout:0/806: rmdir d11/d16/d37/d84 39 2024-03-22T02:29:06.858 INFO:tasks.workunit.client.1.smithi122.stdout:7/995: link d0/f2 d0/f13e 0 2024-03-22T02:29:06.859 INFO:tasks.workunit.client.1.smithi122.stdout:7/996: read - d0/d4/d1d/db8/d114/f124 zero size 2024-03-22T02:29:06.860 INFO:tasks.workunit.client.1.smithi122.stdout:2/934: rename df/d1f/d3d/d52/ddc/ff1 to df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f12c 0 2024-03-22T02:29:06.861 INFO:tasks.workunit.client.1.smithi122.stdout:4/952: mkdir d0/d12/d15/d42/d4e/de/da1/dbf/d103/d138 0 2024-03-22T02:29:06.865 INFO:tasks.workunit.client.1.smithi122.stdout:2/935: creat df/d1f/d3d/d52/f12d x:0 0 0 2024-03-22T02:29:06.868 INFO:tasks.workunit.client.1.smithi122.stdout:7/997: creat d0/d4/d1d/df6/f13f x:0 0 0 2024-03-22T02:29:06.869 INFO:tasks.workunit.client.1.smithi122.stdout:2/936: stat df/d1f/d33/d4d/d94/dc2/d51/d67/f6e 0 2024-03-22T02:29:06.872 INFO:tasks.workunit.client.1.smithi122.stdout:0/807: rename d11/d14/d1d/d27/d2f/dd1/de7 to d11/d16/d113 0 2024-03-22T02:29:06.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:06 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:06 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:06.874 INFO:tasks.workunit.client.1.smithi122.stdout:4/953: link d0/d12/d15/d3d/d90/df3/d113/f120 d0/d12/d15/d3d/d90/df3/d113/f139 0 2024-03-22T02:29:06.874 INFO:tasks.workunit.client.1.smithi122.stdout:7/998: dread - d0/d4/d1d/df6/d5d/ff3 zero size 2024-03-22T02:29:06.875 INFO:tasks.workunit.client.1.smithi122.stdout:2/937: fdatasync df/d1f/f2d 0 2024-03-22T02:29:06.877 INFO:tasks.workunit.client.1.smithi122.stdout:4/954: fsync d0/d12/d61/f87 0 2024-03-22T02:29:06.880 INFO:tasks.workunit.client.1.smithi122.stdout:7/999: creat d0/d4/de9/d5c/f140 x:0 0 0 2024-03-22T02:29:06.882 INFO:tasks.workunit.client.1.smithi122.stdout:2/938: chown df/d1f/d3d/d52/ddc/fb7 0 1 2024-03-22T02:29:06.887 INFO:tasks.workunit.client.1.smithi122.stdout:4/955: link d0/d12/d15/d42/d4e/c8d d0/d12/d61/dce/d118/c13a 0 2024-03-22T02:29:06.889 INFO:tasks.workunit.client.1.smithi122.stdout:2/939: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f70 [0,4194304] 0 2024-03-22T02:29:06.890 INFO:tasks.workunit.client.1.smithi122.stdout:2/940: fsync df/d1f/d11f/f121 0 2024-03-22T02:29:06.891 INFO:tasks.workunit.client.1.smithi122.stdout:0/808: dread d11/d16/d44/d89/f4b [4194304,4194304] 0 2024-03-22T02:29:06.892 INFO:tasks.workunit.client.1.smithi122.stdout:4/956: link d0/d12/l10b d0/d12/d15/d3d/d90/df3/d113/l13b 0 2024-03-22T02:29:06.894 INFO:tasks.workunit.client.1.smithi122.stdout:2/941: creat df/d1f/d33/d4d/d94/dc2/d51/d67/db0/f12e x:0 0 0 2024-03-22T02:29:06.895 INFO:tasks.workunit.client.1.smithi122.stdout:0/809: creat d11/d14/d1d/d8a/f114 x:0 0 0 2024-03-22T02:29:06.896 INFO:tasks.workunit.client.1.smithi122.stdout:4/957: mknod d0/d12/d15/d2d/c13c 0 2024-03-22T02:29:06.896 INFO:tasks.workunit.client.1.smithi122.stdout:0/810: dread - d11/d14/d1d/d27/d9d/daa/fdb zero size 2024-03-22T02:29:06.899 INFO:tasks.workunit.client.1.smithi122.stdout:2/942: rename df/d1f/d33/d4d/d94/dc2/d72/ff3 to df/d1f/d33/d6a/f12f 0 2024-03-22T02:29:06.900 INFO:tasks.workunit.client.1.smithi122.stdout:4/958: mknod d0/d12/d15/d3d/d90/df3/d113/c13d 0 2024-03-22T02:29:06.910 INFO:tasks.workunit.client.1.smithi122.stdout:0/811: link d11/d14/d1d/d27/c62 d11/d16/d37/d84/c115 0 2024-03-22T02:29:06.912 INFO:tasks.workunit.client.1.smithi122.stdout:4/959: rename d0/d12/da2/l132 to d0/d12/d15/d42/d4e/l13e 0 2024-03-22T02:29:06.914 INFO:tasks.workunit.client.1.smithi122.stdout:0/812: mkdir d11/d4f/d5a/d116 0 2024-03-22T02:29:06.915 INFO:tasks.workunit.client.1.smithi122.stdout:4/960: rmdir d0/d25/df2 39 2024-03-22T02:29:06.916 INFO:tasks.workunit.client.1.smithi122.stdout:4/961: chown d0/d12/d15/d3d/d90/cd1 81 1 2024-03-22T02:29:06.917 INFO:tasks.workunit.client.1.smithi122.stdout:2/943: write df/f37 [51450,89195] 0 2024-03-22T02:29:06.920 INFO:tasks.workunit.client.1.smithi122.stdout:0/813: mknod d11/d14/d1d/d27/dc0/dc6/dd0/dea/c117 0 2024-03-22T02:29:06.923 INFO:tasks.workunit.client.1.smithi122.stdout:4/962: dwrite d0/d12/d61/fd0 [0,4194304] 0 2024-03-22T02:29:06.923 INFO:tasks.workunit.client.1.smithi122.stdout:0/814: rmdir d11/d14/d1d/d27/dc0/dc6 39 2024-03-22T02:29:06.929 INFO:tasks.workunit.client.1.smithi122.stdout:2/944: truncate df/d1f/d33/d4d/d94/dc2/d51/fcd 86957 0 2024-03-22T02:29:06.933 INFO:tasks.workunit.client.1.smithi122.stdout:4/963: write d0/d12/d15/f55 [4168966,52753] 0 2024-03-22T02:29:06.934 INFO:tasks.workunit.client.1.smithi122.stdout:0/815: dwrite d11/d16/f45 [0,4194304] 0 2024-03-22T02:29:06.939 INFO:tasks.workunit.client.1.smithi122.stdout:0/816: creat d11/d14/d1d/d27/dc0/dc6/dd0/dea/f118 x:0 0 0 2024-03-22T02:29:06.941 INFO:tasks.workunit.client.1.smithi122.stdout:4/964: dwrite d0/d12/d15/d2d/db5/fc8 [0,4194304] 0 2024-03-22T02:29:06.944 INFO:tasks.workunit.client.1.smithi122.stdout:2/945: truncate f6 3497483 0 2024-03-22T02:29:06.947 INFO:tasks.workunit.client.1.smithi122.stdout:4/965: creat d0/d12/d15/d2d/f13f x:0 0 0 2024-03-22T02:29:06.950 INFO:tasks.workunit.client.1.smithi122.stdout:0/817: creat d11/d14/d23/d24/d33/f119 x:0 0 0 2024-03-22T02:29:06.951 INFO:tasks.workunit.client.1.smithi122.stdout:4/966: symlink d0/d12/d61/dce/l140 0 2024-03-22T02:29:06.951 INFO:tasks.workunit.client.1.smithi122.stdout:0/818: chown d11/d16/d44/f76 9810 1 2024-03-22T02:29:06.953 INFO:tasks.workunit.client.1.smithi122.stdout:4/967: chown d0/d12/l4b 31380 1 2024-03-22T02:29:06.953 INFO:tasks.workunit.client.1.smithi122.stdout:0/819: rmdir d11/d14/d1d/d27/d2f 39 2024-03-22T02:29:06.954 INFO:tasks.workunit.client.1.smithi122.stdout:2/946: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/fde [0,4194304] 0 2024-03-22T02:29:06.957 INFO:tasks.workunit.client.1.smithi122.stdout:2/947: dread - df/d1f/d33/d6a/f90 zero size 2024-03-22T02:29:06.962 INFO:tasks.workunit.client.1.smithi122.stdout:2/948: creat df/f130 x:0 0 0 2024-03-22T02:29:06.963 INFO:tasks.workunit.client.1.smithi122.stdout:4/968: dwrite d0/d12/d61/f92 [0,4194304] 0 2024-03-22T02:29:06.979 INFO:tasks.workunit.client.1.smithi122.stdout:4/969: dwrite d0/d12/f79 [0,4194304] 0 2024-03-22T02:29:06.979 INFO:tasks.workunit.client.1.smithi122.stdout:2/949: dwrite df/d1f/d33/d4d/d94/dc2/d114/d115/dd2/f62 [0,4194304] 0 2024-03-22T02:29:06.982 INFO:tasks.workunit.client.1.smithi122.stdout:4/970: truncate d0/d12/d61/dce/fd4 560604 0 2024-03-22T02:29:06.982 INFO:tasks.workunit.client.1.smithi122.stdout:2/950: fdatasync df/d1f/d33/d6a/f8d 0 2024-03-22T02:29:06.990 INFO:tasks.workunit.client.1.smithi122.stdout:2/951: link df/f13 df/d1f/d33/d4d/d94/dc2/d114/d115/f131 0 2024-03-22T02:29:07.007 INFO:tasks.workunit.client.1.smithi122.stdout:4/971: dread d0/d12/d15/d42/d4e/f5b [0,4194304] 0 2024-03-22T02:29:07.007 INFO:tasks.workunit.client.1.smithi122.stdout:2/952: write df/d1f/d33/f80 [217448,74647] 0 2024-03-22T02:29:07.011 INFO:tasks.workunit.client.1.smithi122.stdout:2/953: mkdir df/d1f/d11f/d132 0 2024-03-22T02:29:07.016 INFO:tasks.workunit.client.1.smithi122.stdout:4/972: dwrite d0/d12/d15/feb [0,4194304] 0 2024-03-22T02:29:07.017 INFO:tasks.workunit.client.1.smithi122.stdout:2/954: rename df/d1f/d33/d4d/d94/fe7 to df/d1f/d3d/d52/ddc/f133 0 2024-03-22T02:29:07.019 INFO:tasks.workunit.client.1.smithi122.stdout:4/973: creat d0/d12/d15/d3d/d90/df3/d113/f141 x:0 0 0 2024-03-22T02:29:07.020 INFO:tasks.workunit.client.1.smithi122.stdout:2/955: fdatasync df/d1f/d33/d4d/f78 0 2024-03-22T02:29:07.021 INFO:tasks.workunit.client.1.smithi122.stdout:4/974: unlink d0/d12/d15/d42/c11a 0 2024-03-22T02:29:07.026 INFO:tasks.workunit.client.1.smithi122.stdout:4/975: unlink d0/d12/d15/d2d/c13c 0 2024-03-22T02:29:07.028 INFO:tasks.workunit.client.1.smithi122.stdout:2/956: write fd [5073327,31557] 0 2024-03-22T02:29:07.029 INFO:tasks.workunit.client.1.smithi122.stdout:4/976: symlink d0/d12/d15/l142 0 2024-03-22T02:29:07.035 INFO:tasks.workunit.client.1.smithi122.stdout:4/977: write d0/d25/fa [1289844,67616] 0 2024-03-22T02:29:07.036 INFO:tasks.workunit.client.1.smithi122.stdout:2/957: write df/d1f/d33/d4d/d94/dc2/d51/fcd [566598,110089] 0 2024-03-22T02:29:07.038 INFO:tasks.workunit.client.1.smithi122.stdout:2/958: unlink df/d1f/f3c 0 2024-03-22T02:29:07.039 INFO:tasks.workunit.client.1.smithi122.stdout:4/978: creat d0/d12/d15/d3d/d90/df3/d113/f143 x:0 0 0 2024-03-22T02:29:07.039 INFO:tasks.workunit.client.1.smithi122.stdout:2/959: chown df/d1f/d33/l11c 175852 1 2024-03-22T02:29:07.040 INFO:tasks.workunit.client.1.smithi122.stdout:2/960: stat df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2 0 2024-03-22T02:29:07.041 INFO:tasks.workunit.client.1.smithi122.stdout:4/979: creat d0/d12/d61/dce/d116/f144 x:0 0 0 2024-03-22T02:29:07.041 INFO:tasks.workunit.client.1.smithi122.stdout:2/961: chown df/d1f/d33/d4d/d94/dc2/d51 3824 1 2024-03-22T02:29:07.042 INFO:tasks.workunit.client.1.smithi122.stdout:2/962: chown df/d1f/d33/d4d/d94/dc2/d114/d115/l8b 5 1 2024-03-22T02:29:07.043 INFO:tasks.workunit.client.1.smithi122.stdout:4/980: stat d0/d12/d15/d42/d4e/de/d91/c105 0 2024-03-22T02:29:07.044 INFO:tasks.workunit.client.1.smithi122.stdout:4/981: creat d0/d7a/f145 x:0 0 0 2024-03-22T02:29:07.046 INFO:tasks.workunit.client.1.smithi122.stdout:2/963: rename df/d1f/d33/c7f to df/d1f/d33/d34/c134 0 2024-03-22T02:29:07.047 INFO:tasks.workunit.client.1.smithi122.stdout:4/982: symlink d0/d12/d15/d3d/d90/l146 0 2024-03-22T02:29:07.048 INFO:tasks.workunit.client.1.smithi122.stdout:2/964: symlink df/d1f/d33/d4d/d94/dc2/d114/l135 0 2024-03-22T02:29:07.049 INFO:tasks.workunit.client.1.smithi122.stdout:4/983: mknod d0/d12/d15/df7/c147 0 2024-03-22T02:29:07.051 INFO:tasks.workunit.client.1.smithi122.stdout:2/965: creat df/d1f/d33/d4d/d94/dc2/f136 x:0 0 0 2024-03-22T02:29:07.051 INFO:tasks.workunit.client.1.smithi122.stdout:4/984: creat d0/d12/d15/da7/f148 x:0 0 0 2024-03-22T02:29:07.053 INFO:tasks.workunit.client.1.smithi122.stdout:2/966: unlink df/d1f/d33/d4d/d94/dc2/lae 0 2024-03-22T02:29:07.055 INFO:tasks.workunit.client.1.smithi122.stdout:4/985: rename d0/d12/d73/fd5 to d0/d12/d15/d42/d4e/dd6/f149 0 2024-03-22T02:29:07.067 INFO:tasks.workunit.client.1.smithi122.stdout:4/986: dread d0/d12/d15/d42/d4e/de/f66 [0,4194304] 0 2024-03-22T02:29:07.070 INFO:tasks.workunit.client.1.smithi122.stdout:4/987: rename d0/d12/d15/d42/c94 to d0/d12/d15/c14a 0 2024-03-22T02:29:07.071 INFO:tasks.workunit.client.1.smithi122.stdout:4/988: readlink d0/d12/l2f 0 2024-03-22T02:29:07.071 INFO:tasks.workunit.client.1.smithi122.stdout:4/989: chown d0/d12/d15/feb 19 1 2024-03-22T02:29:07.073 INFO:tasks.workunit.client.1.smithi122.stdout:4/990: unlink d0/d12/d15/d3d/d90/f48 0 2024-03-22T02:29:07.075 INFO:tasks.workunit.client.1.smithi122.stdout:4/991: mkdir d0/d12/d15/d2d/d14b 0 2024-03-22T02:29:07.076 INFO:tasks.workunit.client.1.smithi122.stdout:4/992: creat d0/d12/d15/d3d/d90/f14c x:0 0 0 2024-03-22T02:29:07.087 INFO:tasks.workunit.client.1.smithi122.stdout:4/993: write d0/d12/d15/fef [469546,62351] 0 2024-03-22T02:29:07.096 INFO:tasks.workunit.client.1.smithi122.stdout:4/994: dwrite d0/d12/d15/d42/d4e/de/d91/fa9 [0,4194304] 0 2024-03-22T02:29:07.099 INFO:tasks.workunit.client.1.smithi122.stdout:4/995: getdents d0/d12/d15/d42/d4e/de/da1/dbf/d103/d138 0 2024-03-22T02:29:07.102 INFO:tasks.workunit.client.1.smithi122.stdout:4/996: unlink d0/d12/d15/c109 0 2024-03-22T02:29:07.104 INFO:tasks.workunit.client.1.smithi122.stdout:4/997: mkdir d0/d12/da2/d11e/d14d 0 2024-03-22T02:29:07.105 INFO:tasks.workunit.client.1.smithi122.stdout:4/998: chown d0/d7a/fb6 1025961003 1 2024-03-22T02:29:07.108 INFO:tasks.workunit.client.1.smithi122.stdout:4/999: rename d0/d12/d15/d42/d4e/de/f8b to d0/d12/d61/dce/f14e 0 2024-03-22T02:29:07.255 INFO:tasks.workunit.client.1.smithi122.stdout:5/890: sync 2024-03-22T02:29:07.255 INFO:tasks.workunit.client.1.smithi122.stdout:5/891: stat d8/d74/fe1 0 2024-03-22T02:29:07.256 INFO:tasks.workunit.client.1.smithi122.stdout:5/892: mkdir d8/d1e/d22/d78/d12d 0 2024-03-22T02:29:07.257 INFO:tasks.workunit.client.1.smithi122.stdout:5/893: chown d8/d1e/d22/d3a/d46/d4f/d9b 3640234 1 2024-03-22T02:29:07.391 INFO:tasks.workunit.client.1.smithi122.stdout:3/943: sync 2024-03-22T02:29:07.403 INFO:tasks.workunit.client.1.smithi122.stdout:3/944: dread d2/d9/dc/df/d38/f3b [0,4194304] 0 2024-03-22T02:29:07.419 INFO:tasks.workunit.client.1.smithi122.stdout:3/945: dread d2/d9/dc/df/d49/fac [0,4194304] 0 2024-03-22T02:29:07.421 INFO:tasks.workunit.client.1.smithi122.stdout:3/946: unlink d2/d9/dc/df/d49/d85/dda/dd5/df9/f106 0 2024-03-22T02:29:07.423 INFO:tasks.workunit.client.1.smithi122.stdout:3/947: dread d2/d9/dc/df/d49/f4f [0,4194304] 0 2024-03-22T02:29:07.428 INFO:tasks.workunit.client.1.smithi122.stdout:3/948: write d2/d9/dc/df/d49/d85/f94 [287134,43832] 0 2024-03-22T02:29:07.430 INFO:tasks.workunit.client.1.smithi122.stdout:3/949: mkdir d2/d9/dc/d17/d28/d13d 0 2024-03-22T02:29:07.430 INFO:tasks.workunit.client.1.smithi122.stdout:5/894: sync 2024-03-22T02:29:07.430 INFO:tasks.workunit.client.1.smithi122.stdout:0/820: sync 2024-03-22T02:29:07.430 INFO:tasks.workunit.client.1.smithi122.stdout:2/967: sync 2024-03-22T02:29:07.431 INFO:tasks.workunit.client.1.smithi122.stdout:1/895: sync 2024-03-22T02:29:07.433 INFO:tasks.workunit.client.1.smithi122.stdout:2/968: fsync df/d1f/d33/d4d/f107 0 2024-03-22T02:29:07.435 INFO:tasks.workunit.client.1.smithi122.stdout:0/821: creat d11/d14/d23/d24/d33/d67/f11a x:0 0 0 2024-03-22T02:29:07.436 INFO:tasks.workunit.client.1.smithi122.stdout:3/950: creat d2/d9/dc/df/d49/d85/dda/dd5/df9/d11d/f13e x:0 0 0 2024-03-22T02:29:07.438 INFO:tasks.workunit.client.1.smithi122.stdout:5/895: write d8/d74/f86 [29985,80331] 0 2024-03-22T02:29:07.438 INFO:tasks.workunit.client.1.smithi122.stdout:0/822: chown d11/d14/d1d/d8a/l78 2337 1 2024-03-22T02:29:07.443 INFO:tasks.workunit.client.1.smithi122.stdout:1/896: dwrite dc/d3b/d8d/d72/d9c/de8/f10c [0,4194304] 0 2024-03-22T02:29:07.444 INFO:tasks.workunit.client.1.smithi122.stdout:0/823: dread - d11/d14/d1d/d27/dc0/dc6/dd0/dea/f101 zero size 2024-03-22T02:29:07.445 INFO:tasks.workunit.client.1.smithi122.stdout:5/896: creat d8/d1e/d22/d3a/d46/f12e x:0 0 0 2024-03-22T02:29:07.445 INFO:tasks.workunit.client.1.smithi122.stdout:3/951: write d2/d9/dc/df/d49/f4f [555698,7504] 0 2024-03-22T02:29:07.447 INFO:tasks.workunit.client.1.smithi122.stdout:2/969: dwrite df/d1f/d33/d6a/d7c/d8e/da0/dd4/f126 [0,4194304] 0 2024-03-22T02:29:07.452 INFO:tasks.workunit.client.1.smithi122.stdout:0/824: symlink d11/d14/d23/d24/d33/d67/l11b 0 2024-03-22T02:29:07.454 INFO:tasks.workunit.client.1.smithi122.stdout:2/970: readlink df/d1f/d3d/d52/ddc/l97 0 2024-03-22T02:29:07.455 INFO:tasks.workunit.client.1.smithi122.stdout:2/971: write df/d1f/d33/d34/f43 [3264520,15371] 0 2024-03-22T02:29:07.458 INFO:tasks.workunit.client.1.smithi122.stdout:2/972: dread - df/d1f/d33/d6a/d7c/d8e/da0/dd4/de2/f104 zero size 2024-03-22T02:29:07.460 INFO:tasks.workunit.client.1.smithi122.stdout:2/973: read - df/d1f/d3d/f11a zero size 2024-03-22T02:29:07.463 INFO:tasks.workunit.client.1.smithi122.stdout:1/897: dwrite dc/d3b/d8d/d72/d9c/db0/d71/fb1 [0,4194304] 0 2024-03-22T02:29:07.467 INFO:tasks.workunit.client.1.smithi122.stdout:2/974: rename df/d1f/lcb to df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/l137 0 2024-03-22T02:29:07.468 INFO:tasks.workunit.client.1.smithi122.stdout:1/898: symlink dc/d2b/d86/d110/l11d 0 2024-03-22T02:29:07.471 INFO:tasks.workunit.client.1.smithi122.stdout:2/975: truncate df/d1f/d33/d4d/d94/dc2/d51/d67/fbf 1620031 0 2024-03-22T02:29:07.472 INFO:tasks.workunit.client.1.smithi122.stdout:0/825: write d11/d14/d1d/d27/d59/fb0 [748737,71445] 0 2024-03-22T02:29:07.473 INFO:tasks.workunit.client.1.smithi122.stdout:1/899: creat dc/d3b/d46/db5/dba/df6/f11e x:0 0 0 2024-03-22T02:29:07.476 INFO:tasks.workunit.client.1.smithi122.stdout:2/976: unlink df/d1f/d33/d4d/f108 0 2024-03-22T02:29:07.477 INFO:tasks.workunit.client.1.smithi122.stdout:1/900: readlink dc/d3b/d8d/d72/d9c/db0/d67/l10b 0 2024-03-22T02:29:07.479 INFO:tasks.workunit.client.1.smithi122.stdout:5/897: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f5d [0,4194304] 0 2024-03-22T02:29:07.480 INFO:tasks.workunit.client.1.smithi122.stdout:2/977: mkdir df/d1f/d9b/d138 0 2024-03-22T02:29:07.486 INFO:tasks.workunit.client.1.smithi122.stdout:1/901: mknod dc/d2b/d86/d10d/c11f 0 2024-03-22T02:29:07.486 INFO:tasks.workunit.client.1.smithi122.stdout:0/826: dwrite d11/d16/d37/d84/d87/df6/f108 [0,4194304] 0 2024-03-22T02:29:07.488 INFO:tasks.workunit.client.1.smithi122.stdout:0/827: chown d11/d16/d37/d84/d87/df6/ffd 265634177 1 2024-03-22T02:29:07.488 INFO:tasks.workunit.client.1.smithi122.stdout:5/898: mknod d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/c12f 0 2024-03-22T02:29:07.492 INFO:tasks.workunit.client.1.smithi122.stdout:2/978: rename df/d1f/d33/d6a/f12f to df/d1f/d33/d6a/d7c/d8e/da0/dd4/da2/f139 0 2024-03-22T02:29:07.497 INFO:tasks.workunit.client.1.smithi122.stdout:2/979: stat df/d1f/d3d/d52/ce0 0 2024-03-22T02:29:07.498 INFO:tasks.workunit.client.1.smithi122.stdout:3/952: dread d2/d9/dc/df/d38/fa1 [0,4194304] 0 2024-03-22T02:29:07.498 INFO:tasks.workunit.client.1.smithi122.stdout:1/902: rename dc/d2b/f2d to dc/d2b/d86/ddd/f120 0 2024-03-22T02:29:07.499 INFO:tasks.workunit.client.1.smithi122.stdout:0/828: dwrite d11/d16/d37/f9a [0,4194304] 0 2024-03-22T02:29:07.500 INFO:tasks.workunit.client.1.smithi122.stdout:0/829: chown d11/d16/d44/dfb 8037917 1 2024-03-22T02:29:07.506 INFO:tasks.workunit.client.1.smithi122.stdout:3/953: rename d2/d9/l18 to d2/d9/d7e/dbd/l13f 0 2024-03-22T02:29:07.509 INFO:tasks.workunit.client.1.smithi122.stdout:0/830: creat d11/d16/d37/dda/f11c x:0 0 0 2024-03-22T02:29:07.516 INFO:tasks.workunit.client.1.smithi122.stdout:0/831: dwrite d11/d16/d37/f9a [0,4194304] 0 2024-03-22T02:29:07.517 INFO:tasks.workunit.client.1.smithi122.stdout:5/899: getdents d8/d1e/d22 0 2024-03-22T02:29:07.521 INFO:tasks.workunit.client.1.smithi122.stdout:2/980: fdatasync df/d1f/d11f/f124 0 2024-03-22T02:29:07.521 INFO:tasks.workunit.client.1.smithi122.stdout:5/900: write d8/d1e/d22/d3a/d46/f12e [100092,109904] 0 2024-03-22T02:29:07.523 INFO:tasks.workunit.client.1.smithi122.stdout:1/903: dwrite dc/d4c/f62 [0,4194304] 0 2024-03-22T02:29:07.524 INFO:tasks.workunit.client.1.smithi122.stdout:3/954: rename d2/d9/dc/d57/da4/dc7/c128 to d2/d9/dc/df/d3c/c140 0 2024-03-22T02:29:07.525 INFO:tasks.workunit.client.1.smithi122.stdout:1/904: chown dc/d3b/fb4 79071 1 2024-03-22T02:29:07.529 INFO:tasks.workunit.client.1.smithi122.stdout:1/905: creat dc/d4c/f121 x:0 0 0 2024-03-22T02:29:07.533 INFO:tasks.workunit.client.1.smithi122.stdout:5/901: dwrite d8/d1e/d22/d3a/d46/d4f/d9b/ddf/ff8 [4194304,4194304] 0 2024-03-22T02:29:07.559 INFO:tasks.workunit.client.1.smithi122.stdout:0/832: rename d11/d14/d23/d24/d33/d67/dc2/fe2 to d11/d14/d1d/d27/dc0/dc6/dd0/dea/f11d 0 2024-03-22T02:29:07.563 INFO:tasks.workunit.client.1.smithi122.stdout:5/902: readlink d8/d1e/d22/d3a/d46/d4f/d55/l82 0 2024-03-22T02:29:07.565 INFO:tasks.workunit.client.1.smithi122.stdout:5/903: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/fbf [4957375,70251] 0 2024-03-22T02:29:07.568 INFO:tasks.workunit.client.1.smithi122.stdout:1/906: link dc/fd dc/d3b/d8d/d72/d9c/de8/d116/f122 0 2024-03-22T02:29:07.571 INFO:tasks.workunit.client.1.smithi122.stdout:3/955: dwrite d2/f7 [0,4194304] 0 2024-03-22T02:29:07.573 INFO:tasks.workunit.client.1.smithi122.stdout:5/904: rename d8/d1e/d22/dd7 to d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130 0 2024-03-22T02:29:07.576 INFO:tasks.workunit.client.1.smithi122.stdout:0/833: dwrite d11/d14/d1d/f6a [0,4194304] 0 2024-03-22T02:29:07.579 INFO:tasks.workunit.client.1.smithi122.stdout:0/834: write f10 [900587,36997] 0 2024-03-22T02:29:07.585 INFO:tasks.workunit.client.1.smithi122.stdout:1/907: link dc/d3b/d8d/d72/c79 dc/d2b/d86/d10d/c123 0 2024-03-22T02:29:07.586 INFO:tasks.workunit.client.1.smithi122.stdout:3/956: dwrite d2/d9/dc/df/f113 [0,4194304] 0 2024-03-22T02:29:07.590 INFO:tasks.workunit.client.1.smithi122.stdout:1/908: symlink dc/d3b/d8d/d72/d9c/db0/ddf/l124 0 2024-03-22T02:29:07.592 INFO:tasks.workunit.client.1.smithi122.stdout:0/835: dwrite d11/d16/d44/f90 [0,4194304] 0 2024-03-22T02:29:07.593 INFO:tasks.workunit.client.1.smithi122.stdout:0/836: stat d11/d14/d23/l7d 0 2024-03-22T02:29:07.598 INFO:tasks.workunit.client.1.smithi122.stdout:3/957: creat d2/d9/d7e/dbd/f141 x:0 0 0 2024-03-22T02:29:07.601 INFO:tasks.workunit.client.1.smithi122.stdout:3/958: truncate d2/d9/dc/d17/d28/d4c/da5/db4/f10f 180058 0 2024-03-22T02:29:07.602 INFO:tasks.workunit.client.1.smithi122.stdout:0/837: mknod d11/d14/d1d/d27/d32/c11e 0 2024-03-22T02:29:07.604 INFO:tasks.workunit.client.1.smithi122.stdout:0/838: dread - d11/d14/d23/d24/d33/f119 zero size 2024-03-22T02:29:07.604 INFO:tasks.workunit.client.1.smithi122.stdout:3/959: creat d2/d9/dc/d17/d2b/ddd/f142 x:0 0 0 2024-03-22T02:29:07.605 INFO:tasks.workunit.client.1.smithi122.stdout:5/905: symlink d8/d1e/d22/d3a/d7a/l131 0 2024-03-22T02:29:07.608 INFO:tasks.workunit.client.1.smithi122.stdout:0/839: rmdir d11/d16 39 2024-03-22T02:29:07.608 INFO:tasks.workunit.client.1.smithi122.stdout:0/840: stat d11/d4f/cb7 0 2024-03-22T02:29:07.615 INFO:tasks.workunit.client.1.smithi122.stdout:5/906: dwrite d8/d1e/d22/d3a/fea [0,4194304] 0 2024-03-22T02:29:07.622 INFO:tasks.workunit.client.1.smithi122.stdout:3/960: truncate d2/d9/dc/d57/f93 846436 0 2024-03-22T02:29:07.625 INFO:tasks.workunit.client.1.smithi122.stdout:5/907: symlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/dcc/l132 0 2024-03-22T02:29:07.628 INFO:tasks.workunit.client.1.smithi122.stdout:5/908: fdatasync d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/ffd 0 2024-03-22T02:29:07.628 INFO:tasks.workunit.client.1.smithi122.stdout:3/961: rename d2/d9/dc/df/f48 to d2/d9/dc/d57/f143 0 2024-03-22T02:29:07.630 INFO:tasks.workunit.client.1.smithi122.stdout:5/909: chown d8/d1e/d22/d3a/d46/d4f/d55/db6/de6 0 1 2024-03-22T02:29:07.632 INFO:tasks.workunit.client.1.smithi122.stdout:5/910: chown d8/d1e/l67 0 1 2024-03-22T02:29:07.635 INFO:tasks.workunit.client.1.smithi122.stdout:5/911: creat d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130/f133 x:0 0 0 2024-03-22T02:29:07.637 INFO:tasks.workunit.client.1.smithi122.stdout:5/912: read d8/d3b/fe8 [32391,29822] 0 2024-03-22T02:29:07.638 INFO:tasks.workunit.client.1.smithi122.stdout:1/909: dread dc/d3b/d8d/d72/d9c/db0/d67/dc6/f48 [4194304,4194304] 0 2024-03-22T02:29:07.643 INFO:tasks.workunit.client.1.smithi122.stdout:5/913: rename d8/d1e/d22/da3/fa5 to d8/d1e/d22/d3a/d39/f134 0 2024-03-22T02:29:07.645 INFO:tasks.workunit.client.1.smithi122.stdout:0/841: dread d11/d14/d1d/d27/d32/f6b [0,4194304] 0 2024-03-22T02:29:07.648 INFO:tasks.workunit.client.1.smithi122.stdout:5/914: creat d8/d3b/d11e/f135 x:0 0 0 2024-03-22T02:29:07.649 INFO:tasks.workunit.client.1.smithi122.stdout:5/915: write d8/d3b/d11e/f135 [4235,3968] 0 2024-03-22T02:29:07.652 INFO:tasks.workunit.client.1.smithi122.stdout:1/910: creat dc/d3b/d8d/f125 x:0 0 0 2024-03-22T02:29:07.655 INFO:tasks.workunit.client.1.smithi122.stdout:3/962: dwrite d2/d9/dc/d17/d2b/d6b/fcb [0,4194304] 0 2024-03-22T02:29:07.655 INFO:tasks.workunit.client.1.smithi122.stdout:5/916: readlink d8/d1e/d22/d3a/d46/d4f/d55/db6/l89 0 2024-03-22T02:29:07.657 INFO:tasks.workunit.client.1.smithi122.stdout:1/911: write dc/d3b/f68 [3315518,101364] 0 2024-03-22T02:29:07.661 INFO:tasks.workunit.client.1.smithi122.stdout:1/912: fdatasync dc/d55/f77 0 2024-03-22T02:29:07.664 INFO:tasks.workunit.client.1.smithi122.stdout:3/963: creat d2/d9/dc/d17/d28/d4c/da5/db4/d118/f144 x:0 0 0 2024-03-22T02:29:07.666 INFO:tasks.workunit.client.1.smithi122.stdout:5/917: dwrite d8/d1e/d22/d3a/d46/f4b [0,4194304] 0 2024-03-22T02:29:07.668 INFO:tasks.workunit.client.1.smithi122.stdout:3/964: stat d2/d9/d7e/l8e 0 2024-03-22T02:29:07.677 INFO:tasks.workunit.client.1.smithi122.stdout:3/965: link d2/d9/dc/df/d49/d85/dda/dd5/cd7 d2/d9/dc/d17/d2b/d6b/c145 0 2024-03-22T02:29:07.681 INFO:tasks.workunit.client.1.smithi122.stdout:3/966: creat d2/d9/dc/f146 x:0 0 0 2024-03-22T02:29:07.682 INFO:tasks.workunit.client.1.smithi122.stdout:5/918: dread d8/d74/fc9 [0,4194304] 0 2024-03-22T02:29:07.684 INFO:tasks.workunit.client.1.smithi122.stdout:3/967: creat d2/d9/dc/d17/d2b/f147 x:0 0 0 2024-03-22T02:29:07.685 INFO:tasks.workunit.client.1.smithi122.stdout:5/919: creat d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/f136 x:0 0 0 2024-03-22T02:29:07.690 INFO:tasks.workunit.client.1.smithi122.stdout:5/920: creat d8/d1e/d22/d3a/d46/d4f/d55/f137 x:0 0 0 2024-03-22T02:29:07.695 INFO:tasks.workunit.client.1.smithi122.stdout:5/921: unlink d8/d1e/d22/c4a 0 2024-03-22T02:29:07.708 INFO:tasks.workunit.client.1.smithi122.stdout:3/968: dread d2/d9/dc/d57/d89/fbf [0,4194304] 0 2024-03-22T02:29:07.710 INFO:tasks.workunit.client.1.smithi122.stdout:1/913: dread dc/d2b/d86/fd7 [0,4194304] 0 2024-03-22T02:29:07.713 INFO:tasks.workunit.client.1.smithi122.stdout:1/914: symlink dc/d4c/l126 0 2024-03-22T02:29:07.718 INFO:tasks.workunit.client.1.smithi122.stdout:5/922: dread d8/d1e/d22/d3a/d46/d4f/d55/d6a/f8d [0,4194304] 0 2024-03-22T02:29:07.728 INFO:tasks.workunit.client.1.smithi122.stdout:1/915: write dc/d3b/d8d/d94/f98 [4232717,62160] 0 2024-03-22T02:29:07.729 INFO:tasks.workunit.client.1.smithi122.stdout:5/923: link d8/d1e/d22/d3a/d46/d4f/ca7 d8/d1e/d22/d3a/d46/d4f/d55/d6a/c138 0 2024-03-22T02:29:07.788 INFO:tasks.workunit.client.1.smithi122.stdout:2/981: sync 2024-03-22T02:29:07.804 INFO:tasks.workunit.client.1.smithi122.stdout:2/982: rmdir df/d1f/d33/d4d/d94/dc2/d51/d67/db0 39 2024-03-22T02:29:07.807 INFO:tasks.workunit.client.1.smithi122.stdout:5/924: creat d8/d1e/d22/d3a/d46/d4f/d9b/ddf/f139 x:0 0 0 2024-03-22T02:29:07.808 INFO:tasks.workunit.client.1.smithi122.stdout:1/916: dread dc/d3b/f81 [0,4194304] 0 2024-03-22T02:29:07.812 INFO:tasks.workunit.client.1.smithi122.stdout:2/983: dwrite df/d1f/d3d/d52/f12d [0,4194304] 0 2024-03-22T02:29:07.817 INFO:tasks.workunit.client.1.smithi122.stdout:1/917: dwrite dc/d3b/d46/fe6 [0,4194304] 0 2024-03-22T02:29:07.818 INFO:tasks.workunit.client.1.smithi122.stdout:5/925: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/feb [4293458,9534] 0 2024-03-22T02:29:07.823 INFO:tasks.workunit.client.1.smithi122.stdout:2/984: dwrite df/d1f/d33/d4d/f10e [0,4194304] 0 2024-03-22T02:29:07.828 INFO:tasks.workunit.client.1.smithi122.stdout:5/926: creat d8/d1e/f13a x:0 0 0 2024-03-22T02:29:07.829 INFO:tasks.workunit.client.1.smithi122.stdout:1/918: link dc/d2b/d86/dbb/fdc dc/d3b/d46/db5/dba/f127 0 2024-03-22T02:29:07.835 INFO:tasks.workunit.client.1.smithi122.stdout:1/919: symlink dc/d3b/d8d/d72/d9c/db0/dab/l128 0 2024-03-22T02:29:07.839 INFO:tasks.workunit.client.1.smithi122.stdout:2/985: dwrite df/d1f/d33/d4d/d94/dc2/d51/d67/db0/f12e [0,4194304] 0 2024-03-22T02:29:07.840 INFO:tasks.workunit.client.1.smithi122.stdout:5/927: dwrite d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/fb2 [0,4194304] 0 2024-03-22T02:29:07.841 INFO:tasks.workunit.client.1.smithi122.stdout:1/920: unlink dc/d3b/d8d/d72/d9c/db0/f7d 0 2024-03-22T02:29:07.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:07 smithi122 ceph-mon[36403]: Reconfiguring grafana.smithi103 (dependencies changed)... 2024-03-22T02:29:07.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:07 smithi122 ceph-mon[36403]: Reconfiguring daemon grafana.smithi103 on smithi103 2024-03-22T02:29:07.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:07 smithi122 ceph-mon[36403]: pgmap v27: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 115 MiB/s rd, 276 MiB/s wr, 645 op/s 2024-03-22T02:29:07.850 INFO:tasks.workunit.client.1.smithi122.stdout:2/986: rename df/d1f/d33/d4d/d94/dc2/d114 to df/d1f/d33/d4d/d94/dc2/d114/d115/dd2/d13a 22 2024-03-22T02:29:07.853 INFO:tasks.workunit.client.1.smithi122.stdout:5/928: link d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/dcc/l11a d8/d1a/d83/dcb/l13b 0 2024-03-22T02:29:07.860 INFO:tasks.workunit.client.1.smithi122.stdout:1/921: rmdir dc/d3b/d8d 39 2024-03-22T02:29:07.863 INFO:tasks.workunit.client.1.smithi122.stdout:2/987: truncate df/d1f/d33/d4d/d94/dc2/d51/d67/db0/f12e 1960701 0 2024-03-22T02:29:07.864 INFO:tasks.workunit.client.1.smithi122.stdout:1/922: write dc/d3b/d46/fa7 [3038057,17575] 0 2024-03-22T02:29:07.865 INFO:tasks.workunit.client.1.smithi122.stdout:2/988: chown df/d1f/d33/d4d/lac 115467 1 2024-03-22T02:29:07.868 INFO:tasks.workunit.client.1.smithi122.stdout:2/989: rmdir df/d1f/d33/d6a/d7c 39 2024-03-22T02:29:07.868 INFO:tasks.workunit.client.1.smithi122.stdout:5/929: dwrite d8/d1e/d22/d3a/d39/fcf [0,4194304] 0 2024-03-22T02:29:07.873 INFO:tasks.workunit.client.1.smithi122.stdout:2/990: symlink df/d1f/d3d/l13b 0 2024-03-22T02:29:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:07 smithi103 ceph-mon[29258]: Reconfiguring grafana.smithi103 (dependencies changed)... 2024-03-22T02:29:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:07 smithi103 ceph-mon[29258]: Reconfiguring daemon grafana.smithi103 on smithi103 2024-03-22T02:29:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:07 smithi103 ceph-mon[29258]: pgmap v27: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 115 MiB/s rd, 276 MiB/s wr, 645 op/s 2024-03-22T02:29:07.875 INFO:tasks.workunit.client.1.smithi122.stdout:1/923: dwrite dc/d4c/f121 [0,4194304] 0 2024-03-22T02:29:07.879 INFO:tasks.workunit.client.1.smithi122.stdout:5/930: link d8/d3b/f69 d8/d1e/d22/da3/ddc/f13c 0 2024-03-22T02:29:07.880 INFO:tasks.workunit.client.1.smithi122.stdout:0/842: sync 2024-03-22T02:29:07.881 INFO:tasks.workunit.client.1.smithi122.stdout:2/991: unlink df/d1f/d33/d6a/d7c/d8e/da0/dd4/d4f/f5e 0 2024-03-22T02:29:07.882 INFO:tasks.workunit.client.1.smithi122.stdout:1/924: dwrite dc/ff [0,4194304] 0 2024-03-22T02:29:07.883 INFO:tasks.workunit.client.1.smithi122.stdout:0/843: chown d11/d14/d23/d24/f25 635707342 1 2024-03-22T02:29:07.883 INFO:tasks.workunit.client.1.smithi122.stdout:5/931: chown d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130/c10f 80 1 2024-03-22T02:29:07.888 INFO:tasks.workunit.client.1.smithi122.stdout:5/932: readlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/l129 0 2024-03-22T02:29:07.889 INFO:tasks.workunit.client.1.smithi122.stdout:5/933: chown d8/d1e/d22/d3a/d46/d4f/d9b/ddf/de7 205 1 2024-03-22T02:29:07.889 INFO:tasks.workunit.client.1.smithi122.stdout:2/992: stat df/d1f/d33/d4d/d94/dc2/d114/d115/dd2/l5b 0 2024-03-22T02:29:07.896 INFO:tasks.workunit.client.1.smithi122.stdout:0/844: dwrite d11/d14/d1d/d27/dc0/dc6/dd0/dea/f11d [0,4194304] 0 2024-03-22T02:29:07.900 INFO:tasks.workunit.client.1.smithi122.stdout:0/845: dread - d11/d14/d1d/d27/d9d/fcf zero size 2024-03-22T02:29:07.901 INFO:tasks.workunit.client.1.smithi122.stdout:5/934: getdents d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3 0 2024-03-22T02:29:07.902 INFO:tasks.workunit.client.1.smithi122.stdout:2/993: write df/d1f/d33/d6a/f90 [284617,76821] 0 2024-03-22T02:29:07.902 INFO:tasks.workunit.client.1.smithi122.stdout:0/846: read d11/d14/d1d/d27/d9d/fcc [3553842,35865] 0 2024-03-22T02:29:07.904 INFO:tasks.workunit.client.1.smithi122.stdout:5/935: readlink d8/d1e/d22/l66 0 2024-03-22T02:29:07.907 INFO:tasks.workunit.client.1.smithi122.stdout:1/925: dwrite dc/f39 [0,4194304] 0 2024-03-22T02:29:07.908 INFO:tasks.workunit.client.1.smithi122.stdout:0/847: mkdir d11/d14/d1d/d27/d9d/daa/d11f 0 2024-03-22T02:29:07.912 INFO:tasks.workunit.client.1.smithi122.stdout:5/936: mknod d8/d1e/d22/d3a/d46/d4f/d55/db6/d4e/d128/c13d 0 2024-03-22T02:29:07.916 INFO:tasks.workunit.client.1.smithi122.stdout:2/994: dwrite df/d1f/d33/d4d/d94/dc2/d51/d67/fbf [0,4194304] 0 2024-03-22T02:29:07.918 INFO:tasks.workunit.client.1.smithi122.stdout:5/937: stat d8/d1e/d22/d78/d12d 0 2024-03-22T02:29:07.918 INFO:tasks.workunit.client.1.smithi122.stdout:2/995: dread - df/d1f/d33/d4d/d94/df0/f105 zero size 2024-03-22T02:29:07.921 INFO:tasks.workunit.client.1.smithi122.stdout:5/938: creat d8/d1e/d51/f13e x:0 0 0 2024-03-22T02:29:07.926 INFO:tasks.workunit.client.1.smithi122.stdout:5/939: mknod d8/d1e/c13f 0 2024-03-22T02:29:07.928 INFO:tasks.workunit.client.1.smithi122.stdout:5/940: creat d8/d1a/d104/f140 x:0 0 0 2024-03-22T02:29:07.930 INFO:tasks.workunit.client.1.smithi122.stdout:5/941: symlink d8/d1e/d22/d3a/d46/d4f/d9b/l141 0 2024-03-22T02:29:07.933 INFO:tasks.workunit.client.1.smithi122.stdout:5/942: creat d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130/d108/f142 x:0 0 0 2024-03-22T02:29:07.935 INFO:tasks.workunit.client.1.smithi122.stdout:5/943: symlink d8/d1e/d22/d3a/d46/d4f/l143 0 2024-03-22T02:29:07.946 INFO:tasks.workunit.client.1.smithi122.stdout:1/926: dwrite dc/d3b/d8d/f8f [0,4194304] 0 2024-03-22T02:29:07.947 INFO:tasks.workunit.client.1.smithi122.stdout:2/996: dwrite df/d1f/d3d/d52/ddc/f133 [0,4194304] 0 2024-03-22T02:29:07.947 INFO:tasks.workunit.client.1.smithi122.stdout:5/944: dwrite d8/d3b/faf [4194304,4194304] 0 2024-03-22T02:29:07.948 INFO:tasks.workunit.client.1.smithi122.stdout:5/945: fdatasync d8/fb0 0 2024-03-22T02:29:07.955 INFO:tasks.workunit.client.1.smithi122.stdout:2/997: chown df/d1f/d33/d4d/d94/fc1 58 1 2024-03-22T02:29:07.958 INFO:tasks.workunit.client.1.smithi122.stdout:1/927: symlink dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/d107/l129 0 2024-03-22T02:29:07.960 INFO:tasks.workunit.client.1.smithi122.stdout:1/928: fsync dc/d55/f75 0 2024-03-22T02:29:07.963 INFO:tasks.workunit.client.1.smithi122.stdout:1/929: unlink dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/d107/ce1 0 2024-03-22T02:29:07.963 INFO:tasks.workunit.client.1.smithi122.stdout:1/930: stat dc/d4c/f92 0 2024-03-22T02:29:07.973 INFO:tasks.workunit.client.1.smithi122.stdout:5/946: dwrite d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/fee [0,4194304] 0 2024-03-22T02:29:07.975 INFO:tasks.workunit.client.1.smithi122.stdout:5/947: chown d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130/d108/f114 155397610 1 2024-03-22T02:29:07.999 INFO:tasks.workunit.client.1.smithi122.stdout:5/948: dread d8/d1e/d22/f107 [0,4194304] 0 2024-03-22T02:29:08.003 INFO:tasks.workunit.client.1.smithi122.stdout:5/949: dwrite d8/d1e/d22/d3a/d46/f6f [0,4194304] 0 2024-03-22T02:29:08.006 INFO:tasks.workunit.client.1.smithi122.stdout:5/950: mkdir d8/d1e/d22/d3a/d144 0 2024-03-22T02:29:08.057 INFO:tasks.workunit.client.1.smithi122.stdout:3/969: sync 2024-03-22T02:29:08.064 INFO:tasks.workunit.client.1.smithi122.stdout:5/951: getdents d8/d1e/d22/d3a/d46 0 2024-03-22T02:29:08.067 INFO:tasks.workunit.client.1.smithi122.stdout:3/970: read d2/d9/dc/d17/d28/d4c/da5/db4/d79/fb9 [2371671,50967] 0 2024-03-22T02:29:08.069 INFO:tasks.workunit.client.1.smithi122.stdout:3/971: dread - d2/d9/dc/d17/f12a zero size 2024-03-22T02:29:08.070 INFO:tasks.workunit.client.1.smithi122.stdout:5/952: dwrite d8/d1e/d22/da3/f120 [0,4194304] 0 2024-03-22T02:29:08.071 INFO:tasks.workunit.client.1.smithi122.stdout:3/972: chown d2/d9/dc/d17/d28/f2d 160498 1 2024-03-22T02:29:08.073 INFO:tasks.workunit.client.1.smithi122.stdout:5/953: creat d8/f145 x:0 0 0 2024-03-22T02:29:08.074 INFO:tasks.workunit.client.1.smithi122.stdout:3/973: symlink d2/d9/dc/d17/d28/d4c/da5/db4/d79/l148 0 2024-03-22T02:29:08.077 INFO:tasks.workunit.client.1.smithi122.stdout:3/974: mknod d2/d9/dc/d57/dab/deb/c149 0 2024-03-22T02:29:08.078 INFO:tasks.workunit.client.1.smithi122.stdout:3/975: write d2/d9/dc/df/d49/d85/dda/f12f [241978,24588] 0 2024-03-22T02:29:08.082 INFO:tasks.workunit.client.1.smithi122.stdout:5/954: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/f110 [0,4194304] 0 2024-03-22T02:29:08.082 INFO:tasks.workunit.client.1.smithi122.stdout:5/955: readlink d8/lb3 0 2024-03-22T02:29:08.084 INFO:tasks.workunit.client.1.smithi122.stdout:5/956: truncate d8/d1e/f13a 977257 0 2024-03-22T02:29:08.088 INFO:tasks.workunit.client.1.smithi122.stdout:3/976: rmdir d2/d9/dc/df/d49/d85/dda/dd5/d13b 0 2024-03-22T02:29:08.088 INFO:tasks.workunit.client.1.smithi122.stdout:5/957: stat d8/d1e/d22/d3a/d46/d4f/d55/db6/ldb 0 2024-03-22T02:29:08.092 INFO:tasks.workunit.client.1.smithi122.stdout:3/977: rename d2/d9/dc/d17/d2b/ddd/f100 to d2/d9/dc/df/d38/f14a 0 2024-03-22T02:29:08.093 INFO:tasks.workunit.client.1.smithi122.stdout:3/978: symlink d2/d8a/df6/l14b 0 2024-03-22T02:29:08.094 INFO:tasks.workunit.client.1.smithi122.stdout:3/979: dread - d2/d9/dc/df/d49/d85/dda/dd5/fcd zero size 2024-03-22T02:29:08.096 INFO:tasks.workunit.client.1.smithi122.stdout:3/980: mkdir d2/d9/dc/df/d49/d85/dda/dd5/d14c 0 2024-03-22T02:29:08.097 INFO:tasks.workunit.client.1.smithi122.stdout:3/981: dread - d2/d9/dc/d17/f116 zero size 2024-03-22T02:29:08.100 INFO:tasks.workunit.client.1.smithi122.stdout:3/982: rename d2/d9/dc/d17/d45/l58 to d2/d9/d123/l14d 0 2024-03-22T02:29:08.101 INFO:tasks.workunit.client.1.smithi122.stdout:3/983: fsync d2/d9/dc/d17/d28/f6c 0 2024-03-22T02:29:08.103 INFO:tasks.workunit.client.1.smithi122.stdout:3/984: chown d2/d9/dc/d17/d28/d4c/da5/db4/d79/fbc 4077115 1 2024-03-22T02:29:08.104 INFO:tasks.workunit.client.1.smithi122.stdout:3/985: chown d2/d9/dc/df/d49/d85/dda/dd5/df9/d11d 0 1 2024-03-22T02:29:08.105 INFO:tasks.workunit.client.1.smithi122.stdout:3/986: dread - d2/d9/d7e/dbd/f141 zero size 2024-03-22T02:29:08.108 INFO:tasks.workunit.client.1.smithi122.stdout:3/987: mknod d2/d9/dc/d17/d2b/ddd/c14e 0 2024-03-22T02:29:08.110 INFO:tasks.workunit.client.1.smithi122.stdout:3/988: readlink d2/d9/dc/d17/d28/d4c/da5/db4/le3 0 2024-03-22T02:29:08.112 INFO:tasks.workunit.client.1.smithi122.stdout:5/958: dread d8/d1a/f34 [0,4194304] 0 2024-03-22T02:29:08.115 INFO:tasks.workunit.client.1.smithi122.stdout:3/989: link d2/d9/dc/d57/d89/l13c d2/d9/dc/df/d49/d85/l14f 0 2024-03-22T02:29:08.116 INFO:tasks.workunit.client.1.smithi122.stdout:5/959: write d8/d1e/d22/da3/ddc/f16 [1970172,103050] 0 2024-03-22T02:29:08.119 INFO:tasks.workunit.client.1.smithi122.stdout:5/960: rmdir d8/d1e/d22/d78 39 2024-03-22T02:29:08.123 INFO:tasks.workunit.client.1.smithi122.stdout:5/961: write d8/d1a/d83/dcb/ffc [1363254,54864] 0 2024-03-22T02:29:08.126 INFO:tasks.workunit.client.1.smithi122.stdout:5/962: truncate d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f37 260465 0 2024-03-22T02:29:08.131 INFO:tasks.workunit.client.1.smithi122.stdout:5/963: write d8/d1e/d22/d3a/f7d [806900,113756] 0 2024-03-22T02:29:08.133 INFO:tasks.workunit.client.1.smithi122.stdout:3/990: dread d2/d9/dc/df/d3c/fe2 [0,4194304] 0 2024-03-22T02:29:08.133 INFO:tasks.workunit.client.1.smithi122.stdout:5/964: symlink d8/d3b/l146 0 2024-03-22T02:29:08.134 INFO:tasks.workunit.client.1.smithi122.stdout:5/965: dread - d8/d1a/d104/f118 zero size 2024-03-22T02:29:08.136 INFO:tasks.workunit.client.1.smithi122.stdout:5/966: write d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d71/f110 [85026,106357] 0 2024-03-22T02:29:08.141 INFO:tasks.workunit.client.1.smithi122.stdout:3/991: dwrite d2/d9/dc/d57/d89/fbf [0,4194304] 0 2024-03-22T02:29:08.143 INFO:tasks.workunit.client.1.smithi122.stdout:3/992: symlink d2/l150 0 2024-03-22T02:29:08.144 INFO:tasks.workunit.client.1.smithi122.stdout:3/993: write d2/d9/d7e/dbd/f141 [988391,66610] 0 2024-03-22T02:29:08.145 INFO:tasks.workunit.client.1.smithi122.stdout:5/967: dwrite d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/f94 [4194304,4194304] 0 2024-03-22T02:29:08.146 INFO:tasks.workunit.client.1.smithi122.stdout:5/968: readlink d8/d1e/d22/d3a/d46/d4f/l143 0 2024-03-22T02:29:08.147 INFO:tasks.workunit.client.1.smithi122.stdout:3/994: mknod d2/d9/dc/d17/d28/d4c/da5/c151 0 2024-03-22T02:29:08.152 INFO:tasks.workunit.client.1.smithi122.stdout:5/969: mkdir d8/d1e/d22/d78/d12d/d147 0 2024-03-22T02:29:08.153 INFO:tasks.workunit.client.1.smithi122.stdout:3/995: link d2/d9/dc/f146 d2/d9/dc/df/d49/f152 0 2024-03-22T02:29:08.154 INFO:tasks.workunit.client.1.smithi122.stdout:5/970: creat d8/d1e/d22/d3a/d46/d4f/f148 x:0 0 0 2024-03-22T02:29:08.155 INFO:tasks.workunit.client.1.smithi122.stdout:3/996: fdatasync d2/d9/dc/df/d49/d85/dda/f12f 0 2024-03-22T02:29:08.156 INFO:tasks.workunit.client.1.smithi122.stdout:5/971: symlink d8/d1e/d51/l149 0 2024-03-22T02:29:08.158 INFO:tasks.workunit.client.1.smithi122.stdout:3/997: symlink d2/d9/dc/l153 0 2024-03-22T02:29:08.159 INFO:tasks.workunit.client.1.smithi122.stdout:5/972: unlink d8/d1e/d22/d3a/d7a/l131 0 2024-03-22T02:29:08.160 INFO:tasks.workunit.client.1.smithi122.stdout:3/998: write d2/d9/dc/d17/d28/d4c/da5/db4/fd3 [1001073,80659] 0 2024-03-22T02:29:08.161 INFO:tasks.workunit.client.1.smithi122.stdout:3/999: write d2/d9/dc/d57/da4/fe4 [419337,75338] 0 2024-03-22T02:29:08.162 INFO:tasks.workunit.client.1.smithi122.stdout:5/973: read d8/d1e/d22/d3a/d46/d4f/f63 [2200911,34198] 0 2024-03-22T02:29:08.165 INFO:tasks.workunit.client.1.smithi122.stdout:5/974: rename d8/d3b/l80 to d8/d1e/d22/d3a/d46/d4f/d55/db6/l14a 0 2024-03-22T02:29:08.167 INFO:tasks.workunit.client.1.smithi122.stdout:5/975: write d8/d1e/d22/d3a/d46/d4f/d55/db6/de6/f100 [435735,12261] 0 2024-03-22T02:29:08.169 INFO:tasks.workunit.client.1.smithi122.stdout:5/976: unlink d8/c10d 0 2024-03-22T02:29:08.170 INFO:tasks.workunit.client.1.smithi122.stdout:5/977: chown d8/d1e/d22/d3a/fea 6 1 2024-03-22T02:29:08.181 INFO:tasks.workunit.client.1.smithi122.stdout:5/978: rmdir d8 39 2024-03-22T02:29:08.181 INFO:tasks.workunit.client.1.smithi122.stdout:0/848: sync 2024-03-22T02:29:08.182 INFO:tasks.workunit.client.1.smithi122.stdout:5/979: dread - d8/f145 zero size 2024-03-22T02:29:08.186 INFO:tasks.workunit.client.1.smithi122.stdout:5/980: write d8/d1e/d22/d3a/d46/faa [81882,46256] 0 2024-03-22T02:29:08.190 INFO:tasks.workunit.client.1.smithi122.stdout:0/849: dwrite fb [0,4194304] 0 2024-03-22T02:29:08.192 INFO:tasks.workunit.client.1.smithi122.stdout:5/981: dread d8/d1e/d22/d3a/fea [0,4194304] 0 2024-03-22T02:29:08.192 INFO:tasks.workunit.client.1.smithi122.stdout:5/982: dread - d8/d74/fe1 zero size 2024-03-22T02:29:08.194 INFO:tasks.workunit.client.1.smithi122.stdout:0/850: truncate d11/d14/d1d/d27/d2f/f53 654413 0 2024-03-22T02:29:08.195 INFO:tasks.workunit.client.1.smithi122.stdout:5/983: chown d8/d1e/d22/d3a/d46/d4f/d55/d6a/d8c/cf0 14 1 2024-03-22T02:29:08.195 INFO:tasks.workunit.client.1.smithi122.stdout:0/851: stat d11/d14/d23/d24/d33/d67/dc2/le5 0 2024-03-22T02:29:08.197 INFO:tasks.workunit.client.1.smithi122.stdout:5/984: mkdir d8/d1e/d22/d3a/d46/d4f/d9b/ddf/d14b 0 2024-03-22T02:29:08.198 INFO:tasks.workunit.client.1.smithi122.stdout:5/985: readlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/l129 0 2024-03-22T02:29:08.199 INFO:tasks.workunit.client.1.smithi122.stdout:5/986: fsync d8/d3b/f62 0 2024-03-22T02:29:08.201 INFO:tasks.workunit.client.1.smithi122.stdout:5/987: truncate d8/d1e/f23 2401848 0 2024-03-22T02:29:08.202 INFO:tasks.workunit.client.1.smithi122.stdout:0/852: write d11/d14/d1d/d27/d2f/d4e/df0/fbd [3523270,129409] 0 2024-03-22T02:29:08.205 INFO:tasks.workunit.client.1.smithi122.stdout:5/988: getdents d8/d1e/d22/d3a/d46/d4f/d55/db6/dc3 0 2024-03-22T02:29:08.207 INFO:tasks.workunit.client.1.smithi122.stdout:5/989: chown d8/d1e/d22/fde 0 1 2024-03-22T02:29:08.208 INFO:tasks.workunit.client.1.smithi122.stdout:0/853: dwrite d11/d14/d23/d24/d33/f8e [0,4194304] 0 2024-03-22T02:29:08.209 INFO:tasks.workunit.client.1.smithi122.stdout:5/990: truncate d8/d1e/d22/d3a/d7a/f12c 847548 0 2024-03-22T02:29:08.210 INFO:tasks.workunit.client.1.smithi122.stdout:0/854: write d11/d14/d23/d24/d33/f8e [826045,55378] 0 2024-03-22T02:29:08.212 INFO:tasks.workunit.client.1.smithi122.stdout:0/855: write d11/d16/d113/ffe [439912,47480] 0 2024-03-22T02:29:08.213 INFO:tasks.workunit.client.1.smithi122.stdout:5/991: unlink d8/d1e/d22/d3a/d46/d4f/d55/dc7/d130/lfe 0 2024-03-22T02:29:08.213 INFO:tasks.workunit.client.1.smithi122.stdout:0/856: fsync d11/d14/d1d/d27/d9d/daa/fdc 0 2024-03-22T02:29:08.215 INFO:tasks.workunit.client.1.smithi122.stdout:1/931: sync 2024-03-22T02:29:08.215 INFO:tasks.workunit.client.1.smithi122.stdout:2/998: sync 2024-03-22T02:29:08.219 INFO:tasks.workunit.client.1.smithi122.stdout:2/999: truncate df/d1f/d33/dfd/f10f 5025621 0 2024-03-22T02:29:08.221 INFO:tasks.workunit.client.1.smithi122.stdout:5/992: dwrite d8/d74/f86 [0,4194304] 0 2024-03-22T02:29:08.222 INFO:tasks.workunit.client.1.smithi122.stdout:0/857: rmdir d11/d14 39 2024-03-22T02:29:08.225 INFO:tasks.workunit.client.1.smithi122.stdout:1/932: creat dc/d3b/d46/de3/f12a x:0 0 0 2024-03-22T02:29:08.227 INFO:tasks.workunit.client.1.smithi122.stdout:5/993: mkdir d8/d3b/d11e/d14c 0 2024-03-22T02:29:08.232 INFO:tasks.workunit.client.1.smithi122.stdout:5/994: rename d8/d1e/d51/l149 to d8/d1e/d22/d78/d11d/l14d 0 2024-03-22T02:29:08.234 INFO:tasks.workunit.client.1.smithi122.stdout:1/933: link dc/f25 dc/d2b/d86/ddd/f12b 0 2024-03-22T02:29:08.235 INFO:tasks.workunit.client.1.smithi122.stdout:0/858: dwrite d11/d14/d23/f2d [0,4194304] 0 2024-03-22T02:29:08.236 INFO:tasks.workunit.client.1.smithi122.stdout:5/995: creat d8/d1e/d22/d3a/d39/f14e x:0 0 0 2024-03-22T02:29:08.237 INFO:tasks.workunit.client.1.smithi122.stdout:1/934: truncate dc/d3b/d8d/fee 24880 0 2024-03-22T02:29:08.237 INFO:tasks.workunit.client.1.smithi122.stdout:5/996: read d8/d74/fc9 [616553,97265] 0 2024-03-22T02:29:08.239 INFO:tasks.workunit.client.1.smithi122.stdout:1/935: chown dc/c3c 12830630 1 2024-03-22T02:29:08.246 INFO:tasks.workunit.client.1.smithi122.stdout:0/859: dwrite d11/d14/ff7 [0,4194304] 0 2024-03-22T02:29:08.248 INFO:tasks.workunit.client.1.smithi122.stdout:5/997: dread d8/d1e/d22/d3a/d46/d4f/d55/dc7/feb [0,4194304] 0 2024-03-22T02:29:08.254 INFO:tasks.workunit.client.1.smithi122.stdout:5/998: mknod d8/d1e/d22/d3a/d46/d4f/d55/dc7/d35/d106/c14f 0 2024-03-22T02:29:08.257 INFO:tasks.workunit.client.1.smithi122.stdout:1/936: dwrite dc/d2b/d86/ddd/f12b [0,4194304] 0 2024-03-22T02:29:08.258 INFO:tasks.workunit.client.1.smithi122.stdout:0/860: getdents d11/d14/d1d/d27/d9d/daa 0 2024-03-22T02:29:08.260 INFO:tasks.workunit.client.1.smithi122.stdout:1/937: dread - dc/d3b/d8d/d72/d9c/de8/ff8 zero size 2024-03-22T02:29:08.260 INFO:tasks.workunit.client.1.smithi122.stdout:0/861: mknod d11/d47/c120 0 2024-03-22T02:29:08.261 INFO:tasks.workunit.client.1.smithi122.stdout:1/938: fdatasync dc/d3b/d8d/d72/d9c/dd3/f102 0 2024-03-22T02:29:08.263 INFO:tasks.workunit.client.1.smithi122.stdout:0/862: stat d11/d14/d23/d24/d33/d67/dc2/fc9 0 2024-03-22T02:29:08.268 INFO:tasks.workunit.client.1.smithi122.stdout:0/863: creat d11/f121 x:0 0 0 2024-03-22T02:29:08.269 INFO:tasks.workunit.client.1.smithi122.stdout:1/939: link dc/d3b/d8d/d72/d9c/db0/le2 dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/l12c 0 2024-03-22T02:29:08.270 INFO:tasks.workunit.client.1.smithi122.stdout:0/864: dread - d11/d16/d37/d84/da3/fdd zero size 2024-03-22T02:29:08.270 INFO:tasks.workunit.client.1.smithi122.stdout:1/940: chown dc/d2b/f57 681850722 1 2024-03-22T02:29:08.271 INFO:tasks.workunit.client.1.smithi122.stdout:0/865: readlink d11/d14/d23/d24/d33/d67/l11b 0 2024-03-22T02:29:08.272 INFO:tasks.workunit.client.1.smithi122.stdout:1/941: fdatasync dc/d3b/d8d/d72/d9c/db0/dab/fde 0 2024-03-22T02:29:08.274 INFO:tasks.workunit.client.1.smithi122.stdout:0/866: creat d11/d16/d37/d84/f122 x:0 0 0 2024-03-22T02:29:08.275 INFO:tasks.workunit.client.1.smithi122.stdout:1/942: rmdir dc/d3b/d8d/d72/d9c/db0/d90 39 2024-03-22T02:29:08.276 INFO:tasks.workunit.client.1.smithi122.stdout:1/943: write dc/f22 [1709370,107931] 0 2024-03-22T02:29:08.279 INFO:tasks.workunit.client.1.smithi122.stdout:1/944: creat dc/d3b/f12d x:0 0 0 2024-03-22T02:29:08.290 INFO:tasks.workunit.client.1.smithi122.stdout:5/999: dread d8/f33 [0,4194304] 0 2024-03-22T02:29:08.292 INFO:tasks.workunit.client.1.smithi122.stdout:1/945: dwrite dc/d3b/fb4 [0,4194304] 0 2024-03-22T02:29:08.293 INFO:tasks.workunit.client.1.smithi122.stdout:1/946: truncate dc/d4c/f119 62376 0 2024-03-22T02:29:08.295 INFO:tasks.workunit.client.1.smithi122.stdout:1/947: readlink dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/l12c 0 2024-03-22T02:29:08.298 INFO:tasks.workunit.client.1.smithi122.stdout:1/948: creat dc/d3b/d8d/d72/f12e x:0 0 0 2024-03-22T02:29:08.301 INFO:tasks.workunit.client.1.smithi122.stdout:1/949: rmdir dc/d3b/d8d/d72/d9c/db0/d71/d106 0 2024-03-22T02:29:08.303 INFO:tasks.workunit.client.1.smithi122.stdout:1/950: mknod dc/d3b/d8d/d72/d9c/db0/d90/c12f 0 2024-03-22T02:29:08.304 INFO:tasks.workunit.client.1.smithi122.stdout:1/951: write dc/d3b/d46/f9e [337536,105730] 0 2024-03-22T02:29:08.305 INFO:tasks.workunit.client.1.smithi122.stdout:1/952: chown dc/d2b/d86/ddd/f12b 921437 1 2024-03-22T02:29:08.306 INFO:tasks.workunit.client.1.smithi122.stdout:1/953: dread - dc/d3b/d8d/d72/d9c/db0/d67/f11c zero size 2024-03-22T02:29:08.473 INFO:tasks.workunit.client.1.smithi122.stdout:1/954: sync 2024-03-22T02:29:08.474 INFO:tasks.workunit.client.1.smithi122.stdout:0/867: sync 2024-03-22T02:29:08.475 INFO:tasks.workunit.client.1.smithi122.stdout:1/955: rmdir dc/d3b/d8d/d72/d9c/dd3 39 2024-03-22T02:29:08.476 INFO:tasks.workunit.client.1.smithi122.stdout:0/868: fsync d11/d14/d1d/d27/d9d/fcc 0 2024-03-22T02:29:08.479 INFO:tasks.workunit.client.1.smithi122.stdout:1/956: rename dc/d3b/lcd to dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/l130 0 2024-03-22T02:29:08.479 INFO:tasks.workunit.client.1.smithi122.stdout:0/869: truncate d11/d14/d1d/d27/d9d/daa/dcd/fe0 3128993 0 2024-03-22T02:29:08.481 INFO:tasks.workunit.client.1.smithi122.stdout:0/870: creat d11/d16/d44/f123 x:0 0 0 2024-03-22T02:29:08.486 INFO:tasks.workunit.client.1.smithi122.stdout:1/957: dwrite dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5/fb7 [0,4194304] 0 2024-03-22T02:29:08.487 INFO:tasks.workunit.client.1.smithi122.stdout:0/871: link d11/d14/d1d/d27/c62 d11/d14/d1d/c124 0 2024-03-22T02:29:08.494 INFO:tasks.workunit.client.1.smithi122.stdout:0/872: creat d11/d14/d1d/d27/dc0/f125 x:0 0 0 2024-03-22T02:29:08.495 INFO:tasks.workunit.client.1.smithi122.stdout:1/958: getdents dc/d3b/d8d/d72/d9c 0 2024-03-22T02:29:08.495 INFO:tasks.workunit.client.1.smithi122.stdout:1/959: dread - dc/d55/faa zero size 2024-03-22T02:29:08.500 INFO:tasks.workunit.client.1.smithi122.stdout:1/960: creat dc/d2b/d86/f131 x:0 0 0 2024-03-22T02:29:08.505 INFO:tasks.workunit.client.1.smithi122.stdout:1/961: getdents dc/d3b/d8d/d72 0 2024-03-22T02:29:08.515 INFO:tasks.workunit.client.1.smithi122.stdout:0/873: write d11/d14/d1d/d27/f5e [136515,61907] 0 2024-03-22T02:29:08.517 INFO:tasks.workunit.client.1.smithi122.stdout:0/874: truncate d11/d14/d1d/d27/d9d/daa/f112 140368 0 2024-03-22T02:29:08.518 INFO:tasks.workunit.client.1.smithi122.stdout:0/875: mkdir d11/d14/d1b/d126 0 2024-03-22T02:29:08.519 INFO:tasks.workunit.client.1.smithi122.stdout:0/876: write d11/d14/d23/d24/d33/f119 [884165,112135] 0 2024-03-22T02:29:08.522 INFO:tasks.workunit.client.1.smithi122.stdout:0/877: getdents d11/d14/d1d/d27/d2f/d4e/df0 0 2024-03-22T02:29:08.523 INFO:tasks.workunit.client.1.smithi122.stdout:0/878: chown d11/d14/d1d/d27/d9d/lb4 897 1 2024-03-22T02:29:08.524 INFO:tasks.workunit.client.1.smithi122.stdout:0/879: write d11/d14/d1d/d27/d9d/fcc [3705689,89225] 0 2024-03-22T02:29:08.525 INFO:tasks.workunit.client.1.smithi122.stdout:0/880: write d11/d16/d37/d84/da3/fb9 [429757,71994] 0 2024-03-22T02:29:08.528 INFO:tasks.workunit.client.1.smithi122.stdout:0/881: creat d11/d14/d1d/d27/d59/f127 x:0 0 0 2024-03-22T02:29:08.530 INFO:tasks.workunit.client.1.smithi122.stdout:0/882: mkdir d11/d16/d37/dda/d128 0 2024-03-22T02:29:08.531 INFO:tasks.workunit.client.1.smithi122.stdout:0/883: chown d11/d14/d1d/d27/d2f/ff2 2333369 1 2024-03-22T02:29:08.532 INFO:tasks.workunit.client.1.smithi122.stdout:0/884: symlink d11/d16/d37/dda/l129 0 2024-03-22T02:29:08.538 INFO:tasks.workunit.client.1.smithi122.stdout:0/885: dwrite d11/d14/d1d/d27/d9d/daa/fdc [4194304,4194304] 0 2024-03-22T02:29:08.539 INFO:tasks.workunit.client.1.smithi122.stdout:0/886: dread - d11/d14/d23/d24/d33/d67/dc2/f10e zero size 2024-03-22T02:29:08.541 INFO:tasks.workunit.client.1.smithi122.stdout:0/887: mkdir d11/d14/d12a 0 2024-03-22T02:29:08.544 INFO:tasks.workunit.client.1.smithi122.stdout:0/888: mknod d11/d14/d1d/d27/dc0/dc6/c12b 0 2024-03-22T02:29:08.639 INFO:tasks.workunit.client.1.smithi122.stdout:1/962: sync 2024-03-22T02:29:08.647 INFO:tasks.workunit.client.1.smithi122.stdout:1/963: write dc/fea [1878901,31559] 0 2024-03-22T02:29:08.650 INFO:tasks.workunit.client.1.smithi122.stdout:1/964: truncate dc/d4c/f52 1319273 0 2024-03-22T02:29:08.652 INFO:tasks.workunit.client.1.smithi122.stdout:1/965: rmdir dc/d3b/d8d/d72/d9c/db0/d90/d113 39 2024-03-22T02:29:08.654 INFO:tasks.workunit.client.1.smithi122.stdout:1/966: creat dc/d4c/f132 x:0 0 0 2024-03-22T02:29:08.657 INFO:tasks.workunit.client.1.smithi122.stdout:0/889: sync 2024-03-22T02:29:08.658 INFO:tasks.workunit.client.1.smithi122.stdout:1/967: truncate dc/d3b/d8d/d72/d9c/db0/f5c 2295779 0 2024-03-22T02:29:08.662 INFO:tasks.workunit.client.1.smithi122.stdout:1/968: unlink dc/d3b/d8d/d72/d9c/db0/d67/l10e 0 2024-03-22T02:29:08.665 INFO:tasks.workunit.client.1.smithi122.stdout:1/969: unlink dc/d3b/l41 0 2024-03-22T02:29:08.668 INFO:tasks.workunit.client.1.smithi122.stdout:0/890: write d11/d14/d23/f6d [2105190,97628] 0 2024-03-22T02:29:08.677 INFO:tasks.workunit.client.1.smithi122.stdout:1/970: truncate dc/f22 3431770 0 2024-03-22T02:29:08.678 INFO:tasks.workunit.client.1.smithi122.stdout:1/971: write dc/d2b/d86/f131 [78916,116510] 0 2024-03-22T02:29:08.680 INFO:tasks.workunit.client.1.smithi122.stdout:1/972: mkdir dc/d2b/d86/d133 0 2024-03-22T02:29:08.685 INFO:tasks.workunit.client.1.smithi122.stdout:1/973: unlink dc/d3b/d8d/ca0 0 2024-03-22T02:29:08.686 INFO:tasks.workunit.client.1.smithi122.stdout:1/974: write dc/d3b/f12d [1047279,114883] 0 2024-03-22T02:29:08.694 INFO:tasks.workunit.client.1.smithi122.stdout:1/975: write dc/d55/f77 [299139,21302] 0 2024-03-22T02:29:08.698 INFO:tasks.workunit.client.1.smithi122.stdout:1/976: getdents dc/d2b/d86/d110 0 2024-03-22T02:29:08.698 INFO:tasks.workunit.client.1.smithi122.stdout:0/891: dread d11/d14/d23/d24/fca [0,4194304] 0 2024-03-22T02:29:08.701 INFO:tasks.workunit.client.1.smithi122.stdout:1/977: creat dc/d3b/d46/de7/f134 x:0 0 0 2024-03-22T02:29:08.704 INFO:tasks.workunit.client.1.smithi122.stdout:1/978: unlink dc/d4c/f8b 0 2024-03-22T02:29:08.706 INFO:tasks.workunit.client.1.smithi122.stdout:0/892: dwrite d11/d14/d1d/d27/d9d/daa/fdb [0,4194304] 0 2024-03-22T02:29:08.707 INFO:tasks.workunit.client.1.smithi122.stdout:1/979: write dc/d3b/d8d/d72/d9c/fd6 [4506951,10013] 0 2024-03-22T02:29:08.709 INFO:tasks.workunit.client.1.smithi122.stdout:0/893: read d11/d16/d113/ffe [338701,18508] 0 2024-03-22T02:29:08.714 INFO:tasks.workunit.client.1.smithi122.stdout:0/894: mknod d11/d14/d1d/d27/dc0/def/c12c 0 2024-03-22T02:29:08.722 INFO:tasks.workunit.client.1.smithi122.stdout:0/895: dwrite d11/d14/d23/d24/fca [0,4194304] 0 2024-03-22T02:29:08.723 INFO:tasks.workunit.client.1.smithi122.stdout:0/896: chown d11/d14/d23/d24/f25 25626445 1 2024-03-22T02:29:08.724 INFO:tasks.workunit.client.1.smithi122.stdout:0/897: write d11/d14/d1d/d27/d2f/f58 [3111988,90930] 0 2024-03-22T02:29:08.727 INFO:tasks.workunit.client.1.smithi122.stdout:1/980: dread dc/d3b/d8d/d72/d9c/db0/d71/fda [4194304,4194304] 0 2024-03-22T02:29:08.728 INFO:tasks.workunit.client.1.smithi122.stdout:1/981: write dc/d3b/d46/de3/fef [605761,41518] 0 2024-03-22T02:29:08.731 INFO:tasks.workunit.client.1.smithi122.stdout:1/982: unlink dc/f88 0 2024-03-22T02:29:08.732 INFO:tasks.workunit.client.1.smithi122.stdout:0/898: link d11/d14/d1d/d27/d2f/d4e/lf5 d11/d14/d23/d24/d33/d67/l12d 0 2024-03-22T02:29:08.734 INFO:tasks.workunit.client.1.smithi122.stdout:1/983: rmdir dc/d3b/d46/de3 39 2024-03-22T02:29:08.735 INFO:tasks.workunit.client.1.smithi122.stdout:0/899: truncate d11/d14/d1b/f2c 3209419 0 2024-03-22T02:29:08.735 INFO:tasks.workunit.client.1.smithi122.stdout:1/984: chown dc/d3b/d8d/d72/d9c/db0/d67/dc6/da5 15 1 2024-03-22T02:29:08.737 INFO:tasks.workunit.client.1.smithi122.stdout:0/900: creat d11/d14/d1d/d8a/f12e x:0 0 0 2024-03-22T02:29:08.739 INFO:tasks.workunit.client.1.smithi122.stdout:1/985: mkdir dc/d3b/d46/de7/d135 0 2024-03-22T02:29:08.740 INFO:tasks.workunit.client.1.smithi122.stdout:0/901: mkdir d11/d16/d37/d84/da3/d12f 0 2024-03-22T02:29:08.741 INFO:tasks.workunit.client.1.smithi122.stdout:0/902: dread - d11/d14/d1d/d27/d2f/d4e/df0/feb zero size 2024-03-22T02:29:08.742 INFO:tasks.workunit.client.1.smithi122.stdout:1/986: creat dc/d3b/d8d/d72/d9c/db0/f136 x:0 0 0 2024-03-22T02:29:08.742 INFO:tasks.workunit.client.1.smithi122.stdout:0/903: fdatasync d11/d14/d23/d24/fca 0 2024-03-22T02:29:08.743 INFO:tasks.workunit.client.1.smithi122.stdout:0/904: fsync d11/d14/d1d/d27/dc0/dc6/dd0/f111 0 2024-03-22T02:29:08.745 INFO:tasks.workunit.client.1.smithi122.stdout:0/905: rmdir d11/d14/d23/d24 39 2024-03-22T02:29:08.746 INFO:tasks.workunit.client.1.smithi122.stdout:1/987: dread dc/f10 [0,4194304] 0 2024-03-22T02:29:08.747 INFO:tasks.workunit.client.1.smithi122.stdout:0/906: creat d11/d14/d1d/d27/dc0/dc6/f130 x:0 0 0 2024-03-22T02:29:08.753 INFO:tasks.workunit.client.1.smithi122.stdout:0/907: link d11/d14/d1d/fa2 d11/d14/d23/d24/f131 0 2024-03-22T02:29:08.755 INFO:tasks.workunit.client.1.smithi122.stdout:0/908: mknod d11/d14/d1d/d27/d9d/daa/c132 0 2024-03-22T02:29:08.756 INFO:tasks.workunit.client.1.smithi122.stdout:1/988: write dc/d3b/d46/db5/dba/fcc [755560,26548] 0 2024-03-22T02:29:08.757 INFO:tasks.workunit.client.1.smithi122.stdout:0/909: write d11/d4f/d5a/f61 [1050508,8168] 0 2024-03-22T02:29:08.758 INFO:tasks.workunit.client.1.smithi122.stdout:1/989: write dc/d3b/d8d/fee [144014,93187] 0 2024-03-22T02:29:08.763 INFO:tasks.workunit.client.1.smithi122.stdout:1/990: getdents dc/d3b/d46/db5/dba 0 2024-03-22T02:29:08.764 INFO:tasks.workunit.client.1.smithi122.stdout:1/991: mkdir dc/d3b/d8d/d72/d9c/d137 0 2024-03-22T02:29:08.765 INFO:tasks.workunit.client.1.smithi122.stdout:0/910: write d11/d14/d23/d24/d33/fd5 [856759,102444] 0 2024-03-22T02:29:08.768 INFO:tasks.workunit.client.1.smithi122.stdout:0/911: unlink d11/d14/d1d/c106 0 2024-03-22T02:29:08.768 INFO:tasks.workunit.client.1.smithi122.stdout:1/992: write dc/fd [7669497,44770] 0 2024-03-22T02:29:08.770 INFO:tasks.workunit.client.1.smithi122.stdout:0/912: truncate d11/d16/d37/d84/d87/df6/ffd 521013 0 2024-03-22T02:29:08.770 INFO:tasks.workunit.client.1.smithi122.stdout:1/993: fdatasync dc/d3b/d46/de3/fef 0 2024-03-22T02:29:08.772 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-22T02:29:08.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-22T02:29:08.773 INFO:tasks.workunit.client.1.smithi122.stdout:1/994: mkdir dc/d3b/d46/db5/d138 0 2024-03-22T02:29:08.775 INFO:tasks.workunit.client.1.smithi122.stdout:1/995: creat dc/d4c/f139 x:0 0 0 2024-03-22T02:29:08.775 INFO:tasks.workunit.client.1.smithi122.stdout:0/913: dread d11/d14/d1d/d27/dc0/dc6/dd0/dea/f11d [0,4194304] 0 2024-03-22T02:29:08.777 INFO:tasks.workunit.client.1.smithi122.stdout:0/914: chown d11/d4f/d5a/ff8 13837 1 2024-03-22T02:29:08.777 INFO:tasks.workunit.client.1.smithi122.stdout:0/915: chown c6 65494 1 2024-03-22T02:29:08.777 INFO:tasks.workunit.client.1.smithi122.stdout:1/996: fsync dc/d4c/f63 0 2024-03-22T02:29:08.782 INFO:tasks.workunit.client.1.smithi122.stdout:1/997: getdents dc/d3b/d8d 0 2024-03-22T02:29:08.784 INFO:tasks.workunit.client.1.smithi122.stdout:0/916: dwrite d11/d4f/d5a/ff8 [0,4194304] 0 2024-03-22T02:29:08.785 INFO:tasks.workunit.client.1.smithi122.stdout:0/917: read d11/d14/ff7 [369194,65470] 0 2024-03-22T02:29:08.787 INFO:tasks.workunit.client.1.smithi122.stdout:1/998: fsync dc/d2b/d86/dbb/fce 0 2024-03-22T02:29:08.789 INFO:tasks.workunit.client.1.smithi122.stdout:1/999: dread dc/d4c/f119 [0,4194304] 0 2024-03-22T02:29:08.790 INFO:tasks.workunit.client.1.smithi122.stdout:0/918: creat d11/d14/d1d/d27/d2f/d4e/f133 x:0 0 0 2024-03-22T02:29:08.792 INFO:tasks.workunit.client.1.smithi122.stdout:0/919: read d11/d14/d1d/d27/d2f/f58 [3322811,78057] 0 2024-03-22T02:29:08.792 INFO:tasks.workunit.client.1.smithi122.stdout:0/920: stat d11/d14/d1d/d27/d9d/daa 0 2024-03-22T02:29:08.795 INFO:tasks.workunit.client.1.smithi122.stdout:0/921: write d11/d14/d1d/d27/d2f/d4e/df0/feb [609258,89086] 0 2024-03-22T02:29:08.804 INFO:tasks.workunit.client.1.smithi122.stdout:0/922: dwrite d11/d16/d37/fbf [0,4194304] 0 2024-03-22T02:29:08.805 INFO:tasks.workunit.client.1.smithi122.stdout:0/923: dread - d11/d16/d37/dda/ffa zero size 2024-03-22T02:29:08.808 INFO:tasks.workunit.client.1.smithi122.stdout:0/924: creat d11/d14/d1d/d27/d2f/dd1/f134 x:0 0 0 2024-03-22T02:29:08.810 INFO:tasks.workunit.client.1.smithi122.stdout:0/925: getdents d11/d4f/d5a/d116 0 2024-03-22T02:29:08.984 INFO:tasks.workunit.client.1.smithi122.stdout:0/926: sync 2024-03-22T02:29:08.989 INFO:tasks.workunit.client.1.smithi122.stdout:0/927: dwrite d11/d14/d1d/d27/d2f/d4e/f97 [0,4194304] 0 2024-03-22T02:29:08.995 INFO:tasks.workunit.client.1.smithi122.stdout:0/928: dread d11/d14/d1d/d27/d9d/daa/fdb [0,4194304] 0 2024-03-22T02:29:08.996 INFO:tasks.workunit.client.1.smithi122.stdout:0/929: mkdir d11/d14/d1b/d135 0 2024-03-22T02:29:09.002 INFO:tasks.workunit.client.1.smithi122.stdout:0/930: dwrite d11/d14/d1d/d27/d9d/daa/f112 [0,4194304] 0 2024-03-22T02:29:09.005 INFO:tasks.workunit.client.1.smithi122.stdout:0/931: mknod d11/d14/d23/c136 0 2024-03-22T02:29:09.008 INFO:tasks.workunit.client.1.smithi122.stdout:0/932: write d11/d14/d1d/d27/d32/f6c [3136388,3785] 0 2024-03-22T02:29:09.010 INFO:tasks.workunit.client.1.smithi122.stdout:0/933: read d11/d16/d37/fab [2991957,64741] 0 2024-03-22T02:29:09.012 INFO:tasks.workunit.client.1.smithi122.stdout:0/934: mkdir d11/d14/d1d/d27/d9d/daa/d137 0 2024-03-22T02:29:09.016 INFO:tasks.workunit.client.1.smithi122.stdout:0/935: read d11/d16/d44/d89/f4b [7203183,86095] 0 2024-03-22T02:29:09.017 INFO:tasks.workunit.client.1.smithi122.stdout:0/936: creat d11/d47/f138 x:0 0 0 2024-03-22T02:29:09.025 INFO:tasks.workunit.client.1.smithi122.stdout:0/937: dwrite d11/d14/d1d/d27/d59/fb1 [0,4194304] 0 2024-03-22T02:29:09.029 INFO:tasks.workunit.client.1.smithi122.stdout:0/938: rename d11/d14/d23/d24/d33 to d11/d16/d37/dda/d139 0 2024-03-22T02:29:09.030 INFO:tasks.workunit.client.1.smithi122.stdout:0/939: mknod d11/d14/d1d/d8a/c13a 0 2024-03-22T02:29:09.042 INFO:tasks.workunit.client.1.smithi122.stdout:0/940: dread d11/d14/d23/f72 [0,4194304] 0 2024-03-22T02:29:09.059 INFO:tasks.workunit.client.1.smithi122.stdout:0/941: dread d11/f3d [0,4194304] 0 2024-03-22T02:29:09.060 INFO:tasks.workunit.client.1.smithi122.stdout:0/942: mknod d11/d14/d1d/d27/d59/c13b 0 2024-03-22T02:29:09.065 INFO:tasks.workunit.client.1.smithi122.stdout:0/943: write d11/d14/f8d [42184,108258] 0 2024-03-22T02:29:09.066 INFO:tasks.workunit.client.1.smithi122.stdout:0/944: mkdir d11/d14/d1d/d27/dc0/dc6/d13c 0 2024-03-22T02:29:09.168 INFO:tasks.workunit.client.1.smithi122.stdout:0/945: sync 2024-03-22T02:29:09.170 INFO:tasks.workunit.client.1.smithi122.stdout:0/946: rename d11/d16/d44/fbe to d11/d14/d1d/d8a/f13d 0 2024-03-22T02:29:09.173 INFO:tasks.workunit.client.1.smithi122.stdout:0/947: dread d11/d16/d113/ffe [0,4194304] 0 2024-03-22T02:29:09.176 INFO:tasks.workunit.client.1.smithi122.stdout:0/948: creat d11/d14/d1d/d27/f13e x:0 0 0 2024-03-22T02:29:09.177 INFO:tasks.workunit.client.1.smithi122.stdout:0/949: rmdir d11/d14/d1d/d27/dc0/dc6/dd0/dea 39 2024-03-22T02:29:09.185 INFO:tasks.workunit.client.1.smithi122.stdout:0/950: rmdir d11 39 2024-03-22T02:29:09.186 INFO:tasks.workunit.client.1.smithi122.stdout:0/951: truncate d11/d16/d37/d84/da3/fdd 812078 0 2024-03-22T02:29:09.188 INFO:tasks.workunit.client.1.smithi122.stdout:0/952: mkdir d11/d14/d23/d24/d13f 0 2024-03-22T02:29:09.201 INFO:tasks.workunit.client.1.smithi122.stdout:0/953: dread d11/d16/d37/dda/d139/d67/f7c [0,4194304] 0 2024-03-22T02:29:09.203 INFO:tasks.workunit.client.1.smithi122.stdout:0/954: creat d11/d14/d12a/f140 x:0 0 0 2024-03-22T02:29:09.214 INFO:tasks.workunit.client.1.smithi122.stdout:0/955: dread d11/d14/d1d/d27/d9d/fcc [0,4194304] 0 2024-03-22T02:29:09.215 INFO:tasks.workunit.client.1.smithi122.stdout:0/956: chown d11/d14/d1d/d27/d2f 443 1 2024-03-22T02:29:09.215 INFO:tasks.workunit.client.1.smithi122.stdout:0/957: fdatasync d11/d47/ff4 0 2024-03-22T02:29:09.216 INFO:tasks.workunit.client.1.smithi122.stdout:0/958: write d11/d14/d23/f2d [1237420,101099] 0 2024-03-22T02:29:09.217 INFO:tasks.workunit.client.1.smithi122.stdout:0/959: chown d11/d16/d37 13058853 1 2024-03-22T02:29:09.219 INFO:tasks.workunit.client.1.smithi122.stdout:0/960: mkdir d11/d16/d113/d141 0 2024-03-22T02:29:09.221 INFO:tasks.workunit.client.1.smithi122.stdout:0/961: write d11/d16/d37/d84/da3/fb9 [482441,125173] 0 2024-03-22T02:29:09.223 INFO:tasks.workunit.client.1.smithi122.stdout:0/962: mkdir d11/d14/d23/d24/d142 0 2024-03-22T02:29:09.228 INFO:tasks.workunit.client.1.smithi122.stdout:0/963: dwrite d11/d14/d1d/d27/d9d/fcb [0,4194304] 0 2024-03-22T02:29:09.231 INFO:tasks.workunit.client.1.smithi122.stdout:0/964: rename d11/d14/d1d/d27/d2f/f3a to d11/d14/d1d/d27/d2f/d4e/df0/f143 0 2024-03-22T02:29:09.253 INFO:tasks.workunit.client.1.smithi122.stdout:0/965: dread d11/d14/d1d/f80 [0,4194304] 0 2024-03-22T02:29:09.255 INFO:tasks.workunit.client.1.smithi122.stdout:0/966: creat d11/d14/d1d/d27/d2f/d4e/f144 x:0 0 0 2024-03-22T02:29:09.259 INFO:tasks.workunit.client.1.smithi122.stdout:0/967: link d11/d4f/d5a/cdf d11/d14/d1d/d27/dc0/c145 0 2024-03-22T02:29:09.262 INFO:tasks.workunit.client.1.smithi122.stdout:0/968: mkdir d11/d14/d23/d24/d146 0 2024-03-22T02:29:09.262 INFO:tasks.workunit.client.1.smithi122.stdout:0/969: readlink d11/d14/d1d/d27/d9d/daa/dcd/lde 0 2024-03-22T02:29:09.264 INFO:tasks.workunit.client.1.smithi122.stdout:0/970: symlink d11/d14/d1d/d27/d9d/daa/d137/l147 0 2024-03-22T02:29:09.266 INFO:tasks.workunit.client.1.smithi122.stdout:0/971: creat d11/d14/d23/f148 x:0 0 0 2024-03-22T02:29:09.269 INFO:tasks.workunit.client.1.smithi122.stdout:0/972: link d11/d14/d1d/d27/dc0/c145 d11/d14/d1d/d27/c149 0 2024-03-22T02:29:09.271 INFO:tasks.workunit.client.1.smithi122.stdout:0/973: creat d11/d14/d1d/d27/dc0/def/f14a x:0 0 0 2024-03-22T02:29:09.272 INFO:tasks.workunit.client.1.smithi122.stdout:0/974: symlink d11/d14/d12a/l14b 0 2024-03-22T02:29:09.275 INFO:tasks.workunit.client.1.smithi122.stdout:0/975: rmdir d11/d14/d23/d24/d13f 0 2024-03-22T02:29:09.277 INFO:tasks.workunit.client.1.smithi122.stdout:0/976: rename d11/d14/fe8 to d11/d14/d1d/d27/d2f/f14c 0 2024-03-22T02:29:09.280 INFO:tasks.workunit.client.1.smithi122.stdout:0/977: rename d11/d16/d44/d89 to d11/d14/d1b/d126/d14d 0 2024-03-22T02:29:09.283 INFO:tasks.workunit.client.1.smithi122.stdout:0/978: symlink d11/d14/d23/d24/d146/l14e 0 2024-03-22T02:29:09.284 INFO:tasks.workunit.client.1.smithi122.stdout:0/979: write d11/fb8 [1093897,99157] 0 2024-03-22T02:29:09.286 INFO:tasks.workunit.client.1.smithi122.stdout:0/980: write d11/d14/d1d/f80 [1787182,79508] 0 2024-03-22T02:29:09.287 INFO:tasks.workunit.client.1.smithi122.stdout:0/981: readlink d11/d47/lfc 0 2024-03-22T02:29:09.289 INFO:tasks.workunit.client.1.smithi122.stdout:0/982: symlink d11/de1/l14f 0 2024-03-22T02:29:09.294 INFO:tasks.workunit.client.1.smithi122.stdout:0/983: write d11/d14/d1d/d27/d2f/f58 [4239426,6644] 0 2024-03-22T02:29:09.299 INFO:tasks.workunit.client.1.smithi122.stdout:0/984: rename d11/d14/d1b/f2c to d11/d14/d23/d24/f150 0 2024-03-22T02:29:09.299 INFO:tasks.workunit.client.1.smithi122.stdout:0/985: chown d11/d47/f138 707696 1 2024-03-22T02:29:09.300 INFO:tasks.workunit.client.1.smithi122.stdout:0/986: chown d11/d16/d37/d84/f122 3171184 1 2024-03-22T02:29:09.303 INFO:tasks.workunit.client.1.smithi122.stdout:0/987: dread d11/d14/d1d/d27/d9d/daa/fdb [0,4194304] 0 2024-03-22T02:29:09.309 INFO:tasks.workunit.client.1.smithi122.stdout:0/988: truncate d11/d16/d37/dda/d139/f79 1571264 0 2024-03-22T02:29:09.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/989: creat d11/d4f/d5a/d116/f151 x:0 0 0 2024-03-22T02:29:09.311 INFO:tasks.workunit.client.1.smithi122.stdout:0/990: write d11/d16/d37/dda/d139/d67/dc2/fc9 [119614,101903] 0 2024-03-22T02:29:09.314 INFO:tasks.workunit.client.1.smithi122.stdout:0/991: write d11/d14/d1d/d27/dc0/dc6/dd0/dea/f11d [1369812,93089] 0 2024-03-22T02:29:09.315 INFO:tasks.workunit.client.1.smithi122.stdout:0/992: symlink d11/d16/d37/d84/da3/l152 0 2024-03-22T02:29:09.321 INFO:tasks.workunit.client.1.smithi122.stdout:0/993: write d11/d16/d37/dda/d139/d67/f7c [1719192,71813] 0 2024-03-22T02:29:09.323 INFO:tasks.workunit.client.1.smithi122.stdout:0/994: write d11/d16/d37/dda/d139/f95 [1414135,109713] 0 2024-03-22T02:29:09.325 INFO:tasks.workunit.client.1.smithi122.stdout:0/995: unlink d11/la1 0 2024-03-22T02:29:09.326 INFO:tasks.workunit.client.1.smithi122.stdout:0/996: write d11/d14/d23/d24/fa8 [1174182,116518] 0 2024-03-22T02:29:09.329 INFO:tasks.workunit.client.1.smithi122.stdout:0/997: creat d11/d16/d37/dda/d139/f153 x:0 0 0 2024-03-22T02:29:09.330 INFO:tasks.workunit.client.1.smithi122.stdout:0/998: mknod d11/c154 0 2024-03-22T02:29:09.332 INFO:tasks.workunit.client.1.smithi122.stdout:0/999: creat d11/d14/d1b/d126/f155 x:0 0 0 2024-03-22T02:29:09.335 INFO:tasks.workunit.client.1.smithi122.stderr:+ rm -rf -- ./tmp.3l0pIfzLJO 2024-03-22T02:29:09.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:09 smithi122 ceph-mon[36403]: pgmap v28: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 64 MiB/s rd, 135 MiB/s wr, 383 op/s 2024-03-22T02:29:09.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:09 smithi103 ceph-mon[29258]: pgmap v28: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 64 MiB/s rd, 135 MiB/s wr, 383 op/s 2024-03-22T02:29:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:11 smithi122 ceph-mon[36403]: pgmap v29: 65 pgs: 65 active+clean; 2.6 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 87 MiB/s rd, 204 MiB/s wr, 558 op/s 2024-03-22T02:29:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:11 smithi103 ceph-mon[29258]: pgmap v29: 65 pgs: 65 active+clean; 2.6 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 87 MiB/s rd, 204 MiB/s wr, 558 op/s 2024-03-22T02:29:13.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:13 smithi103 ceph-mon[29258]: pgmap v30: 65 pgs: 65 active+clean; 2.6 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 58 MiB/s rd, 137 MiB/s wr, 378 op/s 2024-03-22T02:29:13.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:13 smithi122 ceph-mon[36403]: pgmap v30: 65 pgs: 65 active+clean; 2.6 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 58 MiB/s rd, 137 MiB/s wr, 378 op/s 2024-03-22T02:29:15.752 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:15 smithi103 ceph-mon[29258]: pgmap v31: 65 pgs: 65 active+clean; 2.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 58 MiB/s rd, 137 MiB/s wr, 392 op/s 2024-03-22T02:29:15.757 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:15 smithi103 ceph-mon[29258]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:29:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:15 smithi122 ceph-mon[36403]: pgmap v31: 65 pgs: 65 active+clean; 2.5 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 58 MiB/s rd, 137 MiB/s wr, 392 op/s 2024-03-22T02:29:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:15 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:29:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:17 smithi122 ceph-mon[36403]: pgmap v32: 65 pgs: 65 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 56 MiB/s rd, 130 MiB/s wr, 515 op/s 2024-03-22T02:29:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:17 smithi103 ceph-mon[29258]: pgmap v32: 65 pgs: 65 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 56 MiB/s rd, 130 MiB/s wr, 515 op/s 2024-03-22T02:29:18.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.671+0000 7f63569fd700 1 -- 172.21.15.103:0/1202139905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fe060 msgr2=0x7f63500fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:18.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.671+0000 7f63569fd700 1 --2- 172.21.15.103:0/1202139905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fe060 0x7f63500fe4d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6344004940 tx=0x7f6344009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 -- 172.21.15.103:0/1202139905 shutdown_connections 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 --2- 172.21.15.103:0/1202139905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fe060 0x7f63500fe4d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 --2- 172.21.15.103:0/1202139905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fcef0 0x7f63500fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 -- 172.21.15.103:0/1202139905 >> 172.21.15.103:0/1202139905 conn(0x7f63500f85b0 msgr2=0x7f63500fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 -- 172.21.15.103:0/1202139905 shutdown_connections 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 -- 172.21.15.103:0/1202139905 wait complete. 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 Processor -- start 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.672+0000 7f63569fd700 1 -- start start 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f63569fd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f63569fd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 0x7f6350070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f63569fd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63500708c0 con 0x7f63500fcef0 2024-03-22T02:29:18.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f63569fd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350070a30 con 0x7f63500fe060 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 0x7f6350070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40590/0 (socket says 172.21.15.103:40590) 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 -- 172.21.15.103:0/3093630021 learned_addr learned my addr 172.21.15.103:0/3093630021 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 -- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 msgr2=0x7f6350070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 0x7f6350070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 -- 172.21.15.103:0/3093630021 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344005040 con 0x7f63500fcef0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634f7fe700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 0x7f6350070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634ffff700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f634000eb40 tx=0x7f634000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.673+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6340007cc0 con 0x7f63500fcef0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.674+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6350070cb0 con 0x7f63500fcef0 2024-03-22T02:29:18.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.674+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63501159f0 con 0x7f63500fcef0 2024-03-22T02:29:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.674+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f634000f040 con 0x7f63500fcef0 2024-03-22T02:29:18.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.674+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634000ca10 con 0x7f63500fcef0 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.675+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635005f5f0 con 0x7f63500fcef0 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.676+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f634000cb70 con 0x7f63500fcef0 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.676+0000 7f634cff9700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 0x7f633806de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.676+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f634008a140 con 0x7f63500fcef0 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.677+0000 7f634f7fe700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 0x7f633806de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:18.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.677+0000 7f634f7fe700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 0x7f633806de40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6344009b30 tx=0x7f6344009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:18.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:18.684+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f6340050040 con 0x7f63500fcef0 2024-03-22T02:29:18.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:18 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:18 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:19.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.052+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6350102330 con 0x7f633806b990 2024-03-22T02:29:19.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.053+0000 7f634cff9700 1 -- 172.21.15.103:0/3093630021 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f6350102330 con 0x7f633806b990 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 msgr2=0x7f633806de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 0x7f633806de40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6344009b30 tx=0x7f6344009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 msgr2=0x7f6350071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f634000eb40 tx=0x7f634000b040 comp rx=0 tx=0).stop 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 shutdown_connections 2024-03-22T02:29:19.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f633806b990 0x7f633806de40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63500fcef0 0x7f6350071e40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 --2- 172.21.15.103:0/3093630021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63500fe060 0x7f6350070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.055+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 >> 172.21.15.103:0/3093630021 conn(0x7f63500f85b0 msgr2=0x7f63500f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:19.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.056+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 shutdown_connections 2024-03-22T02:29:19.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.056+0000 7f63569fd700 1 -- 172.21.15.103:0/3093630021 wait complete. 2024-03-22T02:29:19.069 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.190+0000 7f1a989ae700 1 -- 172.21.15.103:0/124009701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 msgr2=0x7f1a940ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 --2- 172.21.15.103:0/124009701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a940ff9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1a84004940 tx=0x7f1a84009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 -- 172.21.15.103:0/124009701 shutdown_connections 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 --2- 172.21.15.103:0/124009701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 0x7f1a94102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 --2- 172.21.15.103:0/124009701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a940ff9c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 -- 172.21.15.103:0/124009701 >> 172.21.15.103:0/124009701 conn(0x7f1a940f7430 msgr2=0x7f1a940f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 -- 172.21.15.103:0/124009701 shutdown_connections 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 -- 172.21.15.103:0/124009701 wait complete. 2024-03-22T02:29:19.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 Processor -- start 2024-03-22T02:29:19.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.191+0000 7f1a989ae700 1 -- start start 2024-03-22T02:29:19.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a989ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a989ae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 0x7f1a9410df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a989ae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9410e500 con 0x7f1a940fff00 2024-03-22T02:29:19.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a989ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9410ed50 con 0x7f1a940662e0 2024-03-22T02:29:19.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a9259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a9259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:35590/0 (socket says 172.21.15.103:35590) 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a9259c700 1 -- 172.21.15.103:0/3367692646 learned_addr learned my addr 172.21.15.103:0/3367692646 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.192+0000 7f1a91d9b700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 0x7f1a9410df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a9259c700 1 -- 172.21.15.103:0/3367692646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 msgr2=0x7f1a9410df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a9259c700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 0x7f1a9410df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a9259c700 1 -- 172.21.15.103:0/3367692646 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a84005040 con 0x7f1a940662e0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a9259c700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a84004090 tx=0x7f1a84009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a84021070 con 0x7f1a940662e0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a8401c400 con 0x7f1a940662e0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9410efd0 con 0x7f1a940662e0 2024-03-22T02:29:19.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.193+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9410f510 con 0x7f1a940662e0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.194+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a8400e070 con 0x7f1a940662e0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.194+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9405f5f0 con 0x7f1a940662e0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.195+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f1a8400b760 con 0x7f1a940662e0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.195+0000 7f1a82ffd700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 0x7f1a7c06dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.196+0000 7f1a91d9b700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 0x7f1a7c06dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.196+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f1a8408b690 con 0x7f1a940662e0 2024-03-22T02:29:19.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.196+0000 7f1a91d9b700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 0x7f1a7c06dd70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009b80 tx=0x7f1a8800b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:19.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.200+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f1a84058cf0 con 0x7f1a940662e0 2024-03-22T02:29:19.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.518+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a9410f7f0 con 0x7f1a7c06b8c0 2024-03-22T02:29:19.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.519+0000 7f1a82ffd700 1 -- 172.21.15.103:0/3367692646 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f1a9410f7f0 con 0x7f1a7c06b8c0 2024-03-22T02:29:19.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 msgr2=0x7f1a7c06dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 0x7f1a7c06dd70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009b80 tx=0x7f1a8800b040 comp rx=0 tx=0).stop 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 msgr2=0x7f1a9410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a84004090 tx=0x7f1a84009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 shutdown_connections 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f1a7c06b8c0 0x7f1a7c06dd70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a940662e0 0x7f1a9410d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 --2- 172.21.15.103:0/3367692646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a940fff00 0x7f1a9410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 >> 172.21.15.103:0/3367692646 conn(0x7f1a940f7430 msgr2=0x7f1a940f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 shutdown_connections 2024-03-22T02:29:19.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.522+0000 7f1a989ae700 1 -- 172.21.15.103:0/3367692646 wait complete. 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 -- 172.21.15.103:0/2231928916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 msgr2=0x7fc2700fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 --2- 172.21.15.103:0/2231928916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc2700fc0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc260004940 tx=0x7fc260009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 -- 172.21.15.103:0/2231928916 shutdown_connections 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 --2- 172.21.15.103:0/2231928916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc2700fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 --2- 172.21.15.103:0/2231928916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc2700fc0e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 -- 172.21.15.103:0/2231928916 >> 172.21.15.103:0/2231928916 conn(0x7fc2700f7430 msgr2=0x7fc2700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:19.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 -- 172.21.15.103:0/2231928916 shutdown_connections 2024-03-22T02:29:19.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.655+0000 7fc2755e0700 1 -- 172.21.15.103:0/2231928916 wait complete. 2024-03-22T02:29:19.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 Processor -- start 2024-03-22T02:29:19.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 -- start start 2024-03-22T02:29:19.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc270066c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc270112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270112c40 con 0x7fc2700fbcd0 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.656+0000 7fc2755e0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270112d80 con 0x7fc2700fcf80 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc270066c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc270066c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40620/0 (socket says 172.21.15.103:40620) 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26effd700 1 -- 172.21.15.103:0/337067582 learned_addr learned my addr 172.21.15.103:0/337067582 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26e7fc700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc270112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26e7fc700 1 -- 172.21.15.103:0/337067582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 msgr2=0x7fc270066c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26e7fc700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc270066c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26e7fc700 1 -- 172.21.15.103:0/337067582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc260005040 con 0x7fc2700fcf80 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc26e7fc700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc270112700 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc264009fd0 tx=0x7fc26400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc264004e90 con 0x7fc2700fcf80 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc270112f80 con 0x7fc2700fcf80 2024-03-22T02:29:19.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc26400f040 con 0x7fc2700fcf80 2024-03-22T02:29:19.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26400ca50 con 0x7fc2700fcf80 2024-03-22T02:29:19.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.657+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc270113510 con 0x7fc2700fcf80 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.659+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27004c710 con 0x7fc2700fcf80 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.659+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7fc26400cc00 con 0x7fc2700fcf80 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.659+0000 7fc2577fe700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 0x7fc25806dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.660+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc26408a060 con 0x7fc2700fcf80 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.660+0000 7fc26effd700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 0x7fc25806dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:19.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.660+0000 7fc26effd700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 0x7fc25806dd80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2600043d0 tx=0x7fc260009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:19.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.664+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fc264050040 con 0x7fc2700fcf80 2024-03-22T02:29:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:19 smithi122 ceph-mon[36403]: pgmap v33: 65 pgs: 65 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 24 MiB/s rd, 71 MiB/s wr, 327 op/s 2024-03-22T02:29:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:19 smithi103 ceph-mon[29258]: pgmap v33: 65 pgs: 65 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 24 MiB/s rd, 71 MiB/s wr, 327 op/s 2024-03-22T02:29:19.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.984+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc270101270 con 0x7fc25806b8d0 2024-03-22T02:29:19.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.994+0000 7fc2577fe700 1 -- 172.21.15.103:0/337067582 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fc270101270 con 0x7fc25806b8d0 2024-03-22T02:29:19.995 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 starting - - - - 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (9m) 26s ago 9m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (8m) 50s ago 8m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (7m) 26s ago 8m 84.6M - 9.4.7 954c08fa6188 55f93f687cd7 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (3m) 26s ago 3m 11.7M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (3m) 26s ago 3m 109M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (3m) 50s ago 3m 13.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (3m) 50s ago 3m 100M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (10m) 26s ago 10m 396M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (87s) 50s ago 8m 489M - 19.0.0-2344-gc3a04c0e 9d92c22378ed c01e6142c155 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (10m) 26s ago 10m 50.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (7m) 50s ago 7m 39.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 starting - - - - 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 50s ago 8m 16.3M - 1.5.0 0da6a335fe13 a57d1e60c6fb 2024-03-22T02:29:19.996 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (7m) 26s ago 7m 122M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (6m) 26s ago 6m 134M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (6m) 26s ago 6m 121M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (5m) 50s ago 5m 143M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (5m) 50s ago 5m 135M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (5m) 50s ago 5m 125M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:29:19.997 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (7m) 26s ago 8m 58.6M - 2.43.0 a07b618ecd1d 34dc94fa87d4 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 msgr2=0x7fc25806dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 0x7fc25806dd80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2600043d0 tx=0x7fc260009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 msgr2=0x7fc270112700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc270112700 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc264009fd0 tx=0x7fc26400b040 comp rx=0 tx=0).stop 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 shutdown_connections 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc25806b8d0 0x7fc25806dd80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2700fbcd0 0x7fc270066c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 --2- 172.21.15.103:0/337067582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcf80 0x7fc270112700 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 >> 172.21.15.103:0/337067582 conn(0x7fc2700f7430 msgr2=0x7fc2700f8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 shutdown_connections 2024-03-22T02:29:19.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:19.996+0000 7fc2755e0700 1 -- 172.21.15.103:0/337067582 wait complete. 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 -- 172.21.15.103:0/2038446589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fce40 msgr2=0x7f5d3c0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 --2- 172.21.15.103:0/2038446589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c0fd250 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c004a90 tx=0x7f5d2c009dd0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 -- 172.21.15.103:0/2038446589 shutdown_connections 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 --2- 172.21.15.103:0/2038446589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 --2- 172.21.15.103:0/2038446589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c0fd250 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 -- 172.21.15.103:0/2038446589 >> 172.21.15.103:0/2038446589 conn(0x7f5d3c0f8570 msgr2=0x7f5d3c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 -- 172.21.15.103:0/2038446589 shutdown_connections 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.119+0000 7f5d41018700 1 -- 172.21.15.103:0/2038446589 wait complete. 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 Processor -- start 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 -- start start 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c10e510 con 0x7f5d3c0fe040 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d41018700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c10ed60 con 0x7f5d3c0fce40 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d3a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d3a59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40638/0 (socket says 172.21.15.103:40638) 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d3a59c700 1 -- 172.21.15.103:0/220456933 learned_addr learned my addr 172.21.15.103:0/220456933 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:20.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.120+0000 7f5d3ad9d700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d3a59c700 1 -- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 msgr2=0x7f5d3c10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d3a59c700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d3a59c700 1 -- 172.21.15.103:0/220456933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c004650 con 0x7f5d3c0fe040 2024-03-22T02:29:20.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d3ad9d700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d3a59c700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000eb40 tx=0x7f5d3000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d30007cc0 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c10efe0 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c10f520 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d3000f040 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.121+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d3000ca10 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.122+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d280052e0 con 0x7f5d3c0fe040 2024-03-22T02:29:20.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.123+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7f5d3000cb70 con 0x7f5d3c0fe040 2024-03-22T02:29:20.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.123+0000 7f5d237fe700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 0x7f5d2406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.123+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f5d30089fc0 con 0x7f5d3c0fe040 2024-03-22T02:29:20.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.124+0000 7f5d3ad9d700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 0x7f5d2406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.124+0000 7f5d3ad9d700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 0x7f5d2406dd70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00b8b0 tx=0x7f5d2c016040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:20.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.128+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f5d30050040 con 0x7f5d3c0fe040 2024-03-22T02:29:20.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.451+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5d28005860 con 0x7f5d3c0fe040 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.451+0000 7f5d237fe700 1 -- 172.21.15.103:0/220456933 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f5d30057470 con 0x7f5d3c0fe040 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:29:20.453 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:29:20.454 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 msgr2=0x7f5d2406dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 0x7f5d2406dd70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00b8b0 tx=0x7f5d2c016040 comp rx=0 tx=0).stop 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 msgr2=0x7f5d3c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000eb40 tx=0x7f5d3000b040 comp rx=0 tx=0).stop 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 shutdown_connections 2024-03-22T02:29:20.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f5d2406b8c0 0x7f5d2406dd70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d3c0fce40 0x7f5d3c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 --2- 172.21.15.103:0/220456933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d3c0fe040 0x7f5d3c10df10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.454+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 >> 172.21.15.103:0/220456933 conn(0x7f5d3c0f8570 msgr2=0x7f5d3c0f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:20.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.455+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 shutdown_connections 2024-03-22T02:29:20.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.455+0000 7f5d41018700 1 -- 172.21.15.103:0/220456933 wait complete. 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- 172.21.15.103:0/3586666602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 msgr2=0x7fa0fc1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 --2- 172.21.15.103:0/3586666602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc1023d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec004940 tx=0x7fa0ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- 172.21.15.103:0/3586666602 shutdown_connections 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 --2- 172.21.15.103:0/3586666602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc1023d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 --2- 172.21.15.103:0/3586666602 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 0x7fa0fc0ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- 172.21.15.103:0/3586666602 >> 172.21.15.103:0/3586666602 conn(0x7fa0fc0f74e0 msgr2=0x7fa0fc0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- 172.21.15.103:0/3586666602 shutdown_connections 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- 172.21.15.103:0/3586666602 wait complete. 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 Processor -- start 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- start start 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 0x7fa0fc10da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc10e570 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa103c89700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc10ed70 con 0x7fa0fc066b10 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa101224700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa101224700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40644/0 (socket says 172.21.15.103:40644) 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.578+0000 7fa101224700 1 -- 172.21.15.103:0/3592365211 learned_addr learned my addr 172.21.15.103:0/3592365211 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa101a25700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 0x7fa0fc10da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa101224700 1 -- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 msgr2=0x7fa0fc10da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa101224700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 0x7fa0fc10da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa101224700 1 -- 172.21.15.103:0/3592365211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec005040 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa101224700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec013870 tx=0x7fa0ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0ec021070 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0ec009430 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0ec00e070 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0fc10eff0 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.579+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc10f530 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.580+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc05f5f0 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.580+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) v1 ==== 89380+0+0 (secure 0 0 0) 0x7fa0ec01c750 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.581+0000 7fa0f27fc700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 0x7fa0e806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.581+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa0ec08b640 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.581+0000 7fa101a25700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 0x7fa0e806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:20.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.582+0000 7fa101a25700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 0x7fa0e806dd70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009b80 tx=0x7fa0f800b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:20.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.586+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fa0ec058d60 con 0x7fa0fc0ffff0 2024-03-22T02:29:20.731 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:20 smithi103 ceph-mon[29258]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:20.731 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:20 smithi103 ceph-mon[29258]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:20.731 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:20 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/220456933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:29:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:20 smithi122 ceph-mon[36403]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:20 smithi122 ceph-mon[36403]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:20 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/220456933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:29:20.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.867+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0fc10f810 con 0x7fa0e806b8c0 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.868+0000 7fa0f27fc700 1 -- 172.21.15.103:0/3592365211 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fa0fc10f810 con 0x7fa0e806b8c0 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:29:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "1/2 daemons upgraded", 2024-03-22T02:29:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "", 2024-03-22T02:29:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:29:20.871 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 msgr2=0x7fa0e806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 0x7fa0e806dd70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009b80 tx=0x7fa0f800b040 comp rx=0 tx=0).stop 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 msgr2=0x7fa0fc10df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec013870 tx=0x7fa0ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 shutdown_connections 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fa0e806b8c0 0x7fa0e806dd70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0fc066b10 0x7fa0fc10da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 --2- 172.21.15.103:0/3592365211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0fc0ffff0 0x7fa0fc10df70 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec013870 tx=0x7fa0ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:20.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.872+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 >> 172.21.15.103:0/3592365211 conn(0x7fa0fc0f74e0 msgr2=0x7fa0fc0f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:20.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.873+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 shutdown_connections 2024-03-22T02:29:20.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:20.873+0000 7fa103c89700 1 -- 172.21.15.103:0/3592365211 wait complete. 2024-03-22T02:29:21.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:21 smithi122 ceph-mon[36403]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:21.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:21 smithi122 ceph-mon[36403]: pgmap v34: 65 pgs: 65 active+clean; 941 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 24 MiB/s rd, 72 MiB/s wr, 498 op/s 2024-03-22T02:29:21.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:21 smithi122 ceph-mon[36403]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:21 smithi103 ceph-mon[29258]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:21 smithi103 ceph-mon[29258]: pgmap v34: 65 pgs: 65 active+clean; 941 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 24 MiB/s rd, 72 MiB/s wr, 498 op/s 2024-03-22T02:29:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:21 smithi103 ceph-mon[29258]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:22.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:22 smithi122 ceph-mon[36403]: pgmap v35: 65 pgs: 65 active+clean; 941 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-03-22T02:29:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:22 smithi103 ceph-mon[29258]: pgmap v35: 65 pgs: 65 active+clean; 941 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-03-22T02:29:23.769 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-22T02:29:23.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-22T02:29:25.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:25 smithi103 ceph-mon[29258]: pgmap v36: 65 pgs: 65 active+clean; 892 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 336 op/s 2024-03-22T02:29:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:25 smithi122 ceph-mon[36403]: pgmap v36: 65 pgs: 65 active+clean; 892 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 336 op/s 2024-03-22T02:29:26.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:26 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:26.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:26 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:26.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:26 smithi103 ceph-mon[29258]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:29:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:26 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:26 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:26 smithi122 ceph-mon[36403]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:29:27.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:27 smithi103 ceph-mon[29258]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:29:27.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:27 smithi103 ceph-mon[29258]: pgmap v37: 65 pgs: 65 active+clean; 486 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-22T02:29:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:27 smithi122 ceph-mon[36403]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:29:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:27 smithi122 ceph-mon[36403]: pgmap v37: 65 pgs: 65 active+clean; 486 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-22T02:29:29.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:29 smithi103 ceph-mon[29258]: pgmap v38: 65 pgs: 65 active+clean; 486 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 274 op/s 2024-03-22T02:29:29.650 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:29 smithi122 ceph-mon[36403]: pgmap v38: 65 pgs: 65 active+clean; 486 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 34 KiB/s rd, 1.4 MiB/s wr, 274 op/s 2024-03-22T02:29:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:31 smithi122 ceph-mon[36403]: pgmap v39: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.1 MiB/s wr, 455 op/s 2024-03-22T02:29:31.623 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:31 smithi103 ceph-mon[29258]: pgmap v39: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.1 MiB/s wr, 455 op/s 2024-03-22T02:29:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:33 smithi103 ceph-mon[29258]: pgmap v40: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 284 op/s 2024-03-22T02:29:33.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:33 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:33 smithi122 ceph-mon[36403]: pgmap v40: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 284 op/s 2024-03-22T02:29:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:33 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:35 smithi103 ceph-mon[29258]: pgmap v41: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.3 MiB/s wr, 313 op/s 2024-03-22T02:29:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:35 smithi122 ceph-mon[36403]: pgmap v41: 65 pgs: 65 active+clean; 285 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.3 MiB/s wr, 313 op/s 2024-03-22T02:29:37.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:37 smithi103 ceph-mon[29258]: pgmap v42: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 2.3 MiB/s wr, 526 op/s 2024-03-22T02:29:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:37 smithi122 ceph-mon[36403]: pgmap v42: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 2.3 MiB/s wr, 526 op/s 2024-03-22T02:29:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:39 smithi122 ceph-mon[36403]: pgmap v43: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.8 MiB/s wr, 435 op/s 2024-03-22T02:29:39.855 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:39 smithi103 ceph-mon[29258]: pgmap v43: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.8 MiB/s wr, 435 op/s 2024-03-22T02:29:41.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:41 smithi122 ceph-mon[36403]: pgmap v44: 65 pgs: 65 active+clean; 281 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 65 KiB/s rd, 2.6 MiB/s wr, 684 op/s 2024-03-22T02:29:41.644 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:41 smithi103 ceph-mon[29258]: pgmap v44: 65 pgs: 65 active+clean; 281 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 65 KiB/s rd, 2.6 MiB/s wr, 684 op/s 2024-03-22T02:29:43.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:43 smithi122 ceph-mon[36403]: pgmap v45: 65 pgs: 65 active+clean; 281 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 48 KiB/s rd, 1.8 MiB/s wr, 503 op/s 2024-03-22T02:29:43.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:43 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:43.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:43 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:43.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:43 smithi122 ceph-mon[36403]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-03-22T02:29:43.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:43 smithi122 ceph-mon[36403]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-03-22T02:29:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:43 smithi103 ceph-mon[29258]: pgmap v45: 65 pgs: 65 active+clean; 281 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 48 KiB/s rd, 1.8 MiB/s wr, 503 op/s 2024-03-22T02:29:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:43 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:43 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:43 smithi103 ceph-mon[29258]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-03-22T02:29:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:43 smithi103 ceph-mon[29258]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-03-22T02:29:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: pgmap v46: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 56 KiB/s rd, 1.8 MiB/s wr, 524 op/s 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:29:45.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:45 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: pgmap v46: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 56 KiB/s rd, 1.8 MiB/s wr, 524 op/s 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:29:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:45 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:46 smithi122 ceph-mon[36403]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:29:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:46 smithi122 ceph-mon[36403]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:29:46.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:46 smithi122 ceph-mon[36403]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:29:46.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:46 smithi122 ceph-mon[36403]: Upgrade: Need to upgrade myself (mgr.smithi122.prgtti) 2024-03-22T02:29:46.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:46 smithi122 ceph-mon[36403]: Upgrade: Need to upgrade myself (mgr.smithi122.prgtti) 2024-03-22T02:29:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:46 smithi103 ceph-mon[29258]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:29:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:46 smithi103 ceph-mon[29258]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:29:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:46 smithi103 ceph-mon[29258]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:29:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:46 smithi103 ceph-mon[29258]: Upgrade: Need to upgrade myself (mgr.smithi122.prgtti) 2024-03-22T02:29:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:46 smithi103 ceph-mon[29258]: Upgrade: Need to upgrade myself (mgr.smithi122.prgtti) 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: pgmap v47: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.6 MiB/s wr, 666 op/s 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:29:47.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:47 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:29:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: pgmap v47: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.6 MiB/s wr, 666 op/s 2024-03-22T02:29:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:29:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:29:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:29:47.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:47 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:29:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:48 smithi122 ceph-mon[36403]: Upgrade: Updating mgr.smithi103.pfvmui 2024-03-22T02:29:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:48 smithi122 ceph-mon[36403]: Deploying daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:29:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:48 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:48 smithi103 ceph-mon[29258]: Upgrade: Updating mgr.smithi103.pfvmui 2024-03-22T02:29:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:48 smithi103 ceph-mon[29258]: Deploying daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:29:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:48 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:29:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:49 smithi122 ceph-mon[36403]: pgmap v48: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 1.6 MiB/s wr, 441 op/s 2024-03-22T02:29:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:49 smithi103 ceph-mon[29258]: pgmap v48: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 1.6 MiB/s wr, 441 op/s 2024-03-22T02:29:51.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 -- 172.21.15.103:0/267759440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fcef0 msgr2=0x7f98fc0fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 --2- 172.21.15.103:0/267759440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fcef0 0x7f98fc0fd300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f98ec004940 tx=0x7f98ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:51.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 -- 172.21.15.103:0/267759440 shutdown_connections 2024-03-22T02:29:51.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 --2- 172.21.15.103:0/267759440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fe060 0x7f98fc0fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 --2- 172.21.15.103:0/267759440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fcef0 0x7f98fc0fd300 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 -- 172.21.15.103:0/267759440 >> 172.21.15.103:0/267759440 conn(0x7f98fc0f85b0 msgr2=0x7f98fc0fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 -- 172.21.15.103:0/267759440 shutdown_connections 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.038+0000 7f9902388700 1 -- 172.21.15.103:0/267759440 wait complete. 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 Processor -- start 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 -- start start 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fcef0 0x7f98fc10da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc10e5c0 con 0x7f98fc0fe060 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f9902388700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc10edc0 con 0x7f98fc0fcef0 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f98fb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f98fb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53746/0 (socket says 172.21.15.103:53746) 2024-03-22T02:29:51.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f98fb7fe700 1 -- 172.21.15.103:0/855406594 learned_addr learned my addr 172.21.15.103:0/855406594 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.039+0000 7f98fbfff700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fcef0 0x7f98fc10da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98fb7fe700 1 -- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fcef0 msgr2=0x7f98fc10da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98fb7fe700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fcef0 0x7f98fc10da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98fb7fe700 1 -- 172.21.15.103:0/855406594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec005040 con 0x7f98fc0fe060 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98fb7fe700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f98f0002a50 tx=0x7f98f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f0007cc0 con 0x7f98fc0fe060 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f000f040 con 0x7f98fc0fe060 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc10f0a0 con 0x7f98fc0fe060 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f000da30 con 0x7f98fc0fe060 2024-03-22T02:29:51.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.040+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc10f5e0 con 0x7f98fc0fe060 2024-03-22T02:29:51.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.041+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc05f5f0 con 0x7f98fc0fe060 2024-03-22T02:29:51.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.042+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f98f000db90 con 0x7f98fc0fe060 2024-03-22T02:29:51.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.042+0000 7f98f8ff9700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 0x7f98e406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.042+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f98f008a120 con 0x7f98fc0fe060 2024-03-22T02:29:51.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.042+0000 7f98fbfff700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 0x7f98e406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.043+0000 7f98fbfff700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 0x7f98e406dcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009fd0 tx=0x7f98ec009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.047+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f98f0057780 con 0x7f98fc0fe060 2024-03-22T02:29:51.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.284+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98fc10f960 con 0x7f98e406b800 2024-03-22T02:29:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.286+0000 7f98f8ff9700 1 -- 172.21.15.103:0/855406594 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f98fc10f960 con 0x7f98e406b800 2024-03-22T02:29:51.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 msgr2=0x7f98e406dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 0x7f98e406dcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009fd0 tx=0x7f98ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 msgr2=0x7f98fc10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f98f0002a50 tx=0x7f98f00046b0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 shutdown_connections 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f98e406b800 0x7f98e406dcb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98fc0fcef0 0x7f98fc10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 --2- 172.21.15.103:0/855406594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98fc0fe060 0x7f98fc10dfc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 >> 172.21.15.103:0/855406594 conn(0x7f98fc0f85b0 msgr2=0x7f98fc0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 shutdown_connections 2024-03-22T02:29:51.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.288+0000 7f9902388700 1 -- 172.21.15.103:0/855406594 wait complete. 2024-03-22T02:29:51.299 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:29:51.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 -- 172.21.15.103:0/2097597065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 msgr2=0x7fe0fc0fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 --2- 172.21.15.103:0/2097597065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc0fe810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8004940 tx=0x7fe0f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 -- 172.21.15.103:0/2097597065 shutdown_connections 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 --2- 172.21.15.103:0/2097597065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc0fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 --2- 172.21.15.103:0/2097597065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc0fe810 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.398+0000 7fe1032ce700 1 -- 172.21.15.103:0/2097597065 >> 172.21.15.103:0/2097597065 conn(0x7fe0fc0f6ba0 msgr2=0x7fe0fc0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 -- 172.21.15.103:0/2097597065 shutdown_connections 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 -- 172.21.15.103:0/2097597065 wait complete. 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 Processor -- start 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 -- start start 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc10e4c0 con 0x7fe0fc0fe400 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.399+0000 7fe1032ce700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc10ed10 con 0x7fe0fc0fed50 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53764/0 (socket says 172.21.15.103:53764) 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 -- 172.21.15.103:0/3963060397 learned_addr learned my addr 172.21.15.103:0/3963060397 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe100869700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 -- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 msgr2=0x7fe0fc10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc10dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 -- 172.21.15.103:0/3963060397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8005040 con 0x7fe0fc0fe400 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe100869700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe10106a700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8013870 tx=0x7fe0f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f8021070 con 0x7fe0fc0fe400 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc10ef90 con 0x7fe0fc0fe400 2024-03-22T02:29:51.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0f8009430 con 0x7fe0fc0fe400 2024-03-22T02:29:51.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.400+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f800e070 con 0x7fe0fc0fe400 2024-03-22T02:29:51.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.401+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc10f4d0 con 0x7fe0fc0fe400 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.401+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc05f5f0 con 0x7fe0fc0fe400 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.402+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) v1 ==== 89278+0+0 (secure 0 0 0) 0x7fe0f801c750 con 0x7fe0fc0fe400 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.402+0000 7fe0f1ffb700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 0x7fe0e806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.402+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fe0f808b3b0 con 0x7fe0fc0fe400 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.403+0000 7fe100869700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 0x7fe0e806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.403+0000 7fe100869700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 0x7fe0e806dc50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009b80 tx=0x7fe0ec00b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.407+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fe0f8058cf0 con 0x7fe0fc0fe400 2024-03-22T02:29:51.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.642+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0fc10f800 con 0x7fe0e806b7a0 2024-03-22T02:29:51.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.644+0000 7fe0f1ffb700 1 -- 172.21.15.103:0/3963060397 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe0fc10f800 con 0x7fe0e806b7a0 2024-03-22T02:29:51.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 msgr2=0x7fe0e806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 0x7fe0e806dc50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009b80 tx=0x7fe0ec00b040 comp rx=0 tx=0).stop 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 msgr2=0x7fe0fc10d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8013870 tx=0x7fe0f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 shutdown_connections 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fe0e806b7a0 0x7fe0e806dc50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0fc0fe400 0x7fe0fc10d980 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 --2- 172.21.15.103:0/3963060397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0fc0fed50 0x7fe0fc10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 >> 172.21.15.103:0/3963060397 conn(0x7fe0fc0f6ba0 msgr2=0x7fe0fc0f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 shutdown_connections 2024-03-22T02:29:51.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.646+0000 7fe1032ce700 1 -- 172.21.15.103:0/3963060397 wait complete. 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- 172.21.15.103:0/1267337557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 msgr2=0x7f397c0fd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 --2- 172.21.15.103:0/1267337557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c0fd0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f396c004940 tx=0x7f396c009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- 172.21.15.103:0/1267337557 shutdown_connections 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 --2- 172.21.15.103:0/1267337557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c0fd0d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 --2- 172.21.15.103:0/1267337557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c0fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- 172.21.15.103:0/1267337557 >> 172.21.15.103:0/1267337557 conn(0x7f397c0f6ba0 msgr2=0x7f397c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- 172.21.15.103:0/1267337557 shutdown_connections 2024-03-22T02:29:51.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- 172.21.15.103:0/1267337557 wait complete. 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 Processor -- start 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.759+0000 7f398392f700 1 -- start start 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f398392f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f398392f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f398392f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0fbcb0 con 0x7f397c0fcc60 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f398392f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0fbdf0 con 0x7f397c0fc310 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f3980eca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f3980eca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53780/0 (socket says 172.21.15.103:53780) 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f3980eca700 1 -- 172.21.15.103:0/3287363827 learned_addr learned my addr 172.21.15.103:0/3287363827 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f39816cb700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f39816cb700 1 -- 172.21.15.103:0/3287363827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 msgr2=0x7f397c112c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:51.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f39816cb700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:51.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f39816cb700 1 -- 172.21.15.103:0/3287363827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c005040 con 0x7f397c0fc310 2024-03-22T02:29:51.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f3980eca700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:51.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.760+0000 7f39816cb700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c110700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3978009fd0 tx=0x7f397800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.761+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3978004e90 con 0x7f397c0fc310 2024-03-22T02:29:51.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.761+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c113190 con 0x7f397c0fc310 2024-03-22T02:29:51.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.761+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f397800f040 con 0x7f397c0fc310 2024-03-22T02:29:51.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.761+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397800ca50 con 0x7f397c0fc310 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.761+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c113680 con 0x7f397c0fc310 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.762+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c05f5f0 con 0x7f397c0fc310 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.762+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 26) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f3978007cc0 con 0x7f397c0fc310 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.762+0000 7f3971ffb700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 0x7f396806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.763+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f3978089cd0 con 0x7f397c0fc310 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.763+0000 7f3980eca700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 0x7f396806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:51.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.763+0000 7f3980eca700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 0x7f396806dc60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f396c013740 tx=0x7f396c009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:51.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:51.771+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f3978050040 con 0x7f397c0fc310 2024-03-22T02:29:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:51 smithi122 ceph-mon[36403]: pgmap v49: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.3 MiB/s wr, 638 op/s 2024-03-22T02:29:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:51 smithi122 ceph-mon[36403]: mgrmap e26: smithi122.prgtti(active, since 92s), standbys: smithi103.pfvmui 2024-03-22T02:29:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:51 smithi103 ceph-mon[29258]: pgmap v49: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.3 MiB/s wr, 638 op/s 2024-03-22T02:29:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:51 smithi103 ceph-mon[29258]: mgrmap e26: smithi122.prgtti(active, since 92s), standbys: smithi103.pfvmui 2024-03-22T02:29:52.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.015+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f397c1013c0 con 0x7f396806b7b0 2024-03-22T02:29:52.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.033+0000 7f3971ffb700 1 -- 172.21.15.103:0/3287363827 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f397c1013c0 con 0x7f396806b7b0 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 starting - - - - 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (9m) 58s ago 9m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (8m) 82s ago 8m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 starting - - - - 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (4m) 58s ago 4m 11.7M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (3m) 58s ago 3m 109M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (3m) 82s ago 3m 13.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (4m) 82s ago 4m 100M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:9283 running (11m) 58s ago 11m 396M - 17.2.7-726-g9c4e0c34 837124da804f 58c5fbe605ad 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (119s) 82s ago 8m 489M - 19.0.0-2344-gc3a04c0e 9d92c22378ed c01e6142c155 2024-03-22T02:29:52.034 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (11m) 58s ago 11m 50.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (8m) 82s ago 8m 39.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 starting - - - - 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 starting - - - - 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (7m) 58s ago 7m 122M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (7m) 58s ago 7m 134M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (6m) 58s ago 6m 121M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (6m) 82s ago 6m 143M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (6m) 82s ago 6m 135M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (5m) 82s ago 5m 125M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:29:52.035 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 starting - - - - 2024-03-22T02:29:52.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 msgr2=0x7f396806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 0x7f396806dc60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f396c013740 tx=0x7f396c009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 msgr2=0x7f397c110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c110700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3978009fd0 tx=0x7f397800b040 comp rx=0 tx=0).stop 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 shutdown_connections 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f396806b7b0 0x7f396806dc60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f397c0fc310 0x7f397c110700 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3978009fd0 tx=0x7f397800b040 comp rx=0 tx=0).stop 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 --2- 172.21.15.103:0/3287363827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f397c0fcc60 0x7f397c112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.036+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 >> 172.21.15.103:0/3287363827 conn(0x7f397c0f6ba0 msgr2=0x7f397c0f8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.037+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 shutdown_connections 2024-03-22T02:29:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.037+0000 7f398392f700 1 -- 172.21.15.103:0/3287363827 wait complete. 2024-03-22T02:29:52.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 -- 172.21.15.103:0/2009699378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fcf80 msgr2=0x7f59c00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/2009699378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fcf80 0x7f59c00fd3f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f59b4004880 tx=0x7f59b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 -- 172.21.15.103:0/2009699378 shutdown_connections 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/2009699378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fcf80 0x7f59c00fd3f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/2009699378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fbcd0 0x7f59c00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 -- 172.21.15.103:0/2009699378 >> 172.21.15.103:0/2009699378 conn(0x7f59c00f7430 msgr2=0x7f59c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 -- 172.21.15.103:0/2009699378 shutdown_connections 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.195+0000 7f59c6a4e700 1 -- 172.21.15.103:0/2009699378 wait complete. 2024-03-22T02:29:52.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 Processor -- start 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 -- start start 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fcf80 0x7f59c0112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c0066350 con 0x7f59c00fbcd0 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59c6a4e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c0066490 con 0x7f59c00fcf80 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.196+0000 7f59bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53790/0 (socket says 172.21.15.103:53790) 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 -- 172.21.15.103:0/3210951778 learned_addr learned my addr 172.21.15.103:0/3210951778 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bf7fe700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fcf80 0x7f59c0112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 -- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fcf80 msgr2=0x7f59c0112c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fcf80 0x7f59c0112c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 -- 172.21.15.103:0/3210951778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0005040 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bffff700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f59b00096b0 tx=0x7f59b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b0017070 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59b000f040 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b4005040 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b0013be0 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.197+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c010c930 con 0x7f59c00fbcd0 2024-03-22T02:29:52.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.198+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c005f5f0 con 0x7f59c00fbcd0 2024-03-22T02:29:52.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.199+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) v1 ==== 89278+0+0 (secure 0 0 0) 0x7f59b000e840 con 0x7f59c00fbcd0 2024-03-22T02:29:52.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.199+0000 7f59bcff9700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 0x7f59a806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.199+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f59b008a120 con 0x7f59c00fbcd0 2024-03-22T02:29:52.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.200+0000 7f59bf7fe700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 0x7f59a806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.200+0000 7f59bf7fe700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 0x7f59a806dd30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4009e30 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:52.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.204+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f59b0053cd0 con 0x7f59c00fbcd0 2024-03-22T02:29:52.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.515+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f59c010cd00 con 0x7f59c00fbcd0 2024-03-22T02:29:52.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.515+0000 7f59bcff9700 1 -- 172.21.15.103:0/3210951778 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f59b0057390 con 0x7f59c00fbcd0 2024-03-22T02:29:52.516 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:29:52.517 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:29:52.518 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 13, 2024-03-22T02:29:52.518 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:29:52.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:29:52.518 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:29:52.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 msgr2=0x7f59a806dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 0x7f59a806dd30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 msgr2=0x7f59c0110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f59b00096b0 tx=0x7f59b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 shutdown_connections 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7f59a806b880 0x7f59a806dd30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59c00fbcd0 0x7f59c0110720 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 --2- 172.21.15.103:0/3210951778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59c00fcf80 0x7f59c0112c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 >> 172.21.15.103:0/3210951778 conn(0x7f59c00f7430 msgr2=0x7f59c00f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 shutdown_connections 2024-03-22T02:29:52.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.519+0000 7f59c6a4e700 1 -- 172.21.15.103:0/3210951778 wait complete. 2024-03-22T02:29:52.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 -- 172.21.15.103:0/2036255080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 msgr2=0x7fc8540fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 --2- 172.21.15.103:0/2036255080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc8540fb850 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc844004940 tx=0x7fc844009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 -- 172.21.15.103:0/2036255080 shutdown_connections 2024-03-22T02:29:52.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 --2- 172.21.15.103:0/2036255080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 0x7fc8540fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 --2- 172.21.15.103:0/2036255080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc8540fb850 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 -- 172.21.15.103:0/2036255080 >> 172.21.15.103:0/2036255080 conn(0x7fc8540f6ba0 msgr2=0x7fc8540f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:52.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 -- 172.21.15.103:0/2036255080 shutdown_connections 2024-03-22T02:29:52.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.649+0000 7fc859e55700 1 -- 172.21.15.103:0/2036255080 wait complete. 2024-03-22T02:29:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.650+0000 7fc859e55700 1 Processor -- start 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.650+0000 7fc859e55700 1 -- start start 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc859e55700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc859e55700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 0x7fc8541109d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc859e55700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541026e0 con 0x7fc8540fb440 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc859e55700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc854102820 con 0x7fc8540fc6f0 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53806/0 (socket says 172.21.15.103:53806) 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 -- 172.21.15.103:0/2611766713 learned_addr learned my addr 172.21.15.103:0/2611766713 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc852ffd700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 0x7fc8541109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 -- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 msgr2=0x7fc8541109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 0x7fc8541109d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.651+0000 7fc8537fe700 1 -- 172.21.15.103:0/2611766713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844005040 con 0x7fc8540fb440 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc8537fe700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc844000c00 tx=0x7fc844009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:52.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84400e070 con 0x7fc8540fb440 2024-03-22T02:29:52.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc854110f70 con 0x7fc8540fb440 2024-03-22T02:29:52.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc844009430 con 0x7fc8540fb440 2024-03-22T02:29:52.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84400e070 con 0x7fc8540fb440 2024-03-22T02:29:52.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.652+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc854111450 con 0x7fc8540fb440 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.653+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85405f5f0 con 0x7fc8540fb440 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.653+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) v1 ==== 89278+0+0 (secure 0 0 0) 0x7fc84401c7e0 con 0x7fc8540fb440 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.654+0000 7fc83bfff700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 0x7fc83c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.654+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fc84408b590 con 0x7fc8540fb440 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.654+0000 7fc852ffd700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 0x7fc83c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:52.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.655+0000 7fc852ffd700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 0x7fc83c06dd10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc848009b80 tx=0x7fc84800b040 comp rx=0 tx=0).ready entity=mgr.24383 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:52.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.659+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fc8440551c0 con 0x7fc8540fb440 2024-03-22T02:29:52.740 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:52 smithi103 ceph-mon[29258]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:52.740 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:52 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3210951778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:29:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:52 smithi122 ceph-mon[36403]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:52 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3210951778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:29:52.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.920+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 --> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc854100a60 con 0x7fc83c06b860 2024-03-22T02:29:52.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.922+0000 7fc83bfff700 1 -- 172.21.15.103:0/2611766713 <== mgr.24383 v2:172.21.15.122:6828/2536422926 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fc854100a60 con 0x7fc83c06b860 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "1/2 daemons upgraded", 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mgr daemons", 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:29:52.923 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:29:52.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 msgr2=0x7fc83c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 0x7fc83c06dd10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc848009b80 tx=0x7fc84800b040 comp rx=0 tx=0).stop 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 msgr2=0x7fc85410e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc844000c00 tx=0x7fc844009e30 comp rx=0 tx=0).stop 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 shutdown_connections 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:6828/2536422926,v1:172.21.15.122:6829/2536422926] conn(0x7fc83c06b860 0x7fc83c06dd10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8540fb440 0x7fc85410e480 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 --2- 172.21.15.103:0/2611766713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540fc6f0 0x7fc8541109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 >> 172.21.15.103:0/2611766713 conn(0x7fc8540f6ba0 msgr2=0x7fc8540f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 shutdown_connections 2024-03-22T02:29:52.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:29:52.925+0000 7fc859e55700 1 -- 172.21.15.103:0/2611766713 wait complete. 2024-03-22T02:29:53.773 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:53 smithi103 ceph-mon[29258]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:53.773 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:53 smithi103 ceph-mon[29258]: pgmap v50: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.5 MiB/s wr, 389 op/s 2024-03-22T02:29:53.773 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:53 smithi103 ceph-mon[29258]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:53 smithi122 ceph-mon[36403]: from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:53 smithi122 ceph-mon[36403]: pgmap v50: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.5 MiB/s wr, 389 op/s 2024-03-22T02:29:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:53 smithi122 ceph-mon[36403]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:54 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:54 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:55.713 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:55 smithi103 ceph-mon[29258]: pgmap v51: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.5 MiB/s wr, 406 op/s 2024-03-22T02:29:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:55 smithi122 ceph-mon[36403]: pgmap v51: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.5 MiB/s wr, 406 op/s 2024-03-22T02:29:57.824 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:57 smithi103 ceph-mon[29258]: pgmap v52: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 2.0 MiB/s wr, 516 op/s 2024-03-22T02:29:57.824 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:57.824 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:57 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:57 smithi122 ceph-mon[36403]: pgmap v52: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 2.0 MiB/s wr, 516 op/s 2024-03-22T02:29:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:57.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:57 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:29:59.699 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:29:59 smithi103 ceph-mon[29258]: pgmap v53: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.2 MiB/s wr, 345 op/s 2024-03-22T02:29:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:29:59 smithi122 ceph-mon[36403]: pgmap v53: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 33 KiB/s rd, 1.2 MiB/s wr, 345 op/s 2024-03-22T02:30:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:00 smithi122 ceph-mon[36403]: overall HEALTH_OK 2024-03-22T02:30:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:00 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:00 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:00 smithi103 ceph-mon[29258]: overall HEALTH_OK 2024-03-22T02:30:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:00 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:00 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:01.749 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:01 smithi103 ceph-mon[29258]: pgmap v54: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 47 KiB/s rd, 1.7 MiB/s wr, 474 op/s 2024-03-22T02:30:01.749 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:01 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:01.749 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:01 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:01 smithi122 ceph-mon[36403]: pgmap v54: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 47 KiB/s rd, 1.7 MiB/s wr, 474 op/s 2024-03-22T02:30:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:01 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:01 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: Standby manager daemon smithi103.pfvmui restarted 2024-03-22T02:30:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: Standby manager daemon smithi103.pfvmui started 2024-03-22T02:30:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/crt"}]: dispatch 2024-03-22T02:30:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/key"}]: dispatch 2024-03-22T02:30:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:02 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: Standby manager daemon smithi103.pfvmui restarted 2024-03-22T02:30:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: Standby manager daemon smithi103.pfvmui started 2024-03-22T02:30:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/crt"}]: dispatch 2024-03-22T02:30:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.pfvmui/key"}]: dispatch 2024-03-22T02:30:02.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:02 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:03 smithi122 ceph-mon[36403]: pgmap v55: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.0 MiB/s wr, 277 op/s 2024-03-22T02:30:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:03 smithi122 ceph-mon[36403]: mgrmap e27: smithi122.prgtti(active, since 104s), standbys: smithi103.pfvmui 2024-03-22T02:30:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:03 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:03 smithi103 ceph-mon[29258]: pgmap v55: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.0 MiB/s wr, 277 op/s 2024-03-22T02:30:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:03 smithi103 ceph-mon[29258]: mgrmap e27: smithi122.prgtti(active, since 104s), standbys: smithi103.pfvmui 2024-03-22T02:30:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:03 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:05 smithi122 ceph-mon[36403]: pgmap v56: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.0 MiB/s wr, 277 op/s 2024-03-22T02:30:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:05 smithi103 ceph-mon[29258]: pgmap v56: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 1.0 MiB/s wr, 277 op/s 2024-03-22T02:30:07.707 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:07 smithi103 ceph-mon[29258]: pgmap v57: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 1.5 MiB/s wr, 399 op/s 2024-03-22T02:30:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:07 smithi122 ceph-mon[36403]: pgmap v57: 65 pgs: 65 active+clean; 271 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 1.5 MiB/s wr, 399 op/s 2024-03-22T02:30:09.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: osdmap e48: 6 total, 6 up, 6 in 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd='[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]': finished 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: mgrmap e28: smithi103.pfvmui(active, starting, since 0.00567246s) 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:30:09.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:30:09.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:09.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:09.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:30:09.102 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:30:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 172.21.15.122:0/1480078760' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: osdmap e48: 6 total, 6 up, 6 in 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.24383 ' entity='mgr.smithi122.prgtti' cmd='[{"prefix": "mgr fail", "who": "smithi122.prgtti"}]': finished 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: mgrmap e28: smithi103.pfvmui(active, starting, since 0.00567246s) 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:30:09.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:30:09.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:09.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:09.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:30:09.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:30:10.006 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:09 smithi122 ceph-mon[36403]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T02:30:10.006 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:09 smithi122 ceph-mon[36403]: mgrmap e29: smithi103.pfvmui(active, since 1.01095s) 2024-03-22T02:30:10.006 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:09 smithi122 ceph-mon[36403]: pgmap v3: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:09 smithi103 ceph-mon[29258]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T02:30:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:09 smithi103 ceph-mon[29258]: mgrmap e29: smithi103.pfvmui(active, since 1.01095s) 2024-03-22T02:30:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:09 smithi103 ceph-mon[29258]: pgmap v3: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: [22/Mar/2024:02:30:09] ENGINE Bus STARTING 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: [22/Mar/2024:02:30:09] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: [22/Mar/2024:02:30:09] ENGINE Serving on http://172.21.15.103:8765 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: [22/Mar/2024:02:30:09] ENGINE Bus STARTED 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: Cluster is now healthy 2024-03-22T02:30:11.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:10 smithi122 ceph-mon[36403]: pgmap v4: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: [22/Mar/2024:02:30:09] ENGINE Bus STARTING 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: [22/Mar/2024:02:30:09] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: [22/Mar/2024:02:30:09] ENGINE Serving on http://172.21.15.103:8765 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: [22/Mar/2024:02:30:09] ENGINE Bus STARTED 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: Cluster is now healthy 2024-03-22T02:30:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:10 smithi103 ceph-mon[29258]: pgmap v4: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:11 smithi103 ceph-mon[29258]: mgrmap e30: smithi103.pfvmui(active, since 2s) 2024-03-22T02:30:11.911 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-22T02:30:11.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-22T02:30:12.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:11 smithi122 ceph-mon[36403]: mgrmap e30: smithi103.pfvmui(active, since 2s) 2024-03-22T02:30:12.258 DEBUG:teuthology.parallel:result is None 2024-03-22T02:30:12.939 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:12 smithi122 ceph-mon[36403]: pgmap v5: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:12 smithi103 ceph-mon[29258]: pgmap v5: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:14.007 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:13 smithi122 ceph-mon[36403]: mgrmap e31: smithi103.pfvmui(active, since 4s) 2024-03-22T02:30:14.007 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:13 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:14.007 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:13 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:13 smithi103 ceph-mon[29258]: mgrmap e31: smithi103.pfvmui(active, since 4s) 2024-03-22T02:30:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:13 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:13 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.785 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:15 smithi122 ceph-mon[36403]: pgmap v6: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:15.785 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.785 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.785 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.785 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:15 smithi103 ceph-mon[29258]: pgmap v6: 65 pgs: 65 active+clean; 266 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:16.833 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:16 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:16.833 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:16 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:16.833 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:16 smithi122 ceph-mon[36403]: pgmap v7: 65 pgs: 65 active+clean; 263 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 51 KiB/s rd, 985 KiB/s wr, 272 op/s 2024-03-22T02:30:16.834 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-22T02:30:16.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-22T02:30:16.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:16 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:16.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:16 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:16.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:16 smithi103 ceph-mon[29258]: pgmap v7: 65 pgs: 65 active+clean; 263 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 51 KiB/s rd, 985 KiB/s wr, 272 op/s 2024-03-22T02:30:17.121 DEBUG:teuthology.parallel:result is None 2024-03-22T02:30:17.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T02:30:17.203 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T02:30:17.204 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-22T02:30:17.253 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-22T02:30:17.254 DEBUG:teuthology.parallel:result is None 2024-03-22T02:30:17.872 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:17 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:30:17.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:17 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:30:17.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:17 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:17.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:17 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:30:17.873 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:17 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:17 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:30:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:17 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:30:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:17 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:17 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:30:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:17 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/157573602' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:18.940 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:18 smithi103 ceph-mon[29258]: mgrmap e32: smithi103.pfvmui(active, since 9s), standbys: smithi122.prgtti 2024-03-22T02:30:18.940 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:18 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:18.940 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:18 smithi103 ceph-mon[29258]: pgmap v8: 65 pgs: 65 active+clean; 263 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 766 KiB/s wr, 211 op/s 2024-03-22T02:30:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:18 smithi122 ceph-mon[36403]: mgrmap e32: smithi103.pfvmui(active, since 9s), standbys: smithi122.prgtti 2024-03-22T02:30:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:18 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:30:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:18 smithi122 ceph-mon[36403]: pgmap v8: 65 pgs: 65 active+clean; 263 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 766 KiB/s wr, 211 op/s 2024-03-22T02:30:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:21 smithi122 ceph-mon[36403]: pgmap v9: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 938 KiB/s wr, 304 op/s 2024-03-22T02:30:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:21 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:21 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:21 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:21 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:21 smithi103 ceph-mon[29258]: pgmap v9: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 938 KiB/s wr, 304 op/s 2024-03-22T02:30:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:21 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:21 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:21 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:21 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:22 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:30:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:30:22.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:22.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:22 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 -- 172.21.15.103:0/162629332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fbcd0 msgr2=0x7f62c00fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 --2- 172.21.15.103:0/162629332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fbcd0 0x7f62c00fc0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f62b4004940 tx=0x7f62b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 -- 172.21.15.103:0/162629332 shutdown_connections 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 --2- 172.21.15.103:0/162629332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fcf80 0x7f62c00fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 --2- 172.21.15.103:0/162629332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fbcd0 0x7f62c00fc0e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 -- 172.21.15.103:0/162629332 >> 172.21.15.103:0/162629332 conn(0x7f62c00f7430 msgr2=0x7f62c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 -- 172.21.15.103:0/162629332 shutdown_connections 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.033+0000 7f62c6b65700 1 -- 172.21.15.103:0/162629332 wait complete. 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 Processor -- start 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 -- start start 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fbcd0 0x7f62c010d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c010e4d0 con 0x7f62c00fcf80 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c6b65700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c010ed20 con 0x7f62c00fbcd0 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62bffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59188/0 (socket says 172.21.15.103:59188) 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62bffff700 1 -- 172.21.15.103:0/8107030 learned_addr learned my addr 172.21.15.103:0/8107030 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62c4901700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fbcd0 0x7f62c010d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62bffff700 1 -- 172.21.15.103:0/8107030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fbcd0 msgr2=0x7f62c010d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.034+0000 7f62bffff700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fbcd0 0x7f62c010d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62bffff700 1 -- 172.21.15.103:0/8107030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b4005040 con 0x7f62c00fcf80 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62bffff700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00eb70 tx=0x7f62ac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62ac007cc0 con 0x7f62c00fcf80 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c010efa0 con 0x7f62c00fcf80 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62ac00f040 con 0x7f62c00fcf80 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62ac009640 con 0x7f62c00fcf80 2024-03-22T02:30:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.035+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c010f4e0 con 0x7f62c00fcf80 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.036+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c005f5f0 con 0x7f62c00fcf80 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.036+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 32) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f62ac0097a0 con 0x7f62c00fcf80 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.037+0000 7f62bd7fa700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 0x7f62b0070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.037+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f62ac08d330 con 0x7f62c00fcf80 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.037+0000 7f62c4901700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 0x7f62b0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.037+0000 7f62c4901700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 0x7f62b0070270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009b30 tx=0x7f62b4009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.042+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f62ac05a6e0 con 0x7f62c00fcf80 2024-03-22T02:30:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:23 smithi103 ceph-mon[29258]: pgmap v10: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 870 KiB/s wr, 282 op/s 2024-03-22T02:30:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:23 smithi103 ceph-mon[29258]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:30:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:23 smithi103 ceph-mon[29258]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:30:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:23 smithi103 ceph-mon[29258]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T02:30:23.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.265+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62c010f7a0 con 0x7f62b006ddc0 2024-03-22T02:30:23.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.266+0000 7f62bd7fa700 1 -- 172.21.15.103:0/8107030 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f62c010f7a0 con 0x7f62b006ddc0 2024-03-22T02:30:23.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 msgr2=0x7f62b0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 0x7f62b0070270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009b30 tx=0x7f62b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 msgr2=0x7f62c010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00eb70 tx=0x7f62ac00b040 comp rx=0 tx=0).stop 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 shutdown_connections 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f62b006ddc0 0x7f62b0070270 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62c00fbcd0 0x7f62c010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 --2- 172.21.15.103:0/8107030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62c00fcf80 0x7f62c010ded0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 >> 172.21.15.103:0/8107030 conn(0x7f62c00f7430 msgr2=0x7f62c00f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 shutdown_connections 2024-03-22T02:30:23.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.268+0000 7f62c6b65700 1 -- 172.21.15.103:0/8107030 wait complete. 2024-03-22T02:30:23.279 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:30:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:23 smithi122 ceph-mon[36403]: pgmap v10: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 870 KiB/s wr, 282 op/s 2024-03-22T02:30:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:23 smithi122 ceph-mon[36403]: Reconfiguring prometheus.smithi103 (dependencies changed)... 2024-03-22T02:30:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:23 smithi122 ceph-mon[36403]: Reconfiguring daemon prometheus.smithi103 on smithi103 2024-03-22T02:30:23.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:23 smithi122 ceph-mon[36403]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- 172.21.15.103:0/86599843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 msgr2=0x7f8c840ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/86599843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c840ffa40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c74004940 tx=0x7f8c74009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- 172.21.15.103:0/86599843 shutdown_connections 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/86599843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 0x7f8c84102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/86599843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c840ffa40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- 172.21.15.103:0/86599843 >> 172.21.15.103:0/86599843 conn(0x7f8c840f7430 msgr2=0x7f8c840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- 172.21.15.103:0/86599843 shutdown_connections 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- 172.21.15.103:0/86599843 wait complete. 2024-03-22T02:30:23.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 Processor -- start 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.372+0000 7f8c8a91d700 1 -- start start 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 0x7f8c8410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8410e580 con 0x7f8c84066260 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8410ed80 con 0x7f8c840fff80 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59210/0 (socket says 172.21.15.103:59210) 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 -- 172.21.15.103:0/1585749948 learned_addr learned my addr 172.21.15.103:0/1585749948 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c837fe700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 0x7f8c8410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 -- 172.21.15.103:0/1585749948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 msgr2=0x7f8c8410df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 0x7f8c8410df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 -- 172.21.15.103:0/1585749948 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74005040 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c83fff700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c74000c00 tx=0x7f8c74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c7400e030 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c74013de0 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8410f000 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c7401ce50 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.373+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8410f540 con 0x7f8c84066260 2024-03-22T02:30:23.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.374+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8405f5f0 con 0x7f8c84066260 2024-03-22T02:30:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.375+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 32) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f8c7401c400 con 0x7f8c84066260 2024-03-22T02:30:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.375+0000 7f8c80ff9700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 0x7f8c6c074680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.375+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f8c7408ef60 con 0x7f8c84066260 2024-03-22T02:30:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.375+0000 7f8c837fe700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 0x7f8c6c074680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.376+0000 7f8c837fe700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 0x7f8c6c074680 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009b80 tx=0x7f8c7800b040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.380+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f8c740c3040 con 0x7f8c84066260 2024-03-22T02:30:23.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.599+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c8410f7d0 con 0x7f8c6c0721d0 2024-03-22T02:30:23.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.600+0000 7f8c80ff9700 1 -- 172.21.15.103:0/1585749948 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f8c8410f7d0 con 0x7f8c6c0721d0 2024-03-22T02:30:23.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 msgr2=0x7f8c6c074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 0x7f8c6c074680 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009b80 tx=0x7f8c7800b040 comp rx=0 tx=0).stop 2024-03-22T02:30:23.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 msgr2=0x7f8c8410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c74000c00 tx=0x7f8c74009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 shutdown_connections 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f8c6c0721d0 0x7f8c6c074680 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c84066260 0x7f8c8410da40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 --2- 172.21.15.103:0/1585749948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c840fff80 0x7f8c8410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.603+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 >> 172.21.15.103:0/1585749948 conn(0x7f8c840f7430 msgr2=0x7f8c840f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.604+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 shutdown_connections 2024-03-22T02:30:23.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.604+0000 7f8c8a91d700 1 -- 172.21.15.103:0/1585749948 wait complete. 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.706+0000 7f907545f700 1 -- 172.21.15.103:0/4272941116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90700fce40 msgr2=0x7f90700fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.706+0000 7f907545f700 1 --2- 172.21.15.103:0/4272941116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90700fce40 0x7f90700fd250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9060004940 tx=0x7f9060009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- 172.21.15.103:0/4272941116 shutdown_connections 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 --2- 172.21.15.103:0/4272941116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 0x7f90700fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 --2- 172.21.15.103:0/4272941116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90700fce40 0x7f90700fd250 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- 172.21.15.103:0/4272941116 >> 172.21.15.103:0/4272941116 conn(0x7f90700f8510 msgr2=0x7f90700fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- 172.21.15.103:0/4272941116 shutdown_connections 2024-03-22T02:30:23.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- 172.21.15.103:0/4272941116 wait complete. 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 Processor -- start 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- start start 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 0x7f907010db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907010e550 con 0x7f907010e090 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.707+0000 7f907545f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907010e690 con 0x7f90700fe040 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59222/0 (socket says 172.21.15.103:59222) 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 -- 172.21.15.103:0/2977158789 learned_addr learned my addr 172.21.15.103:0/2977158789 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906effd700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 0x7f907010db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 -- 172.21.15.103:0/2977158789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 msgr2=0x7f907010db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 0x7f907010db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 -- 172.21.15.103:0/2977158789 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060005040 con 0x7f907010e090 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f906e7fc700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9064002a80 tx=0x7f90640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9064007cc0 con 0x7f907010e090 2024-03-22T02:30:23.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f906400f040 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907010f540 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90640096f0 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.708+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907010fad0 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.709+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90701060e0 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.710+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 32) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f906400d690 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.710+0000 7f90577fe700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 0x7f9058070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.710+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f906408ce60 con 0x7f907010e090 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.710+0000 7f906effd700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 0x7f9058070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.711+0000 7f906effd700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 0x7f9058070270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f906000b840 tx=0x7f9060009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.715+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f906405a210 con 0x7f907010e090 2024-03-22T02:30:23.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.928+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9070105a80 con 0x7f905806ddc0 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.938+0000 7f90577fe700 1 -- 172.21.15.103:0/2977158789 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9060007cc0 con 0x7f905806ddc0 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (79s) 9s ago 10m 27.7M - 0.25.0 c8568f914cd2 44d2a25d6238 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (10m) 9s ago 10m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (9m) 10s ago 9m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 unknown 9s ago 9m - - 2024-03-22T02:30:23.939 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (4m) 9s ago 4m 12.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (4m) 9s ago 4m 132M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (4m) 10s ago 4m 13.8M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (4m) 10s ago 4m 131M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (30s) 9s ago 11m 476M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283 running (2m) 10s ago 9m 68.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed c01e6142c155 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (11m) 9s ago 11m 53.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (9m) 10s ago 9m 44.4M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (84s) 9s ago 10m 15.5M - 1.5.0 0da6a335fe13 f7307d2f541a 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (39s) 10s ago 9m 10.2M - 1.5.0 0da6a335fe13 48829ffe4faf 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (8m) 9s ago 8m 205M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (7m) 9s ago 7m 226M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (7m) 9s ago 7m 192M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (7m) 10s ago 7m 255M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:30:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (6m) 10s ago 6m 226M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:30:23.941 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (6m) 10s ago 6m 218M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:30:23.941 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (41s) 9s ago 9m 44.8M - 2.43.0 a07b618ecd1d a65fe440918b 2024-03-22T02:30:23.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 msgr2=0x7f9058070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 0x7f9058070270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f906000b840 tx=0x7f9060009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 msgr2=0x7f907010eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9064002a80 tx=0x7f90640046b0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 shutdown_connections 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f905806ddc0 0x7f9058070270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90700fe040 0x7f907010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 --2- 172.21.15.103:0/2977158789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f907010e090 0x7f907010eee0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 >> 172.21.15.103:0/2977158789 conn(0x7f90700f8510 msgr2=0x7f90700f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 shutdown_connections 2024-03-22T02:30:23.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:23.941+0000 7f907545f700 1 -- 172.21.15.103:0/2977158789 wait complete. 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 -- 172.21.15.103:0/411417533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 msgr2=0x7f92440fcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 --2- 172.21.15.103:0/411417533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f92440fcbb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9238004a50 tx=0x7f923800af40 comp rx=0 tx=0).stop 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 -- 172.21.15.103:0/411417533 shutdown_connections 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 --2- 172.21.15.103:0/411417533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f92440fcbb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 --2- 172.21.15.103:0/411417533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f92440fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 -- 172.21.15.103:0/411417533 >> 172.21.15.103:0/411417533 conn(0x7f92440f6c10 msgr2=0x7f92440f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 -- 172.21.15.103:0/411417533 shutdown_connections 2024-03-22T02:30:24.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.042+0000 7f92494ae700 1 -- 172.21.15.103:0/411417533 wait complete. 2024-03-22T02:30:24.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 Processor -- start 2024-03-22T02:30:24.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 -- start start 2024-03-22T02:30:24.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f924410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924410e4f0 con 0x7f92440fc740 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92494ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9244112e90 con 0x7f92440fb490 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34862/0 (socket says 172.21.15.103:34862) 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 -- 172.21.15.103:0/2191718899 learned_addr learned my addr 172.21.15.103:0/2191718899 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f9242ffd700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f924410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 -- 172.21.15.103:0/2191718899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 msgr2=0x7f924410d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f924410d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f92427fc700 1 -- 172.21.15.103:0/2191718899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9238005040 con 0x7f92440fc740 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.043+0000 7f9242ffd700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f924410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f92427fc700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f923800c3b0 tx=0x7f923800af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f923800f050 con 0x7f92440fc740 2024-03-22T02:30:24.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f923800cad0 con 0x7f92440fc740 2024-03-22T02:30:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9244113090 con 0x7f92440fc740 2024-03-22T02:30:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9238014960 con 0x7f92440fc740 2024-03-22T02:30:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.044+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92441134b0 con 0x7f92440fc740 2024-03-22T02:30:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.045+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924404c710 con 0x7f92440fc740 2024-03-22T02:30:24.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.045+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 32) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f9238014ac0 con 0x7f92440fc740 2024-03-22T02:30:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.046+0000 7f922b7fe700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 0x7f922c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.046+0000 7f9242ffd700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 0x7f922c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.046+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f923808f380 con 0x7f92440fc740 2024-03-22T02:30:24.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.046+0000 7f9242ffd700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 0x7f922c070280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:24.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.050+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f923805c850 con 0x7f92440fc740 2024-03-22T02:30:24.256 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:24 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:24.256 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:24 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:24.257 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:24 smithi103 ceph-mon[29258]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:24.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.336+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9244103770 con 0x7f92440fc740 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.337+0000 7f922b7fe700 1 -- 172.21.15.103:0/2191718899 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f923805c3e0 con 0x7f92440fc740 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:30:24.338 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:30:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 msgr2=0x7f922c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 0x7f922c070280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).stop 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 msgr2=0x7f924410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f923800c3b0 tx=0x7f923800af40 comp rx=0 tx=0).stop 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 shutdown_connections 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f922c06ddd0 0x7f922c070280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92440fb490 0x7f924410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 --2- 172.21.15.103:0/2191718899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92440fc740 0x7f924410de60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 >> 172.21.15.103:0/2191718899 conn(0x7f92440f6c10 msgr2=0x7f92440f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 shutdown_connections 2024-03-22T02:30:24.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.339+0000 7f92494ae700 1 -- 172.21.15.103:0/2191718899 wait complete. 2024-03-22T02:30:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:24 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:24 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:24 smithi122 ceph-mon[36403]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:24.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.442+0000 7fa0b7d88700 1 -- 172.21.15.103:0/1265608790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 msgr2=0x7fa0b00fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.442+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/1265608790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b00fd430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0004940 tx=0x7fa0a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.442+0000 7fa0b7d88700 1 -- 172.21.15.103:0/1265608790 shutdown_connections 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/1265608790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b00fd430 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/1265608790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b00fc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- 172.21.15.103:0/1265608790 >> 172.21.15.103:0/1265608790 conn(0x7fa0b00f7490 msgr2=0x7fa0b00f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- 172.21.15.103:0/1265608790 shutdown_connections 2024-03-22T02:30:24.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- 172.21.15.103:0/1265608790 wait complete. 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 Processor -- start 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- start start 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b0066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0112c60 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.443+0000 7fa0b7d88700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0112da0 con 0x7fa0b00fbd10 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34890/0 (socket says 172.21.15.103:34890) 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 -- 172.21.15.103:0/2687745216 learned_addr learned my addr 172.21.15.103:0/2687745216 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5b24700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b0066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 -- 172.21.15.103:0/2687745216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 msgr2=0x7fa0b0066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b0066cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 -- 172.21.15.103:0/2687745216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0005040 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5b24700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b0066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b5323700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000b3b0 tx=0x7fa0a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a000e040 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0a000bcc0 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0112fa0 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a001c630 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.444+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b0113530 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.445+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b004c710 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.446+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 32) v1 ==== 92035+0+0 (secure 0 0 0) 0x7fa0a0013980 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.446+0000 7fa0a67fc700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 0x7fa09c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.446+0000 7fa0b5b24700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 0x7fa09c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.446+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa0a008e2e0 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.447+0000 7fa0b5b24700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 0x7fa09c070280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0041d0 tx=0x7fa0ac009040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:24.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.451+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fa0a005b710 con 0x7fa0b00fcfc0 2024-03-22T02:30:24.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.669+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b0101300 con 0x7fa09c06ddd0 2024-03-22T02:30:24.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.670+0000 7fa0a67fc700 1 -- 172.21.15.103:0/2687745216 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fa0b0101300 con 0x7fa09c06ddd0 2024-03-22T02:30:24.670 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr" 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "", 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:30:24.671 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 msgr2=0x7fa09c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 0x7fa09c070280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0041d0 tx=0x7fa0ac009040 comp rx=0 tx=0).stop 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 msgr2=0x7fa0b0112720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000b3b0 tx=0x7fa0a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 shutdown_connections 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fa09c06ddd0 0x7fa09c070280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa0b00fbd10 0x7fa0b0066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 --2- 172.21.15.103:0/2687745216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0b00fcfc0 0x7fa0b0112720 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 >> 172.21.15.103:0/2687745216 conn(0x7fa0b00f7490 msgr2=0x7fa0b00f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 shutdown_connections 2024-03-22T02:30:24.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:24.672+0000 7fa0b7d88700 1 -- 172.21.15.103:0/2687745216 wait complete. 2024-03-22T02:30:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:25 smithi103 ceph-mon[29258]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:25 smithi103 ceph-mon[29258]: pgmap v11: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 870 KiB/s wr, 282 op/s 2024-03-22T02:30:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:25 smithi103 ceph-mon[29258]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:25 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2191718899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:25 smithi122 ceph-mon[36403]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:25 smithi122 ceph-mon[36403]: pgmap v11: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 870 KiB/s wr, 282 op/s 2024-03-22T02:30:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:25 smithi122 ceph-mon[36403]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:25 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2191718899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:26 smithi122 ceph-mon[36403]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:26 smithi103 ceph-mon[29258]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:27.316 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:27 smithi103 ceph-mon[29258]: pgmap v12: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 884 KiB/s wr, 283 op/s 2024-03-22T02:30:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:27 smithi122 ceph-mon[36403]: pgmap v12: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 884 KiB/s wr, 283 op/s 2024-03-22T02:30:28.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:28 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:28.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:28 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:28.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:28 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:30:28.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:28 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:28 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:28 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:28 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:30:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:28 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:29.722 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:29 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:30:29.722 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:29 smithi122 ceph-mon[36403]: pgmap v13: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 303 KiB/s wr, 122 op/s 2024-03-22T02:30:29.722 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:30:29.722 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:30:29.722 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:29 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:30:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:29 smithi103 ceph-mon[29258]: pgmap v13: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 303 KiB/s wr, 122 op/s 2024-03-22T02:30:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:30:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:30:29.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:30.818 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:30 smithi122 ceph-mon[36403]: Upgrade: Updating mgr.smithi122.prgtti 2024-03-22T02:30:30.819 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:30 smithi122 ceph-mon[36403]: Deploying daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:30:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:30 smithi103 ceph-mon[29258]: Upgrade: Updating mgr.smithi122.prgtti 2024-03-22T02:30:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:30 smithi103 ceph-mon[29258]: Deploying daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:30:31.836 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:31 smithi103 ceph-mon[29258]: pgmap v14: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 303 KiB/s wr, 123 op/s 2024-03-22T02:30:31.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:31 smithi122 ceph-mon[36403]: pgmap v14: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 13 KiB/s rd, 303 KiB/s wr, 123 op/s 2024-03-22T02:30:33.774 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:33 smithi103 ceph-mon[29258]: pgmap v15: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 259 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:33.774 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:33.774 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:33.774 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:33.803 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:33 smithi122 ceph-mon[36403]: pgmap v15: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 259 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:33.803 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:33.803 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:33.803 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:35 smithi122 ceph-mon[36403]: pgmap v16: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 255 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:35 smithi103 ceph-mon[29258]: pgmap v16: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 255 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:37 smithi122 ceph-mon[36403]: pgmap v17: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 255 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:37 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:37 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:37 smithi103 ceph-mon[29258]: pgmap v17: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 255 B/s rd, 14 KiB/s wr, 1 op/s 2024-03-22T02:30:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:37 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:37 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:39.680 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:39 smithi122 ceph-mon[36403]: pgmap v18: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-22T02:30:39.680 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:39 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:39.681 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:39 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:39.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:39 smithi103 ceph-mon[29258]: pgmap v18: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-22T02:30:39.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:39 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:39.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:39 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:30:40.672 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:40 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:30:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:40 smithi103 ceph-mon[29258]: from='mgr.? 172.21.15.122:0/2907240813' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:30:41.775 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: pgmap v19: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: mgrmap e33: smithi103.pfvmui(active, since 32s), standbys: smithi122.prgtti 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi103.pfvmui"}]: dispatch 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi103.pfvmui"}]': finished 2024-03-22T02:30:41.776 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.prgtti"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.prgtti"}]': finished 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.777 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.778 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:41 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: pgmap v19: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 85 B/s rd, 255 B/s wr, 0 op/s 2024-03-22T02:30:41.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: mgrmap e33: smithi103.pfvmui(active, since 32s), standbys: smithi122.prgtti 2024-03-22T02:30:41.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi103.pfvmui"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi103.pfvmui"}]': finished 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.prgtti"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.prgtti"}]': finished 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:41.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:41 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: pgmap v20: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: pgmap v21: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all mgr 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all rgw 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all iscsi 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all nfs 2024-03-22T02:30:42.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:42 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all nvmeof 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: pgmap v20: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: pgmap v21: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all mgr 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all rgw 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T02:30:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all iscsi 2024-03-22T02:30:42.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all nfs 2024-03-22T02:30:42.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:42 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all nvmeof 2024-03-22T02:30:44.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:44 smithi103 ceph-mon[29258]: Upgrade: Updating node-exporter.smithi103 (1/2) 2024-03-22T02:30:44.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:44 smithi103 ceph-mon[29258]: Deploying daemon node-exporter.smithi103 on smithi103 2024-03-22T02:30:44.844 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:44 smithi103 ceph-mon[29258]: pgmap v22: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:44 smithi122 ceph-mon[36403]: Upgrade: Updating node-exporter.smithi103 (1/2) 2024-03-22T02:30:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:44 smithi122 ceph-mon[36403]: Deploying daemon node-exporter.smithi103 on smithi103 2024-03-22T02:30:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:44 smithi122 ceph-mon[36403]: pgmap v22: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:46 smithi122 ceph-mon[36403]: pgmap v23: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:46 smithi103 ceph-mon[29258]: pgmap v23: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:47 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:47.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:47 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:47.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:47 smithi122 ceph-mon[36403]: pgmap v24: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:47 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:47 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:47 smithi103 ceph-mon[29258]: pgmap v24: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:48.945 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:48 smithi122 ceph-mon[36403]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-03-22T02:30:48.945 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:48 smithi122 ceph-mon[36403]: Deploying daemon node-exporter.smithi122 on smithi122 2024-03-22T02:30:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:48 smithi103 ceph-mon[29258]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-03-22T02:30:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:48 smithi103 ceph-mon[29258]: Deploying daemon node-exporter.smithi122 on smithi122 2024-03-22T02:30:49.710 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:49 smithi122 ceph-mon[36403]: pgmap v25: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:49 smithi103 ceph-mon[29258]: pgmap v25: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:51 smithi122 ceph-mon[36403]: pgmap v26: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:51 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:51 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:51 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:51 smithi103 ceph-mon[29258]: pgmap v26: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:51 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:51 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:51 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:53 smithi122 ceph-mon[36403]: pgmap v27: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:53 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:53 smithi103 ceph-mon[29258]: pgmap v27: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:53 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 -- 172.21.15.103:0/2267792201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fe050 msgr2=0x7fe2c80fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/2267792201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fe050 0x7fe2c80fe4c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8004940 tx=0x7fe2b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 -- 172.21.15.103:0/2267792201 shutdown_connections 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/2267792201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fe050 0x7fe2c80fe4c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/2267792201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fce50 0x7fe2c80fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:54.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 -- 172.21.15.103:0/2267792201 >> 172.21.15.103:0/2267792201 conn(0x7fe2c80f8580 msgr2=0x7fe2c80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 -- 172.21.15.103:0/2267792201 shutdown_connections 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.776+0000 7fe2d06e2700 1 -- 172.21.15.103:0/2267792201 wait complete. 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 Processor -- start 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 -- start start 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fe050 0x7fe2c810dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:54.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810e3e0 con 0x7fe2c80fce50 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2d06e2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810e550 con 0x7fe2c80fe050 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60808/0 (socket says 172.21.15.103:60808) 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 -- 172.21.15.103:0/374427996 learned_addr learned my addr 172.21.15.103:0/374427996 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2cdc7d700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fe050 0x7fe2c810dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:54.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 -- 172.21.15.103:0/374427996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fe050 msgr2=0x7fe2c810dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fe050 0x7fe2c810dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.777+0000 7fe2ce47e700 1 -- 172.21.15.103:0/374427996 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b8005040 con 0x7fe2c80fce50 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2ce47e700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400eb70 tx=0x7fe2c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2c4007cc0 con 0x7fe2c80fce50 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c8112f50 con 0x7fe2c80fce50 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2c400f040 con 0x7fe2c80fce50 2024-03-22T02:30:54.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2c400cb20 con 0x7fe2c80fce50 2024-03-22T02:30:54.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.778+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c8113400 con 0x7fe2c80fce50 2024-03-22T02:30:54.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.779+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c804c710 con 0x7fe2c80fce50 2024-03-22T02:30:54.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.779+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7fe2c400cc80 con 0x7fe2c80fce50 2024-03-22T02:30:54.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.779+0000 7fe2beffd700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 0x7fe2b4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:54.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.780+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe2c408d0a0 con 0x7fe2c80fce50 2024-03-22T02:30:54.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.780+0000 7fe2cdc7d700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 0x7fe2b4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:54.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.780+0000 7fe2cdc7d700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 0x7fe2b4070280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009b30 tx=0x7fe2b8009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:54.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:54.784+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fe2c405a500 con 0x7fe2c80fce50 2024-03-22T02:30:55.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.014+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c8102320 con 0x7fe2b406ddd0 2024-03-22T02:30:55.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.015+0000 7fe2beffd700 1 -- 172.21.15.103:0/374427996 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fe2c8102320 con 0x7fe2b406ddd0 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 msgr2=0x7fe2b4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 0x7fe2b4070280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009b30 tx=0x7fe2b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 msgr2=0x7fe2c810d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400eb70 tx=0x7fe2c400b040 comp rx=0 tx=0).stop 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 shutdown_connections 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe2b406ddd0 0x7fe2b4070280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c80fce50 0x7fe2c810d8a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 --2- 172.21.15.103:0/374427996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c80fe050 0x7fe2c810dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 >> 172.21.15.103:0/374427996 conn(0x7fe2c80f8580 msgr2=0x7fe2c80f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 shutdown_connections 2024-03-22T02:30:55.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.017+0000 7fe2d06e2700 1 -- 172.21.15.103:0/374427996 wait complete. 2024-03-22T02:30:55.029 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:30:55.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:54 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:55.030 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:54 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:55.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 -- 172.21.15.103:0/1615559921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fcf80 msgr2=0x7ff0f80fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 --2- 172.21.15.103:0/1615559921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fcf80 0x7ff0f80fd3f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8004940 tx=0x7ff0e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:55.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 -- 172.21.15.103:0/1615559921 shutdown_connections 2024-03-22T02:30:55.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 --2- 172.21.15.103:0/1615559921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fcf80 0x7ff0f80fd3f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 --2- 172.21.15.103:0/1615559921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.123+0000 7ff1001a4700 1 -- 172.21.15.103:0/1615559921 >> 172.21.15.103:0/1615559921 conn(0x7ff0f80f7430 msgr2=0x7ff0f80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 -- 172.21.15.103:0/1615559921 shutdown_connections 2024-03-22T02:30:55.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 -- 172.21.15.103:0/1615559921 wait complete. 2024-03-22T02:30:55.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 Processor -- start 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 -- start start 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fcf80 0x7ff0f810dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f810e4c0 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff1001a4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f810ed10 con 0x7ff0f80fcf80 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff0fdf40700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff0fdf40700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60824/0 (socket says 172.21.15.103:60824) 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff0fdf40700 1 -- 172.21.15.103:0/2408138309 learned_addr learned my addr 172.21.15.103:0/2408138309 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.124+0000 7ff0fd73f700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fcf80 0x7ff0f810dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0fdf40700 1 -- 172.21.15.103:0/2408138309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fcf80 msgr2=0x7ff0f810dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0fdf40700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fcf80 0x7ff0f810dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0fdf40700 1 -- 172.21.15.103:0/2408138309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8005040 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0fdf40700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400eb40 tx=0x7ff0f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f4007cc0 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0f400f040 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f810ef90 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f400ca10 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.125+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f810f4d0 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.126+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8101b10 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.126+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7ff0f400cc80 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.127+0000 7ff0ee7fc700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 0x7ff0e4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.127+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff0f408d260 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.127+0000 7ff0fd73f700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 0x7ff0e4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.127+0000 7ff0fd73f700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 0x7ff0e4070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009b30 tx=0x7ff0e8009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.131+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7ff0f405a6c0 con 0x7ff0f80fbcd0 2024-03-22T02:30:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:54 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:54 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:55.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.380+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0f8104970 con 0x7ff0e406ddc0 2024-03-22T02:30:55.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.382+0000 7ff0ee7fc700 1 -- 172.21.15.103:0/2408138309 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7ff0e8009710 con 0x7ff0e406ddc0 2024-03-22T02:30:55.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 msgr2=0x7ff0e4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 0x7ff0e4070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009b30 tx=0x7ff0e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 msgr2=0x7ff0f810d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400eb40 tx=0x7ff0f400b040 comp rx=0 tx=0).stop 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 shutdown_connections 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff0e406ddc0 0x7ff0e4070270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0f80fbcd0 0x7ff0f810d980 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 --2- 172.21.15.103:0/2408138309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f80fcf80 0x7ff0f810dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 >> 172.21.15.103:0/2408138309 conn(0x7ff0f80f7430 msgr2=0x7ff0f8065ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 shutdown_connections 2024-03-22T02:30:55.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.384+0000 7ff1001a4700 1 -- 172.21.15.103:0/2408138309 wait complete. 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2919482196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 msgr2=0x7f9ce8101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2919482196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce8101e30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009ce0 tx=0x7f9cd8004d10 comp rx=0 tx=0).stop 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2919482196 shutdown_connections 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2919482196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 0x7f9ce8104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2919482196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce8101e30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2919482196 >> 172.21.15.103:0/2919482196 conn(0x7f9ce80f7440 msgr2=0x7f9ce80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2919482196 shutdown_connections 2024-03-22T02:30:55.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.499+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2919482196 wait complete. 2024-03-22T02:30:55.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 Processor -- start 2024-03-22T02:30:55.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 -- start start 2024-03-22T02:30:55.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 0x7f9ce8110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8110860 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9cecbdc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8110fa0 con 0x7f9ce8102370 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60842/0 (socket says 172.21.15.103:60842) 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 -- 172.21.15.103:0/2328789645 learned_addr learned my addr 172.21.15.103:0/2328789645 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce5d9b700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 0x7f9ce8110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 -- 172.21.15.103:0/2328789645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 msgr2=0x7f9ce8110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 0x7f9ce8110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.500+0000 7f9ce659c700 1 -- 172.21.15.103:0/2328789645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8009940 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9ce659c700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009380 tx=0x7f9cd8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd800e040 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd801cd00 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce8111220 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd8012b90 con 0x7f9ce8066b10 2024-03-22T02:30:55.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.501+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce81117e0 con 0x7f9ce8066b10 2024-03-22T02:30:55.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.502+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce805f5f0 con 0x7f9ce8066b10 2024-03-22T02:30:55.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.502+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f9cd801c830 con 0x7f9ce8066b10 2024-03-22T02:30:55.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.503+0000 7f9cd6ffd700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 0x7f9cd0070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.503+0000 7f9ce5d9b700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 0x7f9cd0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.503+0000 7f9ce5d9b700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 0x7f9cd0070270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009b80 tx=0x7f9cdc00b040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.504+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9cd8091030 con 0x7f9ce8066b10 2024-03-22T02:30:55.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.509+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f9cd805cd40 con 0x7f9ce8066b10 2024-03-22T02:30:55.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.721+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce8111b00 con 0x7f9cd006ddc0 2024-03-22T02:30:55.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.731+0000 7f9cd6ffd700 1 -- 172.21.15.103:0/2328789645 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9ce8111b00 con 0x7f9cd006ddc0 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (110s) 0s ago 10m 27.7M - 0.25.0 c8568f914cd2 44d2a25d6238 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (10m) 0s ago 10m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (9m) 1s ago 9m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 unknown 0s ago 10m - - 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (5m) 0s ago 5m 12.7M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (5m) 0s ago 5m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (4m) 1s ago 4m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (5m) 1s ago 5m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (62s) 0s ago 12m 497M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (23s) 1s ago 9m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (12m) 0s ago 12m 56.0M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (9m) 1s ago 9m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:30:55.733 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (9s) 0s ago 10m 8526k - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (5s) 1s ago 9m 3611k - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (8m) 0s ago 8m 212M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (8m) 0s ago 8m 229M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (7m) 0s ago 7m 197M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (7m) 1s ago 7m 262M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (7m) 1s ago 7m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (6m) 1s ago 6m 217M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:30:55.734 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (28s) 0s ago 10m 50.0M - 2.43.0 a07b618ecd1d a8820f579bf6 2024-03-22T02:30:55.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 msgr2=0x7f9cd0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 0x7f9cd0070270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009b80 tx=0x7f9cdc00b040 comp rx=0 tx=0).stop 2024-03-22T02:30:55.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 msgr2=0x7f9ce810fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009380 tx=0x7f9cd8004d10 comp rx=0 tx=0).stop 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 shutdown_connections 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f9cd006ddc0 0x7f9cd0070270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ce8066b10 0x7f9ce810fd20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 --2- 172.21.15.103:0/2328789645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce8102370 0x7f9ce8110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 >> 172.21.15.103:0/2328789645 conn(0x7f9ce80f7440 msgr2=0x7f9ce80f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 shutdown_connections 2024-03-22T02:30:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.733+0000 7f9cecbdc700 1 -- 172.21.15.103:0/2328789645 wait complete. 2024-03-22T02:30:55.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 -- 172.21.15.103:0/2096720075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 msgr2=0x7fc6000fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 --2- 172.21.15.103:0/2096720075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc6000fe450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec004940 tx=0x7fc5ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 -- 172.21.15.103:0/2096720075 shutdown_connections 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 --2- 172.21.15.103:0/2096720075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc6000fe450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 --2- 172.21.15.103:0/2096720075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc6000fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 -- 172.21.15.103:0/2096720075 >> 172.21.15.103:0/2096720075 conn(0x7fc6000f8550 msgr2=0x7fc6000fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 -- 172.21.15.103:0/2096720075 shutdown_connections 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.837+0000 7fc606c7f700 1 -- 172.21.15.103:0/2096720075 wait complete. 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 Processor -- start 2024-03-22T02:30:55.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 -- start start 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc600071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc600072260 con 0x7fc6000fdfe0 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc606c7f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc600070850 con 0x7fc6000fce70 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc5fffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc604a1b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc600071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc5fffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60856/0 (socket says 172.21.15.103:60856) 2024-03-22T02:30:55.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.838+0000 7fc5fffff700 1 -- 172.21.15.103:0/3294093848 learned_addr learned my addr 172.21.15.103:0/3294093848 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc604a1b700 1 -- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 msgr2=0x7fc600070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc604a1b700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc604a1b700 1 -- 172.21.15.103:0/3294093848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec005040 con 0x7fc6000fce70 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc5fffff700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc604a1b700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc600071c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009fd0 tx=0x7fc5f400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f4004e90 con 0x7fc6000fce70 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5f400f040 con 0x7fc6000fce70 2024-03-22T02:30:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f400ca50 con 0x7fc6000fce70 2024-03-22T02:30:55.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc600070aa0 con 0x7fc6000fce70 2024-03-22T02:30:55.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.839+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc600070f90 con 0x7fc6000fce70 2024-03-22T02:30:55.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.840+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60004c710 con 0x7fc6000fce70 2024-03-22T02:30:55.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.841+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7fc5f4007cc0 con 0x7fc6000fce70 2024-03-22T02:30:55.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.841+0000 7fc5fd7fa700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 0x7fc5f0070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:55.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.841+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc5f408d160 con 0x7fc6000fce70 2024-03-22T02:30:55.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.841+0000 7fc5fffff700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 0x7fc5f0070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:55.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.842+0000 7fc5fffff700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 0x7fc5f0070280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b0d0 tx=0x7fc5ec009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:55.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:55.845+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fc5f405a520 con 0x7fc6000fce70 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.142+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc600115bc0 con 0x7fc6000fce70 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.143+0000 7fc5fd7fa700 1 -- 172.21.15.103:0/3294093848 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fc5f405a0b0 con 0x7fc6000fce70 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:56.145 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:30:56.146 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:30:56.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 msgr2=0x7fc5f0070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 0x7fc5f0070280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00b0d0 tx=0x7fc5ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 msgr2=0x7fc600071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc600071c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009fd0 tx=0x7fc5f400b040 comp rx=0 tx=0).stop 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 shutdown_connections 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc5f006ddd0 0x7fc5f0070280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6000fce70 0x7fc600071c60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 --2- 172.21.15.103:0/3294093848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6000fdfe0 0x7fc600070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 >> 172.21.15.103:0/3294093848 conn(0x7fc6000f8550 msgr2=0x7fc6000f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:56.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 shutdown_connections 2024-03-22T02:30:56.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.146+0000 7fc606c7f700 1 -- 172.21.15.103:0/3294093848 wait complete. 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 -- 172.21.15.103:0/718671777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe400662e0 msgr2=0x7efe400ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 --2- 172.21.15.103:0/718671777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe400662e0 0x7efe400ff9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efe34004940 tx=0x7efe34009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 -- 172.21.15.103:0/718671777 shutdown_connections 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 --2- 172.21.15.103:0/718671777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe40102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 --2- 172.21.15.103:0/718671777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe400662e0 0x7efe400ff9c0 secure :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efe34004940 tx=0x7efe34009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 -- 172.21.15.103:0/718671777 >> 172.21.15.103:0/718671777 conn(0x7efe400f7430 msgr2=0x7efe400f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 -- 172.21.15.103:0/718671777 shutdown_connections 2024-03-22T02:30:56.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 -- 172.21.15.103:0/718671777 wait complete. 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.286+0000 7efe46b14700 1 Processor -- start 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe46b14700 1 -- start start 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe46b14700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe4010fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe46b14700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe46b14700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe40110810 con 0x7efe40110350 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe46b14700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe40110950 con 0x7efe400fff00 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe3ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe3ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60884/0 (socket says 172.21.15.103:60884) 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe448b0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe4010fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.287+0000 7efe3ffff700 1 -- 172.21.15.103:0/3987784261 learned_addr learned my addr 172.21.15.103:0/3987784261 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:30:56.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe448b0700 1 -- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 msgr2=0x7efe401110f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe448b0700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe448b0700 1 -- 172.21.15.103:0/3987784261 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe34005040 con 0x7efe400fff00 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe3ffff700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe448b0700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe4010fe10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe400657d0 tx=0x7efe34009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3400e040 con 0x7efe400fff00 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe3400bbf0 con 0x7efe400fff00 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3401c3f0 con 0x7efe400fff00 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe40111720 con 0x7efe400fff00 2024-03-22T02:30:56.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.288+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe40111c50 con 0x7efe400fff00 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.289+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe400fe620 con 0x7efe400fff00 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.290+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7efe3400bd60 con 0x7efe400fff00 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.290+0000 7efe3d7fa700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 0x7efe30070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.290+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7efe3408e8d0 con 0x7efe400fff00 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.290+0000 7efe3ffff700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 0x7efe30070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:56.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.291+0000 7efe3ffff700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 0x7efe30070270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0041d0 tx=0x7efe2c009040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:56.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.295+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7efe340581b0 con 0x7efe400fff00 2024-03-22T02:30:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: pgmap v28: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:56 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: pgmap v28: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:56 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:56.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.518+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe401058e0 con 0x7efe3006ddc0 2024-03-22T02:30:56.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.519+0000 7efe3d7fa700 1 -- 172.21.15.103:0/3987784261 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7efe401058e0 con 0x7efe3006ddc0 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr" 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:30:56.521 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 msgr2=0x7efe30070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 0x7efe30070270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0041d0 tx=0x7efe2c009040 comp rx=0 tx=0).stop 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 msgr2=0x7efe4010fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe4010fe10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe400657d0 tx=0x7efe34009e30 comp rx=0 tx=0).stop 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 shutdown_connections 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7efe3006ddc0 0x7efe30070270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe400fff00 0x7efe4010fe10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 --2- 172.21.15.103:0/3987784261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe40110350 0x7efe401110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:56.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 >> 172.21.15.103:0/3987784261 conn(0x7efe400f7430 msgr2=0x7efe400fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:56.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 shutdown_connections 2024-03-22T02:30:56.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:30:56.521+0000 7efe46b14700 1 -- 172.21.15.103:0/3987784261 wait complete. 2024-03-22T02:30:57.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:57 smithi122 ceph-mon[36403]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:57.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:57 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3294093848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:57.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:57 smithi122 ceph-mon[36403]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:57.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:57 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:57.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:57 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:57 smithi103 ceph-mon[29258]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:57 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3294093848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:30:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:57 smithi103 ceph-mon[29258]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:30:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:57 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:57 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:58.180 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:58 smithi122 ceph-mon[36403]: pgmap v29: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:58 smithi103 ceph-mon[29258]: pgmap v29: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:30:59.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:59 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:59.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:30:59 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:59 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:30:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:30:59 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:00 smithi122 ceph-mon[36403]: pgmap v30: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:00 smithi103 ceph-mon[29258]: pgmap v30: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: pgmap v31: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: pgmap v32: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: pgmap v33: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:02 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: pgmap v31: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: pgmap v32: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: pgmap v33: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:02 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:04.425 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:04 smithi103 ceph-mon[29258]: Upgrade: Updating prometheus.smithi103 2024-03-22T02:31:04.425 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:04 smithi103 ceph-mon[29258]: pgmap v34: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:04.425 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:04 smithi103 ceph-mon[29258]: Deploying daemon prometheus.smithi103 on smithi103 2024-03-22T02:31:04.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:04 smithi122 ceph-mon[36403]: Upgrade: Updating prometheus.smithi103 2024-03-22T02:31:04.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:04 smithi122 ceph-mon[36403]: pgmap v34: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:04 smithi122 ceph-mon[36403]: Deploying daemon prometheus.smithi103 on smithi103 2024-03-22T02:31:06.581 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:06 smithi103 ceph-mon[29258]: pgmap v35: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:06.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:06 smithi122 ceph-mon[36403]: pgmap v35: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:08.440 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:08 smithi103 ceph-mon[29258]: pgmap v36: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:08.440 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:08.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:08 smithi122 ceph-mon[36403]: pgmap v36: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:08.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: pgmap v37: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.053 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:09 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: pgmap v37: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:10.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:09 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:12.339 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:12 smithi103 ceph-mon[29258]: pgmap v38: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:12.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:12 smithi122 ceph-mon[36403]: pgmap v38: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:14 smithi103 ceph-mon[29258]: pgmap v39: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:14.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:14 smithi122 ceph-mon[36403]: pgmap v39: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:16.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:16.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:15 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:16.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:15 smithi122 ceph-mon[36403]: pgmap v40: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:15 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:15 smithi103 ceph-mon[29258]: pgmap v40: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:17.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:16 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:17.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:16 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:16 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:16 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:18.036 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:17 smithi103 ceph-mon[29258]: pgmap v41: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:17 smithi122 ceph-mon[36403]: pgmap v41: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: pgmap v42: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:20.456 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.457 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:20 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: pgmap v42: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:20.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:20 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:21.406 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:21 smithi103 ceph-mon[29258]: pgmap v43: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:21.407 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:21 smithi103 ceph-mon[29258]: pgmap v44: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:21.407 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:21 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:31:21.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:21 smithi122 ceph-mon[36403]: pgmap v43: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:21.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:21 smithi122 ceph-mon[36403]: pgmap v44: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:21.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:21 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T02:31:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:22 smithi103 ceph-mon[29258]: Upgrade: Updating alertmanager.smithi103 2024-03-22T02:31:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:22 smithi103 ceph-mon[29258]: Deploying daemon alertmanager.smithi103 on smithi103 2024-03-22T02:31:22.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:22 smithi122 ceph-mon[36403]: Upgrade: Updating alertmanager.smithi103 2024-03-22T02:31:22.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:22 smithi122 ceph-mon[36403]: Deploying daemon alertmanager.smithi103 on smithi103 2024-03-22T02:31:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:23 smithi103 ceph-mon[29258]: pgmap v45: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:23.266 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:23 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:23.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:23 smithi122 ceph-mon[36403]: pgmap v45: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:23 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:25.431 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:25 smithi103 ceph-mon[29258]: pgmap v46: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:25 smithi122 ceph-mon[36403]: pgmap v46: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 -- 172.21.15.103:0/838282152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fbcd0 msgr2=0x7fd1e40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 --2- 172.21.15.103:0/838282152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e40fc0e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8004940 tx=0x7fd1d8009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 -- 172.21.15.103:0/838282152 shutdown_connections 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 --2- 172.21.15.103:0/838282152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fcf80 0x7fd1e40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 --2- 172.21.15.103:0/838282152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e40fc0e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 -- 172.21.15.103:0/838282152 >> 172.21.15.103:0/838282152 conn(0x7fd1e40f7430 msgr2=0x7fd1e40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 -- 172.21.15.103:0/838282152 shutdown_connections 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.632+0000 7fd1eada0700 1 -- 172.21.15.103:0/838282152 wait complete. 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 Processor -- start 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 -- start start 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e410c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e410d4e0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1eada0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e410dcd0 con 0x7fd1e40fbcd0 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42432/0 (socket says 172.21.15.103:42432) 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.633+0000 7fd1e3fff700 1 -- 172.21.15.103:0/2187585518 learned_addr learned my addr 172.21.15.103:0/2187585518 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e8b3c700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e410c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e3fff700 1 -- 172.21.15.103:0/2187585518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 msgr2=0x7fd1e410c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e3fff700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e410c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e3fff700 1 -- 172.21.15.103:0/2187585518 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8005040 con 0x7fd1e40fcf80 2024-03-22T02:31:26.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e8b3c700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:31:26.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e3fff700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000b790 tx=0x7fd1d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:26.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d000f770 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e410dfb0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1d000fda0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d000d310 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.634+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e410e570 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.635+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e405f5f0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.636+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7fd1d000d4f0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.636+0000 7fd1e17fa700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 0x7fd1d4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.636+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd1d008d600 con 0x7fd1e40fcf80 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.636+0000 7fd1e8b3c700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 0x7fd1d4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.636+0000 7fd1e8b3c700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 0x7fd1d4070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b3b0 tx=0x7fd1d8009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:26.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.640+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fd1d005abb0 con 0x7fd1e40fcf80 2024-03-22T02:31:26.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.859+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1e410e860 con 0x7fd1d406ddc0 2024-03-22T02:31:26.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.860+0000 7fd1e17fa700 1 -- 172.21.15.103:0/2187585518 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fd1e410e860 con 0x7fd1d406ddc0 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 msgr2=0x7fd1d4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 0x7fd1d4070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800b3b0 tx=0x7fd1d8009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 msgr2=0x7fd1e410ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000b790 tx=0x7fd1d0009250 comp rx=0 tx=0).stop 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 shutdown_connections 2024-03-22T02:31:26.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd1d406ddc0 0x7fd1d4070270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1e40fbcd0 0x7fd1e410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.862+0000 7fd1eada0700 1 --2- 172.21.15.103:0/2187585518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1e40fcf80 0x7fd1e410ce50 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.863+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 >> 172.21.15.103:0/2187585518 conn(0x7fd1e40f7430 msgr2=0x7fd1e40f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:26.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.863+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 shutdown_connections 2024-03-22T02:31:26.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.863+0000 7fd1eada0700 1 -- 172.21.15.103:0/2187585518 wait complete. 2024-03-22T02:31:26.873 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:31:26.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 -- 172.21.15.103:0/2918023544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7198066f10 msgr2=0x7f71980ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 --2- 172.21.15.103:0/2918023544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7198066f10 0x7f71980ffa50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f718c004940 tx=0x7f718c009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:26.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 -- 172.21.15.103:0/2918023544 shutdown_connections 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 --2- 172.21.15.103:0/2918023544 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71980fff90 0x7f7198102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 --2- 172.21.15.103:0/2918023544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7198066f10 0x7f71980ffa50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 -- 172.21.15.103:0/2918023544 >> 172.21.15.103:0/2918023544 conn(0x7f71980f7480 msgr2=0x7f71980f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 -- 172.21.15.103:0/2918023544 shutdown_connections 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.979+0000 7f719d2eb700 1 -- 172.21.15.103:0/2918023544 wait complete. 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 Processor -- start 2024-03-22T02:31:26.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 -- start start 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 0x7f719810d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719810e560 con 0x7f71980fff90 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f719d2eb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198112eb0 con 0x7f7198066f10 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f71967fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f71967fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42460/0 (socket says 172.21.15.103:42460) 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f71967fc700 1 -- 172.21.15.103:0/1247171525 learned_addr learned my addr 172.21.15.103:0/1247171525 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.980+0000 7f7196ffd700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 0x7f719810d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f71967fc700 1 -- 172.21.15.103:0/1247171525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 msgr2=0x7f719810d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:26.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f71967fc700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 0x7f719810d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f71967fc700 1 -- 172.21.15.103:0/1247171525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c005040 con 0x7f71980fff90 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f7196ffd700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 0x7f719810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f71967fc700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f718800b790 tx=0x7f7188009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718800f770 con 0x7f71980fff90 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7198113110 con 0x7f71980fff90 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f718800fda0 con 0x7f71980fff90 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f718800d310 con 0x7f71980fff90 2024-03-22T02:31:26.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.981+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7198113680 con 0x7f71980fff90 2024-03-22T02:31:26.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.982+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719804c710 con 0x7f71980fff90 2024-03-22T02:31:26.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.983+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f718800f8d0 con 0x7f71980fff90 2024-03-22T02:31:26.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.983+0000 7f717f7fe700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 0x7f7180070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:26.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.983+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f718808d4e0 con 0x7f71980fff90 2024-03-22T02:31:26.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.983+0000 7f7196ffd700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 0x7f7180070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:26.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.985+0000 7f7196ffd700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 0x7f7180070280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f718c00b970 tx=0x7f718c009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:26.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:26.988+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f718805aac0 con 0x7f71980fff90 2024-03-22T02:31:27.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.268+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71980f8f70 con 0x7f718006ddd0 2024-03-22T02:31:27.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.269+0000 7f717f7fe700 1 -- 172.21.15.103:0/1247171525 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f71980f8f70 con 0x7f718006ddd0 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 msgr2=0x7f7180070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 0x7f7180070280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f718c00b970 tx=0x7f718c009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 msgr2=0x7f719810ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f718800b790 tx=0x7f7188009250 comp rx=0 tx=0).stop 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 shutdown_connections 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f718006ddd0 0x7f7180070280 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7198066f10 0x7f719810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 --2- 172.21.15.103:0/1247171525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f71980fff90 0x7f719810ded0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.274+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 >> 172.21.15.103:0/1247171525 conn(0x7f71980f7480 msgr2=0x7f71980f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.275+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 shutdown_connections 2024-03-22T02:31:27.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.276+0000 7f719d2eb700 1 -- 172.21.15.103:0/1247171525 wait complete. 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 -- 172.21.15.103:0/1171029737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 msgr2=0x7f1acc102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 --2- 172.21.15.103:0/1171029737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc102320 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1abc004940 tx=0x7f1abc009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 -- 172.21.15.103:0/1171029737 shutdown_connections 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 --2- 172.21.15.103:0/1171029737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc102320 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 --2- 172.21.15.103:0/1171029737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 0x7f1acc0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 -- 172.21.15.103:0/1171029737 >> 172.21.15.103:0/1171029737 conn(0x7f1acc0f7430 msgr2=0x7f1acc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 -- 172.21.15.103:0/1171029737 shutdown_connections 2024-03-22T02:31:27.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.383+0000 7f1ad1524700 1 -- 172.21.15.103:0/1171029737 wait complete. 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 Processor -- start 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 -- start start 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 0x7f1acc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0fbe50 con 0x7f1acc0fff00 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1ad1524700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0fbf90 con 0x7f1acc0662e0 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42478/0 (socket says 172.21.15.103:42478) 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 -- 172.21.15.103:0/412785214 learned_addr learned my addr 172.21.15.103:0/412785214 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1acaffd700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 0x7f1acc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 -- 172.21.15.103:0/412785214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 msgr2=0x7f1acc0fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 0x7f1acc0fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 -- 172.21.15.103:0/412785214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc005040 con 0x7f1acc0fff00 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.384+0000 7f1aca7fc700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009150 tx=0x7f1abc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.385+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1abc00e040 con 0x7f1acc0fff00 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.385+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1abc00bcc0 con 0x7f1acc0fff00 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.385+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1abc01c630 con 0x7f1acc0fff00 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.385+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc0fc210 con 0x7f1acc0fff00 2024-03-22T02:31:27.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.385+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc115a70 con 0x7f1acc0fff00 2024-03-22T02:31:27.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.386+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc05f5f0 con 0x7f1acc0fff00 2024-03-22T02:31:27.388 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:27 smithi103 ceph-mon[29258]: pgmap v47: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:27.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.387+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f1abc013420 con 0x7f1acc0fff00 2024-03-22T02:31:27.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.387+0000 7f1ab37fe700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 0x7f1ab4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.387+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1abc08e220 con 0x7f1acc0fff00 2024-03-22T02:31:27.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.387+0000 7f1acaffd700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 0x7f1ab4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.388+0000 7f1acaffd700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 0x7f1ab4070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009fd0 tx=0x7f1ac0009f60 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:27.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.394+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f1abc057bc0 con 0x7f1acc0fff00 2024-03-22T02:31:27.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:27 smithi122 ceph-mon[36403]: pgmap v47: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:27.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.623+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1acc0f8f70 con 0x7f1ab406ddc0 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.633+0000 7f1ab37fe700 1 -- 172.21.15.103:0/412785214 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1acc0f8f70 con 0x7f1ab406ddc0 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (2m) 12s ago 11m 27.7M - 0.25.0 c8568f914cd2 44d2a25d6238 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (11m) 12s ago 11m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (10m) 33s ago 10m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 unknown 12s ago 10m - - 2024-03-22T02:31:27.634 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (5m) 12s ago 5m 12.8M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (5m) 12s ago 5m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (5m) 33s ago 5m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (5m) 33s ago 5m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (94s) 12s ago 12m 500M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (55s) 33s ago 10m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (12m) 12s ago 12m 54.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (10m) 33s ago 10m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (41s) 12s ago 11m 14.1M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (37s) 33s ago 10m 3611k - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (9m) 12s ago 9m 213M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (8m) 12s ago 8m 229M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (8m) 12s ago 8m 198M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (8m) 33s ago 8m 262M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (7m) 33s ago 7m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:31:27.635 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (7m) 33s ago 7m 217M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:31:27.636 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (19s) 12s ago 10m 40.0M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:31:27.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 msgr2=0x7f1ab4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 0x7f1ab4070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009fd0 tx=0x7f1ac0009f60 comp rx=0 tx=0).stop 2024-03-22T02:31:27.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 msgr2=0x7f1acc0fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009150 tx=0x7f1abc009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 shutdown_connections 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f1ab406ddc0 0x7f1ab4070270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1acc0662e0 0x7f1acc0fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 --2- 172.21.15.103:0/412785214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1acc0fff00 0x7f1acc0fb8c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.635+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 >> 172.21.15.103:0/412785214 conn(0x7f1acc0f7430 msgr2=0x7f1acc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.636+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 shutdown_connections 2024-03-22T02:31:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.636+0000 7f1ad1524700 1 -- 172.21.15.103:0/412785214 wait complete. 2024-03-22T02:31:27.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 -- 172.21.15.103:0/3119680264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fbd00 msgr2=0x7fb10c0fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 --2- 172.21.15.103:0/3119680264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0fc110 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc004940 tx=0x7fb0fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:27.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 -- 172.21.15.103:0/3119680264 shutdown_connections 2024-03-22T02:31:27.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 --2- 172.21.15.103:0/3119680264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 --2- 172.21.15.103:0/3119680264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0fc110 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 -- 172.21.15.103:0/3119680264 >> 172.21.15.103:0/3119680264 conn(0x7fb10c0f7480 msgr2=0x7fb10c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:27.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 -- 172.21.15.103:0/3119680264 shutdown_connections 2024-03-22T02:31:27.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.737+0000 7fb11216c700 1 -- 172.21.15.103:0/3119680264 wait complete. 2024-03-22T02:31:27.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 Processor -- start 2024-03-22T02:31:27.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 -- start start 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c1001f0 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb11216c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0fe830 con 0x7fb10c0fbd00 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb10affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb10affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42498/0 (socket says 172.21.15.103:42498) 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb10affd700 1 -- 172.21.15.103:0/862870593 learned_addr learned my addr 172.21.15.103:0/862870593 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.738+0000 7fb10b7fe700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb10affd700 1 -- 172.21.15.103:0/862870593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fbd00 msgr2=0x7fb10c0ffbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb10affd700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0ffbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb10affd700 1 -- 172.21.15.103:0/862870593 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc005040 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb10affd700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c830 tx=0x7fb100004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1000106f0 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb100010d20 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c0feb10 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1000113a0 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.739+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c0ff050 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.740+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c101b40 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.740+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7fb100012400 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.741+0000 7fb0f3fff700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 0x7fb0f4070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:27.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.741+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb10008d410 con 0x7fb10c0fcfb0 2024-03-22T02:31:27.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.741+0000 7fb10b7fe700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 0x7fb0f4070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:27.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.741+0000 7fb10b7fe700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 0x7fb0f4070280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00b3b0 tx=0x7fb0fc009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:27.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:27.745+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fb10005a9f0 con 0x7fb10c0fcfb0 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.034+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb10c1047f0 con 0x7fb10c0fcfb0 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.034+0000 7fb0f3fff700 1 -- 172.21.15.103:0/862870593 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fb10005a580 con 0x7fb10c0fcfb0 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:31:28.035 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:31:28.036 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:31:28.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 msgr2=0x7fb0f4070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 0x7fb0f4070280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00b3b0 tx=0x7fb0fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 msgr2=0x7fb10c0fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c830 tx=0x7fb100004470 comp rx=0 tx=0).stop 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 shutdown_connections 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb0f406ddd0 0x7fb0f4070280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb10c0fbd00 0x7fb10c0ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 --2- 172.21.15.103:0/862870593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb10c0fcfb0 0x7fb10c0fe2f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 >> 172.21.15.103:0/862870593 conn(0x7fb10c0f7480 msgr2=0x7fb10c066810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 shutdown_connections 2024-03-22T02:31:28.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.036+0000 7fb11216c700 1 -- 172.21.15.103:0/862870593 wait complete. 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 -- 172.21.15.103:0/1975660042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fbcd0 msgr2=0x7f88f80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 --2- 172.21.15.103:0/1975660042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fbcd0 0x7f88f80fc0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f88ec004940 tx=0x7f88ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 -- 172.21.15.103:0/1975660042 shutdown_connections 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 --2- 172.21.15.103:0/1975660042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fcf80 0x7f88f80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 --2- 172.21.15.103:0/1975660042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fbcd0 0x7f88f80fc0e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 -- 172.21.15.103:0/1975660042 >> 172.21.15.103:0/1975660042 conn(0x7f88f80f7430 msgr2=0x7f88f80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 -- 172.21.15.103:0/1975660042 shutdown_connections 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.139+0000 7f88ff044700 1 -- 172.21.15.103:0/1975660042 wait complete. 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 Processor -- start 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 -- start start 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fbcd0 0x7f88f810da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f810e5d0 con 0x7f88f80fcf80 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88ff044700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f810edd0 con 0x7f88f80fbcd0 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42526/0 (socket says 172.21.15.103:42526) 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 -- 172.21.15.103:0/2254065299 learned_addr learned my addr 172.21.15.103:0/2254065299 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88fcde0700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fbcd0 0x7f88f810da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:28.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 -- 172.21.15.103:0/2254065299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fbcd0 msgr2=0x7f88f810da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fbcd0 0x7f88f810da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.140+0000 7f88f7fff700 1 -- 172.21.15.103:0/2254065299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec005040 con 0x7f88f80fcf80 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88f7fff700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f88e4002a50 tx=0x7f88e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e4007cc0 con 0x7f88f80fcf80 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f810f0b0 con 0x7f88f80fcf80 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88e400f040 con 0x7f88f80fcf80 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e400db70 con 0x7f88f80fcf80 2024-03-22T02:31:28.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.141+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f810f5f0 con 0x7f88f80fcf80 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.142+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f8101b10 con 0x7f88f80fcf80 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.142+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 33) v1 ==== 92035+0+0 (secure 0 0 0) 0x7f88e400dd10 con 0x7f88f80fcf80 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.143+0000 7f88f57fa700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 0x7f88e8070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.143+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f88e408cdc0 con 0x7f88f80fcf80 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.143+0000 7f88fcde0700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 0x7f88e8070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.143+0000 7f88fcde0700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 0x7f88e8070270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f88ec000c00 tx=0x7f88ec009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:28.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.148+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f88e405a170 con 0x7f88f80fcf80 2024-03-22T02:31:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:28 smithi103 ceph-mon[29258]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:28 smithi103 ceph-mon[29258]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:28 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/862870593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:28.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.391+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88f810f810 con 0x7f88e806ddc0 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.392+0000 7f88f57fa700 1 -- 172.21.15.103:0/2254065299 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f88f810f810 con 0x7f88e806ddc0 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr" 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:31:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T02:31:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-22T02:31:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:31:28.394 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:31:28.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.394+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 msgr2=0x7f88e8070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.394+0000 7f88ff044700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 0x7f88e8070270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f88ec000c00 tx=0x7f88ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 msgr2=0x7f88f810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f88e4002a50 tx=0x7f88e40046b0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 shutdown_connections 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f88e806ddc0 0x7f88e8070270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f80fbcd0 0x7f88f810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 --2- 172.21.15.103:0/2254065299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88f80fcf80 0x7f88f810dfd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 >> 172.21.15.103:0/2254065299 conn(0x7f88f80f7430 msgr2=0x7f88f8065c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 shutdown_connections 2024-03-22T02:31:28.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:28.395+0000 7f88ff044700 1 -- 172.21.15.103:0/2254065299 wait complete. 2024-03-22T02:31:28.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:28 smithi122 ceph-mon[36403]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:28.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:28 smithi122 ceph-mon[36403]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:28.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:28 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/862870593' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: pgmap v48: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:29.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:29 smithi103 ceph-mon[29258]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: pgmap v48: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:29.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:29 smithi122 ceph-mon[36403]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:31:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:31 smithi122 ceph-mon[36403]: pgmap v49: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:31.610 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:31 smithi103 ceph-mon[29258]: pgmap v49: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:33 smithi122 ceph-mon[36403]: pgmap v50: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:33.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:33 smithi103 ceph-mon[29258]: pgmap v50: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:33.626 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:33.626 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:35 smithi103 ceph-mon[29258]: pgmap v51: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:35 smithi122 ceph-mon[36403]: pgmap v51: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:37.306 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:37 smithi103 ceph-mon[29258]: pgmap v52: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:37.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:37 smithi122 ceph-mon[36403]: pgmap v52: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: pgmap v53: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: pgmap v54: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:38.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: pgmap v55: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.877 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.878 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.878 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:38.878 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:38 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: pgmap v53: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: pgmap v54: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: pgmap v55: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:39.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:38 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:40 smithi103 ceph-mon[29258]: mgrmap e34: smithi103.pfvmui(active, since 91s), standbys: smithi122.prgtti 2024-03-22T02:31:40.603 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:40 smithi122 ceph-mon[36403]: mgrmap e34: smithi103.pfvmui(active, since 91s), standbys: smithi122.prgtti 2024-03-22T02:31:41.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:41 smithi103 ceph-mon[29258]: pgmap v56: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:41 smithi103 ceph-mon[29258]: Upgrade: Updating grafana.smithi103 2024-03-22T02:31:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:41 smithi103 ceph-mon[29258]: Deploying daemon grafana.smithi103 on smithi103 2024-03-22T02:31:41.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:41 smithi122 ceph-mon[36403]: pgmap v56: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:41.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:41 smithi122 ceph-mon[36403]: Upgrade: Updating grafana.smithi103 2024-03-22T02:31:41.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:41 smithi122 ceph-mon[36403]: Deploying daemon grafana.smithi103 on smithi103 2024-03-22T02:31:43.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:43 smithi103 ceph-mon[29258]: pgmap v57: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:43.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:43 smithi122 ceph-mon[36403]: pgmap v57: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:45 smithi103 ceph-mon[29258]: pgmap v58: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:45 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:45 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:45 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:45.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:45 smithi122 ceph-mon[36403]: pgmap v58: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:45.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:45 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:45.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:45 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:45.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:45 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:47.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:47 smithi103 ceph-mon[29258]: pgmap v59: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:47 smithi122 ceph-mon[36403]: pgmap v59: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 761 B/s wr, 0 op/s 2024-03-22T02:31:49.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:49 smithi122 ceph-mon[36403]: pgmap v60: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 B/s wr, 0 op/s 2024-03-22T02:31:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:49 smithi103 ceph-mon[29258]: pgmap v60: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 614 B/s wr, 0 op/s 2024-03-22T02:31:50.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:50 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:50.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:50 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:50.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:50 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:50.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:50 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:51.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:51 smithi103 ceph-mon[29258]: pgmap v61: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:31:51.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:51 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:51.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:51 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:51 smithi122 ceph-mon[36403]: pgmap v61: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-03-22T02:31:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:51 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:51 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:53.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:53 smithi103 ceph-mon[29258]: pgmap v62: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:53.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:53 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:53 smithi122 ceph-mon[36403]: pgmap v62: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:53 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:31:55.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: pgmap v63: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:55.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: pgmap v64: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: Upgrade: Finalizing container_image settings 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T02:31:55.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T02:31:55.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: Upgrade: Complete! 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: Cluster is now healthy 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:55 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: pgmap v63: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: pgmap v64: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: Upgrade: Finalizing container_image settings 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T02:31:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T02:31:55.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: Upgrade: Complete! 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: Cluster is now healthy 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:55.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:55 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:57 smithi103 ceph-mon[29258]: pgmap v65: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:57.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:57 smithi122 ceph-mon[36403]: pgmap v65: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 -- 172.21.15.103:0/2032165387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 msgr2=0x7ff3980fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/2032165387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980fc0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff388004940 tx=0x7ff388009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 -- 172.21.15.103:0/2032165387 shutdown_connections 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/2032165387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 0x7ff3980fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/2032165387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980fc0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 -- 172.21.15.103:0/2032165387 >> 172.21.15.103:0/2032165387 conn(0x7ff3980f7430 msgr2=0x7ff3980f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 -- 172.21.15.103:0/2032165387 shutdown_connections 2024-03-22T02:31:58.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.510+0000 7ff39cc7e700 1 -- 172.21.15.103:0/2032165387 wait complete. 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 Processor -- start 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 -- start start 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 0x7ff3980fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980fe900 con 0x7ff3980fbcd0 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39cc7e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980fea70 con 0x7ff3980fcf80 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51028/0 (socket says 172.21.15.103:51028) 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 -- 172.21.15.103:0/1473325932 learned_addr learned my addr 172.21.15.103:0/1473325932 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff38dd9b700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 0x7ff3980fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 -- 172.21.15.103:0/1473325932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 msgr2=0x7ff3980fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 0x7ff3980fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.511+0000 7ff39659c700 1 -- 172.21.15.103:0/1473325932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388005040 con 0x7ff3980fbcd0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff39659c700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff388009150 tx=0x7ff388009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff38800e040 con 0x7ff3980fbcd0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff38800bcc0 con 0x7ff3980fbcd0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff38801c630 con 0x7ff3980fbcd0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3980fecf0 con 0x7ff3980fbcd0 2024-03-22T02:31:58.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.512+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff398115a30 con 0x7ff3980fbcd0 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.513+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39805f5f0 con 0x7ff3980fbcd0 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.514+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7ff388013420 con 0x7ff3980fbcd0 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.514+0000 7ff38f7fe700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 0x7ff384070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.514+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff38808df90 con 0x7ff3980fbcd0 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.514+0000 7ff38dd9b700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 0x7ff384070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:58.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.515+0000 7ff38dd9b700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 0x7ff384070150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff380009b10 tx=0x7ff380008040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:58.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.519+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7ff388057b50 con 0x7ff3980fbcd0 2024-03-22T02:31:58.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.734+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff398104980 con 0x7ff38406dca0 2024-03-22T02:31:58.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.735+0000 7ff38f7fe700 1 -- 172.21.15.103:0/1473325932 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7ff398104980 con 0x7ff38406dca0 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 msgr2=0x7ff384070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 0x7ff384070150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff380009b10 tx=0x7ff380008040 comp rx=0 tx=0).stop 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 msgr2=0x7ff3980ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff388009150 tx=0x7ff388009e30 comp rx=0 tx=0).stop 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 shutdown_connections 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7ff38406dca0 0x7ff384070150 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3980fbcd0 0x7ff3980ffdf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 --2- 172.21.15.103:0/1473325932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3980fcf80 0x7ff3980fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 >> 172.21.15.103:0/1473325932 conn(0x7ff3980f7430 msgr2=0x7ff3980f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 shutdown_connections 2024-03-22T02:31:58.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:31:58.737+0000 7ff39cc7e700 1 -- 172.21.15.103:0/1473325932 wait complete. 2024-03-22T02:31:59.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:59 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:59.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:31:59 smithi122 ceph-mon[36403]: pgmap v66: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:59 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:31:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:31:59 smithi103 ceph-mon[29258]: pgmap v66: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:31:59.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-22T02:31:59.702 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:00.112 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:00 smithi103 ceph-mon[29258]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:00 smithi122 ceph-mon[36403]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:01 smithi103 ceph-mon[29258]: pgmap v67: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:01 smithi122 ceph-mon[36403]: pgmap v67: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- 172.21.15.103:0/2756695548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fde10 msgr2=0x7fe1c00fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 --2- 172.21.15.103:0/2756695548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fde10 0x7fe1c00fe1e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc004940 tx=0x7fe1bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- 172.21.15.103:0/2756695548 shutdown_connections 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 --2- 172.21.15.103:0/2756695548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 --2- 172.21.15.103:0/2756695548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fde10 0x7fe1c00fe1e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- 172.21.15.103:0/2756695548 >> 172.21.15.103:0/2756695548 conn(0x7fe1c00f7870 msgr2=0x7fe1c00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:01.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- 172.21.15.103:0/2756695548 shutdown_connections 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- 172.21.15.103:0/2756695548 wait complete. 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 Processor -- start 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- start start 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fde10 0x7fe1c0113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0114520 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.383+0000 7fe1c7627700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c010db40 con 0x7fe1c00fde10 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51042/0 (socket says 172.21.15.103:51042) 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 -- 172.21.15.103:0/872036963 learned_addr learned my addr 172.21.15.103:0/872036963 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c53c3700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fde10 0x7fe1c0113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 -- 172.21.15.103:0/872036963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fde10 msgr2=0x7fe1c0113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:01.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fde10 0x7fe1c0113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 -- 172.21.15.103:0/872036963 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc005040 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c4bc2700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0002a50 tx=0x7fe1b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b0007cc0 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1b000f040 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c010ddc0 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b000da30 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.384+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c010e300 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.385+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0066ad0 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.386+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fe1b000db90 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.386+0000 7fe1b5ffb700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 0x7fe1ac070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.386+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe1b008d1c0 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.386+0000 7fe1c53c3700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 0x7fe1ac070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:01.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.386+0000 7fe1c53c3700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 0x7fe1ac070150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc004f90 tx=0x7fe1bc009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:01.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.391+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fe1b0056c60 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.669+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe1c0109ea0 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.670+0000 7fe1b5ffb700 1 -- 172.21.15.103:0/872036963 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fe1b005a320 con 0x7fe1c00fe7b0 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 msgr2=0x7fe1ac070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 0x7fe1ac070150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc004f90 tx=0x7fe1bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 msgr2=0x7fe1c0113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0002a50 tx=0x7fe1b00046b0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 shutdown_connections 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe1ac06dca0 0x7fe1ac070150 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c00fde10 0x7fe1c0113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 --2- 172.21.15.103:0/872036963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1c00fe7b0 0x7fe1c0113ed0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 >> 172.21.15.103:0/872036963 conn(0x7fe1c00f7870 msgr2=0x7fe1c00f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 shutdown_connections 2024-03-22T02:32:01.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:01.672+0000 7fe1c7627700 1 -- 172.21.15.103:0/872036963 wait complete. 2024-03-22T02:32:01.684 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:32:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:02 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/872036963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:02.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:02 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/872036963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:02.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-22T02:32:02.650 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:03 smithi122 ceph-mon[36403]: pgmap v68: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:03 smithi103 ceph-mon[29258]: pgmap v68: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:04.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 -- 172.21.15.103:0/1599136412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 msgr2=0x7f0c880fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1599136412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c880fce10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c004940 tx=0x7f0c7c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 -- 172.21.15.103:0/1599136412 shutdown_connections 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1599136412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c880fce10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1599136412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.307+0000 7f0c8d778700 1 -- 172.21.15.103:0/1599136412 >> 172.21.15.103:0/1599136412 conn(0x7f0c880f89b0 msgr2=0x7f0c880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 -- 172.21.15.103:0/1599136412 shutdown_connections 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 -- 172.21.15.103:0/1599136412 wait complete. 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 Processor -- start 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 -- start start 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c88113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88114510 con 0x7f0c881037a0 2024-03-22T02:32:04.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c8d778700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8810db30 con 0x7f0c88102e00 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.308+0000 7f0c867fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39400/0 (socket says 172.21.15.103:39400) 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 -- 172.21.15.103:0/1050097107 learned_addr learned my addr 172.21.15.103:0/1050097107 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c86ffd700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c88113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 -- 172.21.15.103:0/1050097107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 msgr2=0x7f0c88113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c88113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 -- 172.21.15.103:0/1050097107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c005040 con 0x7f0c881037a0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c86ffd700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c88113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c867fc700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009180 tx=0x7f0c7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c7c00e040 con 0x7f0c881037a0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c7c00bbc0 con 0x7f0c881037a0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8810ddb0 con 0x7f0c881037a0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c7c01c3f0 con 0x7f0c881037a0 2024-03-22T02:32:04.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.309+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8810e370 con 0x7f0c881037a0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.310+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c881052d0 con 0x7f0c881037a0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.311+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f0c7c00bd30 con 0x7f0c881037a0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.311+0000 7f0c6f7fe700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 0x7f0c70070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.311+0000 7f0c86ffd700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 0x7f0c70070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.311+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0c7c08e750 con 0x7f0c881037a0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.312+0000 7f0c86ffd700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 0x7f0c70070150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007fd0 tx=0x7f0c78009040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:04.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.316+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f0c7c058270 con 0x7f0c881037a0 2024-03-22T02:32:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.597+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0c8810efd0 con 0x7f0c881037a0 2024-03-22T02:32:04.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.598+0000 7f0c6f7fe700 1 -- 172.21.15.103:0/1050097107 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f0c7c05b930 con 0x7f0c881037a0 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 msgr2=0x7f0c70070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 0x7f0c70070150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c78007fd0 tx=0x7f0c78009040 comp rx=0 tx=0).stop 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 msgr2=0x7f0c88113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009180 tx=0x7f0c7c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 shutdown_connections 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f0c7006dca0 0x7f0c70070150 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0c88102e00 0x7f0c88113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 --2- 172.21.15.103:0/1050097107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0c881037a0 0x7f0c88113ec0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 >> 172.21.15.103:0/1050097107 conn(0x7f0c880f89b0 msgr2=0x7f0c881021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:04.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 shutdown_connections 2024-03-22T02:32:04.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:04.600+0000 7f0c8d778700 1 -- 172.21.15.103:0/1050097107 wait complete. 2024-03-22T02:32:04.611 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)" 2024-03-22T02:32:05.249 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:05 smithi103 ceph-mon[29258]: pgmap v69: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:05.249 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:05 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1050097107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:05.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-22T02:32:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:05 smithi122 ceph-mon[36403]: pgmap v69: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:05 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1050097107' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:05.540 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 -- 172.21.15.103:0/4030639029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 msgr2=0x7feb4c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 --2- 172.21.15.103:0/4030639029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c1031b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb44004940 tx=0x7feb44009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 -- 172.21.15.103:0/4030639029 shutdown_connections 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 --2- 172.21.15.103:0/4030639029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb4c103780 0x7feb4c0fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 --2- 172.21.15.103:0/4030639029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c1031b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 -- 172.21.15.103:0/4030639029 >> 172.21.15.103:0/4030639029 conn(0x7feb4c0f8970 msgr2=0x7feb4c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:07.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 -- 172.21.15.103:0/4030639029 shutdown_connections 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.175+0000 7feb516af700 1 -- 172.21.15.103:0/4030639029 wait complete. 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 Processor -- start 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 -- start start 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb4c103780 0x7feb4c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c070e20 con 0x7feb4c102de0 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb516af700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c070f60 con 0x7feb4c103780 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39408/0 (socket says 172.21.15.103:39408) 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 -- 172.21.15.103:0/1868705096 learned_addr learned my addr 172.21.15.103:0/1868705096 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 -- 172.21.15.103:0/1868705096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb4c103780 msgr2=0x7feb4c070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb4c103780 0x7feb4c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.176+0000 7feb4affd700 1 -- 172.21.15.103:0/1868705096 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44005040 con 0x7feb4c102de0 2024-03-22T02:32:07.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb4affd700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7feb4400b3b0 tx=0x7feb44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4400e040 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb4400bbc0 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c119f40 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4401c3f0 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.177+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c11a3a0 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.178+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c1000f0 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.178+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7feb4400bd30 con 0x7feb4c102de0 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.179+0000 7feb337fe700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 0x7feb34070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:07.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.179+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7feb4408e6a0 con 0x7feb4c102de0 2024-03-22T02:32:07.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.179+0000 7feb4a7fc700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 0x7feb34070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:07.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.179+0000 7feb4a7fc700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 0x7feb34070150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feb4c071c00 tx=0x7feb40009370 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:07.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.183+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7feb44058140 con 0x7feb4c102de0 2024-03-22T02:32:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:07 smithi122 ceph-mon[36403]: pgmap v70: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:07 smithi103 ceph-mon[29258]: pgmap v70: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:07.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.461+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb4c109f50 con 0x7feb4c102de0 2024-03-22T02:32:07.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.462+0000 7feb337fe700 1 -- 172.21.15.103:0/1868705096 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7feb4405b800 con 0x7feb4c102de0 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 msgr2=0x7feb34070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 0x7feb34070150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feb4c071c00 tx=0x7feb40009370 comp rx=0 tx=0).stop 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 msgr2=0x7feb4c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7feb4400b3b0 tx=0x7feb44009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 shutdown_connections 2024-03-22T02:32:07.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7feb3406dca0 0x7feb34070150 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb4c102de0 0x7feb4c0702c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 --2- 172.21.15.103:0/1868705096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb4c103780 0x7feb4c070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:07.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 >> 172.21.15.103:0/1868705096 conn(0x7feb4c0f8970 msgr2=0x7feb4c107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:07.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 shutdown_connections 2024-03-22T02:32:07.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:07.464+0000 7feb516af700 1 -- 172.21.15.103:0/1868705096 wait complete. 2024-03-22T02:32:07.475 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:32:08.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-22T02:32:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:08 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1868705096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:08 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:08.364 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:08 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1868705096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:08 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:09 smithi122 ceph-mon[36403]: pgmap v71: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:09 smithi103 ceph-mon[29258]: pgmap v71: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:10.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/1339063386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 msgr2=0x7fb5f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/1339063386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f80fce10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0004940 tx=0x7fb5e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/1339063386 shutdown_connections 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/1339063386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f80fce10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/1339063386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 0x7fb5f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/1339063386 >> 172.21.15.103:0/1339063386 conn(0x7fb5f80f89b0 msgr2=0x7fb5f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/1339063386 shutdown_connections 2024-03-22T02:32:10.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.132+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/1339063386 wait complete. 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 Processor -- start 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 -- start start 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 0x7fb5f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8072210 con 0x7fb5f81037a0 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5fcc4a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8070d40 con 0x7fb5f8102e00 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:10.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39426/0 (socket says 172.21.15.103:39426) 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 -- 172.21.15.103:0/3617372245 learned_addr learned my addr 172.21.15.103:0/3617372245 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5f659c700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 0x7fb5f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 -- 172.21.15.103:0/3617372245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 msgr2=0x7fb5f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 0x7fb5f80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 -- 172.21.15.103:0/3617372245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0005040 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.133+0000 7fb5edd9b700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009180 tx=0x7fb5e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.134+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e000e040 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.134+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5e0013420 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.134+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f8070fc0 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.134+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e001c610 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.134+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f811a070 con 0x7fb5f81037a0 2024-03-22T02:32:10.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.135+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f81052d0 con 0x7fb5f81037a0 2024-03-22T02:32:10.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.135+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fb5e0013a00 con 0x7fb5f81037a0 2024-03-22T02:32:10.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.136+0000 7fb5ef7fe700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 0x7fb5e4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:10.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.136+0000 7fb5f659c700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 0x7fb5e4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:10.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.136+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb5e008e530 con 0x7fb5f81037a0 2024-03-22T02:32:10.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.136+0000 7fb5f659c700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 0x7fb5e4070150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009fd0 tx=0x7fb5e800b040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:10.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.141+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fb5e0058100 con 0x7fb5f81037a0 2024-03-22T02:32:10.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:10.357+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5f8109ee0 con 0x7fb5e406dca0 2024-03-22T02:32:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:11 smithi122 ceph-mon[36403]: pgmap v72: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:11 smithi122 ceph-mon[36403]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:11 smithi103 ceph-mon[29258]: pgmap v72: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:11 smithi103 ceph-mon[29258]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:12.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.377+0000 7fb5ef7fe700 1 -- 172.21.15.103:0/3617372245 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5092 (secure 0 0 0) 0x7fb5f8109ee0 con 0x7fb5e406dca0 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 msgr2=0x7fb5e4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 0x7fb5e4070150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009fd0 tx=0x7fb5e800b040 comp rx=0 tx=0).stop 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 msgr2=0x7fb5f8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009180 tx=0x7fb5e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 shutdown_connections 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.379+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fb5e406dca0 0x7fb5e4070150 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.380+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f8102e00 0x7fb5f80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.380+0000 7fb5fcc4a700 1 --2- 172.21.15.103:0/3617372245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81037a0 0x7fb5f8070800 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.380+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 >> 172.21.15.103:0/3617372245 conn(0x7fb5f80f89b0 msgr2=0x7fb5f8102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.380+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 shutdown_connections 2024-03-22T02:32:12.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:12.380+0000 7fb5fcc4a700 1 -- 172.21.15.103:0/3617372245 wait complete. 2024-03-22T02:32:12.391 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:32:12.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch ps' 2024-03-22T02:32:13.252 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:13 smithi122 ceph-mon[36403]: pgmap v73: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:13 smithi103 ceph-mon[29258]: pgmap v73: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 -- 172.21.15.103:0/1806693128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 msgr2=0x7f4e541031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/1806693128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e541031d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004940 tx=0x7f4e3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 -- 172.21.15.103:0/1806693128 shutdown_connections 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/1806693128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 0x7f4e540fb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/1806693128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e541031d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:14.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 -- 172.21.15.103:0/1806693128 >> 172.21.15.103:0/1806693128 conn(0x7f4e540f6bd0 msgr2=0x7f4e540f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:14.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 -- 172.21.15.103:0/1806693128 shutdown_connections 2024-03-22T02:32:14.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 -- 172.21.15.103:0/1806693128 wait complete. 2024-03-22T02:32:14.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.854+0000 7f4e5a05a700 1 Processor -- start 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e5a05a700 1 -- start start 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e5a05a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 0x7f4e54113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e5a05a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e5a05a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e541144c0 con 0x7f4e54102e00 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e5a05a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5410dac0 con 0x7f4e540faee0 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38172/0 (socket says 172.21.15.103:38172) 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 -- 172.21.15.103:0/2852799447 learned_addr learned my addr 172.21.15.103:0/2852799447 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e537fe700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 0x7f4e54113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 -- 172.21.15.103:0/2852799447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 msgr2=0x7f4e54113930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:14.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 0x7f4e54113930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.855+0000 7f4e52ffd700 1 -- 172.21.15.103:0/2852799447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c005040 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e52ffd700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e44002a50 tx=0x7f4e440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e44007cc0 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5410dda0 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e4400f040 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e4400da30 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.856+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5410e360 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.857+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e540fc320 con 0x7f4e54102e00 2024-03-22T02:32:14.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.857+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f4e4400db90 con 0x7f4e54102e00 2024-03-22T02:32:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.858+0000 7f4e4bfff700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 0x7f4e40070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.858+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4e4408d1c0 con 0x7f4e54102e00 2024-03-22T02:32:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.858+0000 7f4e537fe700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 0x7f4e40070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:14.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.858+0000 7f4e537fe700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 0x7f4e40070150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004f90 tx=0x7f4e3c009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:14.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:14.863+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f4e4405a790 con 0x7f4e54102e00 2024-03-22T02:32:15.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.069+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e54109ec0 con 0x7f4e4006dca0 2024-03-22T02:32:15.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.079+0000 7f4e4bfff700 1 -- 172.21.15.103:0/2852799447 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f4e54109ec0 con 0x7f4e4006dca0 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (47s) 25s ago 11m 16.8M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (12m) 25s ago 12m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (11m) 81s ago 11m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (31s) 25s ago 11m 51.8M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (6m) 25s ago 6m 12.9M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (6m) 25s ago 6m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (6m) 81s ago 6m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (6m) 81s ago 6m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (2m) 25s ago 13m 508M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (102s) 81s ago 10m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (13m) 25s ago 13m 55.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (10m) 81s ago 10m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (89s) 25s ago 12m 16.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (84s) 81s ago 11m 3611k - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:32:15.080 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (10m) 25s ago 10m 214M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (9m) 25s ago 9m 230M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (9m) 25s ago 9m 198M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (8m) 81s ago 8m 262M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (8m) 81s ago 8m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (7m) 81s ago 7m 217M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:32:15.081 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (66s) 25s ago 11m 49.4M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 msgr2=0x7f4e40070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 0x7f4e40070150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004f90 tx=0x7f4e3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 msgr2=0x7f4e54113e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e44002a50 tx=0x7f4e440046b0 comp rx=0 tx=0).stop 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 shutdown_connections 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f4e4006dca0 0x7f4e40070150 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:15.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e540faee0 0x7f4e54113930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 --2- 172.21.15.103:0/2852799447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e54102e00 0x7f4e54113e70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 >> 172.21.15.103:0/2852799447 conn(0x7f4e540f6bd0 msgr2=0x7f4e540f7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 shutdown_connections 2024-03-22T02:32:15.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:15.081+0000 7f4e5a05a700 1 -- 172.21.15.103:0/2852799447 wait complete. 2024-03-22T02:32:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:15 smithi122 ceph-mon[36403]: pgmap v74: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:15 smithi103 ceph-mon[29258]: pgmap v74: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:15.716 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-22T02:32:15.717 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:32:15.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-03-22T02:32:15.981 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:16 smithi122 ceph-mon[36403]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:16.353 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:16 smithi103 ceph-mon[29258]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:17 smithi122 ceph-mon[36403]: pgmap v75: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:17 smithi103 ceph-mon[29258]: pgmap v75: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:17.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 -- 172.21.15.103:0/215395134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070c70 msgr2=0x7f37c4109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 --2- 172.21.15.103:0/215395134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070c70 0x7f37c4109120 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f37ac004970 tx=0x7f37ac009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 -- 172.21.15.103:0/215395134 shutdown_connections 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 --2- 172.21.15.103:0/215395134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070c70 0x7f37c4109120 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 --2- 172.21.15.103:0/215395134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070360 0x7f37c4070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 -- 172.21.15.103:0/215395134 >> 172.21.15.103:0/215395134 conn(0x7f37c40f8980 msgr2=0x7f37c40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 -- 172.21.15.103:0/215395134 shutdown_connections 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.648+0000 7f37c98c8700 1 -- 172.21.15.103:0/215395134 wait complete. 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 Processor -- start 2024-03-22T02:32:17.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 -- start start 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 0x7f37c41162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41168f0 con 0x7f37c4070360 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c98c8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c410fcc0 con 0x7f37c4070c70 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38188/0 (socket says 172.21.15.103:38188) 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37c2ffd700 1 -- 172.21.15.103:0/2392515214 learned_addr learned my addr 172.21.15.103:0/2392515214 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:17.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.649+0000 7f37ba7fc700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 0x7f37c41162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c2ffd700 1 -- 172.21.15.103:0/2392515214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 msgr2=0x7f37c41162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c2ffd700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 0x7f37c41162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c2ffd700 1 -- 172.21.15.103:0/2392515214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac005040 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37ba7fc700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 0x7f37c41162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c2ffd700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f37b400cb10 tx=0x7f37b4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37b4004200 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c410ffa0 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37b400acc0 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37b4003d90 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.650+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c41104e0 con 0x7f37c4070360 2024-03-22T02:32:17.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.651+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c4071aa0 con 0x7f37c4070360 2024-03-22T02:32:17.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.651+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f37b4003430 con 0x7f37c4070360 2024-03-22T02:32:17.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.652+0000 7f37bbfff700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 0x7f37b0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.652+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f37b408da10 con 0x7f37c4070360 2024-03-22T02:32:17.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.652+0000 7f37ba7fc700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 0x7f37b0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.652+0000 7f37ba7fc700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 0x7f37b0070150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00b370 tx=0x7f37ac009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.657+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f37b405afe0 con 0x7f37c4070360 2024-03-22T02:32:17.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.854+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f37c405f5f0 con 0x7f37c4070360 2024-03-22T02:32:17.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.858+0000 7f37bbfff700 1 -- 172.21.15.103:0/2392515214 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v46)=0 v46) v1 ==== 125+0+0 (secure 0 0 0) 0x7f37b405ab70 con 0x7f37c4070360 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 msgr2=0x7f37b0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 0x7f37b0070150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00b370 tx=0x7f37ac009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 msgr2=0x7f37c4115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f37b400cb10 tx=0x7f37b4007f90 comp rx=0 tx=0).stop 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 shutdown_connections 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f37b006dca0 0x7f37b0070150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37c4070360 0x7f37c4115d60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 --2- 172.21.15.103:0/2392515214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37c4070c70 0x7f37c41162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 >> 172.21.15.103:0/2392515214 conn(0x7f37c40f8980 msgr2=0x7f37c40f9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 shutdown_connections 2024-03-22T02:32:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:17.861+0000 7f37c98c8700 1 -- 172.21.15.103:0/2392515214 wait complete. 2024-03-22T02:32:18.470 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-22T02:32:18.471 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:32:18.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-22T02:32:18.739 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/2392515214' entity='client.admin' 2024-03-22T02:32:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: pgmap v76: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:18 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/2392515214' entity='client.admin' 2024-03-22T02:32:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: pgmap v76: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:18 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:20.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- 172.21.15.103:0/1576803049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fe7c0 msgr2=0x7f3758106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:20.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 --2- 172.21.15.103:0/1576803049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fe7c0 0x7f3758106e90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3740004970 tx=0x7f3740009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- 172.21.15.103:0/1576803049 shutdown_connections 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 --2- 172.21.15.103:0/1576803049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fe7c0 0x7f3758106e90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 --2- 172.21.15.103:0/1576803049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fde20 0x7f37580fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- 172.21.15.103:0/1576803049 >> 172.21.15.103:0/1576803049 conn(0x7f37580f7860 msgr2=0x7f37580f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- 172.21.15.103:0/1576803049 shutdown_connections 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- 172.21.15.103:0/1576803049 wait complete. 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 Processor -- start 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.430+0000 7f375d502700 1 -- start start 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f375d502700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f375d502700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fe7c0 0x7f37580fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f375d502700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37580fdcc0 con 0x7f37580fde20 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f375d502700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37580fc760 con 0x7f37580fe7c0 2024-03-22T02:32:20.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38210/0 (socket says 172.21.15.103:38210) 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 -- 172.21.15.103:0/739000312 learned_addr learned my addr 172.21.15.103:0/739000312 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f37567fc700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fe7c0 0x7f37580fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 -- 172.21.15.103:0/739000312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fe7c0 msgr2=0x7f37580fc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fe7c0 0x7f37580fc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 -- 172.21.15.103:0/739000312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740005040 con 0x7f37580fde20 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f3756ffd700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f374800bbb0 tx=0x7f3748009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37480092e0 con 0x7f37580fde20 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f374800cbd0 con 0x7f37580fde20 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37580fca40 con 0x7f37580fde20 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.431+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3748007400 con 0x7f37580fde20 2024-03-22T02:32:20.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.432+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375811a140 con 0x7f37580fde20 2024-03-22T02:32:20.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.433+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3758066a50 con 0x7f37580fde20 2024-03-22T02:32:20.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.433+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f3748012070 con 0x7f37580fde20 2024-03-22T02:32:20.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.433+0000 7f374f7fe700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 0x7f3744070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:20.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.433+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f374808d0d0 con 0x7f37580fde20 2024-03-22T02:32:20.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.434+0000 7f37567fc700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 0x7f3744070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:20.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.434+0000 7f37567fc700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 0x7f3744070150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3740004f50 tx=0x7f3740009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:20.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.438+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f3748056b70 con 0x7f37580fde20 2024-03-22T02:32:20.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.634+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f375804c710 con 0x7f37580fde20 2024-03-22T02:32:20.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.635+0000 7f374f7fe700 1 -- 172.21.15.103:0/739000312 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v46)=0 v46) v1 ==== 155+0+0 (secure 0 0 0) 0x7f374805a230 con 0x7f37580fde20 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 msgr2=0x7f3744070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 0x7f3744070150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3740004f50 tx=0x7f3740009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 msgr2=0x7f37580fbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f374800bbb0 tx=0x7f3748009840 comp rx=0 tx=0).stop 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 shutdown_connections 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f374406dca0 0x7f3744070150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37580fde20 0x7f37580fbce0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 --2- 172.21.15.103:0/739000312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f37580fe7c0 0x7f37580fc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:20.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 >> 172.21.15.103:0/739000312 conn(0x7f37580f7860 msgr2=0x7f3758101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:20.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 shutdown_connections 2024-03-22T02:32:20.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:20.637+0000 7f375d502700 1 -- 172.21.15.103:0/739000312 wait complete. 2024-03-22T02:32:21.258 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:21 smithi103 ceph-mon[29258]: pgmap v77: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:21.284 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-22T02:32:21.542 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:21.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:21 smithi122 ceph-mon[36403]: pgmap v77: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:23.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 -- 172.21.15.103:0/2684316643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 msgr2=0x7f34d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 --2- 172.21.15.103:0/2684316643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d41031d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f34d0004940 tx=0x7f34d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 -- 172.21.15.103:0/2684316643 shutdown_connections 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 --2- 172.21.15.103:0/2684316643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 0x7f34d40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 --2- 172.21.15.103:0/2684316643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d41031d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.230+0000 7f34dc177700 1 -- 172.21.15.103:0/2684316643 >> 172.21.15.103:0/2684316643 conn(0x7f34d40f89b0 msgr2=0x7f34d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 -- 172.21.15.103:0/2684316643 shutdown_connections 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 -- 172.21.15.103:0/2684316643 wait complete. 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 Processor -- start 2024-03-22T02:32:23.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 -- start start 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 0x7f34d4113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.231+0000 7f34dc177700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4114510 con 0x7f34d4102e00 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34dc177700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d410db10 con 0x7f34d41037a0 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38220/0 (socket says 172.21.15.103:38220) 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 -- 172.21.15.103:0/706587561 learned_addr learned my addr 172.21.15.103:0/706587561 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9712700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 0x7f34d4113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 -- 172.21.15.103:0/706587561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 msgr2=0x7f34d4113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 0x7f34d4113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 -- 172.21.15.103:0/706587561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0005040 con 0x7f34d4102e00 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34d9f13700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f34d000b8b0 tx=0x7f34d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d000e040 con 0x7f34d4102e00 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34d0004290 con 0x7f34d4102e00 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d001c3f0 con 0x7f34d4102e00 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d410dd90 con 0x7f34d4102e00 2024-03-22T02:32:23.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.232+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d410e300 con 0x7f34d4102e00 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.233+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4100110 con 0x7f34d4102e00 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.234+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f34d00137b0 con 0x7f34d4102e00 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.234+0000 7f34ca7fc700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 0x7f34c0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.234+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f34d008e770 con 0x7f34d4102e00 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.234+0000 7f34d9712700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 0x7f34c0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:23.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.235+0000 7f34d9712700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 0x7f34c0070150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34c4007fd0 tx=0x7f34c4009040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:23.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.239+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f34d005bdc0 con 0x7f34d4102e00 2024-03-22T02:32:23.239 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:23 smithi103 ceph-mon[29258]: pgmap v78: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:23.240 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:23 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:23.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.432+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f34d410efd0 con 0x7f34d4102e00 2024-03-22T02:32:23.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.433+0000 7f34ca7fc700 1 -- 172.21.15.103:0/706587561 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v46)=0 v46) v1 ==== 163+0+0 (secure 0 0 0) 0x7f34d005b950 con 0x7f34d4102e00 2024-03-22T02:32:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 msgr2=0x7f34c0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 0x7f34c0070150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34c4007fd0 tx=0x7f34c4009040 comp rx=0 tx=0).stop 2024-03-22T02:32:23.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 msgr2=0x7f34d4113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f34d000b8b0 tx=0x7f34d0009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 shutdown_connections 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f34c006dca0 0x7f34c0070150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34d4102e00 0x7f34d4113980 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 --2- 172.21.15.103:0/706587561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34d41037a0 0x7f34d4113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 >> 172.21.15.103:0/706587561 conn(0x7f34d40f89b0 msgr2=0x7f34d4107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 shutdown_connections 2024-03-22T02:32:23.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:23.435+0000 7f34dc177700 1 -- 172.21.15.103:0/706587561 wait complete. 2024-03-22T02:32:23.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:23 smithi122 ceph-mon[36403]: pgmap v78: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:23 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:24.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-22T02:32:24.303 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:25.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:25 smithi103 ceph-mon[29258]: pgmap v79: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:25 smithi122 ceph-mon[36403]: pgmap v79: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 -- 172.21.15.103:0/385866138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96701037a0 msgr2=0x7f96700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 --2- 172.21.15.103:0/385866138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96701037a0 0x7f96700fce10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9660004940 tx=0x7f9660009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 -- 172.21.15.103:0/385866138 shutdown_connections 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 --2- 172.21.15.103:0/385866138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96701037a0 0x7f96700fce10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 --2- 172.21.15.103:0/385866138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9670102e00 0x7f96701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 -- 172.21.15.103:0/385866138 >> 172.21.15.103:0/385866138 conn(0x7f96700f89b0 msgr2=0x7f96700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 -- 172.21.15.103:0/385866138 shutdown_connections 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.860+0000 7f96782cb700 1 -- 172.21.15.103:0/385866138 wait complete. 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 Processor -- start 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 -- start start 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:25.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96701037a0 0x7f96701160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9670116730 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f96782cb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967010fd60 con 0x7f96701037a0 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60418/0 (socket says 172.21.15.103:60418) 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 -- 172.21.15.103:0/3805422269 learned_addr learned my addr 172.21.15.103:0/3805422269 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 -- 172.21.15.103:0/3805422269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96701037a0 msgr2=0x7f96701160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96701037a0 0x7f96701160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.861+0000 7f9676067700 1 -- 172.21.15.103:0/3805422269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660005040 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f9676067700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c8c0 tx=0x7f966c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f966c00a980 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f966c00ce60 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9670110040 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f966c0115e0 con 0x7f9670102e00 2024-03-22T02:32:25.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.862+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9670110580 con 0x7f9670102e00 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.863+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9670102120 con 0x7f9670102e00 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.863+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f966c012400 con 0x7f9670102e00 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.864+0000 7f9666ffd700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 0x7f965c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.864+0000 7f9675866700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 0x7f965c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.864+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f966c08d410 con 0x7f9670102e00 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.864+0000 7f9675866700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 0x7f965c070150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9660004f90 tx=0x7f9660009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:25.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:25.868+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f966c05ab00 con 0x7f9670102e00 2024-03-22T02:32:26.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.064+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f96700fe550 con 0x7f9670102e00 2024-03-22T02:32:26.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.065+0000 7f9666ffd700 1 -- 172.21.15.103:0/3805422269 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v46)=0 v46) v1 ==== 135+0+0 (secure 0 0 0) 0x7f966c0126b0 con 0x7f9670102e00 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 msgr2=0x7f965c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 0x7f965c070150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9660004f90 tx=0x7f9660009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 msgr2=0x7f9670115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c8c0 tx=0x7f966c0044c0 comp rx=0 tx=0).stop 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 shutdown_connections 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f965c06dca0 0x7f965c070150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:26.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9670102e00 0x7f9670115ba0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:26.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 --2- 172.21.15.103:0/3805422269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96701037a0 0x7f96701160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:26.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 >> 172.21.15.103:0/3805422269 conn(0x7f96700f89b0 msgr2=0x7f9670070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:26.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 shutdown_connections 2024-03-22T02:32:26.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:26.067+0000 7f96782cb700 1 -- 172.21.15.103:0/3805422269 wait complete. 2024-03-22T02:32:26.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-22T02:32:26.941 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:27 smithi103 ceph-mon[29258]: pgmap v80: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:27.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:27 smithi122 ceph-mon[36403]: pgmap v80: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:28.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 -- 172.21.15.103:0/2291439551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 msgr2=0x7fe1a8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:28.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/2291439551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a8102140 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe19c004940 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:28.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 -- 172.21.15.103:0/2291439551 shutdown_connections 2024-03-22T02:32:28.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/2291439551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1a8102710 0x7fe1a80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/2291439551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a8102140 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 -- 172.21.15.103:0/2291439551 >> 172.21.15.103:0/2291439551 conn(0x7fe1a8072bc0 msgr2=0x7fe1a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 -- 172.21.15.103:0/2291439551 shutdown_connections 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.517+0000 7fe1b05b5700 1 -- 172.21.15.103:0/2291439551 wait complete. 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 Processor -- start 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 -- start start 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1a8102710 0x7fe1a810f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a810f760 con 0x7fe1a8101d70 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a810f8a0 con 0x7fe1a8102710 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:28.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60440/0 (socket says 172.21.15.103:60440) 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 -- 172.21.15.103:0/3677751788 learned_addr learned my addr 172.21.15.103:0/3677751788 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 -- 172.21.15.103:0/3677751788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1a8102710 msgr2=0x7fe1a810f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1a8102710 0x7fe1a810f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 -- 172.21.15.103:0/3677751788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c005040 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1ae351700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00b3b0 tx=0x7fe19c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19c00e040 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe19c00bbc0 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8117b60 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe19c01c3f0 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.518+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a81180f0 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.519+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a80ff080 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.520+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fe19c00bd30 con 0x7fe1a8101d70 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.520+0000 7fe19affd700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 0x7fe194070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:28.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.520+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe19c08e5c0 con 0x7fe1a8101d70 2024-03-22T02:32:28.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.520+0000 7fe1adb50700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 0x7fe194070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:28.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.521+0000 7fe1adb50700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 0x7fe194070150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006fd0 tx=0x7fe1a4006db0 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:28.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.525+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fe19c0580e0 con 0x7fe1a8101d70 2024-03-22T02:32:28.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.743+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1a8108ee0 con 0x7fe19406dca0 2024-03-22T02:32:28.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.748+0000 7fe19affd700 1 -- 172.21.15.103:0/3677751788 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fe1a8108ee0 con 0x7fe19406dca0 2024-03-22T02:32:28.749 INFO:teuthology.orchestra.run.smithi103.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:32:28.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.750+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 msgr2=0x7fe194070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:28.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.750+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 0x7fe194070150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006fd0 tx=0x7fe1a4006db0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.750+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 msgr2=0x7fe1a810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.750+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00b3b0 tx=0x7fe19c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 shutdown_connections 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fe19406dca0 0x7fe194070150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1a8101d70 0x7fe1a810ec50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 --2- 172.21.15.103:0/3677751788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1a8102710 0x7fe1a810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 >> 172.21.15.103:0/3677751788 conn(0x7fe1a8072bc0 msgr2=0x7fe1a80faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 shutdown_connections 2024-03-22T02:32:28.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:28.751+0000 7fe1b05b5700 1 -- 172.21.15.103:0/3677751788 wait complete. 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: pgmap v81: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:29.354 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:29 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:29.382 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-22T02:32:29.383 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:32:29.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: pgmap v81: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:29.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:29 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:29.647 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:32:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:30 smithi103 ceph-mon[29258]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:30 smithi103 ceph-mon[29258]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:32:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:30 smithi103 ceph-mon[29258]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:32:30.412 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:30 smithi122 ceph-mon[36403]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:30.412 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:30 smithi122 ceph-mon[36403]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:32:30.412 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:30 smithi122 ceph-mon[36403]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:32:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:31 smithi103 ceph-mon[29258]: pgmap v82: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:31.578 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:31 smithi122 ceph-mon[36403]: pgmap v82: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:31.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 -- 172.21.15.103:0/30556096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fe7c0 msgr2=0x7f5e70106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 --2- 172.21.15.103:0/30556096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fe7c0 0x7f5e70106e00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5e60004940 tx=0x7f5e60009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:31.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 -- 172.21.15.103:0/30556096 shutdown_connections 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 --2- 172.21.15.103:0/30556096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fe7c0 0x7f5e70106e00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 --2- 172.21.15.103:0/30556096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e700fde20 0x7f5e700fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 -- 172.21.15.103:0/30556096 >> 172.21.15.103:0/30556096 conn(0x7f5e700f7870 msgr2=0x7f5e700f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 -- 172.21.15.103:0/30556096 shutdown_connections 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 -- 172.21.15.103:0/30556096 wait complete. 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.652+0000 7f5e7793e700 1 Processor -- start 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 -- start start 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e700fe7c0 0x7f5e70113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70114590 con 0x7f5e700fde20 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7010db00 con 0x7f5e700fe7c0 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60466/0 (socket says 172.21.15.103:60466) 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 -- 172.21.15.103:0/2163583428 learned_addr learned my addr 172.21.15.103:0/2163583428 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:31.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 -- 172.21.15.103:0/2163583428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e700fe7c0 msgr2=0x7f5e70113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e700fe7c0 0x7f5e70113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 -- 172.21.15.103:0/2163583428 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60005040 con 0x7f5e700fde20 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e756da700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00b790 tx=0x7f5e6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6c00f770 con 0x7f5e700fde20 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e6c00fda0 con 0x7f5e700fde20 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7010dde0 con 0x7f5e700fde20 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.653+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6c00d310 con 0x7f5e700fde20 2024-03-22T02:32:31.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.654+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7010e320 con 0x7f5e700fde20 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.655+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70101770 con 0x7f5e700fde20 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.655+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f5e6c010420 con 0x7f5e700fde20 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.655+0000 7f5e65ffb700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 0x7f5e5c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.656+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5e6c08d450 con 0x7f5e700fde20 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.656+0000 7f5e74ed9700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 0x7f5e5c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.656+0000 7f5e74ed9700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 0x7f5e5c070150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e60004f90 tx=0x7f5e60009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.660+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f5e6c05aaa0 con 0x7f5e700fde20 2024-03-22T02:32:31.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.880+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e70109f30 con 0x7f5e5c06dca0 2024-03-22T02:32:31.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.882+0000 7f5e65ffb700 1 -- 172.21.15.103:0/2163583428 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f5e70109f30 con 0x7f5e5c06dca0 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 msgr2=0x7f5e5c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 0x7f5e5c070150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e60004f90 tx=0x7f5e60009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 msgr2=0x7f5e70113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00b790 tx=0x7f5e6c009250 comp rx=0 tx=0).stop 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 shutdown_connections 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f5e5c06dca0 0x7f5e5c070150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e700fde20 0x7f5e70113970 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 --2- 172.21.15.103:0/2163583428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e700fe7c0 0x7f5e70113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 >> 172.21.15.103:0/2163583428 conn(0x7f5e700f7870 msgr2=0x7f5e700ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 shutdown_connections 2024-03-22T02:32:31.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.884+0000 7f5e7793e700 1 -- 172.21.15.103:0/2163583428 wait complete. 2024-03-22T02:32:31.896 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:32:31.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 -- 172.21.15.103:0/3717707157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 msgr2=0x7f67580fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 --2- 172.21.15.103:0/3717707157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f67580fb820 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f674c004940 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:31.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 -- 172.21.15.103:0/3717707157 shutdown_connections 2024-03-22T02:32:31.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 --2- 172.21.15.103:0/3717707157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f67580fb820 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 --2- 172.21.15.103:0/3717707157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 -- 172.21.15.103:0/3717707157 >> 172.21.15.103:0/3717707157 conn(0x7f6758072310 msgr2=0x7f6758072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 -- 172.21.15.103:0/3717707157 shutdown_connections 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.986+0000 7f675e6fe700 1 -- 172.21.15.103:0/3717707157 wait complete. 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 Processor -- start 2024-03-22T02:32:31.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 -- start start 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67581144f0 con 0x7f67580fb3b0 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f675e6fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675810daf0 con 0x7f6758102f70 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60492/0 (socket says 172.21.15.103:60492) 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 -- 172.21.15.103:0/2511148610 learned_addr learned my addr 172.21.15.103:0/2511148610 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f67577fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 -- 172.21.15.103:0/2511148610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 msgr2=0x7f6758113ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758113ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f6757fff700 1 -- 172.21.15.103:0/2511148610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c005040 con 0x7f67580fb3b0 2024-03-22T02:32:31.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.987+0000 7f67577fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758113ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f6757fff700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f674800c830 tx=0x7f6748004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674800a760 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675810ddd0 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f674800ad90 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67480113a0 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.988+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675810e390 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.989+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6758105350 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.989+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f6748012400 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.990+0000 7f6754ff9700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 0x7f6740070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.990+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f674808d460 con 0x7f67580fb3b0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.990+0000 7f67577fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 0x7f6740070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.990+0000 7f67577fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 0x7f6740070150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f674c000c00 tx=0x7f674c009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:31.994+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f674805aab0 con 0x7f67580fb3b0 2024-03-22T02:32:32.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.209+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f675810f020 con 0x7f674006dca0 2024-03-22T02:32:32.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.210+0000 7f6754ff9700 1 -- 172.21.15.103:0/2511148610 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f675810f020 con 0x7f674006dca0 2024-03-22T02:32:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 msgr2=0x7f6740070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 0x7f6740070150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f674c000c00 tx=0x7f674c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 msgr2=0x7f6758113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f674800c830 tx=0x7f6748004470 comp rx=0 tx=0).stop 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 shutdown_connections 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f674006dca0 0x7f6740070150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67580fb3b0 0x7f6758113960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 --2- 172.21.15.103:0/2511148610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6758102f70 0x7f6758113ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.212+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 >> 172.21.15.103:0/2511148610 conn(0x7f6758072310 msgr2=0x7f67580f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.213+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 shutdown_connections 2024-03-22T02:32:32.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.213+0000 7f675e6fe700 1 -- 172.21.15.103:0/2511148610 wait complete. 2024-03-22T02:32:32.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 -- 172.21.15.103:0/3509979078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 msgr2=0x7fc4180fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 --2- 172.21.15.103:0/3509979078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc4180fbd80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc40c004940 tx=0x7fc40c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:32.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 -- 172.21.15.103:0/3509979078 shutdown_connections 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 --2- 172.21.15.103:0/3509979078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc4180fbd80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 --2- 172.21.15.103:0/3509979078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 0x7fc418102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 -- 172.21.15.103:0/3509979078 >> 172.21.15.103:0/3509979078 conn(0x7fc418072bc0 msgr2=0x7fc418072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 -- 172.21.15.103:0/3509979078 shutdown_connections 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 -- 172.21.15.103:0/3509979078 wait complete. 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.317+0000 7fc41df9d700 1 Processor -- start 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc41df9d700 1 -- start start 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc41df9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 0x7fc418112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc41df9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc41df9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418113630 con 0x7fc418102710 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc41df9d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41810ca20 con 0x7fc418101d70 2024-03-22T02:32:32.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60510/0 (socket says 172.21.15.103:60510) 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc4177fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 0x7fc418112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 -- 172.21.15.103:0/182907809 learned_addr learned my addr 172.21.15.103:0/182907809 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 -- 172.21.15.103:0/182907809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 msgr2=0x7fc418112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 0x7fc418112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.318+0000 7fc416ffd700 1 -- 172.21.15.103:0/182907809 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c005040 con 0x7fc418102710 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc416ffd700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009100 tx=0x7fc40c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40c00e040 con 0x7fc418102710 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41810cca0 con 0x7fc418102710 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc40c009af0 con 0x7fc418102710 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40c01c3f0 con 0x7fc418102710 2024-03-22T02:32:32.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.319+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41810d1e0 con 0x7fc418102710 2024-03-22T02:32:32.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.320+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4180fd4c0 con 0x7fc418102710 2024-03-22T02:32:32.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.320+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fc40c00baf0 con 0x7fc418102710 2024-03-22T02:32:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.320+0000 7fc3fffff700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 0x7fc400072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.321+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc40c08e670 con 0x7fc418102710 2024-03-22T02:32:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.321+0000 7fc4177fe700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 0x7fc400072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.321+0000 7fc4177fe700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 0x7fc400072360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc408004eb0 tx=0x7fc40800b040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:32.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.325+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fc40c05bcc0 con 0x7fc418102710 2024-03-22T02:32:32.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.532+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc418108ec0 con 0x7fc40006feb0 2024-03-22T02:32:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.542+0000 7fc3fffff700 1 -- 172.21.15.103:0/182907809 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc418108ec0 con 0x7fc40006feb0 2024-03-22T02:32:32.544 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:32:32.544 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (64s) 43s ago 12m 16.8M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:32:32.544 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (12m) 43s ago 12m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:32:32.544 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (11m) 98s ago 11m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (48s) 43s ago 11m 51.8M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (6m) 43s ago 6m 12.9M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (6m) 43s ago 6m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (6m) 98s ago 6m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (6m) 98s ago 6m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (2m) 43s ago 13m 508M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (2m) 98s ago 11m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (13m) 43s ago 13m 55.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 79436f03000f 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (11m) 98s ago 11m 43.7M 2048M 17.2.7-726-g9c4e0c34 837124da804f 5d70e3dc6601 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (106s) 43s ago 12m 16.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (102s) 98s ago 11m 3611k - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (10m) 43s ago 10m 214M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:32:32.545 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (10m) 43s ago 10m 230M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:32:32.546 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (9m) 43s ago 9m 198M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:32:32.546 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (9m) 98s ago 9m 262M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:32:32.546 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (8m) 98s ago 8m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:32:32.546 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (8m) 98s ago 8m 217M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:32:32.546 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (84s) 43s ago 11m 49.4M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.544+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 msgr2=0x7fc400072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.544+0000 7fc41df9d700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 0x7fc400072360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc408004eb0 tx=0x7fc40800b040 comp rx=0 tx=0).stop 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.544+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 msgr2=0x7fc418112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.544+0000 7fc41df9d700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009100 tx=0x7fc40c009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 shutdown_connections 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc40006feb0 0x7fc400072360 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc418101d70 0x7fc418112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 --2- 172.21.15.103:0/182907809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc418102710 0x7fc418112fe0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 >> 172.21.15.103:0/182907809 conn(0x7fc418072bc0 msgr2=0x7fc4180fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 shutdown_connections 2024-03-22T02:32:32.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.545+0000 7fc41df9d700 1 -- 172.21.15.103:0/182907809 wait complete. 2024-03-22T02:32:32.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 -- 172.21.15.103:0/3243831945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 msgr2=0x7fd6200fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 --2- 172.21.15.103:0/3243831945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd6200fe260 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd618004940 tx=0x7fd618009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 -- 172.21.15.103:0/3243831945 shutdown_connections 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 --2- 172.21.15.103:0/3243831945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd620106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 --2- 172.21.15.103:0/3243831945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd6200fe260 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 -- 172.21.15.103:0/3243831945 >> 172.21.15.103:0/3243831945 conn(0x7fd620072bc0 msgr2=0x7fd620072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 -- 172.21.15.103:0/3243831945 shutdown_connections 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.645+0000 7fd6277d2700 1 -- 172.21.15.103:0/3243831945 wait complete. 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 Processor -- start 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 -- start start 2024-03-22T02:32:32.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd6201140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd620114720 con 0x7fd6200fde90 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62010db30 con 0x7fd6200fe830 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60528/0 (socket says 172.21.15.103:60528) 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 -- 172.21.15.103:0/938569885 learned_addr learned my addr 172.21.15.103:0/938569885 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd624d6d700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd6201140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 -- 172.21.15.103:0/938569885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 msgr2=0x7fd6201140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd6201140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 -- 172.21.15.103:0/938569885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618005040 con 0x7fd6200fde90 2024-03-22T02:32:32.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd624d6d700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd6201140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd62556e700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd618009360 tx=0x7fd618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd61800e040 con 0x7fd6200fde90 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.646+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62010ddb0 con 0x7fd6200fde90 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.647+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd618013420 con 0x7fd6200fde90 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.647+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd61801c630 con 0x7fd6200fde90 2024-03-22T02:32:32.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.647+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62010e2f0 con 0x7fd6200fde90 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.648+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd620101980 con 0x7fd6200fde90 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.648+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fd618013a90 con 0x7fd6200fde90 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.648+0000 7fd615ffb700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 0x7fd60c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.649+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd61808e620 con 0x7fd6200fde90 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.649+0000 7fd624d6d700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 0x7fd60c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:32.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.649+0000 7fd624d6d700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 0x7fd60c070150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd610007fd0 tx=0x7fd610009040 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:32.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.653+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fd618058140 con 0x7fd6200fde90 2024-03-22T02:32:32.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.943+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd62004bed0 con 0x7fd6200fde90 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:32:32.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 12, 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:32:32.947 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.943+0000 7fd615ffb700 1 -- 172.21.15.103:0/938569885 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fd61805b800 con 0x7fd6200fde90 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.945+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 msgr2=0x7fd60c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.945+0000 7fd6277d2700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 0x7fd60c070150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd610007fd0 tx=0x7fd610009040 comp rx=0 tx=0).stop 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.945+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 msgr2=0x7fd620113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.945+0000 7fd6277d2700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd618009360 tx=0x7fd618009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 shutdown_connections 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fd60c06dca0 0x7fd60c070150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6200fde90 0x7fd620113b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 --2- 172.21.15.103:0/938569885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6200fe830 0x7fd6201140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 >> 172.21.15.103:0/938569885 conn(0x7fd620072bc0 msgr2=0x7fd6200ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 shutdown_connections 2024-03-22T02:32:32.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:32.946+0000 7fd6277d2700 1 -- 172.21.15.103:0/938569885 wait complete. 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.058+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3786650805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 msgr2=0x7fc2f00fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.058+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3786650805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f00fe1e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec004940 tx=0x7fc2ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3786650805 shutdown_connections 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3786650805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f0106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3786650805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f00fe1e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3786650805 >> 172.21.15.103:0/3786650805 conn(0x7fc2f00f7880 msgr2=0x7fc2f00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3786650805 shutdown_connections 2024-03-22T02:32:33.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.059+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3786650805 wait complete. 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 Processor -- start 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 -- start start 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f01143e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0114a30 con 0x7fc2f00fde10 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f7c49700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f010de50 con 0x7fc2f00fe7b0 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60544/0 (socket says 172.21.15.103:60544) 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 -- 172.21.15.103:0/3188676216 learned_addr learned my addr 172.21.15.103:0/3188676216 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f51e4700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f01143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 -- 172.21.15.103:0/3188676216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 msgr2=0x7fc2f01143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f01143e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f59e5700 1 -- 172.21.15.103:0/3188676216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec005040 con 0x7fc2f00fde10 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.060+0000 7fc2f51e4700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f01143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2f59e5700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009380 tx=0x7fc2ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2ec00e040 con 0x7fc2f00fde10 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f010e0d0 con 0x7fc2f00fde10 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2ec009af0 con 0x7fc2f00fde10 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2ec01c630 con 0x7fc2f00fde10 2024-03-22T02:32:33.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.061+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f010e610 con 0x7fc2f00fde10 2024-03-22T02:32:33.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.062+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0066770 con 0x7fc2f00fde10 2024-03-22T02:32:33.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.062+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7fc2ec01c8a0 con 0x7fc2f00fde10 2024-03-22T02:32:33.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.063+0000 7fc2e67fc700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 0x7fc2dc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.063+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc2ec08e710 con 0x7fc2f00fde10 2024-03-22T02:32:33.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.063+0000 7fc2f51e4700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 0x7fc2dc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.063+0000 7fc2f51e4700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 0x7fc2dc070150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fd0 tx=0x7fc2e0009ee0 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.072+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7fc2ec05bd60 con 0x7fc2f00fde10 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: pgmap v83: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T02:32:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:33 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/938569885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:33.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.338+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2f004bed0 con 0x7fc2f00fde10 2024-03-22T02:32:33.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.339+0000 7fc2e67fc700 1 -- 172.21.15.103:0/3188676216 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1858 (secure 0 0 0) 0x7fc2ec00bbc0 con 0x7fc2f00fde10 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:e19 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 19 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:29:14.569041+0000 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:32:33.341 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:32:33.342 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:32:33.343 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:32:33.344 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:33.344 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 msgr2=0x7fc2dc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 0x7fc2dc070150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fd0 tx=0x7fc2e0009ee0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 msgr2=0x7fc2f0113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009380 tx=0x7fc2ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 shutdown_connections 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7fc2dc06dca0 0x7fc2dc070150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f00fde10 0x7fc2f0113ea0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 --2- 172.21.15.103:0/3188676216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2f00fe7b0 0x7fc2f01143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 >> 172.21.15.103:0/3188676216 conn(0x7fc2f00f7880 msgr2=0x7fc2f00f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 shutdown_connections 2024-03-22T02:32:33.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.341+0000 7fc2f7c49700 1 -- 172.21.15.103:0/3188676216 wait complete. 2024-03-22T02:32:33.346 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 19 2024-03-22T02:32:33.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 -- 172.21.15.103:0/3317029624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 msgr2=0x7f311c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 --2- 172.21.15.103:0/3317029624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c106eb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3110004940 tx=0x7f3110009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 -- 172.21.15.103:0/3317029624 shutdown_connections 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 --2- 172.21.15.103:0/3317029624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c106eb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 --2- 172.21.15.103:0/3317029624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.455+0000 7f31224d3700 1 -- 172.21.15.103:0/3317029624 >> 172.21.15.103:0/3317029624 conn(0x7f311c0f7860 msgr2=0x7f311c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.456+0000 7f31224d3700 1 -- 172.21.15.103:0/3317029624 shutdown_connections 2024-03-22T02:32:33.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.456+0000 7f31224d3700 1 -- 172.21.15.103:0/3317029624 wait complete. 2024-03-22T02:32:33.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.456+0000 7f31224d3700 1 Processor -- start 2024-03-22T02:32:33.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f31224d3700 1 -- start start 2024-03-22T02:32:33.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f31224d3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f31224d3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f31224d3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0fdc50 con 0x7f311c0fe7e0 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f31224d3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0fc780 con 0x7f311c0fde40 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60568/0 (socket says 172.21.15.103:60568) 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f311b7fe700 1 -- 172.21.15.103:0/417718663 learned_addr learned my addr 172.21.15.103:0/417718663 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.457+0000 7f311bfff700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f311bfff700 1 -- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 msgr2=0x7f311c0fc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f311bfff700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f311bfff700 1 -- 172.21.15.103:0/417718663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110005040 con 0x7f311c0fde40 2024-03-22T02:32:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f311b7fe700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f311bfff700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fbd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f310c002820 tx=0x7f310c004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310c007cc0 con 0x7f311c0fde40 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f310c00f040 con 0x7f311c0fde40 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310c013920 con 0x7f311c0fde40 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c0fca60 con 0x7f311c0fde40 2024-03-22T02:32:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.458+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c11a120 con 0x7f311c0fde40 2024-03-22T02:32:33.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.460+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c066a30 con 0x7f311c0fde40 2024-03-22T02:32:33.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.460+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f310c013ac0 con 0x7f311c0fde40 2024-03-22T02:32:33.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.460+0000 7f3118ff9700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 0x7f3104070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.460+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f310c08d050 con 0x7f311c0fde40 2024-03-22T02:32:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.461+0000 7f311b7fe700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 0x7f3104070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.461+0000 7f311b7fe700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 0x7f3104070150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f311c0fd430 tx=0x7f3110009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.466+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f310c056be0 con 0x7f311c0fde40 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: pgmap v83: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T02:32:33.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:33 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/938569885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:33.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.711+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 --> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f311c0f9420 con 0x7f310406dca0 2024-03-22T02:32:33.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.713+0000 7f3118ff9700 1 -- 172.21.15.103:0/417718663 <== mgr.14702 v2:172.21.15.103:6800/375109873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f311c0f9420 con 0x7f310406dca0 2024-03-22T02:32:33.714 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:32:33.714 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:32:33.714 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr" 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "2/21 daemons upgraded", 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:32:33.715 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:32:33.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 msgr2=0x7f3104070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 0x7f3104070150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f311c0fd430 tx=0x7f3110009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 msgr2=0x7f311c0fbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fbd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f310c002820 tx=0x7f310c004570 comp rx=0 tx=0).stop 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 shutdown_connections 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f310406dca0 0x7f3104070150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f311c0fde40 0x7f311c0fbd00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 --2- 172.21.15.103:0/417718663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f311c0fe7e0 0x7f311c0fc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.715+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 >> 172.21.15.103:0/417718663 conn(0x7f311c0f7860 msgr2=0x7f311c0f8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.716+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 shutdown_connections 2024-03-22T02:32:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.716+0000 7f31224d3700 1 -- 172.21.15.103:0/417718663 wait complete. 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 -- 172.21.15.103:0/1421321228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070c10 msgr2=0x7f69b4109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1421321228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070c10 0x7f69b4109180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69a4004970 tx=0x7f69a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 -- 172.21.15.103:0/1421321228 shutdown_connections 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1421321228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070c10 0x7f69b4109180 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1421321228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070300 0x7f69b40706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 -- 172.21.15.103:0/1421321228 >> 172.21.15.103:0/1421321228 conn(0x7f69b40f8980 msgr2=0x7f69b40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 -- 172.21.15.103:0/1421321228 shutdown_connections 2024-03-22T02:32:33.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.821+0000 7f69bb46a700 1 -- 172.21.15.103:0/1421321228 wait complete. 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 Processor -- start 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 -- start start 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070c10 0x7f69b4116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b41168d0 con 0x7f69b4070300 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69bb46a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b410fca0 con 0x7f69b4070c10 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44490/0 (socket says 172.21.15.103:44490) 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 -- 172.21.15.103:0/1915444499 learned_addr learned my addr 172.21.15.103:0/1915444499 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b8a05700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070c10 0x7f69b4116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 -- 172.21.15.103:0/1915444499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070c10 msgr2=0x7f69b4116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070c10 0x7f69b4116280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.822+0000 7f69b9206700 1 -- 172.21.15.103:0/1915444499 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4005040 con 0x7f69b4070300 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69b9206700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c8c0 tx=0x7f69b00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69b000a760 con 0x7f69b4070300 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69b000ad90 con 0x7f69b4070300 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b410ff80 con 0x7f69b4070300 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69b00113a0 con 0x7f69b4070300 2024-03-22T02:32:33.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.823+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b4110440 con 0x7f69b4070300 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.824+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b4071a40 con 0x7f69b4070300 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.825+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 34) v1 ==== 91933+0+0 (secure 0 0 0) 0x7f69b0012400 con 0x7f69b4070300 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.825+0000 7f69a9ffb700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 0x7f69a0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.825+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f69b008d420 con 0x7f69b4070300 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.825+0000 7f69b8a05700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 0x7f69a0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:33.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.826+0000 7f69b8a05700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 0x7f69a0070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f69a4004f50 tx=0x7f69a4009e30 comp rx=0 tx=0).ready entity=mgr.14702 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:33.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:33.830+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184357 (secure 0 0 0) 0x7f69b005aa70 con 0x7f69b4070300 2024-03-22T02:32:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.135+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f69b405f5f0 con 0x7f69b4070300 2024-03-22T02:32:34.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.136+0000 7f69a9ffb700 1 -- 172.21.15.103:0/1915444499 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f69b005a600 con 0x7f69b4070300 2024-03-22T02:32:34.138 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 msgr2=0x7f69a0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 0x7f69a0070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f69a4004f50 tx=0x7f69a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 msgr2=0x7f69b4115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c8c0 tx=0x7f69b00044c0 comp rx=0 tx=0).stop 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 shutdown_connections 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:6800/375109873,v1:172.21.15.103:6801/375109873] conn(0x7f69a006dca0 0x7f69a0070150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b4070300 0x7f69b4115d40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 --2- 172.21.15.103:0/1915444499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b4070c10 0x7f69b4116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 >> 172.21.15.103:0/1915444499 conn(0x7f69b40f8980 msgr2=0x7f69b40f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 shutdown_connections 2024-03-22T02:32:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:32:34.138+0000 7f69bb46a700 1 -- 172.21.15.103:0/1915444499 wait complete. 2024-03-22T02:32:34.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:34 smithi103 ceph-mon[29258]: Upgrade: Target is version 19.0.0-2344-gc3a04c0e (squid) 2024-03-22T02:32:34.527 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:34 smithi103 ceph-mon[29258]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0'] 2024-03-22T02:32:34.527 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:34 smithi103 ceph-mon[29258]: Upgrade: Setting container_image for all mgr 2024-03-22T02:32:34.527 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:34 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/3188676216' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:32:34.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:34 smithi122 ceph-mon[36403]: Upgrade: Target is version 19.0.0-2344-gc3a04c0e (squid) 2024-03-22T02:32:34.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:34 smithi122 ceph-mon[36403]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0'] 2024-03-22T02:32:34.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:34 smithi122 ceph-mon[36403]: Upgrade: Setting container_image for all mgr 2024-03-22T02:32:34.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:34 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/3188676216' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:32:35.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: pgmap v84: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='client.? 172.21.15.103:0/1915444499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: Upgrade: Updating mon.smithi103 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:32:35.381 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:35 smithi103 ceph-mon[29258]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:35.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: pgmap v84: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='client.? 172.21.15.103:0/1915444499' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: Upgrade: Updating mon.smithi103 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:32:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:35 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:36.234 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:36 smithi103 ceph-mon[29258]: Deploying daemon mon.smithi103 on smithi103 2024-03-22T02:32:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:36 smithi122 ceph-mon[36403]: Deploying daemon mon.smithi103 on smithi103 2024-03-22T02:32:36.783 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:36 smithi103 systemd[1]: Stopping Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:32:36.784 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:36 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[29254]: 2024-03-22T02:32:36.770+0000 7f0577e0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi103 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:32:36.784 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:36 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[29254]: 2024-03-22T02:32:36.770+0000 7f0577e0f700 -1 mon.smithi103@0(leader) e2 *** Got Signal Terminated *** 2024-03-22T02:32:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:36 smithi103 podman[115034]: 2024-03-22 02:32:36.946013298 +0000 UTC m=+0.274695922 container died 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-22T02:32:37.581 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 podman[115034]: 2024-03-22 02:32:37.313509906 +0000 UTC m=+0.642192522 container cleanup 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=quincy-9c4e0c3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:32:37.581 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 bash[115034]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103 2024-03-22T02:32:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 podman[115057]: 2024-03-22 02:32:37.581389593 +0000 UTC m=+0.633456430 container remove 79436f03000f8f8ccd180b9d99a51ced8f64fafb2f8c677d375cb716697e3348 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-22T02:32:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service: Deactivated successfully. 2024-03-22T02:32:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 systemd[1]: Stopped Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:32:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service: Consumed 12.841s CPU time. 2024-03-22T02:32:38.272 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:37 smithi103 systemd[1]: Starting Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:32:38.272 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 podman[115192]: 2024-03-22 02:32:38.020017243 +0000 UTC m=+0.017825132 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:32:38.272 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 podman[115192]: 2024-03-22 02:32:38.141140428 +0000 UTC m=+0.138948321 container create 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5) 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 podman[115192]: 2024-03-22 02:32:38.39381498 +0000 UTC m=+0.391622871 container init 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 podman[115192]: 2024-03-22 02:32:38.397142136 +0000 UTC m=+0.394950029 container start 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: pidfile_write: ignore empty --pid-file 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: load: jerasure load: lrc 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: RocksDB version: 7.9.2 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Git sha 0 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: DB SUMMARY 2024-03-22T02:32:38.660 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: DB Session ID: EO05C1UED5XUS9JY8HZI 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1166 Bytes 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi103/store.db dir, Total Num: 1, files: 000033.sst 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi103/store.db: 000031.log size: 2013841 ; 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T02:32:38.661 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.env: 0x564796cbbc60 2024-03-22T02:32:38.662 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T02:32:38.662 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.info_log: 0x564797a999a0 2024-03-22T02:32:38.662 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.statistics: (nil) 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.use_fsync: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:32:38.663 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.db_log_dir: 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.wal_dir: 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.write_buffer_manager: 0x564797a6a1e0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:32:38.664 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.unordered_write: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.row_cache: None 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.wal_filter: None 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:32:38.665 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.wal_compression: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:32:38.666 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_open_files: -1 2024-03-22T02:32:38.667 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Compression algorithms supported: 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kZSTD supported: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:32:38.668 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000015 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.merge_operator: 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_filter: None 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564797a99aa0) 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:32:38.669 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_type: 0 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_index_type: 0 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_shortening: 1 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: checksum: 4 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: no_block_cache: 0 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache: 0x564797a9d090 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_options: 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: capacity : 536870912 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: num_shard_bits : 4 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: strict_capacity_limit : 0 2024-03-22T02:32:38.670 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_compressed: (nil) 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: persistent_cache: (nil) 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size: 4096 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size_deviation: 10 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_restart_interval: 16 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_block_restart_interval: 1 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: metadata_block_size: 4096 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: partition_filters: 0 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: use_delta_encoding: 1 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: filter_policy: bloomfilter 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: whole_key_filtering: 1 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: verify_compression: 0 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: format_version: 5 2024-03-22T02:32:38.671 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_index_compression: 1 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_align: 0 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: max_auto_readahead_size: 262144 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: prepopulate_block_cache: 0 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: initial_auto_readahead_size: 8192 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression: NoCompression 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.num_levels: 7 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:32:38.672 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:32:38.673 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:32:38.674 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:32:38.675 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:32:38.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:32:38.677 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.ttl: 2592000 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:32:38.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12002, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9990aa1b-a7d4-407c-ba6c-57d7353e84cf 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074758426741, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-03-22T02:32:38.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074758433373, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1768423, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11996, "largest_seqno": 13502, "table_properties": {"data_size": 1761409, "index_size": 4271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1733, "raw_key_size": 16278, "raw_average_key_size": 23, "raw_value_size": 1746951, "raw_average_value_size": 2554, "num_data_blocks": 194, "num_entries": 684, "num_filter_entries": 684, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711074758, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9990aa1b-a7d4-407c-ba6c-57d7353e84cf", "db_session_id": "EO05C1UED5XUS9JY8HZI", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074758433481, "job": 1, "event": "recovery_finished"} 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi103/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564797b8c000 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: DB pointer 0x564797b76000 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** DB Stats ** 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:32:38.680 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L0 1/0 1.69 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L6 1/0 6.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Sum 2/0 8.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 513.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T02:32:38.681 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative compaction: 0.00 GB write, 151.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval compaction: 0.00 GB write, 151.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Block cache BinnedLRUCache@0x564797a9d090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-03-22T02:32:38.682 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: starting mon.smithi103 rank 0 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi103 fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???) e2 preinit fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).mds e19 new map 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).mds e19 print_map 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: e19 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: legacy client fscid: 1 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Filesystem 'cephfs' (1) 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: fs_name cephfs 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: epoch 19 2024-03-22T02:32:38.683 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: modified 2024-03-22T02:29:14.569041+0000 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: tableserver 0 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: root 0 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: session_timeout 60 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: session_autoclose 300 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: max_file_size 1099511627776 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: max_xattr_size 65536 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: required_client_features {} 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: last_failure 0 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: last_failure_osd_epoch 46 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: max_mds 2 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: in 0,1 2024-03-22T02:32:38.684 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: up {0=14472,1=24271} 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: failed 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: damaged 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: stopped 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_pools [3] 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: metadata_pool 2 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: inline_data disabled 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: balancer 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: bal_rank_mask -1 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: standby_count_wanted 1 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: qdb_cluster leader: 0 members: 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: [mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: [mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Standby daemons: 2024-03-22T02:32:38.685 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: [mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: [mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 ceph-mon[115211]: mon.smithi103@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 bash[115192]: 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f 2024-03-22T02:32:38.686 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:38 smithi103 systemd[1]: Started Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: pgmap v86: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: mon.smithi103 calling monitor election 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: monmap epoch 2 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:32:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: last_changed 2024-03-22T02:21:20.985271+0000 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: created 2024-03-22T02:18:38.790476+0000 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: min_mon_release 17 (quincy) 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: election_strategy: 1 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.smithi103 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: osdmap e48: 6 total, 6 up, 6 in 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: mgrmap e34: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: overall HEALTH_OK 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: from='mgr.14702 ' entity='' 2024-03-22T02:32:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:39 smithi122 ceph-mon[36403]: mgrmap e35: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:32:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:32:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: from='mgr.14702 172.21.15.103:0/1681946067' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: pgmap v86: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: mon.smithi103 calling monitor election 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: monmap epoch 2 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: last_changed 2024-03-22T02:21:20.985271+0000 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: created 2024-03-22T02:18:38.790476+0000 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: min_mon_release 17 (quincy) 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: election_strategy: 1 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.smithi103 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: osdmap e48: 6 total, 6 up, 6 in 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: mgrmap e34: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: overall HEALTH_OK 2024-03-22T02:32:40.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: from='mgr.14702 ' entity='' 2024-03-22T02:32:40.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:39 smithi103 ceph-mon[115211]: mgrmap e35: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:46.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:32:46.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:32:46.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:32:46.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:32:46.990 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:32:46.990 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:46 smithi122 ceph-mon[36403]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: Standby manager daemon smithi122.prgtti restarted 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: Standby manager daemon smithi122.prgtti started 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/crt"}]: dispatch 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.prgtti/key"}]: dispatch 2024-03-22T02:32:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:46 smithi103 ceph-mon[115211]: from='mgr.? 172.21.15.122:0/3300141131' entity='mgr.smithi122.prgtti' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:32:48.081 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: mgrmap e36: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:48.081 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: Active manager daemon smithi103.pfvmui restarted 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: osdmap e49: 6 total, 6 up, 6 in 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: mgrmap e37: smithi103.pfvmui(active, starting, since 0.0104801s), standbys: smithi122.prgtti 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:32:48.082 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:32:48.083 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: mgrmap e36: smithi103.pfvmui(active, since 2m), standbys: smithi122.prgtti 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: Active manager daemon smithi103.pfvmui restarted 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: Activating manager daemon smithi103.pfvmui 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: osdmap e49: 6 total, 6 up, 6 in 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: mgrmap e37: smithi103.pfvmui(active, starting, since 0.0104801s), standbys: smithi122.prgtti 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:32:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi103.pfvmui", "id": "smithi103.pfvmui"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr metadata", "who": "smithi122.prgtti", "id": "smithi122.prgtti"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: Manager daemon smithi103.pfvmui is now available 2024-03-22T02:32:48.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:32:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:32:48.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:47 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi103.pfvmui/trash_purge_schedule"}]: dispatch 2024-03-22T02:32:48.903 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: [22/Mar/2024:02:32:47] ENGINE Bus STARTING 2024-03-22T02:32:48.903 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: [22/Mar/2024:02:32:47] ENGINE Serving on http://172.21.15.103:8765 2024-03-22T02:32:48.903 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: [22/Mar/2024:02:32:47] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:32:48.903 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: [22/Mar/2024:02:32:47] ENGINE Bus STARTED 2024-03-22T02:32:48.904 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: mgrmap e38: smithi103.pfvmui(active, since 1.01372s), standbys: smithi122.prgtti 2024-03-22T02:32:48.904 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:48 smithi122 ceph-mon[36403]: pgmap v3: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: [22/Mar/2024:02:32:47] ENGINE Bus STARTING 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: [22/Mar/2024:02:32:47] ENGINE Serving on http://172.21.15.103:8765 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: [22/Mar/2024:02:32:47] ENGINE Serving on https://172.21.15.103:7150 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: [22/Mar/2024:02:32:47] ENGINE Bus STARTED 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: mgrmap e38: smithi103.pfvmui(active, since 1.01372s), standbys: smithi122.prgtti 2024-03-22T02:32:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:48 smithi103 ceph-mon[115211]: pgmap v3: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:50.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:49 smithi122 ceph-mon[36403]: pgmap v4: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:50.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:49 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:50.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:49 smithi122 ceph-mon[36403]: mgrmap e39: smithi103.pfvmui(active, since 2s), standbys: smithi122.prgtti 2024-03-22T02:32:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:49 smithi103 ceph-mon[115211]: pgmap v4: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:49 smithi103 ceph-mon[115211]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:49 smithi103 ceph-mon[115211]: mgrmap e39: smithi103.pfvmui(active, since 2s), standbys: smithi122.prgtti 2024-03-22T02:32:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:51 smithi103 ceph-mon[115211]: pgmap v5: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:51.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:51 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:51.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:51 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:51.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:51 smithi122 ceph-mon[36403]: pgmap v5: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:52.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:52 smithi122 ceph-mon[36403]: mgrmap e40: smithi103.pfvmui(active, since 4s), standbys: smithi122.prgtti 2024-03-22T02:32:52.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:52 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:52.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:52 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:52.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:52 smithi103 ceph-mon[115211]: mgrmap e40: smithi103.pfvmui(active, since 4s), standbys: smithi122.prgtti 2024-03-22T02:32:52.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:52 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:52.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:52 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:53.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:53.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:53.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:53 smithi103 ceph-mon[115211]: pgmap v6: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:53.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:53 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:53.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:53 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:53.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:53 smithi122 ceph-mon[36403]: pgmap v6: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:32:54.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:54 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:54.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:54 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:54.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:54 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:54.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:54 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:55.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:55 smithi122 ceph-mon[36403]: pgmap v7: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-22T02:32:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:55 smithi103 ceph-mon[115211]: pgmap v7: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-22T02:32:56.545 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:56 smithi103 ceph-mon[115211]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:56.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:56 smithi122 ceph-mon[36403]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:32:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: Detected new or changed devices on smithi103 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: pgmap v8: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:57.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: Detected new or changed devices on smithi103 2024-03-22T02:32:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: pgmap v8: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:57 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:58 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: Updating smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: Updating smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:32:58.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T02:32:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:59 smithi103 ceph-mon[115211]: pgmap v9: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T02:32:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:32:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:32:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:32:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:59.886 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:59 smithi122 ceph-mon[36403]: pgmap v9: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T02:32:59.886 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:59 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:32:59.886 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:59 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:32:59.886 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:59 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:32:59.887 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:32:59 smithi122 ceph-mon[36403]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:00 smithi122 ceph-mon[36403]: Upgrade: Updating mon.smithi122 2024-03-22T02:33:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:00 smithi122 ceph-mon[36403]: Deploying daemon mon.smithi122 on smithi122 2024-03-22T02:33:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:00 smithi103 ceph-mon[115211]: Upgrade: Updating mon.smithi122 2024-03-22T02:33:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:00 smithi103 ceph-mon[115211]: Deploying daemon mon.smithi122 on smithi122 2024-03-22T02:33:01.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:33:01.790 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122[36374]: 2024-03-22T02:33:01.421+0000 7fb88a8e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:33:01.791 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122[36374]: 2024-03-22T02:33:01.421+0000 7fb88a8e9700 -1 mon.smithi122@1(peon) e2 *** Got Signal Terminated *** 2024-03-22T02:33:01.791 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 podman[83207]: 2024-03-22 02:33:01.524608902 +0000 UTC m=+0.232338822 container died 5d70e3dc66015de0d338ec7604b358a8b2e8e080f2fcb9d17ae196d9ddcc678f (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-22T02:33:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 podman[83207]: 2024-03-22 02:33:01.888518501 +0000 UTC m=+0.596248423 container cleanup 5d70e3dc66015de0d338ec7604b358a8b2e8e080f2fcb9d17ae196d9ddcc678f (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-9c4e0c3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:33:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:01 smithi122 bash[83207]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122 2024-03-22T02:33:02.532 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 podman[83221]: 2024-03-22 02:33:02.181569483 +0000 UTC m=+0.659917128 container remove 5d70e3dc66015de0d338ec7604b358a8b2e8e080f2fcb9d17ae196d9ddcc678f (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:33:02.533 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi122.service: Deactivated successfully. 2024-03-22T02:33:02.533 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 systemd[1]: Stopped Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:33:02.533 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi122.service: Consumed 6.924s CPU time. 2024-03-22T02:33:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 systemd[1]: Starting Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:33:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 podman[83349]: 2024-03-22 02:33:02.649671192 +0000 UTC m=+0.018112952 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:33:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:02 smithi122 podman[83349]: 2024-03-22 02:33:02.782577138 +0000 UTC m=+0.151018873 container create b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True) 2024-03-22T02:33:03.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 podman[83349]: 2024-03-22 02:33:03.018664928 +0000 UTC m=+0.387106693 container init b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T02:33:03.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 podman[83349]: 2024-03-22 02:33:03.021713679 +0000 UTC m=+0.390155416 container start b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:33:03.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:33:03.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T02:33:03.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: pidfile_write: ignore empty --pid-file 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: load: jerasure load: lrc 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: RocksDB version: 7.9.2 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Git sha 0 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: DB SUMMARY 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: DB Session ID: IIAOXQGRMLLF82NIWOPJ 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1091 Bytes 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi122/store.db dir, Total Num: 1, files: 000027.sst 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi122/store.db: 000025.log size: 5892948 ; 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:33:03.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.env: 0x55eb71b6dc60 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.info_log: 0x55eb730499a0 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.statistics: (nil) 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.use_fsync: 0 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:33:03.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.db_log_dir: 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.wal_dir: 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:33:03.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.write_buffer_manager: 0x55eb7301a1e0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.unordered_write: 0 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:33:03.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:33:03.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:33:03.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.row_cache: None 2024-03-22T02:33:03.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.wal_filter: None 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.wal_compression: 0 2024-03-22T02:33:03.355 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:33:03.356 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_open_files: -1 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:33:03.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Compression algorithms supported: 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kZSTD supported: 0 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T02:33:03.358 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.merge_operator: 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_filter: None 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb73049aa0) 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_type: 0 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_index_type: 0 2024-03-22T02:33:03.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_shortening: 1 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: checksum: 4 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: no_block_cache: 0 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache: 0x55eb7304d090 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_options: 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: capacity : 536870912 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_shard_bits : 4 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: strict_capacity_limit : 0 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_compressed: (nil) 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: persistent_cache: (nil) 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size: 4096 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size_deviation: 10 2024-03-22T02:33:03.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_restart_interval: 16 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_block_restart_interval: 1 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_block_size: 4096 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: partition_filters: 0 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: use_delta_encoding: 1 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: filter_policy: bloomfilter 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: whole_key_filtering: 1 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: verify_compression: 0 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: format_version: 5 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_index_compression: 1 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_align: 0 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_auto_readahead_size: 262144 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: prepopulate_block_cache: 0 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: initial_auto_readahead_size: 8192 2024-03-22T02:33:03.361 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression: NoCompression 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.num_levels: 7 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:33:03.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:33:03.363 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:33:03.364 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:33:03.365 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:33:03.366 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.ttl: 2592000 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:33:03.367 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12245, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2ce51e26-cf47-41cf-8720-61274ec1c2ac 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074783052283, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074783067258, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3584175, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12242, "largest_seqno": 14428, "table_properties": {"data_size": 3575004, "index_size": 5847, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2309, "raw_key_size": 21571, "raw_average_key_size": 23, "raw_value_size": 3555680, "raw_average_value_size": 3907, "num_data_blocks": 264, "num_entries": 910, "num_filter_entries": 910, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711074783, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2ce51e26-cf47-41cf-8720-61274ec1c2ac", "db_session_id": "IIAOXQGRMLLF82NIWOPJ", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711074783067390, "job": 1, "event": "recovery_finished"} 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-22T02:33:03.368 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi122/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb7313c000 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: DB pointer 0x55eb73126000 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** DB Stats ** 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:33:03.369 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L0 1/0 3.42 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 700.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L6 1/0 6.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Sum 2/0 10.31 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 700.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 700.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 700.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative compaction: 0.00 GB write, 173.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval compaction: 0.00 GB write, 173.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache BinnedLRUCache@0x55eb7304d090#2 capacity: 512.00 MB usage: 8.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.371 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: starting mon.smithi122 rank 1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi122 fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???) e2 preinit fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).mds e21 new map 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).mds e21 print_map 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: e21 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: legacy client fscid: 1 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Filesystem 'cephfs' (1) 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: fs_name cephfs 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: epoch 21 2024-03-22T02:33:03.372 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: tableserver 0 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: root 0 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_timeout 60 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_autoclose 300 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_file_size 1099511627776 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_xattr_size 65536 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: required_client_features {} 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure 0 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure_osd_epoch 46 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_mds 2 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: in 0,1 2024-03-22T02:33:03.373 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: up {0=14472,1=24271} 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: failed 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: damaged 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: stopped 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_pools [3] 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_pool 2 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: inline_data disabled 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: balancer 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: bal_rank_mask -1 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: standby_count_wanted 1 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.374 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Standby daemons: 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 ceph-mon[83368]: mon.smithi122@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 bash[83349]: b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f 2024-03-22T02:33:03.375 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:03 smithi122 systemd[1]: Started Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 -- 172.21.15.103:0/58088202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 msgr2=0x7f86000fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 --2- 172.21.15.103:0/58088202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f86000fce30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f85f0004940 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 -- 172.21.15.103:0/58088202 shutdown_connections 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 --2- 172.21.15.103:0/58088202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f86000fce30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 --2- 172.21.15.103:0/58088202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8600102e20 0x7f86001031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 -- 172.21.15.103:0/58088202 >> 172.21.15.103:0/58088202 conn(0x7f86000f89b0 msgr2=0x7f86000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 -- 172.21.15.103:0/58088202 shutdown_connections 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.254+0000 7f8607c2a700 1 -- 172.21.15.103:0/58088202 wait complete. 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 Processor -- start 2024-03-22T02:33:04.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 -- start start 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8600102e20 0x7f86000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600072210 con 0x7f86001037c0 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f8607c2a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600070d40 con 0x7f8600102e20 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46476/0 (socket says 172.21.15.103:46476) 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 -- 172.21.15.103:0/2264109716 learned_addr learned my addr 172.21.15.103:0/2264109716 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 -- 172.21.15.103:0/2264109716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8600102e20 msgr2=0x7f86000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8600102e20 0x7f86000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 -- 172.21.15.103:0/2264109716 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0005040 con 0x7f86001037c0 2024-03-22T02:33:04.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f86051c5700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009100 tx=0x7f85f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.255+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f000e040 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.256+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85f0013420 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.256+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600070fc0 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.256+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f001c610 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.256+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860011a0f0 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.257+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86001052f0 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.257+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f85f0013590 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.257+0000 7f85f67fc700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 0x7f85ec070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.258+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f85f008f280 con 0x7f86001037c0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.258+0000 7f86059c6700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 0x7f85ec070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.258+0000 7f86059c6700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 0x7f85ec070980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0041d0 tx=0x7f85fc004160 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.263+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85f0058380 con 0x7f86001037c0 2024-03-22T02:33:04.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.475+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8600109f00 con 0x7f85ec06e4d0 2024-03-22T02:33:04.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.476+0000 7f85f67fc700 1 -- 172.21.15.103:0/2264109716 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f8600109f00 con 0x7f85ec06e4d0 2024-03-22T02:33:04.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 msgr2=0x7f85ec070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 0x7f85ec070980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0041d0 tx=0x7f85fc004160 comp rx=0 tx=0).stop 2024-03-22T02:33:04.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 msgr2=0x7f8600070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009100 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 shutdown_connections 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85ec06e4d0 0x7f85ec070980 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8600102e20 0x7f86000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 --2- 172.21.15.103:0/2264109716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86001037c0 0x7f8600070800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 >> 172.21.15.103:0/2264109716 conn(0x7f86000f89b0 msgr2=0x7f86001021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.478+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 shutdown_connections 2024-03-22T02:33:04.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.479+0000 7f8607c2a700 1 -- 172.21.15.103:0/2264109716 wait complete. 2024-03-22T02:33:04.489 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:33:04.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.580+0000 7f7b8c23f700 1 -- 172.21.15.103:0/3774491608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841026c0 msgr2=0x7f7b840fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.580+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/3774491608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841026c0 0x7f7b840fbd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004940 tx=0x7f7b74009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 -- 172.21.15.103:0/3774491608 shutdown_connections 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/3774491608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841026c0 0x7f7b840fbd30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/3774491608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b84101d20 0x7f7b841020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 -- 172.21.15.103:0/3774491608 >> 172.21.15.103:0/3774491608 conn(0x7f7b840f7870 msgr2=0x7f7b840f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 -- 172.21.15.103:0/3774491608 shutdown_connections 2024-03-22T02:33:04.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 -- 172.21.15.103:0/3774491608 wait complete. 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 Processor -- start 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.581+0000 7f7b8c23f700 1 -- start start 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b8c23f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b8c23f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b841026c0 0x7f7b84112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b8c23f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b84113440 con 0x7f7b84101d20 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b8c23f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8410ca40 con 0x7f7b841026c0 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46488/0 (socket says 172.21.15.103:46488) 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 -- 172.21.15.103:0/2162117776 learned_addr learned my addr 172.21.15.103:0/2162117776 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b897da700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b841026c0 0x7f7b84112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 -- 172.21.15.103:0/2162117776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b841026c0 msgr2=0x7f7b84112df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b841026c0 0x7f7b84112df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 -- 172.21.15.103:0/2162117776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74005040 con 0x7f7b84101d20 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.582+0000 7f7b89fdb700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000ead0 tx=0x7f7b80004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.583+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b8000cc60 con 0x7f7b84101d20 2024-03-22T02:33:04.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.583+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b8000cdc0 con 0x7f7b84101d20 2024-03-22T02:33:04.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.583+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b800109f0 con 0x7f7b84101d20 2024-03-22T02:33:04.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.583+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8410ccc0 con 0x7f7b84101d20 2024-03-22T02:33:04.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.583+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8410d280 con 0x7f7b84101d20 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.584+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b840fd470 con 0x7f7b84101d20 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.584+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f7b80010b50 con 0x7f7b84101d20 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.585+0000 7f7b7a7fc700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 0x7f7b70070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.585+0000 7f7b897da700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 0x7f7b70070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.585+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f7b800907b0 con 0x7f7b84101d20 2024-03-22T02:33:04.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.585+0000 7f7b897da700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 0x7f7b70070980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b8410dc00 tx=0x7f7b74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.589+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b8005d480 con 0x7f7b84101d20 2024-03-22T02:33:04.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.664+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b80095020 con 0x7f7b84101d20 2024-03-22T02:33:04.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.819+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b84108e00 con 0x7f7b7006e4d0 2024-03-22T02:33:04.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.820+0000 7f7b7a7fc700 1 -- 172.21.15.103:0/2162117776 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f7b84108e00 con 0x7f7b7006e4d0 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 msgr2=0x7f7b70070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 0x7f7b70070980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b8410dc00 tx=0x7f7b74009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 msgr2=0x7f7b841128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000ead0 tx=0x7f7b80004500 comp rx=0 tx=0).stop 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 shutdown_connections 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b7006e4d0 0x7f7b70070980 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101d20 0x7f7b841128b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 --2- 172.21.15.103:0/2162117776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b841026c0 0x7f7b84112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 >> 172.21.15.103:0/2162117776 conn(0x7f7b840f7870 msgr2=0x7f7b840f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:04.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 shutdown_connections 2024-03-22T02:33:04.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.822+0000 7f7b8c23f700 1 -- 172.21.15.103:0/2162117776 wait complete. 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 -- 172.21.15.103:0/2262893235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 msgr2=0x7f01041031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 --2- 172.21.15.103:0/2262893235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f01041031e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f00f4004940 tx=0x7f00f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 -- 172.21.15.103:0/2262893235 shutdown_connections 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 --2- 172.21.15.103:0/2262893235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01040fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 --2- 172.21.15.103:0/2262893235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f01041031e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 -- 172.21.15.103:0/2262893235 >> 172.21.15.103:0/2262893235 conn(0x7f01040f89a0 msgr2=0x7f01040fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:04.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 -- 172.21.15.103:0/2262893235 shutdown_connections 2024-03-22T02:33:04.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.924+0000 7f010b53b700 1 -- 172.21.15.103:0/2262893235 wait complete. 2024-03-22T02:33:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 Processor -- start 2024-03-22T02:33:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 -- start start 2024-03-22T02:33:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01041146f0 con 0x7f0104102e10 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f010b53b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010410dae0 con 0x7f01041037b0 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f01092d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f01092d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46516/0 (socket says 172.21.15.103:46516) 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f01092d7700 1 -- 172.21.15.103:0/1269283554 learned_addr learned my addr 172.21.15.103:0/1269283554 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f0108ad6700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f0108ad6700 1 -- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 msgr2=0x7f01041140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f0108ad6700 1 -- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 msgr2=0x7f01041140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.925+0000 7f0108ad6700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f0108ad6700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f01092d7700 1 -- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 msgr2=0x7f01041140a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f01092d7700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f01092d7700 1 -- 172.21.15.103:0/1269283554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f4005040 con 0x7f0104102e10 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f01092d7700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f00f400b3b0 tx=0x7f00f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f400e040 con 0x7f0104102e10 2024-03-22T02:33:04.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00f400bbc0 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010410dd60 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f401c3f0 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.926+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010410e320 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.927+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0104100120 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.927+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f00f4013420 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.928+0000 7f00f9ffb700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 0x7f00f0070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.928+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f00f408f280 con 0x7f0104102e10 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.928+0000 7f0108ad6700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 0x7f00f0070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.928+0000 7f0108ad6700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 0x7f00f0070970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f010410eca0 tx=0x7f0100009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:04.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:04.933+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f00f405be30 con 0x7f0104102e10 2024-03-22T02:33:05.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.138+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f010410f1b0 con 0x7f00f006e4c0 2024-03-22T02:33:05.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.148+0000 7f00f9ffb700 1 -- 172.21.15.103:0/1269283554 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f010410f1b0 con 0x7f00f006e4c0 2024-03-22T02:33:05.148 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (97s) 13s ago 12m 25.8M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (13m) 13s ago 13m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (11m) 15s ago 11m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (81s) 13s ago 12m 76.6M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (7m) 13s ago 7m 13.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (7m) 13s ago 7m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (7m) 15s ago 7m 14.7M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (7m) 15s ago 7m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (3m) 13s ago 14m 473M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (2m) 15s ago 11m 446M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (27s) 13s ago 14m 27.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 starting - - - 2048M 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (2m) 13s ago 12m 16.1M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:33:05.149 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 15s ago 11m 16.8M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (11m) 13s ago 11m 215M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (10m) 13s ago 10m 231M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (10m) 13s ago 10m 199M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (9m) 15s ago 9m 263M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (9m) 15s ago 9m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (8m) 15s ago 8m 219M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:33:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (116s) 13s ago 12m 58.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:33:05.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 msgr2=0x7f00f0070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 0x7f00f0070970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f010410eca0 tx=0x7f0100009f70 comp rx=0 tx=0).stop 2024-03-22T02:33:05.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 msgr2=0x7f0104113b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f00f400b3b0 tx=0x7f00f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:05.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 shutdown_connections 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f00f006e4c0 0x7f00f0070970 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0104102e10 0x7f0104113b60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 --2- 172.21.15.103:0/1269283554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01041037b0 0x7f01041140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.150+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 >> 172.21.15.103:0/1269283554 conn(0x7f01040f89a0 msgr2=0x7f0104106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.151+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 shutdown_connections 2024-03-22T02:33:05.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.151+0000 7f010b53b700 1 -- 172.21.15.103:0/1269283554 wait complete. 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- 172.21.15.103:0/1935604145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fe810 msgr2=0x7f7300106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 --2- 172.21.15.103:0/1935604145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fe810 0x7f7300106ee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f72ec004940 tx=0x7f72ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- 172.21.15.103:0/1935604145 shutdown_connections 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 --2- 172.21.15.103:0/1935604145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fe810 0x7f7300106ee0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 --2- 172.21.15.103:0/1935604145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fde70 0x7f73000fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- 172.21.15.103:0/1935604145 >> 172.21.15.103:0/1935604145 conn(0x7f73000f78f0 msgr2=0x7f73000f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- 172.21.15.103:0/1935604145 shutdown_connections 2024-03-22T02:33:05.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- 172.21.15.103:0/1935604145 wait complete. 2024-03-22T02:33:05.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 Processor -- start 2024-03-22T02:33:05.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.250+0000 7f730711a700 1 -- start start 2024-03-22T02:33:05.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f730711a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f730711a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f730711a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7300114510 con 0x7f73000fde70 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f730711a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730010db10 con 0x7f73000fe810 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46526/0 (socket says 172.21.15.103:46526) 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 -- 172.21.15.103:0/981168334 learned_addr learned my addr 172.21.15.103:0/981168334 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f72fffff700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f72fffff700 1 -- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 msgr2=0x7f7300113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f72fffff700 1 -- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 msgr2=0x7f7300113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f72fffff700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f72fffff700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:33:05.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 -- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 msgr2=0x7f7300113ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 -- 172.21.15.103:0/981168334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec005040 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.251+0000 7f7304eb6700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72f40041d0 tx=0x7f72f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.252+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72f4007cc0 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.252+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72f400f040 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.252+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730010ddf0 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.252+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72f400dbe0 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.252+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730010e330 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.253+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7300101900 con 0x7f73000fde70 2024-03-22T02:33:05.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.253+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f72f400dd40 con 0x7f73000fde70 2024-03-22T02:33:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.253+0000 7f72fd7fa700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 0x7f72f0070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.254+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f72f408d780 con 0x7f73000fde70 2024-03-22T02:33:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.254+0000 7f72fffff700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 0x7f72f0070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.254+0000 7f72fffff700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 0x7f72f0070980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f730010ecd0 tx=0x7f72ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:05.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.261+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f72f405a330 con 0x7f73000fde70 2024-03-22T02:33:05.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.539+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f730004c740 con 0x7f73000fde70 2024-03-22T02:33:05.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.540+0000 7f72fd7fa700 1 -- 172.21.15.103:0/981168334 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f72f4059ec0 con 0x7f73000fde70 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:33:05.541 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 10, 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:33:05.542 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 msgr2=0x7f72f0070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 0x7f72f0070980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f730010ecd0 tx=0x7f72ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 msgr2=0x7f7300113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72f40041d0 tx=0x7f72f40046b0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 shutdown_connections 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f72f006e4d0 0x7f72f0070980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73000fde70 0x7f7300113980 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 --2- 172.21.15.103:0/981168334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73000fe810 0x7f7300113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 >> 172.21.15.103:0/981168334 conn(0x7f73000f78f0 msgr2=0x7f73000ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:05.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 shutdown_connections 2024-03-22T02:33:05.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.542+0000 7f730711a700 1 -- 172.21.15.103:0/981168334 wait complete. 2024-03-22T02:33:05.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- 172.21.15.103:0/3681088233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a941037a0 msgr2=0x7f3a940fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 --2- 172.21.15.103:0/3681088233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a941037a0 0x7f3a940fce10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3a84004940 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:05.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- 172.21.15.103:0/3681088233 shutdown_connections 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 --2- 172.21.15.103:0/3681088233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a941037a0 0x7f3a940fce10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 --2- 172.21.15.103:0/3681088233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a94102e00 0x7f3a941031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- 172.21.15.103:0/3681088233 >> 172.21.15.103:0/3681088233 conn(0x7f3a940f89b0 msgr2=0x7f3a940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- 172.21.15.103:0/3681088233 shutdown_connections 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- 172.21.15.103:0/3681088233 wait complete. 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 Processor -- start 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.643+0000 7f3a9c029700 1 -- start start 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94070d40 con 0x7f3a94102e00 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94070e80 con 0x7f3a941037a0 2024-03-22T02:33:05.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46550/0 (socket says 172.21.15.103:46550) 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 -- 172.21.15.103:0/648302253 learned_addr learned my addr 172.21.15.103:0/648302253 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a995c4700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a995c4700 1 -- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 msgr2=0x7f3a94070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a995c4700 1 -- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 msgr2=0x7f3a94070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a995c4700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 -- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 msgr2=0x7f3a94070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a995c4700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 -- 172.21.15.103:0/648302253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a84005040 con 0x7f3a94102e00 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a99dc5700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000cb10 tx=0x7f3a90007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a90004200 con 0x7f3a94102e00 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a94071130 con 0x7f3a94102e00 2024-03-22T02:33:05.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a9000acc0 con 0x7f3a94102e00 2024-03-22T02:33:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a90003d90 con 0x7f3a94102e00 2024-03-22T02:33:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.644+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9411a340 con 0x7f3a94102e00 2024-03-22T02:33:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.645+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a941052d0 con 0x7f3a94102e00 2024-03-22T02:33:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.646+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f3a90012420 con 0x7f3a94102e00 2024-03-22T02:33:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.646+0000 7f3a8a7fc700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 0x7f3a80070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.646+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3a9008e2e0 con 0x7f3a94102e00 2024-03-22T02:33:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.646+0000 7f3a995c4700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 0x7f3a80070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.647+0000 7f3a995c4700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 0x7f3a80070970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a94071bb0 tx=0x7f3a84009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:05.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.651+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a900574e0 con 0x7f3a94102e00 2024-03-22T02:33:05.739 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:05.739 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:33:05.739 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: mon.smithi122 calling monitor election 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: mon.smithi103 calling monitor election 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: monmap epoch 3 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: last_changed 2024-03-22T02:33:04.649576+0000 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: created 2024-03-22T02:18:38.790476+0000 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: min_mon_release 19 (squid) 2024-03-22T02:33:05.740 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: election_strategy: 1 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.smithi103 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: osdmap e49: 6 total, 6 up, 6 in 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: mgrmap e40: smithi103.pfvmui(active, since 17s), standbys: smithi122.prgtti 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: overall HEALTH_OK 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:05.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:05.742 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:05.742 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: pgmap v12: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-22T02:33:05.742 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:05.742 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:05 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/981168334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: mon.smithi122 calling monitor election 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: mon.smithi103 calling monitor election 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: mon.smithi103 is new leader, mons smithi103,smithi122 in quorum (ranks 0,1) 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: monmap epoch 3 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: last_changed 2024-03-22T02:33:04.649576+0000 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: created 2024-03-22T02:18:38.790476+0000 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: min_mon_release 19 (squid) 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: election_strategy: 1 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.smithi103 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: fsmap cephfs:2 {0=cephfs.smithi103.xhmfjp=up:active,1=cephfs.smithi122.ztrkgp=up:active} 2 up:standby 2024-03-22T02:33:05.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: osdmap e49: 6 total, 6 up, 6 in 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: mgrmap e40: smithi103.pfvmui(active, since 17s), standbys: smithi122.prgtti 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: overall HEALTH_OK 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: pgmap v12: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:05.876 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:05 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/981168334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:05.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.899+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3a940719c0 con 0x7f3a94102e00 2024-03-22T02:33:05.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.900+0000 7f3a8a7fc700 1 -- 172.21.15.103:0/648302253 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7f3a9005aba0 con 0x7f3a94102e00 2024-03-22T02:33:05.900 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:33:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:33:05.902 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:05.903 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 msgr2=0x7f3a80070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 0x7f3a80070970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a94071bb0 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 msgr2=0x7f3a940702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000cb10 tx=0x7f3a90007f90 comp rx=0 tx=0).stop 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 shutdown_connections 2024-03-22T02:33:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3a8006e4c0 0x7f3a80070970 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3a94102e00 0x7f3a940702c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 --2- 172.21.15.103:0/648302253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a941037a0 0x7f3a94070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 >> 172.21.15.103:0/648302253 conn(0x7f3a940f89b0 msgr2=0x7f3a94102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 shutdown_connections 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:05.902+0000 7f3a9c029700 1 -- 172.21.15.103:0/648302253 wait complete. 2024-03-22T02:33:05.905 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.003+0000 7f89707fb700 1 -- 172.21.15.103:0/3313658890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 msgr2=0x7f89680fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.003+0000 7f89707fb700 1 --2- 172.21.15.103:0/3313658890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f89680fce30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8958004940 tx=0x7f8958009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- 172.21.15.103:0/3313658890 shutdown_connections 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 --2- 172.21.15.103:0/3313658890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f89680fce30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 --2- 172.21.15.103:0/3313658890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- 172.21.15.103:0/3313658890 >> 172.21.15.103:0/3313658890 conn(0x7f89680f89b0 msgr2=0x7f89680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- 172.21.15.103:0/3313658890 shutdown_connections 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- 172.21.15.103:0/3313658890 wait complete. 2024-03-22T02:33:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 Processor -- start 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- start start 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968114530 con 0x7f89681037c0 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f89707fb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896810db30 con 0x7f8968102e20 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f896dd96700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f896dd96700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46576/0 (socket says 172.21.15.103:46576) 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f896dd96700 1 -- 172.21.15.103:0/1432012848 learned_addr learned my addr 172.21.15.103:0/1432012848 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.004+0000 7f896e597700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896e597700 1 -- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 msgr2=0x7f89681139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896e597700 1 -- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 msgr2=0x7f89681139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896e597700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896e597700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896dd96700 1 -- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 msgr2=0x7f89681139a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896dd96700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896dd96700 1 -- 172.21.15.103:0/1432012848 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8958005040 con 0x7f89681037c0 2024-03-22T02:33:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f896dd96700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8958009100 tx=0x7f8958009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:06.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895800e040 con 0x7f89681037c0 2024-03-22T02:33:06.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8958013420 con 0x7f89681037c0 2024-03-22T02:33:06.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896810ddb0 con 0x7f89681037c0 2024-03-22T02:33:06.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895801c610 con 0x7f89681037c0 2024-03-22T02:33:06.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.005+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896810e370 con 0x7f89681037c0 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.006+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89681052f0 con 0x7f89681037c0 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.006+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f8958013590 con 0x7f89681037c0 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.007+0000 7f895effd700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 0x7f8954070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.007+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f895808f280 con 0x7f89681037c0 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.007+0000 7f896e597700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 0x7f8954070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.007+0000 7f896e597700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 0x7f8954070980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8964009e50 tx=0x7f8964009e80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:06.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.012+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f895805beb0 con 0x7f89681037c0 2024-03-22T02:33:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.225+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f896810eff0 con 0x7f895406e4d0 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.226+0000 7f895effd700 1 -- 172.21.15.103:0/1432012848 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f896810eff0 con 0x7f895406e4d0 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr" 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "3/21 daemons upgraded", 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:33:06.227 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 msgr2=0x7f8954070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 0x7f8954070980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8964009e50 tx=0x7f8964009e80 comp rx=0 tx=0).stop 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 msgr2=0x7f8968113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8958009100 tx=0x7f8958009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 shutdown_connections 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e4d0 0x7f8954070980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8968102e20 0x7f89681139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 --2- 172.21.15.103:0/1432012848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89681037c0 0x7f8968113ee0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 >> 172.21.15.103:0/1432012848 conn(0x7f89680f89b0 msgr2=0x7f89681021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 shutdown_connections 2024-03-22T02:33:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.228+0000 7f89707fb700 1 -- 172.21.15.103:0/1432012848 wait complete. 2024-03-22T02:33:06.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3215597742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 msgr2=0x7f1ff80fc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3215597742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff80fc5e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8004940 tx=0x7f1fe8009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3215597742 shutdown_connections 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3215597742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff80fc5e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3215597742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff81021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3215597742 >> 172.21.15.103:0/3215597742 conn(0x7f1ff80714c0 msgr2=0x7f1ff80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3215597742 shutdown_connections 2024-03-22T02:33:06.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3215597742 wait complete. 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 Processor -- start 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.330+0000 7f1fffc8d700 1 -- start start 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fffc8d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fffc8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fffc8d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff810f580 con 0x7f1ff8102770 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fffc8d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff810f6c0 con 0x7f1ff8101dd0 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46592/0 (socket says 172.21.15.103:46592) 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 -- 172.21.15.103:0/3397045949 learned_addr learned my addr 172.21.15.103:0/3397045949 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffda29700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffda29700 1 -- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 msgr2=0x7f1ff810eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffda29700 1 -- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 msgr2=0x7f1ff810eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffda29700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffda29700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 -- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 msgr2=0x7f1ff810eb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 -- 172.21.15.103:0/3397045949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8005040 con 0x7f1ff8102770 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1ffd228700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009100 tx=0x7f1fe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fe800e040 con 0x7f1ff8102770 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fe8013420 con 0x7f1ff8102770 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fe801c610 con 0x7f1ff8102770 2024-03-22T02:33:06.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.331+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff810f940 con 0x7f1ff8102770 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.332+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff8117e70 con 0x7f1ff8102770 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.332+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff81042a0 con 0x7f1ff8102770 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.333+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f1fe8013590 con 0x7f1ff8102770 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.333+0000 7f1fee7fc700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 0x7f1fe4070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.334+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1fe808f250 con 0x7f1ff8102770 2024-03-22T02:33:06.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.334+0000 7f1ffda29700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 0x7f1fe4070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:06.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.334+0000 7f1ffda29700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 0x7f1fe4070980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4007fd0 tx=0x7f1ff4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:06.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.338+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1fe8058350 con 0x7f1ff8102770 2024-03-22T02:33:06.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.618+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ff80fba70 con 0x7f1ff8102770 2024-03-22T02:33:06.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.618+0000 7f1fee7fc700 1 -- 172.21.15.103:0/3397045949 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1fe805ba10 con 0x7f1ff8102770 2024-03-22T02:33:06.619 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 msgr2=0x7f1fe4070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 0x7f1fe4070980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4007fd0 tx=0x7f1ff4009ee0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 msgr2=0x7f1ff810f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009100 tx=0x7f1fe8009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 shutdown_connections 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1fe406e4d0 0x7f1fe4070980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ff8101dd0 0x7f1ff810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 --2- 172.21.15.103:0/3397045949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ff8102770 0x7f1ff810f040 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 >> 172.21.15.103:0/3397045949 conn(0x7f1ff80714c0 msgr2=0x7f1ff80fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:06.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 shutdown_connections 2024-03-22T02:33:06.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:06.620+0000 7f1fffc8d700 1 -- 172.21.15.103:0/3397045949 wait complete. 2024-03-22T02:33:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:06 smithi122 ceph-mon[83368]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:06 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/648302253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:33:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:06 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3397045949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:33:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:06 smithi103 ceph-mon[115211]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:06 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/648302253' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:33:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:06 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3397045949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:33:08.078 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:07 smithi103 ceph-mon[115211]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:08.079 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:07 smithi103 ceph-mon[115211]: pgmap v13: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:08.079 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:08.079 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:07 smithi122 ceph-mon[83368]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:07 smithi122 ceph-mon[83368]: pgmap v13: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:09 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:10.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:09 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:10.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:09 smithi122 ceph-mon[83368]: pgmap v14: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:09 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:09 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:09 smithi103 ceph-mon[115211]: pgmap v14: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:12.084 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:11 smithi122 ceph-mon[83368]: pgmap v15: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:11 smithi103 ceph-mon[115211]: pgmap v15: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: Detected new or changed devices on smithi122 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: Reconfiguring mon.smithi103 (monmap changed)... 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:13.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: Reconfiguring daemon mon.smithi103 on smithi103 2024-03-22T02:33:13.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:13 smithi103 ceph-mon[115211]: pgmap v16: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: Detected new or changed devices on smithi122 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: Reconfiguring mon.smithi103 (monmap changed)... 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: Reconfiguring daemon mon.smithi103 on smithi103 2024-03-22T02:33:13.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:13 smithi122 ceph-mon[83368]: pgmap v16: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:33:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:33:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.pfvmui", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:33:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:33:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: Reconfiguring mgr.smithi103.pfvmui (monmap changed)... 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: Reconfiguring daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: pgmap v17: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:16.283 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: Reconfiguring mgr.smithi103.pfvmui (monmap changed)... 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: Reconfiguring daemon mgr.smithi103.pfvmui on smithi103 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: pgmap v17: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:16 smithi122 ceph-mon[83368]: Reconfiguring crash.smithi103 (monmap changed)... 2024-03-22T02:33:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:16 smithi122 ceph-mon[83368]: Reconfiguring daemon crash.smithi103 on smithi103 2024-03-22T02:33:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:16 smithi103 ceph-mon[115211]: Reconfiguring crash.smithi103 (monmap changed)... 2024-03-22T02:33:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:16 smithi103 ceph-mon[115211]: Reconfiguring daemon crash.smithi103 on smithi103 2024-03-22T02:33:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:18.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:17 smithi103 ceph-mon[115211]: pgmap v18: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:18.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:18.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:18.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:33:18.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:17 smithi122 ceph-mon[83368]: pgmap v18: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:33:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: Reconfiguring osd.0 (monmap changed)... 2024-03-22T02:33:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.0 on smithi103 2024-03-22T02:33:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:33:19.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: Reconfiguring osd.0 (monmap changed)... 2024-03-22T02:33:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.0 on smithi103 2024-03-22T02:33:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:33:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:19 smithi103 ceph-mon[115211]: pgmap v19: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:19 smithi103 ceph-mon[115211]: Reconfiguring osd.1 (monmap changed)... 2024-03-22T02:33:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:19 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.1 on smithi103 2024-03-22T02:33:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:19 smithi122 ceph-mon[83368]: pgmap v19: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:19 smithi122 ceph-mon[83368]: Reconfiguring osd.1 (monmap changed)... 2024-03-22T02:33:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:19 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.1 on smithi103 2024-03-22T02:33:21.766 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:21.766 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:21.767 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: Reconfiguring osd.2 (monmap changed)... 2024-03-22T02:33:21.767 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:33:21.767 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:21.767 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.2 on smithi103 2024-03-22T02:33:21.767 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:21 smithi103 ceph-mon[115211]: pgmap v20: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: Reconfiguring osd.2 (monmap changed)... 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.2 on smithi103 2024-03-22T02:33:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:21 smithi122 ceph-mon[83368]: pgmap v20: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: Reconfiguring mds.cephfs.smithi103.gtuatp (monmap changed)... 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: Reconfiguring daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:33:23.452 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:23 smithi103 ceph-mon[115211]: pgmap v21: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:23.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: Reconfiguring mds.cephfs.smithi103.gtuatp (monmap changed)... 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: Reconfiguring daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:33:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:23 smithi122 ceph-mon[83368]: pgmap v21: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:25.292 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: Reconfiguring mds.cephfs.smithi103.xhmfjp (monmap changed)... 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: Reconfiguring daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:33:25.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:25 smithi103 ceph-mon[115211]: pgmap v22: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: Reconfiguring mds.cephfs.smithi103.xhmfjp (monmap changed)... 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: Reconfiguring daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:33:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:25 smithi122 ceph-mon[83368]: pgmap v22: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: Reconfiguring crash.smithi122 (monmap changed)... 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:26 smithi122 ceph-mon[83368]: Reconfiguring daemon crash.smithi122 on smithi122 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: Reconfiguring crash.smithi122 (monmap changed)... 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:26 smithi103 ceph-mon[115211]: Reconfiguring daemon crash.smithi122 on smithi122 2024-03-22T02:33:27.988 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: pgmap v23: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:27.988 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:27.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:27.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:33:27.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:33:27.989 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: pgmap v23: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.prgtti", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:33:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:29.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:28 smithi122 ceph-mon[83368]: Reconfiguring mgr.smithi122.prgtti (monmap changed)... 2024-03-22T02:33:29.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:28 smithi122 ceph-mon[83368]: Reconfiguring daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:33:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:28 smithi103 ceph-mon[115211]: Reconfiguring mgr.smithi122.prgtti (monmap changed)... 2024-03-22T02:33:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:28 smithi103 ceph-mon[115211]: Reconfiguring daemon mgr.smithi122.prgtti on smithi122 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: pgmap v24: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:33:29.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: pgmap v24: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:33:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:30 smithi103 ceph-mon[115211]: Reconfiguring mon.smithi122 (monmap changed)... 2024-03-22T02:33:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:30 smithi103 ceph-mon[115211]: Reconfiguring daemon mon.smithi122 on smithi122 2024-03-22T02:33:30.983 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:30 smithi122 ceph-mon[83368]: Reconfiguring mon.smithi122 (monmap changed)... 2024-03-22T02:33:30.984 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:30 smithi122 ceph-mon[83368]: Reconfiguring daemon mon.smithi122 on smithi122 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: Reconfiguring osd.3 (monmap changed)... 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.3 on smithi122 2024-03-22T02:33:31.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:31 smithi122 ceph-mon[83368]: pgmap v25: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: Reconfiguring osd.3 (monmap changed)... 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.3 on smithi122 2024-03-22T02:33:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:31 smithi103 ceph-mon[115211]: pgmap v25: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:33:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:33:33.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:33 smithi103 ceph-mon[115211]: Reconfiguring osd.4 (monmap changed)... 2024-03-22T02:33:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:33 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.4 on smithi122 2024-03-22T02:33:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:33 smithi103 ceph-mon[115211]: pgmap v26: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:33 smithi122 ceph-mon[83368]: Reconfiguring osd.4 (monmap changed)... 2024-03-22T02:33:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:33 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.4 on smithi122 2024-03-22T02:33:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:33 smithi122 ceph-mon[83368]: pgmap v26: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: Reconfiguring osd.5 (monmap changed)... 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: Reconfiguring daemon osd.5 on smithi122 2024-03-22T02:33:35.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:35 smithi122 ceph-mon[83368]: pgmap v27: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: Reconfiguring osd.5 (monmap changed)... 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:35.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: Reconfiguring daemon osd.5 on smithi122 2024-03-22T02:33:35.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:35 smithi103 ceph-mon[115211]: pgmap v27: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:36.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 -- 172.21.15.103:0/1966281971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c071f90 msgr2=0x7f3f0c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:36.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1966281971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c071f90 0x7f3f0c070360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009d10 tx=0x7f3f08004d10 comp rx=0 tx=0).stop 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 -- 172.21.15.103:0/1966281971 shutdown_connections 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1966281971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1966281971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c071f90 0x7f3f0c070360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 -- 172.21.15.103:0/1966281971 >> 172.21.15.103:0/1966281971 conn(0x7f3f0c0f89b0 msgr2=0x7f3f0c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 -- 172.21.15.103:0/1966281971 shutdown_connections 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.731+0000 7f3f12be7700 1 -- 172.21.15.103:0/1966281971 wait complete. 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 Processor -- start 2024-03-22T02:33:36.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 -- start start 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c071f90 0x7f3f0c11a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c11ac60 con 0x7f3f0c0708a0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f12be7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c1142f0 con 0x7f3f0c071f90 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f10983700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f10983700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33498/0 (socket says 172.21.15.103:33498) 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f10983700 1 -- 172.21.15.103:0/1530145099 learned_addr learned my addr 172.21.15.103:0/1530145099 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.732+0000 7f3f03fff700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c071f90 0x7f3f0c11a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f10983700 1 -- 172.21.15.103:0/1530145099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c071f90 msgr2=0x7f3f0c11a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f10983700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c071f90 0x7f3f0c11a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f10983700 1 -- 172.21.15.103:0/1530145099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f08009940 con 0x7f3f0c0708a0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f10983700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f08008df0 tx=0x7f3f08004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0800e040 con 0x7f3f0c0708a0 2024-03-22T02:33:36.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f0800be50 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c114570 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0801c740 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.733+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c114ab0 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.734+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c071aa0 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.735+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f3f08012b70 con 0x7f3f0c0708a0 2024-03-22T02:33:36.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.735+0000 7f3f017fa700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 0x7f3ef4070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:36.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.735+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3f0808fb00 con 0x7f3f0c0708a0 2024-03-22T02:33:36.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.735+0000 7f3f03fff700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 0x7f3ef4070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:36.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.736+0000 7f3f03fff700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 0x7f3ef4070980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c10cf70 tx=0x7f3efc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:36.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.740+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f08058b80 con 0x7f3f0c0708a0 2024-03-22T02:33:36.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.955+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f0c110520 con 0x7f3ef406e4d0 2024-03-22T02:33:36.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.957+0000 7f3f017fa700 1 -- 172.21.15.103:0/1530145099 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f3f0c110520 con 0x7f3ef406e4d0 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 msgr2=0x7f3ef4070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 0x7f3ef4070980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c10cf70 tx=0x7f3efc00b040 comp rx=0 tx=0).stop 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 msgr2=0x7f3f0c11a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f08008df0 tx=0x7f3f08004d10 comp rx=0 tx=0).stop 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 shutdown_connections 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ef406e4d0 0x7f3ef4070980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f0c0708a0 0x7f3f0c11a0d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 --2- 172.21.15.103:0/1530145099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f0c071f90 0x7f3f0c11a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:36.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 >> 172.21.15.103:0/1530145099 conn(0x7f3f0c0f89b0 msgr2=0x7f3f0c0f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 shutdown_connections 2024-03-22T02:33:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:36.959+0000 7f3f12be7700 1 -- 172.21.15.103:0/1530145099 wait complete. 2024-03-22T02:33:36.971 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- 172.21.15.103:0/665880400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 msgr2=0x7f1408102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 --2- 172.21.15.103:0/665880400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f1408102180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1404004940 tx=0x7f1404009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- 172.21.15.103:0/665880400 shutdown_connections 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 --2- 172.21.15.103:0/665880400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f14080fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 --2- 172.21.15.103:0/665880400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f1408102180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- 172.21.15.103:0/665880400 >> 172.21.15.103:0/665880400 conn(0x7f1408072310 msgr2=0x7f1408072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- 172.21.15.103:0/665880400 shutdown_connections 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- 172.21.15.103:0/665880400 wait complete. 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 Processor -- start 2024-03-22T02:33:37.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.065+0000 7f140f326700 1 -- start start 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140f326700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f140810ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140f326700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140f326700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140810f640 con 0x7f1408101db0 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140f326700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140810f780 con 0x7f14080fc4c0 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33526/0 (socket says 172.21.15.103:33526) 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 -- 172.21.15.103:0/1447719575 learned_addr learned my addr 172.21.15.103:0/1447719575 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140d0c2700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f140810ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 -- 172.21.15.103:0/1447719575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 msgr2=0x7f140810ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f140810ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 -- 172.21.15.103:0/1447719575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1404005040 con 0x7f1408101db0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140d0c2700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f140810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140c8c1700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13f8002a50 tx=0x7f13f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f8007cc0 con 0x7f1408101db0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1408117c30 con 0x7f1408101db0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13f800f040 con 0x7f1408101db0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.066+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f800db70 con 0x7f1408101db0 2024-03-22T02:33:37.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.067+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1408117f80 con 0x7f1408101db0 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.067+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14080ff090 con 0x7f1408101db0 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.068+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f13f800dd10 con 0x7f1408101db0 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.068+0000 7f13fdffb700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 0x7f13f4070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.069+0000 7f140d0c2700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 0x7f13f4070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.069+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f13f808d6e0 con 0x7f1408101db0 2024-03-22T02:33:37.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.069+0000 7f140d0c2700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 0x7f13f4070980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1404004560 tx=0x7f1404009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.073+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13f8056760 con 0x7f1408101db0 2024-03-22T02:33:37.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.289+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14080fb9c0 con 0x7f13f406e4d0 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: Reconfiguring mds.cephfs.smithi122.ztrkgp (monmap changed)... 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: Reconfiguring daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: pgmap v28: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:37.291 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:37 smithi103 ceph-mon[115211]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:37.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.290+0000 7f13fdffb700 1 -- 172.21.15.103:0/1447719575 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f14080fb9c0 con 0x7f13f406e4d0 2024-03-22T02:33:37.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 msgr2=0x7f13f4070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 0x7f13f4070980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1404004560 tx=0x7f1404009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 msgr2=0x7f140810f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13f8002a50 tx=0x7f13f80046b0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 shutdown_connections 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f13f406e4d0 0x7f13f4070980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14080fc4c0 0x7f140810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 --2- 172.21.15.103:0/1447719575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1408101db0 0x7f140810f100 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 >> 172.21.15.103:0/1447719575 conn(0x7f1408072310 msgr2=0x7f14080fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 shutdown_connections 2024-03-22T02:33:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.293+0000 7f140f326700 1 -- 172.21.15.103:0/1447719575 wait complete. 2024-03-22T02:33:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: Reconfiguring mds.cephfs.smithi122.ztrkgp (monmap changed)... 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: Reconfiguring daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: pgmap v28: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:37 smithi122 ceph-mon[83368]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:37.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- 172.21.15.103:0/2093481774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9464102e00 msgr2=0x7f94641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 --2- 172.21.15.103:0/2093481774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9464102e00 0x7f94641031d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9460004940 tx=0x7f9460009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- 172.21.15.103:0/2093481774 shutdown_connections 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 --2- 172.21.15.103:0/2093481774 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94641037a0 0x7f94640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 --2- 172.21.15.103:0/2093481774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9464102e00 0x7f94641031d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- 172.21.15.103:0/2093481774 >> 172.21.15.103:0/2093481774 conn(0x7f94640f89b0 msgr2=0x7f94640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- 172.21.15.103:0/2093481774 shutdown_connections 2024-03-22T02:33:37.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- 172.21.15.103:0/2093481774 wait complete. 2024-03-22T02:33:37.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 Processor -- start 2024-03-22T02:33:37.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.394+0000 7f946c3a9700 1 -- start start 2024-03-22T02:33:37.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9464102e00 0x7f9464113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464114520 con 0x7f94641037a0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946410db70 con 0x7f9464102e00 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33546/0 (socket says 172.21.15.103:33546) 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 -- 172.21.15.103:0/184433989 learned_addr learned my addr 172.21.15.103:0/184433989 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946a145700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9464102e00 0x7f9464113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 -- 172.21.15.103:0/184433989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9464102e00 msgr2=0x7f9464113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9464102e00 0x7f9464113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 -- 172.21.15.103:0/184433989 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9460005040 con 0x7f94641037a0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f9469944700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9454002a50 tx=0x7f9454004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9454007cc0 con 0x7f94641037a0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946410de50 con 0x7f94641037a0 2024-03-22T02:33:37.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f945400f040 con 0x7f94641037a0 2024-03-22T02:33:37.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f945400d9c0 con 0x7f94641037a0 2024-03-22T02:33:37.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.395+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946410e410 con 0x7f94641037a0 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.396+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9464100110 con 0x7f94641037a0 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.397+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f9454009730 con 0x7f94641037a0 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.397+0000 7f945affd700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 0x7f9450070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.397+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f945408dd20 con 0x7f94641037a0 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.397+0000 7f946a145700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 0x7f9450070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.398+0000 7f946a145700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 0x7f9450070980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9460000c00 tx=0x7f9460009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.402+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f945405a8d0 con 0x7f94641037a0 2024-03-22T02:33:37.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.605+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f946410f050 con 0x7f945006e4d0 2024-03-22T02:33:37.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.615+0000 7f945affd700 1 -- 172.21.15.103:0/184433989 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f946410f050 con 0x7f945006e4d0 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (2m) 45s ago 13m 25.8M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (13m) 45s ago 13m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 9b44c8fa0346 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (12m) 29s ago 12m 6665k - 17.2.7-726-g9c4e0c34 837124da804f 87d208f66717 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (113s) 45s ago 13m 76.6M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (7m) 45s ago 7m 13.4M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (7m) 45s ago 7m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (7m) 29s ago 7m 15.1M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (7m) 29s ago 7m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (3m) 45s ago 14m 473M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (3m) 29s ago 12m 448M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (59s) 45s ago 14m 27.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:33:37.617 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (34s) 29s ago 12m 25.5M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (2m) 45s ago 13m 16.1M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 29s ago 12m 16.9M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (11m) 45s ago 11m 215M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (11m) 45s ago 11m 231M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (10m) 45s ago 10m 199M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (10m) 29s ago 10m 264M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (9m) 29s ago 9m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (9m) 29s ago 9m 219M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:33:37.618 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (2m) 45s ago 12m 58.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:33:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 msgr2=0x7f9450070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 0x7f9450070980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9460000c00 tx=0x7f9460009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 msgr2=0x7f9464113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9454002a50 tx=0x7f9454004600 comp rx=0 tx=0).stop 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 shutdown_connections 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f945006e4d0 0x7f9450070980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9464102e00 0x7f9464113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.617+0000 7f946c3a9700 1 --2- 172.21.15.103:0/184433989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94641037a0 0x7f9464113ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.618+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 >> 172.21.15.103:0/184433989 conn(0x7f94640f89b0 msgr2=0x7f94641073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.618+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 shutdown_connections 2024-03-22T02:33:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.618+0000 7f946c3a9700 1 -- 172.21.15.103:0/184433989 wait complete. 2024-03-22T02:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 -- 172.21.15.103:0/3144963890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 msgr2=0x7f92d00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 --2- 172.21.15.103:0/3144963890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d00fbd30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92c0004940 tx=0x7f92c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 -- 172.21.15.103:0/3144963890 shutdown_connections 2024-03-22T02:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 --2- 172.21.15.103:0/3144963890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d00fbd30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 --2- 172.21.15.103:0/3144963890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 0x7f92d01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 -- 172.21.15.103:0/3144963890 >> 172.21.15.103:0/3144963890 conn(0x7f92d00f7870 msgr2=0x7f92d00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 -- 172.21.15.103:0/3144963890 shutdown_connections 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.717+0000 7f92d75c5700 1 -- 172.21.15.103:0/3144963890 wait complete. 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 Processor -- start 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 -- start start 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 0x7f92d01128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d0113440 con 0x7f92d01026c0 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d75c5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d010ca40 con 0x7f92d0101d20 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33566/0 (socket says 172.21.15.103:33566) 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 -- 172.21.15.103:0/1802043336 learned_addr learned my addr 172.21.15.103:0/1802043336 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d5361700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 0x7f92d01128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 -- 172.21.15.103:0/1802043336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 msgr2=0x7f92d01128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:37.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 0x7f92d01128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 -- 172.21.15.103:0/1802043336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c0005040 con 0x7f92d01026c0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.718+0000 7f92d4b60700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009100 tx=0x7f92c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.719+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92c000e040 con 0x7f92d01026c0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.719+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92c0004290 con 0x7f92d01026c0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.719+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d010ccc0 con 0x7f92d01026c0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.719+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92c0013660 con 0x7f92d01026c0 2024-03-22T02:33:37.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.719+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d010d220 con 0x7f92d01026c0 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.720+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d00fd470 con 0x7f92d01026c0 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.720+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f92c000bd00 con 0x7f92d01026c0 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.721+0000 7f92c5ffb700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 0x7f92bc070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.721+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f92c0091f30 con 0x7f92d01026c0 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.721+0000 7f92d5361700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 0x7f92bc070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:37.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.721+0000 7f92d5361700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 0x7f92bc070980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92cc004eb0 tx=0x7f92cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:37.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:37.725+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f92c00c6740 con 0x7f92d01026c0 2024-03-22T02:33:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.009+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f92d0108e00 con 0x7f92d01026c0 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.009+0000 7f92c5ffb700 1 -- 172.21.15.103:0/1802043336 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f92c005eae0 con 0x7f92d01026c0 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:33:38.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 10, 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:33:38.012 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 msgr2=0x7f92bc070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 0x7f92bc070980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92cc004eb0 tx=0x7f92cc00b040 comp rx=0 tx=0).stop 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 msgr2=0x7f92d0112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009100 tx=0x7f92c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 shutdown_connections 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f92bc06e4d0 0x7f92bc070980 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0101d20 0x7f92d01128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 --2- 172.21.15.103:0/1802043336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92d01026c0 0x7f92d0112df0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 >> 172.21.15.103:0/1802043336 conn(0x7f92d00f7870 msgr2=0x7f92d00f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 shutdown_connections 2024-03-22T02:33:38.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.011+0000 7f92d75c5700 1 -- 172.21.15.103:0/1802043336 wait complete. 2024-03-22T02:33:38.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- 172.21.15.103:0/456861268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f94101d20 msgr2=0x7f6f941020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/456861268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f94101d20 0x7f6f941020f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f90004940 tx=0x7f6f90009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- 172.21.15.103:0/456861268 shutdown_connections 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/456861268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f941026c0 0x7f6f940fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/456861268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f94101d20 0x7f6f941020f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- 172.21.15.103:0/456861268 >> 172.21.15.103:0/456861268 conn(0x7f6f940f78b0 msgr2=0x7f6f940f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- 172.21.15.103:0/456861268 shutdown_connections 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- 172.21.15.103:0/456861268 wait complete. 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 Processor -- start 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- start start 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f94101d20 0x7f6f9410eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9410f5c0 con 0x7f6f941026c0 2024-03-22T02:33:38.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.113+0000 7f6f9b98f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9410f700 con 0x7f6f94101d20 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33584/0 (socket says 172.21.15.103:33584) 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 -- 172.21.15.103:0/2362690367 learned_addr learned my addr 172.21.15.103:0/2362690367 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f9972b700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f94101d20 0x7f6f9410eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 -- 172.21.15.103:0/2362690367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f94101d20 msgr2=0x7f6f9410eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f94101d20 0x7f6f9410eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 -- 172.21.15.103:0/2362690367 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90005040 con 0x7f6f941026c0 2024-03-22T02:33:38.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f98f2a700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f84002a50 tx=0x7f6f84004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f84007cc0 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f8400f040 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f8400d9c0 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.114+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94117bb0 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.115+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94117f80 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.115+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f940fd470 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.116+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f6f84009730 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.116+0000 7f6f89ffb700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 0x7f6f80070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.116+0000 7f6f9972b700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 0x7f6f80070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.116+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6f8408dd20 con 0x7f6f941026c0 2024-03-22T02:33:38.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.117+0000 7f6f9972b700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 0x7f6f80070980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6f90004f90 tx=0x7f6f90009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.121+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f84056da0 con 0x7f6f941026c0 2024-03-22T02:33:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.367+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6f94118170 con 0x7f6f941026c0 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.368+0000 7f6f89ffb700 1 -- 172.21.15.103:0/2362690367 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7f6f8405a460 con 0x7f6f941026c0 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:33:38.370 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:33:38.371 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:33:38.372 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:38.373 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 msgr2=0x7f6f80070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 0x7f6f80070980 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6f90004f90 tx=0x7f6f90009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 msgr2=0x7f6f9410f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f84002a50 tx=0x7f6f84004600 comp rx=0 tx=0).stop 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 shutdown_connections 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6f8006e4d0 0x7f6f80070980 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f94101d20 0x7f6f9410eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 --2- 172.21.15.103:0/2362690367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f941026c0 0x7f6f9410f080 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.370+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 >> 172.21.15.103:0/2362690367 conn(0x7f6f940f78b0 msgr2=0x7f6f94104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.371+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 shutdown_connections 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.371+0000 7f6f9b98f700 1 -- 172.21.15.103:0/2362690367 wait complete. 2024-03-22T02:33:38.374 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 -- 172.21.15.103:0/202529672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 msgr2=0x7ff53c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 --2- 172.21.15.103:0/202529672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c1020f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff52c004940 tx=0x7ff52c009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 -- 172.21.15.103:0/202529672 shutdown_connections 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 --2- 172.21.15.103:0/202529672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 0x7ff53c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 --2- 172.21.15.103:0/202529672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c1020f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 -- 172.21.15.103:0/202529672 >> 172.21.15.103:0/202529672 conn(0x7ff53c0f78d0 msgr2=0x7ff53c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 -- 172.21.15.103:0/202529672 shutdown_connections 2024-03-22T02:33:38.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 -- 172.21.15.103:0/202529672 wait complete. 2024-03-22T02:33:38.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.473+0000 7ff541948700 1 Processor -- start 2024-03-22T02:33:38.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff541948700 1 -- start start 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff541948700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff541948700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 0x7ff53c10f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff541948700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c10f710 con 0x7ff53c101d20 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff541948700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c10f850 con 0x7ff53c1026c0 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33596/0 (socket says 172.21.15.103:33596) 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 -- 172.21.15.103:0/4122006122 learned_addr learned my addr 172.21.15.103:0/4122006122 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff5327fc700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 0x7ff53c10f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 -- 172.21.15.103:0/4122006122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 msgr2=0x7ff53c10f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 0x7ff53c10f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 -- 172.21.15.103:0/4122006122 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c005040 con 0x7ff53c101d20 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.474+0000 7ff53affd700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00b8b0 tx=0x7ff52c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.475+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52c00e040 con 0x7ff53c101d20 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.475+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff52c004290 con 0x7ff53c101d20 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.475+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c117bf0 con 0x7ff53c101d20 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.475+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52c01c3f0 con 0x7ff53c101d20 2024-03-22T02:33:38.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.475+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c118100 con 0x7ff53c101d20 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.476+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c0fd470 con 0x7ff53c101d20 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.476+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7ff52c00baf0 con 0x7ff53c101d20 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.476+0000 7ff533fff700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 0x7ff528070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.477+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff52c08f190 con 0x7ff53c101d20 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.477+0000 7ff5327fc700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 0x7ff528070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.477+0000 7ff5327fc700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 0x7ff528070970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff53c066890 tx=0x7ff524009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.481+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff52c058270 con 0x7ff53c101d20 2024-03-22T02:33:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.697+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff53c066640 con 0x7ff52806e4c0 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.698+0000 7ff533fff700 1 -- 172.21.15.103:0/4122006122 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7ff53c066640 con 0x7ff52806e4c0 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:33:38.700 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "4/21 daemons upgraded", 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:33:38.701 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.700+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 msgr2=0x7ff528070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.700+0000 7ff541948700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 0x7ff528070970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff53c066890 tx=0x7ff524009040 comp rx=0 tx=0).stop 2024-03-22T02:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.700+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 msgr2=0x7ff53c10eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.700+0000 7ff541948700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00b8b0 tx=0x7ff52c009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 shutdown_connections 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff52806e4c0 0x7ff528070970 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff53c101d20 0x7ff53c10eb80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 --2- 172.21.15.103:0/4122006122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff53c1026c0 0x7ff53c10f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 >> 172.21.15.103:0/4122006122 conn(0x7ff53c0f78d0 msgr2=0x7ff53c0f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 shutdown_connections 2024-03-22T02:33:38.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.701+0000 7ff541948700 1 -- 172.21.15.103:0/4122006122 wait complete. 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 -- 172.21.15.103:0/1008496326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 msgr2=0x7fbb341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/1008496326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb341031d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbb24004940 tx=0x7fbb24009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 -- 172.21.15.103:0/1008496326 shutdown_connections 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/1008496326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 0x7fbb340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/1008496326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb341031d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 -- 172.21.15.103:0/1008496326 >> 172.21.15.103:0/1008496326 conn(0x7fbb340f89b0 msgr2=0x7fbb340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 -- 172.21.15.103:0/1008496326 shutdown_connections 2024-03-22T02:33:38.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.802+0000 7fbb3bcae700 1 -- 172.21.15.103:0/1008496326 wait complete. 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 Processor -- start 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 -- start start 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 0x7fbb34070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb34070d90 con 0x7fbb34102e00 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb3bcae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb34070ed0 con 0x7fbb341037a0 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33618/0 (socket says 172.21.15.103:33618) 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 -- 172.21.15.103:0/3926731028 learned_addr learned my addr 172.21.15.103:0/3926731028 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39249700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 0x7fbb34070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 -- 172.21.15.103:0/3926731028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 msgr2=0x7fbb34070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 0x7fbb34070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.803+0000 7fbb39a4a700 1 -- 172.21.15.103:0/3926731028 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24005040 con 0x7fbb34102e00 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb39a4a700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009b30 tx=0x7fbb24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb24021070 con 0x7fbb34102e00 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb34071120 con 0x7fbb34102e00 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb24013c00 con 0x7fbb34102e00 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb2400e070 con 0x7fbb34102e00 2024-03-22T02:33:38.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.804+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3411a1e0 con 0x7fbb34102e00 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.805+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb34100110 con 0x7fbb34102e00 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.805+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fbb2401cad0 con 0x7fbb34102e00 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.806+0000 7fbb2a7fc700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 0x7fbb20070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.806+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fbb2408f350 con 0x7fbb34102e00 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.806+0000 7fbb39249700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 0x7fbb20070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:38.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.806+0000 7fbb39249700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 0x7fbb20070970 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb34071c00 tx=0x7fbb3000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:38.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:38.810+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb24058450 con 0x7fbb34102e00 2024-03-22T02:33:39.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.088+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbb3404bed0 con 0x7fbb34102e00 2024-03-22T02:33:39.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.088+0000 7fbb2a7fc700 1 -- 172.21.15.103:0/3926731028 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fbb24004490 con 0x7fbb34102e00 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: Reconfiguring mds.cephfs.smithi122.lvsegg (monmap changed)... 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: Reconfiguring daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1802043336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:39.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:38 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2362690367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:33:39.091 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:33:39.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.090+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 msgr2=0x7fbb20070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:39.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.090+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 0x7fbb20070970 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb34071c00 tx=0x7fbb3000b040 comp rx=0 tx=0).stop 2024-03-22T02:33:39.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 msgr2=0x7fbb340702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009b30 tx=0x7fbb24009e30 comp rx=0 tx=0).stop 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 shutdown_connections 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbb2006e4c0 0x7fbb20070970 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb34102e00 0x7fbb340702c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 --2- 172.21.15.103:0/3926731028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb341037a0 0x7fbb34070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 >> 172.21.15.103:0/3926731028 conn(0x7fbb340f89b0 msgr2=0x7fbb341073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 shutdown_connections 2024-03-22T02:33:39.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:33:39.091+0000 7fbb3bcae700 1 -- 172.21.15.103:0/3926731028 wait complete. 2024-03-22T02:33:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: Reconfiguring mds.cephfs.smithi122.lvsegg (monmap changed)... 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: Reconfiguring daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1802043336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:38 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2362690367' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:33:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: pgmap v29: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3926731028' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi103"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi103"}]': finished 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-03-22T02:33:40.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: pgmap v29: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3926731028' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi103"}]: dispatch 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi103"}]': finished 2024-03-22T02:33:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-03-22T02:33:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-03-22T02:33:41.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:40 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all mon 2024-03-22T02:33:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:40 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all mon 2024-03-22T02:33:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:41 smithi103 ceph-mon[115211]: pgmap v30: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:41 smithi122 ceph-mon[83368]: pgmap v30: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:42.940 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:42 smithi103 ceph-mon[115211]: Upgrade: Updating crash.smithi103 (1/2) 2024-03-22T02:33:42.940 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:42 smithi103 ceph-mon[115211]: Deploying daemon crash.smithi103 on smithi103 2024-03-22T02:33:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:42 smithi122 ceph-mon[83368]: Upgrade: Updating crash.smithi103 (1/2) 2024-03-22T02:33:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:42 smithi122 ceph-mon[83368]: Deploying daemon crash.smithi103 on smithi103 2024-03-22T02:33:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:43 smithi103 ceph-mon[115211]: pgmap v31: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:43 smithi122 ceph-mon[83368]: pgmap v31: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:45 smithi122 ceph-mon[83368]: pgmap v32: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:46.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:45 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:46.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:45 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:45 smithi103 ceph-mon[115211]: pgmap v32: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:45 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:45 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: Upgrade: Updating crash.smithi122 (2/2) 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: Deploying daemon crash.smithi122 on smithi122 2024-03-22T02:33:47.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: pgmap v33: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:47.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: Upgrade: Updating crash.smithi122 (2/2) 2024-03-22T02:33:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-22T02:33:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:47.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: Deploying daemon crash.smithi122 on smithi122 2024-03-22T02:33:47.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: pgmap v33: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:47.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:33:50.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:49 smithi122 ceph-mon[83368]: pgmap v34: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:49 smithi103 ceph-mon[115211]: pgmap v34: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:51.307 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:51.308 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:51.308 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:51.308 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:51 smithi122 ceph-mon[83368]: pgmap v35: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:51 smithi103 ceph-mon[115211]: pgmap v35: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:54.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:53 smithi103 ceph-mon[115211]: pgmap v36: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:54.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:54.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:53 smithi122 ceph-mon[83368]: pgmap v36: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:55.485 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:55 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:55.485 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:55 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:55.485 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:55 smithi122 ceph-mon[83368]: pgmap v37: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:55 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:55 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:55.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:55 smithi103 ceph-mon[115211]: pgmap v37: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:56 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:56 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:56 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:56 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:57 smithi122 ceph-mon[83368]: pgmap v38: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:57 smithi103 ceph-mon[115211]: pgmap v38: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:58.709 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:58.709 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:58.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:33:59.770 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:33:59 smithi103 ceph-mon[115211]: pgmap v39: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:33:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:33:59 smithi122 ceph-mon[83368]: pgmap v39: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:01.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all crash 2024-03-22T02:34:01.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi103"}]: dispatch 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi103"}]': finished 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: Upgrade: osd.0 is safe to restart 2024-03-22T02:34:01.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:01 smithi122 ceph-mon[83368]: pgmap v40: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all crash 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi103"}]: dispatch 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi103"}]': finished 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: Upgrade: osd.0 is safe to restart 2024-03-22T02:34:01.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:01 smithi103 ceph-mon[115211]: pgmap v40: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:34:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:34:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:03 smithi103 ceph-mon[115211]: Upgrade: Updating osd.0 2024-03-22T02:34:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:03 smithi103 ceph-mon[115211]: Deploying daemon osd.0 on smithi103 2024-03-22T02:34:03.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:03 smithi103 ceph-mon[115211]: pgmap v41: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:03 smithi122 ceph-mon[83368]: Upgrade: Updating osd.0 2024-03-22T02:34:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:03 smithi122 ceph-mon[83368]: Deploying daemon osd.0 on smithi103 2024-03-22T02:34:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:03 smithi122 ceph-mon[83368]: pgmap v41: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:05.624 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:05 smithi103 systemd[1]: Stopping Ceph osd.0 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:34:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:05 smithi103 ceph-mon[115211]: pgmap v42: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:05 smithi103 ceph-mon[115211]: osd.0 marked itself down and dead 2024-03-22T02:34:06.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:05 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[48258]: 2024-03-22T02:34:05.818+0000 7fd3708be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:34:06.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:05 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[48258]: 2024-03-22T02:34:05.818+0000 7fd3708be700 -1 osd.0 49 *** Got signal Terminated *** 2024-03-22T02:34:06.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:05 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[48258]: 2024-03-22T02:34:05.818+0000 7fd3708be700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:34:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:05 smithi122 ceph-mon[83368]: pgmap v42: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:05 smithi122 ceph-mon[83368]: osd.0 marked itself down and dead 2024-03-22T02:34:06.453 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:06 smithi103 podman[126058]: 2024-03-22 02:34:06.196398273 +0000 UTC m=+0.490175758 container died 5d49479f80b739275ba32ca8f1ddb3d921093802475dd564942cd028afc2586e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-22T02:34:06.869 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:06 smithi103 podman[126058]: 2024-03-22 02:34:06.527404536 +0000 UTC m=+0.821182020 container cleanup 5d49479f80b739275ba32ca8f1ddb3d921093802475dd564942cd028afc2586e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2) 2024-03-22T02:34:06.869 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:06 smithi103 bash[126058]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0 2024-03-22T02:34:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:06 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:34:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:06 smithi103 ceph-mon[115211]: osdmap e50: 6 total, 5 up, 6 in 2024-03-22T02:34:07.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:06 smithi103 podman[126077]: 2024-03-22 02:34:06.920775402 +0000 UTC m=+0.718814324 container remove 5d49479f80b739275ba32ca8f1ddb3d921093802475dd564942cd028afc2586e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T02:34:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:06 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:34:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:06 smithi122 ceph-mon[83368]: osdmap e50: 6 total, 5 up, 6 in 2024-03-22T02:34:07.667 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.28819322 +0000 UTC m=+0.018022686 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:07.667 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.397305406 +0000 UTC m=+0.127134910 container create 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T02:34:07.667 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.59152249 +0000 UTC m=+0.321351963 container init 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-22T02:34:07.667 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.594707225 +0000 UTC m=+0.324536696 container start 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:34:07.667 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.666658562 +0000 UTC m=+0.396488034 container attach 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T02:34:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:07 smithi103 ceph-mon[115211]: pgmap v44: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:07 smithi103 ceph-mon[115211]: osdmap e51: 6 total, 5 up, 6 in 2024-03-22T02:34:08.125 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:07 smithi103 podman[126170]: 2024-03-22 02:34:07.885203769 +0000 UTC m=+0.615033239 container died 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-22T02:34:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:07 smithi122 ceph-mon[83368]: pgmap v44: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:07 smithi122 ceph-mon[83368]: osdmap e51: 6 total, 5 up, 6 in 2024-03-22T02:34:08.660 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 podman[126196]: 2024-03-22 02:34:08.512895261 +0000 UTC m=+0.622490997 container remove 05a84ee805e14476559391fd267d936b0188387030a8df2b3f456f57a0376b1e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:34:08.660 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-22T02:34:08.660 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 systemd[1]: Stopped Ceph osd.0 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:34:08.660 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.0.service: Consumed 1min 395ms CPU time. 2024-03-22T02:34:08.933 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 systemd[1]: Starting Ceph osd.0 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:34:09.200 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:08 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded (PG_DEGRADED) 2024-03-22T02:34:09.201 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 podman[126298]: 2024-03-22 02:34:08.864296898 +0000 UTC m=+0.019705714 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:09.201 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:08 smithi103 podman[126298]: 2024-03-22 02:34:08.990206562 +0000 UTC m=+0.145615364 container create 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:34:09.201 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 podman[126298]: 2024-03-22 02:34:09.176300687 +0000 UTC m=+0.331709501 container init 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:34:09.201 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 podman[126298]: 2024-03-22 02:34:09.180612791 +0000 UTC m=+0.336021613 container start 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 -- 172.21.15.103:0/2341240957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 msgr2=0x7ff36c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 --2- 172.21.15.103:0/2341240957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c102140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff364009ce0 tx=0x7ff364004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 -- 172.21.15.103:0/2341240957 shutdown_connections 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 --2- 172.21.15.103:0/2341240957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 0x7ff36c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 --2- 172.21.15.103:0/2341240957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c102140 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 -- 172.21.15.103:0/2341240957 >> 172.21.15.103:0/2341240957 conn(0x7ff36c072bc0 msgr2=0x7ff36c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 -- 172.21.15.103:0/2341240957 shutdown_connections 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 -- 172.21.15.103:0/2341240957 wait complete. 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.199+0000 7ff374315700 1 Processor -- start 2024-03-22T02:34:09.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff374315700 1 -- start start 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff374315700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff374315700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 0x7ff36c1130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff374315700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c1137b0 con 0x7ff36c101d70 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff374315700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c10cb10 con 0x7ff36c102710 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff3720b1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff3720b1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59200/0 (socket says 172.21.15.103:59200) 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff3720b1700 1 -- 172.21.15.103:0/775146425 learned_addr learned my addr 172.21.15.103:0/775146425 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.200+0000 7ff3718b0700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 0x7ff36c1130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff3720b1700 1 -- 172.21.15.103:0/775146425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 msgr2=0x7ff36c1130d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff3720b1700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 0x7ff36c1130d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff3720b1700 1 -- 172.21.15.103:0/775146425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff364009940 con 0x7ff36c101d70 2024-03-22T02:34:09.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff3720b1700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff36400be90 tx=0x7ff364004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff36400e040 con 0x7ff36c101d70 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff364004610 con 0x7ff36c101d70 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c10cd90 con 0x7ff36c101d70 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff36401c6d0 con 0x7ff36c101d70 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.201+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c10d350 con 0x7ff36c101d70 2024-03-22T02:34:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.202+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c0fd4c0 con 0x7ff36c101d70 2024-03-22T02:34:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.203+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7ff364012b00 con 0x7ff36c101d70 2024-03-22T02:34:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.203+0000 7ff362ffd700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 0x7ff358070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.204+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff36408fa10 con 0x7ff36c101d70 2024-03-22T02:34:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.204+0000 7ff3718b0700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 0x7ff358070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.204+0000 7ff3718b0700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 0x7ff358070970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff36c10dcd0 tx=0x7ff35c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.208+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff36405c640 con 0x7ff36c101d70 2024-03-22T02:34:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:08 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded (PG_DEGRADED) 2024-03-22T02:34:09.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.434+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff36c04bed0 con 0x7ff35806e4c0 2024-03-22T02:34:09.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.435+0000 7ff362ffd700 1 -- 172.21.15.103:0/775146425 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff36c04bed0 con 0x7ff35806e4c0 2024-03-22T02:34:09.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 msgr2=0x7ff358070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 0x7ff358070970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff36c10dcd0 tx=0x7ff35c0091a0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 msgr2=0x7ff36c112b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff36400be90 tx=0x7ff364004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 shutdown_connections 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff35806e4c0 0x7ff358070970 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff36c101d70 0x7ff36c112b90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 --2- 172.21.15.103:0/775146425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff36c102710 0x7ff36c1130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.437+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 >> 172.21.15.103:0/775146425 conn(0x7ff36c072bc0 msgr2=0x7ff36c0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.438+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 shutdown_connections 2024-03-22T02:34:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.438+0000 7ff374315700 1 -- 172.21.15.103:0/775146425 wait complete. 2024-03-22T02:34:09.449 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:34:09.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 -- 172.21.15.103:0/38384189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 msgr2=0x7fbf400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 --2- 172.21.15.103:0/38384189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf400fce10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf30004940 tx=0x7fbf30009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:09.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 -- 172.21.15.103:0/38384189 shutdown_connections 2024-03-22T02:34:09.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 --2- 172.21.15.103:0/38384189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf400fce10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 --2- 172.21.15.103:0/38384189 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 -- 172.21.15.103:0/38384189 >> 172.21.15.103:0/38384189 conn(0x7fbf400f89b0 msgr2=0x7fbf400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:09.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 -- 172.21.15.103:0/38384189 shutdown_connections 2024-03-22T02:34:09.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.545+0000 7fbf4666e700 1 -- 172.21.15.103:0/38384189 wait complete. 2024-03-22T02:34:09.547 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 podman[126298]: 2024-03-22 02:34:09.293326753 +0000 UTC m=+0.448735559 container attach 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-22T02:34:09.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 Processor -- start 2024-03-22T02:34:09.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 -- start start 2024-03-22T02:34:09.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf40115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40116990 con 0x7fbf401037a0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf4666e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4010fcd0 con 0x7fbf40102e00 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59220/0 (socket says 172.21.15.103:59220) 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf3f7fe700 1 -- 172.21.15.103:0/4215313094 learned_addr learned my addr 172.21.15.103:0/4215313094 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.546+0000 7fbf3ffff700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf40115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3f7fe700 1 -- 172.21.15.103:0/4215313094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 msgr2=0x7fbf40115d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3f7fe700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf40115d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3f7fe700 1 -- 172.21.15.103:0/4215313094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30005040 con 0x7fbf401037a0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3ffff700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf40115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3f7fe700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009100 tx=0x7fbf30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf3000e040 con 0x7fbf401037a0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4010fef0 con 0x7fbf401037a0 2024-03-22T02:34:09.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf30013420 con 0x7fbf401037a0 2024-03-22T02:34:09.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf401104b0 con 0x7fbf401037a0 2024-03-22T02:34:09.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.547+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf3001c610 con 0x7fbf401037a0 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.548+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf40107300 con 0x7fbf401037a0 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.549+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fbf30013590 con 0x7fbf401037a0 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.549+0000 7fbf3cff9700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 0x7fbf28070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.549+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fbf3008f280 con 0x7fbf401037a0 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.549+0000 7fbf3ffff700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 0x7fbf28070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.550+0000 7fbf3ffff700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 0x7fbf28070970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009a80 tx=0x7fbf34008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.554+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf3005beb0 con 0x7fbf401037a0 2024-03-22T02:34:09.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.783+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf40109f50 con 0x7fbf2806e4c0 2024-03-22T02:34:09.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.784+0000 7fbf3cff9700 1 -- 172.21.15.103:0/4215313094 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbf40109f50 con 0x7fbf2806e4c0 2024-03-22T02:34:09.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 msgr2=0x7fbf28070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 0x7fbf28070970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009a80 tx=0x7fbf34008040 comp rx=0 tx=0).stop 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 msgr2=0x7fbf401162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009100 tx=0x7fbf30009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 shutdown_connections 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbf2806e4c0 0x7fbf28070970 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf40102e00 0x7fbf40115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 --2- 172.21.15.103:0/4215313094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf401037a0 0x7fbf401162b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 >> 172.21.15.103:0/4215313094 conn(0x7fbf400f89b0 msgr2=0x7fbf40070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 shutdown_connections 2024-03-22T02:34:09.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.787+0000 7fbf4666e700 1 -- 172.21.15.103:0/4215313094 wait complete. 2024-03-22T02:34:09.874 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 bash[126298]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate[126313]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-22T02:34:09.875 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:09 smithi103 podman[126298]: 2024-03-22 02:34:09.665800941 +0000 UTC m=+0.821209739 container died 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True) 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 -- 172.21.15.103:0/3755388909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 msgr2=0x7fab140fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 --2- 172.21.15.103:0/3755388909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab140fe260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab04004940 tx=0x7fab04009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 -- 172.21.15.103:0/3755388909 shutdown_connections 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 --2- 172.21.15.103:0/3755388909 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab14106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 --2- 172.21.15.103:0/3755388909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab140fe260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 -- 172.21.15.103:0/3755388909 >> 172.21.15.103:0/3755388909 conn(0x7fab14072bc0 msgr2=0x7fab14072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:09.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 -- 172.21.15.103:0/3755388909 shutdown_connections 2024-03-22T02:34:09.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.889+0000 7fab1c255700 1 -- 172.21.15.103:0/3755388909 wait complete. 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 Processor -- start 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 -- start start 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab141140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab14114730 con 0x7fab140fde90 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab1c255700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1410db40 con 0x7fab140fe830 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59240/0 (socket says 172.21.15.103:59240) 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 -- 172.21.15.103:0/565565732 learned_addr learned my addr 172.21.15.103:0/565565732 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab197f0700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab141140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 -- 172.21.15.103:0/565565732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 msgr2=0x7fab141140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:09.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab141140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 -- 172.21.15.103:0/565565732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab04005040 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab197f0700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab141140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab19ff1700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab0400b3b0 tx=0x7fab04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.890+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0400e040 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.891+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab04013420 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.891+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1410ddc0 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.891+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0401c610 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.891+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1410e300 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.891+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14065e30 con 0x7fab140fde90 2024-03-22T02:34:09.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.892+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fab04013590 con 0x7fab140fde90 2024-03-22T02:34:09.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.893+0000 7fab0a7fc700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 0x7fab00070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:09.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.893+0000 7fab197f0700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 0x7fab00070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:09.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.893+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fab0408f270 con 0x7fab140fde90 2024-03-22T02:34:09.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.893+0000 7fab197f0700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 0x7fab00070970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab10009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:09.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:09.897+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab04058370 con 0x7fab140fde90 2024-03-22T02:34:10.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.109+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fab1404bed0 con 0x7fab0006e4c0 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.119+0000 7fab0a7fc700 1 -- 172.21.15.103:0/565565732 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fab1404bed0 con 0x7fab0006e4c0 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (2m) 14s ago 13m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (25s) 14s ago 14m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (20s) 17s ago 13m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (2m) 14s ago 13m 76.9M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:34:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (8m) 14s ago 8m 13.7M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (8m) 14s ago 8m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (8m) 17s ago 8m 15.3M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (8m) 17s ago 8m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (4m) 14s ago 15m 495M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (3m) 17s ago 12m 449M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (92s) 14s ago 15m 41.0M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (67s) 17s ago 12m 35.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (3m) 14s ago 14m 16.2M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 17s ago 12m 16.8M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (12m) 14s ago 12m 216M 1557M 17.2.7-726-g9c4e0c34 837124da804f 5d49479f80b7 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (11m) 14s ago 11m 231M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (11m) 14s ago 11m 200M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (10m) 17s ago 10m 265M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (10m) 17s ago 10m 231M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:34:10.121 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (9m) 17s ago 9m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:34:10.122 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (3m) 14s ago 13m 59.6M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:34:10.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 msgr2=0x7fab00070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 0x7fab00070970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fab10009fd0 tx=0x7fab10009f20 comp rx=0 tx=0).stop 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 msgr2=0x7fab14113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab0400b3b0 tx=0x7fab04009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 shutdown_connections 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fab0006e4c0 0x7fab00070970 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab140fde90 0x7fab14113ba0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 --2- 172.21.15.103:0/565565732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab140fe830 0x7fab141140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.121+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 >> 172.21.15.103:0/565565732 conn(0x7fab14072bc0 msgr2=0x7fab14103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.122+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 shutdown_connections 2024-03-22T02:34:10.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.122+0000 7fab1c255700 1 -- 172.21.15.103:0/565565732 wait complete. 2024-03-22T02:34:10.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 -- 172.21.15.103:0/2053668090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4140fb4a0 msgr2=0x7ff4140fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 --2- 172.21.15.103:0/2053668090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4140fb4a0 0x7ff4140fb910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff404004940 tx=0x7ff404009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:10.224 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:09 smithi103 ceph-mon[115211]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:10.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 -- 172.21.15.103:0/2053668090 shutdown_connections 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 --2- 172.21.15.103:0/2053668090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4140fb4a0 0x7ff4140fb910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 --2- 172.21.15.103:0/2053668090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4141014f0 0x7ff4141018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 -- 172.21.15.103:0/2053668090 >> 172.21.15.103:0/2053668090 conn(0x7ff4140714c0 msgr2=0x7ff4140718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 -- 172.21.15.103:0/2053668090 shutdown_connections 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.223+0000 7ff41a757700 1 -- 172.21.15.103:0/2053668090 wait complete. 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 Processor -- start 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 -- start start 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4140fb4a0 0x7ff414112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141135f0 con 0x7ff4141014f0 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff41a757700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41410c9e0 con 0x7ff4140fb4a0 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff4137fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff4137fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59250/0 (socket says 172.21.15.103:59250) 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.224+0000 7ff4137fe700 1 -- 172.21.15.103:0/22304841 learned_addr learned my addr 172.21.15.103:0/22304841 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff413fff700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4140fb4a0 0x7ff414112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff4137fe700 1 -- 172.21.15.103:0/22304841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4140fb4a0 msgr2=0x7ff414112a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff4137fe700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4140fb4a0 0x7ff414112a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff4137fe700 1 -- 172.21.15.103:0/22304841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff404005040 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff4137fe700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff40400b3b0 tx=0x7ff404009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff40400e040 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff404013420 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41410cc60 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff40401c630 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.225+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41410d220 con 0x7ff4141014f0 2024-03-22T02:34:10.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.226+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4140fcbd0 con 0x7ff4141014f0 2024-03-22T02:34:10.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.227+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7ff404013590 con 0x7ff4141014f0 2024-03-22T02:34:10.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.227+0000 7ff410ff9700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 0x7ff3fc070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.227+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff40408f270 con 0x7ff4141014f0 2024-03-22T02:34:10.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.227+0000 7ff413fff700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 0x7ff3fc070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.228+0000 7ff413fff700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 0x7ff3fc070970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff408004f40 tx=0x7ff40800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:10.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.232+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff40405be00 con 0x7ff4141014f0 2024-03-22T02:34:10.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:09 smithi122 ceph-mon[83368]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:10.512 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 podman[126484]: 2024-03-22 02:34:10.326464691 +0000 UTC m=+0.651932144 container remove 85d19e837fe1a712431595e7c23f65b3a14581a9d6e5632f92a2acd1b12761f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T02:34:10.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.537+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff41405f5f0 con 0x7ff4141014f0 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.537+0000 7ff410ff9700 1 -- 172.21.15.103:0/22304841 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7ff40405b990 con 0x7ff4141014f0 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:34:10.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 5 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 9, 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:34:10.539 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:34:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 msgr2=0x7ff3fc070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 0x7ff3fc070970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff408004f40 tx=0x7ff40800b040 comp rx=0 tx=0).stop 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 msgr2=0x7ff414112fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff40400b3b0 tx=0x7ff404009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 shutdown_connections 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3fc06e4c0 0x7ff3fc070970 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4140fb4a0 0x7ff414112a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 --2- 172.21.15.103:0/22304841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4141014f0 0x7ff414112fa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 >> 172.21.15.103:0/22304841 conn(0x7ff4140714c0 msgr2=0x7ff4140f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 shutdown_connections 2024-03-22T02:34:10.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.540+0000 7ff41a757700 1 -- 172.21.15.103:0/22304841 wait complete. 2024-03-22T02:34:10.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.642+0000 7f18b2a3d700 1 -- 172.21.15.103:0/3387199905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 msgr2=0x7f18ac106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.642+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/3387199905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac106e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18a0004940 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.642+0000 7f18b2a3d700 1 -- 172.21.15.103:0/3387199905 shutdown_connections 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.642+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/3387199905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac106e00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.642+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/3387199905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac0fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- 172.21.15.103:0/3387199905 >> 172.21.15.103:0/3387199905 conn(0x7f18ac0f7870 msgr2=0x7f18ac0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- 172.21.15.103:0/3387199905 shutdown_connections 2024-03-22T02:34:10.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- 172.21.15.103:0/3387199905 wait complete. 2024-03-22T02:34:10.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 Processor -- start 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- start start 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac11ab10 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.643+0000 7f18b2a3d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1140f0 con 0x7f18ac0fde20 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59272/0 (socket says 172.21.15.103:59272) 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 -- 172.21.15.103:0/2338979972 learned_addr learned my addr 172.21.15.103:0/2338979972 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18abfff700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 -- 172.21.15.103:0/2338979972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 msgr2=0x7f18ac119f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac119f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 -- 172.21.15.103:0/2338979972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0005040 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18abfff700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac119f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18ab7fe700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009380 tx=0x7f18a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a000e040 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a0009af0 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.644+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac114370 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.645+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a001c3f0 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.645+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac1148b0 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.645+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac066b50 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.646+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f18a0013420 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.646+0000 7f18a8ff9700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 0x7f1894070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.646+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f18a008f2e0 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.646+0000 7f18abfff700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 0x7f1894070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.647+0000 7f18abfff700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 0x7f1894070980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f189c010520 tx=0x7f189c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:10.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.653+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18a005be90 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.771 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 podman[126593]: 2024-03-22 02:34:10.543250021 +0000 UTC m=+0.111975259 container create fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T02:34:10.772 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 podman[126593]: 2024-03-22 02:34:10.450024364 +0000 UTC m=+0.018749607 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:10.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.908+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f18ac109eb0 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.908+0000 7f18a8ff9700 1 -- 172.21.15.103:0/2338979972 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7f18a005ba20 con 0x7f18ac0fe7c0 2024-03-22T02:34:10.909 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:34:10.909 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:34:10.909 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:34:10.909 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:34:10.910 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:34:10.911 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:10.912 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:10.913 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:34:10.913 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:10.913 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:10.913 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 msgr2=0x7f1894070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 0x7f1894070980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f189c010520 tx=0x7f189c00b040 comp rx=0 tx=0).stop 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 msgr2=0x7f18ac11a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009380 tx=0x7f18a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 shutdown_connections 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f189406e4d0 0x7f1894070980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18ac0fde20 0x7f18ac119f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 --2- 172.21.15.103:0/2338979972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18ac0fe7c0 0x7f18ac11a4c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 >> 172.21.15.103:0/2338979972 conn(0x7f18ac0f7870 msgr2=0x7f18ac0fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:10.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 shutdown_connections 2024-03-22T02:34:10.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:10.910+0000 7f18b2a3d700 1 -- 172.21.15.103:0/2338979972 wait complete. 2024-03-22T02:34:10.915 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:34:11.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:10 smithi103 ceph-mon[115211]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:11.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:10 smithi103 ceph-mon[115211]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:11.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:10 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/22304841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:11.025 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:10 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2338979972' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:34:11.025 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 podman[126593]: 2024-03-22 02:34:10.771376762 +0000 UTC m=+0.340102009 container init fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-22T02:34:11.025 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 podman[126593]: 2024-03-22 02:34:10.774626836 +0000 UTC m=+0.343352085 container start fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-22T02:34:11.025 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 ceph-osd[126634]: -- 172.21.15.103:0/2476262995 <== mon.0 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b2a9cf680 con 0x556b2a999000 2024-03-22T02:34:11.025 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 bash[126593]: fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 2024-03-22T02:34:11.025 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:10 smithi103 systemd[1]: Started Ceph osd.0 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:34:11.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 -- 172.21.15.103:0/2256340607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 msgr2=0x7f5300103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 --2- 172.21.15.103:0/2256340607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300103200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009ce0 tx=0x7f52f8004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:11.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 -- 172.21.15.103:0/2256340607 shutdown_connections 2024-03-22T02:34:11.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 --2- 172.21.15.103:0/2256340607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 0x7f53000fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 --2- 172.21.15.103:0/2256340607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300103200 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 -- 172.21.15.103:0/2256340607 >> 172.21.15.103:0/2256340607 conn(0x7f53000f8a20 msgr2=0x7f53000fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 -- 172.21.15.103:0/2256340607 shutdown_connections 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.023+0000 7f530797b700 1 -- 172.21.15.103:0/2256340607 wait complete. 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 Processor -- start 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 -- start start 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 0x7f5300113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53001144e0 con 0x7f5300102e30 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f530797b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530010dae0 con 0x7f53001037d0 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f5305717700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f5305717700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59304/0 (socket says 172.21.15.103:59304) 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f5305717700 1 -- 172.21.15.103:0/2291699811 learned_addr learned my addr 172.21.15.103:0/2291699811 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.024+0000 7f5304f16700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 0x7f5300113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f5305717700 1 -- 172.21.15.103:0/2291699811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 msgr2=0x7f5300113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f5305717700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 0x7f5300113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f5305717700 1 -- 172.21.15.103:0/2291699811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f8009940 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f5305717700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52f800ba50 tx=0x7f52f8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f800e040 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530010dd60 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52f801cb20 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52f8012990 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.025+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530010e320 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.026+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5300100140 con 0x7f5300102e30 2024-03-22T02:34:11.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.027+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7f52f801cc90 con 0x7f5300102e30 2024-03-22T02:34:11.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.027+0000 7f52f5ffb700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 0x7f52ec070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.027+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f52f8090270 con 0x7f5300102e30 2024-03-22T02:34:11.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.027+0000 7f5304f16700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 0x7f52ec070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.028+0000 7f5304f16700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 0x7f52ec070980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f530010eca0 tx=0x7f52f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.033+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52f805cea0 con 0x7f5300102e30 2024-03-22T02:34:11.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.259+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f530010f010 con 0x7f52ec06e4d0 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.260+0000 7f52f5ffb700 1 -- 172.21.15.103:0/2291699811 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f530010f010 con 0x7f52ec06e4d0 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:34:11.261 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "6/21 daemons upgraded", 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:34:11.262 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:34:11.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 msgr2=0x7f52ec070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 0x7f52ec070980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f530010eca0 tx=0x7f52f000b040 comp rx=0 tx=0).stop 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 msgr2=0x7f5300113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52f800ba50 tx=0x7f52f8004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 shutdown_connections 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f52ec06e4d0 0x7f52ec070980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5300102e30 0x7f5300113950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 --2- 172.21.15.103:0/2291699811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53001037d0 0x7f5300113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 >> 172.21.15.103:0/2291699811 conn(0x7f53000f8a20 msgr2=0x7f53001073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 shutdown_connections 2024-03-22T02:34:11.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.263+0000 7f530797b700 1 -- 172.21.15.103:0/2291699811 wait complete. 2024-03-22T02:34:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:10 smithi122 ceph-mon[83368]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:11.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:10 smithi122 ceph-mon[83368]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:11.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:10 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/22304841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:11.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:10 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2338979972' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:34:11.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- 172.21.15.103:0/1510512592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 msgr2=0x7fadb81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 --2- 172.21.15.103:0/1510512592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb81031e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fada8004940 tx=0x7fada8009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:11.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- 172.21.15.103:0/1510512592 shutdown_connections 2024-03-22T02:34:11.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 --2- 172.21.15.103:0/1510512592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 0x7fadb80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 --2- 172.21.15.103:0/1510512592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb81031e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- 172.21.15.103:0/1510512592 >> 172.21.15.103:0/1510512592 conn(0x7fadb80f8980 msgr2=0x7fadb80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- 172.21.15.103:0/1510512592 shutdown_connections 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- 172.21.15.103:0/1510512592 wait complete. 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 Processor -- start 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.370+0000 7fadbd33a700 1 -- start start 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 0x7fadb8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8070d90 con 0x7fadb8102e10 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8070ed0 con 0x7fadb81037b0 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59332/0 (socket says 172.21.15.103:59332) 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 -- 172.21.15.103:0/4251562707 learned_addr learned my addr 172.21.15.103:0/4251562707 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadaffff700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 0x7fadb8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 -- 172.21.15.103:0/4251562707 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 msgr2=0x7fadb8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 0x7fadb8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 -- 172.21.15.103:0/4251562707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8005040 con 0x7fadb8102e10 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadb6ffd700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fada800b3b0 tx=0x7fada8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada800e040 con 0x7fadb8102e10 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fada8013420 con 0x7fadb8102e10 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb8071120 con 0x7fadb8102e10 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada801c630 con 0x7fadb8102e10 2024-03-22T02:34:11.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.371+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb811a310 con 0x7fadb8102e10 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.373+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb8100120 con 0x7fadb8102e10 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.373+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 40) v1 ==== 92321+0+0 (secure 0 0 0) 0x7fada8013590 con 0x7fadb8102e10 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.373+0000 7fadaf7fe700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 0x7fada4070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.374+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fada808f1a0 con 0x7fadb8102e10 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.374+0000 7fadaffff700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 0x7fada4070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.374+0000 7fadaffff700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 0x7fada4070970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadb8071c00 tx=0x7fada0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.378+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fada80582a0 con 0x7fadb8102e10 2024-03-22T02:34:11.624 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T02:34:11.315+0000 7f40d2e627c0 -1 Falling back to public interface 2024-03-22T02:34:11.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.685+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fadb80fe560 con 0x7fadb8102e10 2024-03-22T02:34:11.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.685+0000 7fadaf7fe700 1 -- 172.21.15.103:0/4251562707 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7fada805b960 con 0x7fadb8102e10 2024-03-22T02:34:11.686 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded 2024-03-22T02:34:11.686 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: osd.0 (root=default,host=smithi103) is down 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-03-22T02:34:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-03-22T02:34:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 msgr2=0x7fada4070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 0x7fada4070970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadb8071c00 tx=0x7fada0009040 comp rx=0 tx=0).stop 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 msgr2=0x7fadb80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fada800b3b0 tx=0x7fada8009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 shutdown_connections 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fada406e4c0 0x7fada4070970 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadb8102e10 0x7fadb80702c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 --2- 172.21.15.103:0/4251562707 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb81037b0 0x7fadb8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.687+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 >> 172.21.15.103:0/4251562707 conn(0x7fadb80f8980 msgr2=0x7fadb8107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.688+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 shutdown_connections 2024-03-22T02:34:11.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:11.688+0000 7fadbd33a700 1 -- 172.21.15.103:0/4251562707 wait complete. 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:11 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4251562707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:11 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4251562707' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:34:13.154 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:12 smithi103 ceph-mon[115211]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:12 smithi122 ceph-mon[83368]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:13 smithi122 ceph-mon[83368]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:13 smithi103 ceph-mon[115211]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:14 smithi122 ceph-mon[83368]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:14 smithi103 ceph-mon[115211]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:15.874 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:15 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T02:34:15.562+0000 7f40d2e627c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-22T02:34:16.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:16 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T02:34:16.100+0000 7f40d2e627c0 -1 osd.0 49 log_to_monitors true 2024-03-22T02:34:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:34:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:17.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T02:34:17.079+0000 7f40c42ba700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/321 objects degraded (19.315%) 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:18 smithi122 ceph-mon[83368]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:34:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:18 smithi122 ceph-mon[83368]: osdmap e52: 6 total, 5 up, 6 in 2024-03-22T02:34:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:18 smithi122 ceph-mon[83368]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:34:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:18 smithi103 ceph-mon[115211]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:34:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:18 smithi103 ceph-mon[115211]: osdmap e52: 6 total, 5 up, 6 in 2024-03-22T02:34:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:18 smithi103 ceph-mon[115211]: from='osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:34:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876] boot 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: osdmap e53: 6 total, 6 up, 6 in 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: mgrmap e41: smithi103.pfvmui(active, since 91s), standbys: smithi122.prgtti 2024-03-22T02:34:19.315 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:19 smithi103 ceph-mon[115211]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 767 B/s wr, 0 op/s; 62/321 objects degraded (19.315%) 2024-03-22T02:34:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:34:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: osd.0 [v2:172.21.15.103:6802/1332641876,v1:172.21.15.103:6803/1332641876] boot 2024-03-22T02:34:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: osdmap e53: 6 total, 6 up, 6 in 2024-03-22T02:34:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:34:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: mgrmap e41: smithi103.pfvmui(active, since 91s), standbys: smithi122.prgtti 2024-03-22T02:34:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:19 smithi122 ceph-mon[83368]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 767 B/s wr, 0 op/s; 62/321 objects degraded (19.315%) 2024-03-22T02:34:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:20 smithi122 ceph-mon[83368]: osdmap e54: 6 total, 6 up, 6 in 2024-03-22T02:34:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:20 smithi103 ceph-mon[115211]: osdmap e54: 6 total, 6 up, 6 in 2024-03-22T02:34:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 62/321 objects degraded (19.315%) 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:21 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 62/321 objects degraded (19.315%) 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:22.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:21 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:22 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:22 smithi122 ceph-mon[83368]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-22T02:34:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:22 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:22 smithi103 ceph-mon[115211]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-22T02:34:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:23 smithi122 ceph-mon[83368]: pgmap v56: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 46 B/s, 0 objects/s recovering 2024-03-22T02:34:24.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:23 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded) 2024-03-22T02:34:24.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:23 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:34:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:23 smithi103 ceph-mon[115211]: pgmap v56: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s; 46 B/s, 0 objects/s recovering 2024-03-22T02:34:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:23 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/321 objects degraded (19.315%), 19 pgs degraded) 2024-03-22T02:34:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:23 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:34:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:25 smithi122 ceph-mon[83368]: pgmap v57: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 792 B/s wr, 0 op/s; 36 B/s, 0 objects/s recovering 2024-03-22T02:34:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:25 smithi103 ceph-mon[115211]: pgmap v57: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 792 B/s wr, 0 op/s; 36 B/s, 0 objects/s recovering 2024-03-22T02:34:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:27 smithi122 ceph-mon[83368]: pgmap v58: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 32 B/s, 0 objects/s recovering 2024-03-22T02:34:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:27 smithi103 ceph-mon[115211]: pgmap v58: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 32 B/s, 0 objects/s recovering 2024-03-22T02:34:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:29 smithi122 ceph-mon[83368]: pgmap v59: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 28 B/s, 0 objects/s recovering 2024-03-22T02:34:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:29 smithi103 ceph-mon[115211]: pgmap v59: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 28 B/s, 0 objects/s recovering 2024-03-22T02:34:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:31 smithi103 ceph-mon[115211]: pgmap v60: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-22T02:34:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:31 smithi122 ceph-mon[83368]: pgmap v60: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-22T02:34:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:33.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:33 smithi122 ceph-mon[83368]: pgmap v61: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-22T02:34:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:33 smithi103 ceph-mon[115211]: pgmap v61: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 23 B/s, 0 objects/s recovering 2024-03-22T02:34:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:35 smithi122 ceph-mon[83368]: pgmap v62: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:35 smithi103 ceph-mon[115211]: pgmap v62: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:36 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:36 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:37 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:38.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:37 smithi122 ceph-mon[83368]: Upgrade: osd.1 is safe to restart 2024-03-22T02:34:38.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:37 smithi122 ceph-mon[83368]: pgmap v63: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:37 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T02:34:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:37 smithi103 ceph-mon[115211]: Upgrade: osd.1 is safe to restart 2024-03-22T02:34:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:37 smithi103 ceph-mon[115211]: pgmap v63: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:39.216 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:38 smithi103 ceph-mon[115211]: Upgrade: Updating osd.1 2024-03-22T02:34:39.216 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:39.216 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:34:39.217 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:38 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:38 smithi122 ceph-mon[83368]: Upgrade: Updating osd.1 2024-03-22T02:34:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:34:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:38 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:40.238 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:39 smithi103 ceph-mon[115211]: Deploying daemon osd.1 on smithi103 2024-03-22T02:34:40.238 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:39 smithi103 ceph-mon[115211]: pgmap v64: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:39 smithi122 ceph-mon[83368]: Deploying daemon osd.1 on smithi103 2024-03-22T02:34:40.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:39 smithi122 ceph-mon[83368]: pgmap v64: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:41.110 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:40 smithi103 ceph-mon[115211]: pgmap v65: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:40 smithi122 ceph-mon[83368]: pgmap v65: 65 pgs: 65 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:41.782 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:41 smithi103 systemd[1]: Stopping Ceph osd.1 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 -- 172.21.15.103:0/3751169658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 msgr2=0x7f05340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 --2- 172.21.15.103:0/3751169658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05340fbd80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0520004940 tx=0x7f0520009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 -- 172.21.15.103:0/3751169658 shutdown_connections 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 --2- 172.21.15.103:0/3751169658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05340fbd80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 --2- 172.21.15.103:0/3751169658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 -- 172.21.15.103:0/3751169658 >> 172.21.15.103:0/3751169658 conn(0x7f0534072bc0 msgr2=0x7f0534072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 -- 172.21.15.103:0/3751169658 shutdown_connections 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 -- 172.21.15.103:0/3751169658 wait complete. 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.793+0000 7f053ad55700 1 Processor -- start 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f053ad55700 1 -- start start 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f053ad55700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f053ad55700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:41.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f053ad55700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534114710 con 0x7f0534102710 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f053ad55700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053410db00 con 0x7f0534101d70 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44902/0 (socket says 172.21.15.103:44902) 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 -- 172.21.15.103:0/754840046 learned_addr learned my addr 172.21.15.103:0/754840046 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0538af1700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 -- 172.21.15.103:0/754840046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 msgr2=0x7f0534113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 -- 172.21.15.103:0/754840046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520005040 con 0x7f0534102710 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0538af1700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.794+0000 7f0533fff700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0520009100 tx=0x7f0520009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.795+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f052000e040 con 0x7f0534102710 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.795+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0520013420 con 0x7f0534102710 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.795+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053410dd80 con 0x7f0534102710 2024-03-22T02:34:41.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.795+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f052001c610 con 0x7f0534102710 2024-03-22T02:34:41.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.795+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053410e340 con 0x7f0534102710 2024-03-22T02:34:41.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.796+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05340fd4c0 con 0x7f0534102710 2024-03-22T02:34:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.796+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0520013590 con 0x7f0534102710 2024-03-22T02:34:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.797+0000 7f05317fa700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 0x7f0524070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.797+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f052008eea0 con 0x7f0534102710 2024-03-22T02:34:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.797+0000 7f0538af1700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 0x7f0524070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:41.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.797+0000 7f0538af1700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 0x7f0524070850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0528004d80 tx=0x7f052800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:41.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:41.801+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f052005bd90 con 0x7f0534102710 2024-03-22T02:34:42.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.027+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f05340666a0 con 0x7f052406e3a0 2024-03-22T02:34:42.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.028+0000 7f05317fa700 1 -- 172.21.15.103:0/754840046 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f05340666a0 con 0x7f052406e3a0 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 msgr2=0x7f0524070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 0x7f0524070850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0528004d80 tx=0x7f052800b040 comp rx=0 tx=0).stop 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 msgr2=0x7f05341140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0520009100 tx=0x7f0520009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 shutdown_connections 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f052406e3a0 0x7f0524070850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0534101d70 0x7f0534113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 --2- 172.21.15.103:0/754840046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0534102710 0x7f05341140c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.030+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 >> 172.21.15.103:0/754840046 conn(0x7f0534072bc0 msgr2=0x7f0534100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.031+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 shutdown_connections 2024-03-22T02:34:42.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.031+0000 7f053ad55700 1 -- 172.21.15.103:0/754840046 wait complete. 2024-03-22T02:34:42.042 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:34:42.042 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:41 smithi103 ceph-mon[115211]: osd.1 marked itself down and dead 2024-03-22T02:34:42.043 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:41 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[55397]: 2024-03-22T02:34:41.781+0000 7ffa1b6c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:34:42.043 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:41 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[55397]: 2024-03-22T02:34:41.781+0000 7ffa1b6c6700 -1 osd.1 54 *** Got signal Terminated *** 2024-03-22T02:34:42.043 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:41 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[55397]: 2024-03-22T02:34:41.781+0000 7ffa1b6c6700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:34:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- 172.21.15.103:0/3941067812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 msgr2=0x7f76fc1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 --2- 172.21.15.103:0/3941067812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc1020f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f76f0004940 tx=0x7f76f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- 172.21.15.103:0/3941067812 shutdown_connections 2024-03-22T02:34:42.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 --2- 172.21.15.103:0/3941067812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 --2- 172.21.15.103:0/3941067812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc1020f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- 172.21.15.103:0/3941067812 >> 172.21.15.103:0/3941067812 conn(0x7f76fc0f7870 msgr2=0x7f76fc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- 172.21.15.103:0/3941067812 shutdown_connections 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- 172.21.15.103:0/3941067812 wait complete. 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 Processor -- start 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.135+0000 7f7703101700 1 -- start start 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc10f5e0 con 0x7f76fc101d20 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc10f720 con 0x7f76fc1026c0 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44916/0 (socket says 172.21.15.103:44916) 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 -- 172.21.15.103:0/2888586064 learned_addr learned my addr 172.21.15.103:0/2888586064 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:42.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f76fbfff700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 -- 172.21.15.103:0/2888586064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 msgr2=0x7f76fc10f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc10f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 -- 172.21.15.103:0/2888586064 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f0005040 con 0x7f76fc101d20 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f76fbfff700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7700e9d700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f76f000b3b0 tx=0x7f76f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f000e040 con 0x7f76fc101d20 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76f000bbc0 con 0x7f76fc101d20 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc10f9a0 con 0x7f76fc101d20 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f001c3f0 con 0x7f76fc101d20 2024-03-22T02:34:42.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.136+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc117f70 con 0x7f76fc101d20 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.137+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc0fd470 con 0x7f76fc101d20 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.138+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f76f000bd30 con 0x7f76fc101d20 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.138+0000 7f76f97fa700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 0x7f76ec070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.138+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f76f008ef40 con 0x7f76fc101d20 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.138+0000 7f76fbfff700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 0x7f76ec070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.139+0000 7f76fbfff700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 0x7f76ec070850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.143+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76f00c3740 con 0x7f76fc101d20 2024-03-22T02:34:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:41 smithi122 ceph-mon[83368]: osd.1 marked itself down and dead 2024-03-22T02:34:42.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.363+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f76fc0668a0 con 0x7f76ec06e3a0 2024-03-22T02:34:42.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.364+0000 7f76f97fa700 1 -- 172.21.15.103:0/2888586064 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f76fc0668a0 con 0x7f76ec06e3a0 2024-03-22T02:34:42.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.366+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 msgr2=0x7f76ec070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.366+0000 7f7703101700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 0x7f76ec070850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8008040 comp rx=0 tx=0).stop 2024-03-22T02:34:42.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.366+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 msgr2=0x7f76fc10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.366+0000 7f7703101700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f76f000b3b0 tx=0x7f76f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 shutdown_connections 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f76ec06e3a0 0x7f76ec070850 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76fc101d20 0x7f76fc10eb60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 --2- 172.21.15.103:0/2888586064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76fc1026c0 0x7f76fc10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 >> 172.21.15.103:0/2888586064 conn(0x7f76fc0f7870 msgr2=0x7f76fc0f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 shutdown_connections 2024-03-22T02:34:42.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.367+0000 7f7703101700 1 -- 172.21.15.103:0/2888586064 wait complete. 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- 172.21.15.103:0/397829705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28102710 msgr2=0x7f6d280fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/397829705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28102710 0x7f6d280fbd80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c004940 tx=0x7f6d1c009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- 172.21.15.103:0/397829705 shutdown_connections 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/397829705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28102710 0x7f6d280fbd80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/397829705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28101d70 0x7f6d28102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- 172.21.15.103:0/397829705 >> 172.21.15.103:0/397829705 conn(0x7f6d28072bc0 msgr2=0x7f6d28072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- 172.21.15.103:0/397829705 shutdown_connections 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- 172.21.15.103:0/397829705 wait complete. 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 Processor -- start 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.474+0000 7f6d2cf01700 1 -- start start 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28102710 0x7f6d28112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d28113630 con 0x7f6d28101d70 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2810ca20 con 0x7f6d28102710 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44934/0 (socket says 172.21.15.103:44934) 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 -- 172.21.15.103:0/3537449768 learned_addr learned my addr 172.21.15.103:0/3537449768 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d25d9b700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28102710 0x7f6d28112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 -- 172.21.15.103:0/3537449768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28102710 msgr2=0x7f6d28112fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28102710 0x7f6d28112fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 -- 172.21.15.103:0/3537449768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c005040 con 0x7f6d28101d70 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2659c700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c8c0 tx=0x7f6d180044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d1800a980 con 0x7f6d28101d70 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d1800ce60 con 0x7f6d28101d70 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2810cd00 con 0x7f6d28101d70 2024-03-22T02:34:42.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d180115e0 con 0x7f6d28101d70 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.475+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2810d2c0 con 0x7f6d28101d70 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.476+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d28104240 con 0x7f6d28101d70 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.477+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6d18012400 con 0x7f6d28101d70 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.477+0000 7f6d16ffd700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 0x7f6d10070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.477+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6d1808de30 con 0x7f6d28101d70 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.477+0000 7f6d25d9b700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 0x7f6d10070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.478+0000 7f6d25d9b700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 0x7f6d10070850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d2810dbe0 tx=0x7f6d1c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.482+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d1805ac80 con 0x7f6d28101d70 2024-03-22T02:34:42.624 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:42 smithi103 podman[131828]: 2024-03-22 02:34:42.453477509 +0000 UTC m=+0.799974773 container died 48c9166868c5252a34a307b7f35c39747ee9c1fb65d47a77e6cefc267f61ef11 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, RELEASE=quincy-9c4e0c3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:34:42.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.696+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d2810e0f0 con 0x7f6d1006e3a0 2024-03-22T02:34:42.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.706+0000 7f6d16ffd700 1 -- 172.21.15.103:0/3537449768 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f6d2810e0f0 con 0x7f6d1006e3a0 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (3m) 26s ago 14m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (58s) 26s ago 14m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (52s) 49s ago 13m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (2m) 26s ago 14m 81.9M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (8m) 26s ago 8m 13.8M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (8m) 26s ago 8m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (8m) 49s ago 8m 15.3M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (8m) 49s ago 8m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:34:42.707 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (4m) 26s ago 15m 497M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (4m) 49s ago 13m 449M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (2m) 26s ago 16m 42.8M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (100s) 49s ago 13m 35.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (3m) 26s ago 14m 15.9M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 49s ago 13m 16.8M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (32s) 26s ago 12m 14.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (12m) 26s ago 12m 230M 1557M 17.2.7-726-g9c4e0c34 837124da804f 48c9166868c5 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (11m) 26s ago 11m 199M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (11m) 49s ago 11m 265M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (10m) 49s ago 10m 231M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (10m) 49s ago 10m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:34:42.708 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (3m) 26s ago 13m 60.2M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 msgr2=0x7f6d10070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 0x7f6d10070850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d2810dbe0 tx=0x7f6d1c009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 msgr2=0x7f6d28112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c8c0 tx=0x7f6d180044c0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 shutdown_connections 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d1006e3a0 0x7f6d10070850 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d28101d70 0x7f6d28112aa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 --2- 172.21.15.103:0/3537449768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d28102710 0x7f6d28112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 >> 172.21.15.103:0/3537449768 conn(0x7f6d28072bc0 msgr2=0x7f6d280fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 shutdown_connections 2024-03-22T02:34:42.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.708+0000 7f6d2cf01700 1 -- 172.21.15.103:0/3537449768 wait complete. 2024-03-22T02:34:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- 172.21.15.103:0/3372043716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28102710 msgr2=0x7f0d280fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/3372043716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28102710 0x7f0d280fbd80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c004940 tx=0x7f0d1c009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- 172.21.15.103:0/3372043716 shutdown_connections 2024-03-22T02:34:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/3372043716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28102710 0x7f0d280fbd80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/3372043716 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28101d70 0x7f0d28102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- 172.21.15.103:0/3372043716 >> 172.21.15.103:0/3372043716 conn(0x7f0d28072bc0 msgr2=0x7f0d28072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- 172.21.15.103:0/3372043716 shutdown_connections 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- 172.21.15.103:0/3372043716 wait complete. 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 Processor -- start 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.809+0000 7f0d2e54c700 1 -- start start 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28102710 0x7f0d28113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d28114630 con 0x7f0d28101d70 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2810da60 con 0x7f0d28102710 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44958/0 (socket says 172.21.15.103:44958) 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 -- 172.21.15.103:0/2613704839 learned_addr learned my addr 172.21.15.103:0/2613704839 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d277fe700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28102710 0x7f0d28113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 -- 172.21.15.103:0/2613704839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28102710 msgr2=0x7f0d28113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28102710 0x7f0d28113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 -- 172.21.15.103:0/2613704839 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c005040 con 0x7f0d28101d70 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d27fff700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800eb40 tx=0x7f0d1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d18007cc0 con 0x7f0d28101d70 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2810dce0 con 0x7f0d28101d70 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d1800f040 con 0x7f0d28101d70 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d18009620 con 0x7f0d28101d70 2024-03-22T02:34:42.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.810+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2810e220 con 0x7f0d28101d70 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.811+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d280fd4c0 con 0x7f0d28101d70 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.812+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0d18009780 con 0x7f0d28101d70 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.812+0000 7f0d24ff9700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 0x7f0d10070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.812+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0d1808d9d0 con 0x7f0d28101d70 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.812+0000 7f0d277fe700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 0x7f0d10070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.813+0000 7f0d277fe700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 0x7f0d10070860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d280fa560 tx=0x7f0d1c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:42.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:42.817+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d1805a850 con 0x7f0d28101d70 2024-03-22T02:34:42.982 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:42 smithi103 podman[131828]: 2024-03-22 02:34:42.856865123 +0000 UTC m=+1.203362381 container cleanup 48c9166868c5252a34a307b7f35c39747ee9c1fb65d47a77e6cefc267f61ef11 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-22T02:34:42.982 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:42 smithi103 bash[131828]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1 2024-03-22T02:34:43.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.134+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0d2804bed0 con 0x7f0d28101d70 2024-03-22T02:34:43.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.135+0000 7f0d24ff9700 1 -- 172.21.15.103:0/2613704839 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f0d1805a3e0 con 0x7f0d28101d70 2024-03-22T02:34:43.135 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 8, 2024-03-22T02:34:43.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 5 2024-03-22T02:34:43.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:34:43.137 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:34:43.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 msgr2=0x7f0d10070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 0x7f0d10070860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d280fa560 tx=0x7f0d1c009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 msgr2=0x7f0d28113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800eb40 tx=0x7f0d1800b040 comp rx=0 tx=0).stop 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 shutdown_connections 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0d1006e3b0 0x7f0d10070860 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d28101d70 0x7f0d28113aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 --2- 172.21.15.103:0/2613704839 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d28102710 0x7f0d28113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 >> 172.21.15.103:0/2613704839 conn(0x7f0d28072bc0 msgr2=0x7f0d28065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 shutdown_connections 2024-03-22T02:34:43.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.137+0000 7f0d2e54c700 1 -- 172.21.15.103:0/2613704839 wait complete. 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 -- 172.21.15.103:0/1996301470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf4102e10 msgr2=0x7fcdf41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 --2- 172.21.15.103:0/1996301470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf4102e10 0x7fcdf41031e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009ce0 tx=0x7fcdec004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 -- 172.21.15.103:0/1996301470 shutdown_connections 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 --2- 172.21.15.103:0/1996301470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf41037b0 0x7fcdf40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 --2- 172.21.15.103:0/1996301470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf4102e10 0x7fcdf41031e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 -- 172.21.15.103:0/1996301470 >> 172.21.15.103:0/1996301470 conn(0x7fcdf40f8980 msgr2=0x7fcdf40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.243 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[131896]: 2024-03-22 02:34:43.183749346 +0000 UTC m=+0.728372453 container remove 48c9166868c5252a34a307b7f35c39747ee9c1fb65d47a77e6cefc267f61ef11 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, org.label-schema.schema-version=1.0) 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: osdmap e55: 6 total, 5 up, 6 in 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.243 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:42 smithi103 ceph-mon[115211]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:43.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 -- 172.21.15.103:0/1996301470 shutdown_connections 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.241+0000 7fcdf9846700 1 -- 172.21.15.103:0/1996301470 wait complete. 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 Processor -- start 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 -- start start 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf4102e10 0x7fcdf4113d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41148d0 con 0x7fcdf41037b0 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf9846700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410dca0 con 0x7fcdf4102e10 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44976/0 (socket says 172.21.15.103:44976) 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.242+0000 7fcdf27fc700 1 -- 172.21.15.103:0/2514601342 learned_addr learned my addr 172.21.15.103:0/2514601342 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf2ffd700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf4102e10 0x7fcdf4113d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf27fc700 1 -- 172.21.15.103:0/2514601342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf4102e10 msgr2=0x7fcdf4113d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf27fc700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf4102e10 0x7fcdf4113d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf27fc700 1 -- 172.21.15.103:0/2514601342 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009940 con 0x7fcdf41037b0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf27fc700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00c890 tx=0x7fcddc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc00a700 con 0x7fcdf41037b0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcddc00ad30 con 0x7fcdf41037b0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf410df80 con 0x7fcdf41037b0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcddc0113a0 con 0x7fcdf41037b0 2024-03-22T02:34:43.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.243+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf410e540 con 0x7fcdf41037b0 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.245+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf4100120 con 0x7fcdf41037b0 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.245+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcddc011610 con 0x7fcdf41037b0 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.246+0000 7fcdeb7fe700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 0x7fcde00708a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.246+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fcddc08de50 con 0x7fcdf41037b0 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.246+0000 7fcdf2ffd700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 0x7fcde00708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.246+0000 7fcdf2ffd700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 0x7fcde00708a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00b740 tx=0x7fcdec004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.251+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcddc05ad50 con 0x7fcdf41037b0 2024-03-22T02:34:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:34:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: osdmap e55: 6 total, 5 up, 6 in 2024-03-22T02:34:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:42 smithi122 ceph-mon[83368]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:43.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.510+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcdf410f390 con 0x7fcdf41037b0 2024-03-22T02:34:43.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.510+0000 7fcdeb7fe700 1 -- 172.21.15.103:0/2514601342 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7fcddc05a8e0 con 0x7fcdf41037b0 2024-03-22T02:34:43.511 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:34:43.512 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:34:43.513 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:34:43.514 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:34:43.515 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:43.515 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:34:43.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 msgr2=0x7fcde00708a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 0x7fcde00708a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00b740 tx=0x7fcdec004d10 comp rx=0 tx=0).stop 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 msgr2=0x7fcdf4114280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00c890 tx=0x7fcddc0044c0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 shutdown_connections 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcde006e3f0 0x7fcde00708a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdf4102e10 0x7fcdf4113d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 --2- 172.21.15.103:0/2514601342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41037b0 0x7fcdf4114280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 >> 172.21.15.103:0/2514601342 conn(0x7fcdf40f8980 msgr2=0x7fcdf4107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 shutdown_connections 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.513+0000 7fcdf9846700 1 -- 172.21.15.103:0/2514601342 wait complete. 2024-03-22T02:34:43.516 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:34:43.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 -- 172.21.15.103:0/3787338689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fde90 msgr2=0x7f9f040fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 --2- 172.21.15.103:0/3787338689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fde90 0x7f9f040fe260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4004940 tx=0x7f9ef4009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:43.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 -- 172.21.15.103:0/3787338689 shutdown_connections 2024-03-22T02:34:43.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 --2- 172.21.15.103:0/3787338689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fe830 0x7f9f04106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 --2- 172.21.15.103:0/3787338689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fde90 0x7f9f040fe260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 -- 172.21.15.103:0/3787338689 >> 172.21.15.103:0/3787338689 conn(0x7f9f04072bc0 msgr2=0x7f9f04072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 -- 172.21.15.103:0/3787338689 shutdown_connections 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.617+0000 7f9f0a064700 1 -- 172.21.15.103:0/3787338689 wait complete. 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 Processor -- start 2024-03-22T02:34:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 -- start start 2024-03-22T02:34:43.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 0x7f9f04115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04116970 con 0x7f9f040fe830 2024-03-22T02:34:43.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f0a064700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0410fd40 con 0x7f9f040fde90 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45004/0 (socket says 172.21.15.103:45004) 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 -- 172.21.15.103:0/588141204 learned_addr learned my addr 172.21.15.103:0/588141204 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f037fe700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 0x7f9f04115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 -- 172.21.15.103:0/588141204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 msgr2=0x7f9f04115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 0x7f9f04115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.618+0000 7f9f02ffd700 1 -- 172.21.15.103:0/588141204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4005040 con 0x7f9f040fe830 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9f037fe700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 0x7f9f04115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9f02ffd700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8002a50 tx=0x7f9ef80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ef8007cc0 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f04110020 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ef800f040 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f04110560 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.619+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ef800db70 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.620+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f04106a20 con 0x7f9f040fe830 2024-03-22T02:34:43.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.621+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9ef801c400 con 0x7f9f040fe830 2024-03-22T02:34:43.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.622+0000 7f9eebfff700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 0x7f9eec070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.622+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f9ef808d4e0 con 0x7f9f040fe830 2024-03-22T02:34:43.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.622+0000 7f9f037fe700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 0x7f9eec070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.622+0000 7f9f037fe700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 0x7f9eec070850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400b3b0 tx=0x7f9ef4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.627+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ef805a330 con 0x7f9f040fe830 2024-03-22T02:34:43.780 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[132048]: 2024-03-22 02:34:43.518311443 +0000 UTC m=+0.115226299 container create d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-22T02:34:43.780 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[132048]: 2024-03-22 02:34:43.422176923 +0000 UTC m=+0.019091804 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:43.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.860+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f0405f690 con 0x7f9eec06e3a0 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.862+0000 7f9eebfff700 1 -- 172.21.15.103:0/588141204 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9f0405f690 con 0x7f9eec06e3a0 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "7/21 daemons upgraded", 2024-03-22T02:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:34:43.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.865+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 msgr2=0x7f9eec070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.865+0000 7f9f0a064700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 0x7f9eec070850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400b3b0 tx=0x7f9ef4009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.865+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 msgr2=0x7f9f04116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.865+0000 7f9f0a064700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8002a50 tx=0x7f9ef80046b0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 shutdown_connections 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9eec06e3a0 0x7f9eec070850 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f040fde90 0x7f9f04115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 --2- 172.21.15.103:0/588141204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f040fe830 0x7f9f04116320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 >> 172.21.15.103:0/588141204 conn(0x7f9f04072bc0 msgr2=0x7f9f040fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 shutdown_connections 2024-03-22T02:34:43.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.866+0000 7f9f0a064700 1 -- 172.21.15.103:0/588141204 wait complete. 2024-03-22T02:34:43.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 -- 172.21.15.103:0/1901574707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fe830 msgr2=0x7ffbb0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1901574707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fe830 0x7ffbb0106f00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffba0004940 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 -- 172.21.15.103:0/1901574707 shutdown_connections 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1901574707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fe830 0x7ffbb0106f00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1901574707 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fde90 0x7ffbb00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.976+0000 7ffbb751e700 1 -- 172.21.15.103:0/1901574707 >> 172.21.15.103:0/1901574707 conn(0x7ffbb0072bc0 msgr2=0x7ffbb0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.977+0000 7ffbb751e700 1 -- 172.21.15.103:0/1901574707 shutdown_connections 2024-03-22T02:34:43.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.977+0000 7ffbb751e700 1 -- 172.21.15.103:0/1901574707 wait complete. 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.977+0000 7ffbb751e700 1 Processor -- start 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.977+0000 7ffbb751e700 1 -- start start 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb751e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb751e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fe830 0x7ffbb01140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb751e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0114730 con 0x7ffbb00fde90 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb751e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb010db20 con 0x7ffbb00fe830 2024-03-22T02:34:43.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42902/0 (socket says 172.21.15.103:42902) 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 -- 172.21.15.103:0/1394683826 learned_addr learned my addr 172.21.15.103:0/1394683826 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb4ab9700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fe830 0x7ffbb01140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 -- 172.21.15.103:0/1394683826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fe830 msgr2=0x7ffbb01140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fe830 0x7ffbb01140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 -- 172.21.15.103:0/1394683826 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0005040 con 0x7ffbb00fde90 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.978+0000 7ffbb52ba700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbac002a50 tx=0x7ffbac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.979+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac007cc0 con 0x7ffbb00fde90 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.979+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb010de00 con 0x7ffbb00fde90 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.979+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbac00f040 con 0x7ffbb00fde90 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.979+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac00da30 con 0x7ffbb00fde90 2024-03-22T02:34:43.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.979+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb010e340 con 0x7ffbb00fde90 2024-03-22T02:34:43.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.980+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0065e30 con 0x7ffbb00fde90 2024-03-22T02:34:43.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.981+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ffbac00db90 con 0x7ffbb00fde90 2024-03-22T02:34:43.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.981+0000 7ffba5ffb700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 0x7ffb9c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:43.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.981+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ffbac08dae0 con 0x7ffbb00fde90 2024-03-22T02:34:43.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.981+0000 7ffbb4ab9700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 0x7ffb9c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:43.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.982+0000 7ffbb4ab9700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 0x7ffb9c070850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffbb010ece0 tx=0x7ffba0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:43.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:43.987+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffbac05a8b0 con 0x7ffbb00fde90 2024-03-22T02:34:44.089 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:43 smithi103 ceph-mon[115211]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-03-22T02:34:44.089 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:43 smithi103 ceph-mon[115211]: osdmap e56: 6 total, 5 up, 6 in 2024-03-22T02:34:44.090 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:43 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2613704839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:44.090 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:43 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2514601342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:34:44.090 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:43 smithi103 ceph-mon[115211]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:44.090 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[132048]: 2024-03-22 02:34:43.780253244 +0000 UTC m=+0.377168126 container init d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-22T02:34:44.090 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[132048]: 2024-03-22 02:34:43.784990474 +0000 UTC m=+0.381905337 container start d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:34:44.090 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:43 smithi103 podman[132048]: 2024-03-22 02:34:43.913354614 +0000 UTC m=+0.510269477 container attach d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T02:34:44.090 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 podman[132048]: 2024-03-22 02:34:44.089360441 +0000 UTC m=+0.686275288 container died d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T02:34:44.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.285+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffbb0109f70 con 0x7ffbb00fde90 2024-03-22T02:34:44.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.286+0000 7ffba5ffb700 1 -- 172.21.15.103:0/1394683826 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1135 (secure 0 0 0) 0x7ffbac05a440 con 0x7ffbb00fde90 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 12 pgs peering 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: osd.1 (root=default,host=smithi103) is down 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 12 pgs peering 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.4 is stuck peering for 4m, current state peering, last acting [0,4] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.6 is stuck peering for 4m, current state peering, last acting [3,4] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.9 is stuck peering for 4m, current state peering, last acting [4,0] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.a is stuck peering for 4m, current state peering, last acting [4,3] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.d is stuck peering for 4m, current state peering, last acting [3,2] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.15 is stuck peering for 4m, current state peering, last acting [3,0] 2024-03-22T02:34:44.288 INFO:teuthology.orchestra.run.smithi103.stdout: pg 2.1b is stuck peering for 6m, current state peering, last acting [0,5] 2024-03-22T02:34:44.289 INFO:teuthology.orchestra.run.smithi103.stdout: pg 3.0 is stuck peering for 4m, current state peering, last acting [2,4] 2024-03-22T02:34:44.289 INFO:teuthology.orchestra.run.smithi103.stdout: pg 3.2 is stuck peering for 4m, current state peering, last acting [5,3] 2024-03-22T02:34:44.289 INFO:teuthology.orchestra.run.smithi103.stdout: pg 3.4 is stuck peering for 4m, current state peering, last acting [2,3] 2024-03-22T02:34:44.289 INFO:teuthology.orchestra.run.smithi103.stdout: pg 3.b is stuck peering for 4m, current state peering, last acting [0,4] 2024-03-22T02:34:44.289 INFO:teuthology.orchestra.run.smithi103.stdout: pg 3.19 is stuck peering for 4m, current state peering, last acting [4,3] 2024-03-22T02:34:44.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 msgr2=0x7ffb9c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:44.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 0x7ffb9c070850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffbb010ece0 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 msgr2=0x7ffbb0113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbac002a50 tx=0x7ffbac0046b0 comp rx=0 tx=0).stop 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 shutdown_connections 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffb9c06e3a0 0x7ffb9c070850 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbb00fde90 0x7ffbb0113ba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 --2- 172.21.15.103:0/1394683826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffbb00fe830 0x7ffbb01140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 >> 172.21.15.103:0/1394683826 conn(0x7ffbb0072bc0 msgr2=0x7ffbb01037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 shutdown_connections 2024-03-22T02:34:44.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:34:44.289+0000 7ffbb751e700 1 -- 172.21.15.103:0/1394683826 wait complete. 2024-03-22T02:34:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:43 smithi122 ceph-mon[83368]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-03-22T02:34:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:43 smithi122 ceph-mon[83368]: osdmap e56: 6 total, 5 up, 6 in 2024-03-22T02:34:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:43 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2613704839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:43 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2514601342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:34:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:43 smithi122 ceph-mon[83368]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:34:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:44 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1394683826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:34:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:44 smithi103 ceph-mon[115211]: pgmap v69: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:45.124 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 podman[132129]: 2024-03-22 02:34:44.751832921 +0000 UTC m=+0.653625410 container remove d755261f3afb313d7779e9caf536a1bd3f15b71732a2acfe8edcb8eeb835f53e (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-22T02:34:45.124 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-22T02:34:45.124 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 systemd[1]: Stopped Ceph osd.1 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:34:45.124 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.1.service: Consumed 1min 7.681s CPU time. 2024-03-22T02:34:45.125 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:44 smithi103 systemd[1]: Starting Ceph osd.1 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:34:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:44 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1394683826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:34:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:44 smithi122 ceph-mon[83368]: pgmap v69: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:45.507 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.121144656 +0000 UTC m=+0.018276642 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:45.507 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.262060476 +0000 UTC m=+0.159192451 container create 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T02:34:45.507 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.506742291 +0000 UTC m=+0.403874289 container init 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:34:45.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.510075476 +0000 UTC m=+0.407207474 container start 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-22T02:34:45.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.606955364 +0000 UTC m=+0.504087354 container attach 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T02:34:46.353 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 bash[132238]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate[132253]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-22T02:34:46.354 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:45 smithi103 podman[132238]: 2024-03-22 02:34:45.978724638 +0000 UTC m=+0.875856612 container died 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:34:47.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:46 smithi103 podman[132376]: 2024-03-22 02:34:46.736993799 +0000 UTC m=+0.753370574 container remove 31245edf27d833387a7621ce55c728bb3c654c97db43b6be01782e0711c1d5ca (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:34:47.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:46 smithi103 podman[132438]: 2024-03-22 02:34:46.855097444 +0000 UTC m=+0.018758237 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:34:47.102 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:46 smithi103 podman[132438]: 2024-03-22 02:34:46.963163364 +0000 UTC m=+0.126824157 container create 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T02:34:47.374 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 podman[132438]: 2024-03-22 02:34:47.216479004 +0000 UTC m=+0.380139800 container init 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T02:34:47.374 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 podman[132438]: 2024-03-22 02:34:47.219356272 +0000 UTC m=+0.383017072 container start 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:34:47.374 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-osd[132457]: -- 172.21.15.103:0/1265362218 <== mon.0 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562794ba7680 con 0x562794b70c00 2024-03-22T02:34:47.374 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 bash[132438]: 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 2024-03-22T02:34:47.375 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 systemd[1]: Started Ceph osd.1 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:34:48.126 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T02:34:47.765+0000 7fa8015377c0 -1 Falling back to public interface 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: pgmap v70: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: pgmap v70: 65 pgs: 34 peering, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:34:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:34:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:49.207 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:48 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 49/321 objects degraded (15.265%), 16 pgs degraded (PG_DEGRADED) 2024-03-22T02:34:49.208 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:48 smithi103 ceph-mon[115211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs peering) 2024-03-22T02:34:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:48 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 49/321 objects degraded (15.265%), 16 pgs degraded (PG_DEGRADED) 2024-03-22T02:34:49.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:48 smithi122 ceph-mon[83368]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs peering) 2024-03-22T02:34:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:49 smithi103 ceph-mon[115211]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:49 smithi122 ceph-mon[83368]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:52.322 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:51 smithi103 ceph-mon[115211]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:51 smithi122 ceph-mon[83368]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:52.626 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:52 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T02:34:52.321+0000 7fa8015377c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-22T02:34:53.374 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:52 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T02:34:52.937+0000 7fa8015377c0 -1 osd.1 54 log_to_monitors true 2024-03-22T02:34:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:53.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:53.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:53 smithi122 ceph-mon[83368]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:53.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:53 smithi122 ceph-mon[83368]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:34:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:53 smithi103 ceph-mon[115211]: pgmap v73: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:53 smithi103 ceph-mon[115211]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:34:54.678 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T02:34:54.485+0000 7fa7ee987700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:34:54.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:34:54.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: osdmap e57: 6 total, 5 up, 6 in 2024-03-22T02:34:54.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:34:54.678 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:54.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:54.679 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:54 smithi103 ceph-mon[115211]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' 2024-03-22T02:34:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:34:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: osdmap e57: 6 total, 5 up, 6 in 2024-03-22T02:34:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:34:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:54 smithi122 ceph-mon[83368]: from='osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030]' entity='osd.1' 2024-03-22T02:34:55.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:55 smithi103 ceph-mon[115211]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:55.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:55 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:34:55.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:55 smithi103 ceph-mon[115211]: osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030] boot 2024-03-22T02:34:55.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:55 smithi103 ceph-mon[115211]: osdmap e58: 6 total, 6 up, 6 in 2024-03-22T02:34:55.819 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:55 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:34:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:55 smithi122 ceph-mon[83368]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:55 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:34:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:55 smithi122 ceph-mon[83368]: osd.1 [v2:172.21.15.103:6810/292218030,v1:172.21.15.103:6811/292218030] boot 2024-03-22T02:34:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:55 smithi122 ceph-mon[83368]: osdmap e58: 6 total, 6 up, 6 in 2024-03-22T02:34:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:55 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:34:57.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:57 smithi103 ceph-mon[115211]: osdmap e59: 6 total, 6 up, 6 in 2024-03-22T02:34:57.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:57 smithi103 ceph-mon[115211]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:57 smithi122 ceph-mon[83368]: osdmap e59: 6 total, 6 up, 6 in 2024-03-22T02:34:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:57 smithi122 ceph-mon[83368]: pgmap v78: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 49/321 objects degraded (15.265%) 2024-03-22T02:34:59.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:34:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:58 smithi122 ceph-mon[83368]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T02:34:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:34:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:58 smithi103 ceph-mon[115211]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T02:34:59.999 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:34:59 smithi122 ceph-mon[83368]: pgmap v79: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:34:59 smithi103 ceph-mon[115211]: pgmap v79: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:00 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/321 objects degraded (15.265%), 16 pgs degraded) 2024-03-22T02:35:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:00 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:35:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:00 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/321 objects degraded (15.265%), 16 pgs degraded) 2024-03-22T02:35:01.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:00 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:35:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:01 smithi122 ceph-mon[83368]: pgmap v80: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:01 smithi103 ceph-mon[115211]: pgmap v80: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:03 smithi122 ceph-mon[83368]: pgmap v81: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:03 smithi103 ceph-mon[115211]: pgmap v81: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:05 smithi122 ceph-mon[83368]: pgmap v82: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:05 smithi103 ceph-mon[115211]: pgmap v82: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:07 smithi122 ceph-mon[83368]: pgmap v83: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:07 smithi103 ceph-mon[115211]: pgmap v83: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:09 smithi122 ceph-mon[83368]: pgmap v84: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:09 smithi103 ceph-mon[115211]: pgmap v84: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:11 smithi122 ceph-mon[83368]: pgmap v85: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:11 smithi103 ceph-mon[115211]: pgmap v85: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:12 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:35:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:12 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:35:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:13 smithi103 ceph-mon[115211]: pgmap v86: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:14.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:13 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:35:14.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:13 smithi103 ceph-mon[115211]: Upgrade: osd.2 is safe to restart 2024-03-22T02:35:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:13 smithi122 ceph-mon[83368]: pgmap v86: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:13 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T02:35:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:13 smithi122 ceph-mon[83368]: Upgrade: osd.2 is safe to restart 2024-03-22T02:35:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.398+0000 7fa408afa700 1 -- 172.21.15.103:0/291542356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4041037c0 msgr2=0x7fa4040fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.398+0000 7fa408afa700 1 --2- 172.21.15.103:0/291542356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4041037c0 0x7fa4040fce30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4004940 tx=0x7fa3f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- 172.21.15.103:0/291542356 shutdown_connections 2024-03-22T02:35:14.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 --2- 172.21.15.103:0/291542356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4041037c0 0x7fa4040fce30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 --2- 172.21.15.103:0/291542356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa404102e20 0x7fa4041031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- 172.21.15.103:0/291542356 >> 172.21.15.103:0/291542356 conn(0x7fa4040f89b0 msgr2=0x7fa4040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- 172.21.15.103:0/291542356 shutdown_connections 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- 172.21.15.103:0/291542356 wait complete. 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 Processor -- start 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- start start 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 0x7fa404113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa404114590 con 0x7fa404102e20 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.399+0000 7fa408afa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40410db20 con 0x7fa4041037c0 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49982/0 (socket says 172.21.15.103:49982) 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 -- 172.21.15.103:0/2326312680 learned_addr learned my addr 172.21.15.103:0/2326312680 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:14.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa401d9b700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 0x7fa404113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 -- 172.21.15.103:0/2326312680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 msgr2=0x7fa404113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 0x7fa404113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 -- 172.21.15.103:0/2326312680 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4005040 con 0x7fa404102e20 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa401d9b700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 0x7fa404113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa40259c700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800b790 tx=0x7fa3f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f800f770 con 0x7fa404102e20 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3f800fda0 con 0x7fa404102e20 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40410de00 con 0x7fa404102e20 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f800d310 con 0x7fa404102e20 2024-03-22T02:35:14.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.400+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40410e3c0 con 0x7fa404102e20 2024-03-22T02:35:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.401+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4040fe570 con 0x7fa404102e20 2024-03-22T02:35:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.402+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa3f8010420 con 0x7fa404102e20 2024-03-22T02:35:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.402+0000 7fa3f2ffd700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 0x7fa3ec070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.402+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa3f808dc90 con 0x7fa404102e20 2024-03-22T02:35:14.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.402+0000 7fa401d9b700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 0x7fa3ec070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.403+0000 7fa401d9b700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 0x7fa3ec070850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009fd0 tx=0x7fa3f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:14.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.407+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3f8057050 con 0x7fa404102e20 2024-03-22T02:35:14.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.634+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa40410f030 con 0x7fa3ec06e3a0 2024-03-22T02:35:14.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.636+0000 7fa3f2ffd700 1 -- 172.21.15.103:0/2326312680 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa40410f030 con 0x7fa3ec06e3a0 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 msgr2=0x7fa3ec070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 0x7fa3ec070850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009fd0 tx=0x7fa3f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 msgr2=0x7fa404113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800b790 tx=0x7fa3f8009250 comp rx=0 tx=0).stop 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 shutdown_connections 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa3ec06e3a0 0x7fa3ec070850 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa404102e20 0x7fa404113970 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 --2- 172.21.15.103:0/2326312680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4041037c0 0x7fa404113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 >> 172.21.15.103:0/2326312680 conn(0x7fa4040f89b0 msgr2=0x7fa4040f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:14.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 shutdown_connections 2024-03-22T02:35:14.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.638+0000 7fa408afa700 1 -- 172.21.15.103:0/2326312680 wait complete. 2024-03-22T02:35:14.651 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 -- 172.21.15.103:0/2096054977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 msgr2=0x7f4fd8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2096054977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8102140 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc004940 tx=0x7f4fcc009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 -- 172.21.15.103:0/2096054977 shutdown_connections 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2096054977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 0x7f4fd80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2096054977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8102140 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 -- 172.21.15.103:0/2096054977 >> 172.21.15.103:0/2096054977 conn(0x7f4fd8072bc0 msgr2=0x7f4fd8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 -- 172.21.15.103:0/2096054977 shutdown_connections 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.746+0000 7f4fdec63700 1 -- 172.21.15.103:0/2096054977 wait complete. 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 Processor -- start 2024-03-22T02:35:14.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 -- start start 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 0x7f4fd8114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd81146b0 con 0x7f4fd8101d70 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.747+0000 7f4fdec63700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd810daa0 con 0x7f4fd8102710 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49990/0 (socket says 172.21.15.103:49990) 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 -- 172.21.15.103:0/2078274538 learned_addr learned my addr 172.21.15.103:0/2078274538 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fd7fff700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 0x7f4fd8114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 -- 172.21.15.103:0/2078274538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 msgr2=0x7f4fd8114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 0x7f4fd8114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 -- 172.21.15.103:0/2078274538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc005040 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdc9ff700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00b3b0 tx=0x7f4fcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fcc00e040 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd810dd20 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fcc013420 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fcc01c630 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.748+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd810e290 con 0x7f4fd8101d70 2024-03-22T02:35:14.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.750+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd8100090 con 0x7f4fd8101d70 2024-03-22T02:35:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.750+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4fcc01c8b0 con 0x7f4fd8101d70 2024-03-22T02:35:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.750+0000 7f4fd57fa700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 0x7f4fc8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.751+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4fcc08ef10 con 0x7f4fd8101d70 2024-03-22T02:35:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.751+0000 7f4fd7fff700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 0x7f4fc8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:14.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.751+0000 7f4fd7fff700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 0x7f4fc8070850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4fd810ec60 tx=0x7f4fc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:14.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.755+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4fcc05bd60 con 0x7f4fd8101d70 2024-03-22T02:35:14.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.983+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fd810f1e0 con 0x7f4fc806e3a0 2024-03-22T02:35:14.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.985+0000 7f4fd57fa700 1 -- 172.21.15.103:0/2078274538 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4fd810f1e0 con 0x7f4fc806e3a0 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 msgr2=0x7f4fc8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 0x7f4fc8070850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4fd810ec60 tx=0x7f4fc400b040 comp rx=0 tx=0).stop 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 msgr2=0x7f4fd8113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00b3b0 tx=0x7f4fcc009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 shutdown_connections 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4fc806e3a0 0x7f4fc8070850 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fd8101d70 0x7f4fd8113b20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 --2- 172.21.15.103:0/2078274538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fd8102710 0x7f4fd8114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 >> 172.21.15.103:0/2078274538 conn(0x7f4fd8072bc0 msgr2=0x7f4fd8105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 shutdown_connections 2024-03-22T02:35:14.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:14.987+0000 7f4fdec63700 1 -- 172.21.15.103:0/2078274538 wait complete. 2024-03-22T02:35:15.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- 172.21.15.103:0/1045521722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8101d70 msgr2=0x7f28c8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/1045521722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8101d70 0x7f28c8102140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004940 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- 172.21.15.103:0/1045521722 shutdown_connections 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/1045521722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8102710 0x7f28c80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/1045521722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8101d70 0x7f28c8102140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- 172.21.15.103:0/1045521722 >> 172.21.15.103:0/1045521722 conn(0x7f28c8072bc0 msgr2=0x7f28c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- 172.21.15.103:0/1045521722 shutdown_connections 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- 172.21.15.103:0/1045521722 wait complete. 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 Processor -- start 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.104+0000 7f28ccdf9700 1 -- start start 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28ccdf9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8101d70 0x7f28c810ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28ccdf9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28ccdf9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c810f700 con 0x7f28c8102710 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28ccdf9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c810f840 con 0x7f28c8101d70 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50014/0 (socket says 172.21.15.103:50014) 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 -- 172.21.15.103:0/2319462790 learned_addr learned my addr 172.21.15.103:0/2319462790 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c659c700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8101d70 0x7f28c810ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 -- 172.21.15.103:0/2319462790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8101d70 msgr2=0x7f28c810ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8101d70 0x7f28c810ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 -- 172.21.15.103:0/2319462790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8005040 con 0x7f28c8102710 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28c5d9b700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28bc002a50 tx=0x7f28bc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc007cc0 con 0x7f28c8102710 2024-03-22T02:35:15.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8117b30 con 0x7f28c8102710 2024-03-22T02:35:15.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.105+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28bc00f040 con 0x7f28c8102710 2024-03-22T02:35:15.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.106+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc00d9c0 con 0x7f28c8102710 2024-03-22T02:35:15.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.106+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c81180c0 con 0x7f28c8102710 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.107+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c80ff080 con 0x7f28c8102710 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.107+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f28bc009730 con 0x7f28c8102710 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.107+0000 7f28b6ffd700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 0x7f28b0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.108+0000 7f28c659c700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 0x7f28b0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.108+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f28bc08d910 con 0x7f28c8102710 2024-03-22T02:35:15.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.108+0000 7f28c659c700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 0x7f28b0070850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004f90 tx=0x7f28b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.112+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f28bc056c30 con 0x7f28c8102710 2024-03-22T02:35:15.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.326+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28c8108e70 con 0x7f28b006e3a0 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.336+0000 7f28b6ffd700 1 -- 172.21.15.103:0/2319462790 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f28c8108e70 con 0x7f28b006e3a0 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (3m) 22s ago 14m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (90s) 22s ago 15m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (85s) 82s ago 14m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (3m) 22s ago 14m 81.4M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (9m) 22s ago 9m 14.0M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (9m) 22s ago 9m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:35:15.338 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (9m) 82s ago 9m 15.3M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (9m) 82s ago 9m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (5m) 22s ago 16m 499M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (4m) 82s ago 13m 449M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (2m) 22s ago 16m 44.6M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (2m) 82s ago 13m 35.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (4m) 22s ago 15m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 82s ago 14m 16.8M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (64s) 22s ago 13m 86.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (28s) 22s ago 12m 12.6M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (12m) 22s ago 12m 200M 1557M 17.2.7-726-g9c4e0c34 837124da804f 784e4a76b444 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (11m) 82s ago 11m 265M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (11m) 82s ago 11m 231M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (10m) 82s ago 10m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:35:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (4m) 22s ago 14m 62.6M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:35:15.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 msgr2=0x7f28b0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 0x7f28b0070850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28b8004f90 tx=0x7f28b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 msgr2=0x7f28c810f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28bc002a50 tx=0x7f28bc004600 comp rx=0 tx=0).stop 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 shutdown_connections 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28b006e3a0 0x7f28b0070850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28c8101d70 0x7f28c810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 --2- 172.21.15.103:0/2319462790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102710 0x7f28c810f1c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.338+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 >> 172.21.15.103:0/2319462790 conn(0x7f28c8072bc0 msgr2=0x7f28c80fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.339+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 shutdown_connections 2024-03-22T02:35:15.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.339+0000 7f28ccdf9700 1 -- 172.21.15.103:0/2319462790 wait complete. 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 -- 172.21.15.103:0/1873304073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8102710 msgr2=0x7f8ea80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 --2- 172.21.15.103:0/1873304073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8102710 0x7f8ea80fbd80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c004940 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 -- 172.21.15.103:0/1873304073 shutdown_connections 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 --2- 172.21.15.103:0/1873304073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8102710 0x7f8ea80fbd80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 --2- 172.21.15.103:0/1873304073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8101d70 0x7f8ea8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 -- 172.21.15.103:0/1873304073 >> 172.21.15.103:0/1873304073 conn(0x7f8ea8072bc0 msgr2=0x7f8ea8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 -- 172.21.15.103:0/1873304073 shutdown_connections 2024-03-22T02:35:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.440+0000 7f8ead795700 1 -- 172.21.15.103:0/1873304073 wait complete. 2024-03-22T02:35:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 Processor -- start 2024-03-22T02:35:15.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 -- start start 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8102710 0x7f8ea810be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810c480 con 0x7f8ea8101d70 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.441+0000 7f8ead795700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810c5f0 con 0x7f8ea8102710 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50032/0 (socket says 172.21.15.103:50032) 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 -- 172.21.15.103:0/2870701387 learned_addr learned my addr 172.21.15.103:0/2870701387 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:15.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea67fc700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8102710 0x7f8ea810be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 -- 172.21.15.103:0/2870701387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8102710 msgr2=0x7f8ea810be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8102710 0x7f8ea810be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 -- 172.21.15.103:0/2870701387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8ea8101d70 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.442+0000 7f8ea6ffd700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e98002a50 tx=0x7f8e980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.443+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e98007cc0 con 0x7f8ea8101d70 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.443+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e9800f040 con 0x7f8ea8101d70 2024-03-22T02:35:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.443+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea811e1d0 con 0x7f8ea8101d70 2024-03-22T02:35:15.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.443+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9800da30 con 0x7f8ea8101d70 2024-03-22T02:35:15.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.443+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea811e6e0 con 0x7f8ea8101d70 2024-03-22T02:35:15.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.444+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea80fd4c0 con 0x7f8ea8101d70 2024-03-22T02:35:15.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.445+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8e9800db90 con 0x7f8ea8101d70 2024-03-22T02:35:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.445+0000 7f8e8f7fe700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 0x7f8e90070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.445+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8e9808da10 con 0x7f8ea8101d70 2024-03-22T02:35:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.445+0000 7f8ea67fc700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 0x7f8e90070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.446+0000 7f8ea67fc700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 0x7f8e90070850 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80fa680 tx=0x7f8e9c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.450+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e98056d30 con 0x7f8ea8101d70 2024-03-22T02:35:15.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.742+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8ea804bed0 con 0x7f8ea8101d70 2024-03-22T02:35:15.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.743+0000 7f8e8f7fe700 1 -- 172.21.15.103:0/2870701387 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f8e9805a3f0 con 0x7f8ea8101d70 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4, 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:35:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 8, 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:35:15.746 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 msgr2=0x7f8e90070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 0x7f8e90070850 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80fa680 tx=0x7f8e9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 msgr2=0x7f8ea810b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8e98002a50 tx=0x7f8e980046b0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 shutdown_connections 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8e9006e3a0 0x7f8e90070850 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea8101d70 0x7f8ea810b8f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 --2- 172.21.15.103:0/2870701387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ea8102710 0x7f8ea810be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 >> 172.21.15.103:0/2870701387 conn(0x7f8ea8072bc0 msgr2=0x7f8ea8065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 shutdown_connections 2024-03-22T02:35:15.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.745+0000 7f8ead795700 1 -- 172.21.15.103:0/2870701387 wait complete. 2024-03-22T02:35:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: Upgrade: Updating osd.2 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: Deploying daemon osd.2 on smithi103 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:15.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:15 smithi122 ceph-mon[83368]: pgmap v87: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/1089632895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef0101d20 msgr2=0x7f3ef01020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/1089632895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef0101d20 0x7f3ef01020f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009ce0 tx=0x7f3ee8004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/1089632895 shutdown_connections 2024-03-22T02:35:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/1089632895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef01026c0 0x7f3ef00fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/1089632895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef0101d20 0x7f3ef01020f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/1089632895 >> 172.21.15.103:0/1089632895 conn(0x7f3ef00f7870 msgr2=0x7f3ef00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/1089632895 shutdown_connections 2024-03-22T02:35:15.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.848+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/1089632895 wait complete. 2024-03-22T02:35:15.851 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: Upgrade: Updating osd.2 2024-03-22T02:35:15.851 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:15.851 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:35:15.852 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:15.852 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: Deploying daemon osd.2 on smithi103 2024-03-22T02:35:15.852 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:15.852 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:15 smithi103 ceph-mon[115211]: pgmap v87: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 Processor -- start 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 -- start start 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef0101d20 0x7f3ef01139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0114580 con 0x7f3ef01026c0 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3ef6dcf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef010db80 con 0x7f3ef0101d20 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.849+0000 7f3eeffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50042/0 (socket says 172.21.15.103:50042) 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 -- 172.21.15.103:0/835635857 learned_addr learned my addr 172.21.15.103:0/835635857 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3ef4b6b700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef0101d20 0x7f3ef01139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 -- 172.21.15.103:0/835635857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef0101d20 msgr2=0x7f3ef01139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef0101d20 0x7f3ef01139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 -- 172.21.15.103:0/835635857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee8009940 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eeffff700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000eb40 tx=0x7f3ee000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ee0007cc0 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ee000f040 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef010de00 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ee000c9d0 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.850+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef010e3c0 con 0x7f3ef01026c0 2024-03-22T02:35:15.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.851+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef00fd470 con 0x7f3ef01026c0 2024-03-22T02:35:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.852+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3ee000cc40 con 0x7f3ef01026c0 2024-03-22T02:35:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.852+0000 7f3eed7fa700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 0x7f3edc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.852+0000 7f3ef4b6b700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 0x7f3edc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.852+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3ee008d770 con 0x7f3ef01026c0 2024-03-22T02:35:15.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.853+0000 7f3ef4b6b700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 0x7f3edc070850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8004f80 tx=0x7f3ee8004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:15.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:15.857+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ee005a5f0 con 0x7f3ef01026c0 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.103+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3ef010df90 con 0x7f3ef01026c0 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.103+0000 7f3eed7fa700 1 -- 172.21.15.103:0/835635857 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7f3ef010df90 con 0x7f3ef01026c0 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:35:16.106 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:35:16.107 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 msgr2=0x7f3edc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 0x7f3edc070850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8004f80 tx=0x7f3ee8004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 msgr2=0x7f3ef0113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000eb40 tx=0x7f3ee000b040 comp rx=0 tx=0).stop 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 shutdown_connections 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3edc06e3a0 0x7f3edc070850 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ef0101d20 0x7f3ef01139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 --2- 172.21.15.103:0/835635857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ef01026c0 0x7f3ef0113f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 >> 172.21.15.103:0/835635857 conn(0x7f3ef00f7870 msgr2=0x7f3ef00f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:16.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 shutdown_connections 2024-03-22T02:35:16.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.106+0000 7f3ef6dcf700 1 -- 172.21.15.103:0/835635857 wait complete. 2024-03-22T02:35:16.110 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:35:16.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 -- 172.21.15.103:0/373170241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 msgr2=0x7f665c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 --2- 172.21.15.103:0/373170241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c0fb910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6650004940 tx=0x7f6650009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 -- 172.21.15.103:0/373170241 shutdown_connections 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 --2- 172.21.15.103:0/373170241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c0fb910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 --2- 172.21.15.103:0/373170241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 0x7f665c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 -- 172.21.15.103:0/373170241 >> 172.21.15.103:0/373170241 conn(0x7f665c0714c0 msgr2=0x7f665c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 -- 172.21.15.103:0/373170241 shutdown_connections 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 -- 172.21.15.103:0/373170241 wait complete. 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.210+0000 7f66620fa700 1 Processor -- start 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f66620fa700 1 -- start start 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f66620fa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f66620fa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 0x7f665c116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f66620fa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c116870 con 0x7f665c0fb4a0 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f66620fa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c10fc40 con 0x7f665c102e70 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50068/0 (socket says 172.21.15.103:50068) 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 -- 172.21.15.103:0/2923872029 learned_addr learned my addr 172.21.15.103:0/2923872029 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665affd700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 0x7f665c116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 -- 172.21.15.103:0/2923872029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 msgr2=0x7f665c116220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 0x7f665c116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 -- 172.21.15.103:0/2923872029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6650005040 con 0x7f665c0fb4a0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.211+0000 7f665b7fe700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f665400ec90 tx=0x7f665400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.212+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f665400eea0 con 0x7f665c0fb4a0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.212+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f665400f040 con 0x7f665c0fb4a0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.212+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6654009640 con 0x7f665c0fb4a0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.212+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c10ff20 con 0x7f665c0fb4a0 2024-03-22T02:35:16.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.212+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c110460 con 0x7f665c0fb4a0 2024-03-22T02:35:16.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.213+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c106940 con 0x7f665c0fb4a0 2024-03-22T02:35:16.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.213+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f665400c940 con 0x7f665c0fb4a0 2024-03-22T02:35:16.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.213+0000 7f6643fff700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 0x7f6644070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.214+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f665408e2e0 con 0x7f665c0fb4a0 2024-03-22T02:35:16.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.214+0000 7f665affd700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 0x7f6644070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.214+0000 7f665affd700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 0x7f6644070850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f665000b0d0 tx=0x7f6650009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:16.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.218+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f665405b0b0 con 0x7f665c0fb4a0 2024-03-22T02:35:16.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.446+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f665c109fb0 con 0x7f664406e3a0 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.447+0000 7f6643fff700 1 -- 172.21.15.103:0/2923872029 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f665c109fb0 con 0x7f664406e3a0 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:35:16.449 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "8/21 daemons upgraded", 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:35:16.450 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:35:16.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.449+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 msgr2=0x7f6644070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.449+0000 7f66620fa700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 0x7f6644070850 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f665000b0d0 tx=0x7f6650009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:16.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.449+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 msgr2=0x7f665c115ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.449+0000 7f66620fa700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f665400ec90 tx=0x7f665400b040 comp rx=0 tx=0).stop 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 shutdown_connections 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664406e3a0 0x7f6644070850 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fb4a0 0x7f665c115ce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 --2- 172.21.15.103:0/2923872029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c102e70 0x7f665c116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 >> 172.21.15.103:0/2923872029 conn(0x7f665c0714c0 msgr2=0x7f665c0fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 shutdown_connections 2024-03-22T02:35:16.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.450+0000 7f66620fa700 1 -- 172.21.15.103:0/2923872029 wait complete. 2024-03-22T02:35:16.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 -- 172.21.15.103:0/1328475517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 msgr2=0x7f5d500fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 --2- 172.21.15.103:0/1328475517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d500fbd10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d44004940 tx=0x7f5d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 -- 172.21.15.103:0/1328475517 shutdown_connections 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 --2- 172.21.15.103:0/1328475517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d500fbd10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 --2- 172.21.15.103:0/1328475517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 0x7f5d501020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 -- 172.21.15.103:0/1328475517 >> 172.21.15.103:0/1328475517 conn(0x7f5d500f7830 msgr2=0x7f5d500f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:16.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.552+0000 7f5d55049700 1 -- 172.21.15.103:0/1328475517 shutdown_connections 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 -- 172.21.15.103:0/1328475517 wait complete. 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 Processor -- start 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 -- start start 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 0x7f5d50115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d501167d0 con 0x7f5d501026a0 2024-03-22T02:35:16.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d55049700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5010fd20 con 0x7f5d50101d00 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d4e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d4e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50092/0 (socket says 172.21.15.103:50092) 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.553+0000 7f5d4e59c700 1 -- 172.21.15.103:0/3687054423 learned_addr learned my addr 172.21.15.103:0/3687054423 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d4ed9d700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 0x7f5d50115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d4e59c700 1 -- 172.21.15.103:0/3687054423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 msgr2=0x7f5d50115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d4e59c700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 0x7f5d50115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d4e59c700 1 -- 172.21.15.103:0/3687054423 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d44005040 con 0x7f5d501026a0 2024-03-22T02:35:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d4e59c700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009100 tx=0x7f5d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4400e040 con 0x7f5d501026a0 2024-03-22T02:35:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d44009af0 con 0x7f5d501026a0 2024-03-22T02:35:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5010ff40 con 0x7f5d501026a0 2024-03-22T02:35:16.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d4401c3f0 con 0x7f5d501026a0 2024-03-22T02:35:16.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.554+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d50110500 con 0x7f5d501026a0 2024-03-22T02:35:16.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.555+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d500fd450 con 0x7f5d501026a0 2024-03-22T02:35:16.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.556+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5d4400baf0 con 0x7f5d501026a0 2024-03-22T02:35:16.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.556+0000 7f5d377fe700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 0x7f5d38070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:16.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.557+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f5d4408f060 con 0x7f5d501026a0 2024-03-22T02:35:16.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.557+0000 7f5d4ed9d700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 0x7f5d38070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:16.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.557+0000 7f5d4ed9d700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 0x7f5d38070850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009b60 tx=0x7f5d400091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:16.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.561+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d4405bf50 con 0x7f5d501026a0 2024-03-22T02:35:16.734 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:16 smithi103 ceph-mon[115211]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:16.734 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:16 smithi103 ceph-mon[115211]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:16.734 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2870701387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:16.734 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/835635857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:35:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:16 smithi122 ceph-mon[83368]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:16 smithi122 ceph-mon[83368]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2870701387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:16.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/835635857' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:35:16.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.856+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5d50108eb0 con 0x7f5d501026a0 2024-03-22T02:35:16.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.856+0000 7f5d377fe700 1 -- 172.21.15.103:0/3687054423 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f5d4405bae0 con 0x7f5d501026a0 2024-03-22T02:35:16.858 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 msgr2=0x7f5d38070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 0x7f5d38070850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009b60 tx=0x7f5d400091a0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 msgr2=0x7f5d501160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009100 tx=0x7f5d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 shutdown_connections 2024-03-22T02:35:16.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d3806e3a0 0x7f5d38070850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d50101d00 0x7f5d50115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 --2- 172.21.15.103:0/3687054423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d501026a0 0x7f5d501160f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:16.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 >> 172.21.15.103:0/3687054423 conn(0x7f5d500f7830 msgr2=0x7f5d500f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:16.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 shutdown_connections 2024-03-22T02:35:16.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:16.858+0000 7f5d55049700 1 -- 172.21.15.103:0/3687054423 wait complete. 2024-03-22T02:35:17.812 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:17 smithi103 ceph-mon[115211]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:17.812 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:17 smithi103 ceph-mon[115211]: pgmap v88: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:17.812 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:17 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3687054423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:35:17.813 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:17 smithi122 ceph-mon[83368]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:17 smithi122 ceph-mon[83368]: pgmap v88: 65 pgs: 65 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:17 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3687054423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:35:17.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:18.124 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:18 smithi103 systemd[1]: Stopping Ceph osd.2 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:35:18.595 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:18 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[63196]: 2024-03-22T02:35:18.251+0000 7efd37da5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:35:18.596 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:18 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[63196]: 2024-03-22T02:35:18.251+0000 7efd37da5700 -1 osd.2 59 *** Got signal Terminated *** 2024-03-22T02:35:18.596 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:18 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[63196]: 2024-03-22T02:35:18.251+0000 7efd37da5700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:35:18.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:18 smithi122 ceph-mon[83368]: osd.2 marked itself down and dead 2024-03-22T02:35:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:18 smithi103 ceph-mon[115211]: osd.2 marked itself down and dead 2024-03-22T02:35:19.370 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:19 smithi103 podman[137792]: 2024-03-22 02:35:19.099887296 +0000 UTC m=+0.951126493 container died 784e4a76b4445fbf4da46cb01b27b2625ad1d4b59cc06c784ef6b6d5f1d506a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-9c4e0c3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:35:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:19 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:35:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:19 smithi103 ceph-mon[115211]: osdmap e60: 6 total, 5 up, 6 in 2024-03-22T02:35:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:19 smithi103 ceph-mon[115211]: pgmap v90: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:19.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:19 smithi103 podman[137792]: 2024-03-22 02:35:19.526966017 +0000 UTC m=+1.378205218 container cleanup 784e4a76b4445fbf4da46cb01b27b2625ad1d4b59cc06c784ef6b6d5f1d506a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:35:19.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:19 smithi103 bash[137792]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2 2024-03-22T02:35:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:19 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:35:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:19 smithi122 ceph-mon[83368]: osdmap e60: 6 total, 5 up, 6 in 2024-03-22T02:35:19.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:19 smithi122 ceph-mon[83368]: pgmap v90: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:20.055 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:19 smithi103 podman[137809]: 2024-03-22 02:35:19.753870959 +0000 UTC m=+0.657433186 container remove 784e4a76b4445fbf4da46cb01b27b2625ad1d4b59cc06c784ef6b6d5f1d506a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3) 2024-03-22T02:35:20.366 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:20.055067018 +0000 UTC m=+0.103443402 container create 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T02:35:20.367 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:19.970723958 +0000 UTC m=+0.019100345 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:20.367 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:20.282764309 +0000 UTC m=+0.331140704 container init 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:35:20.367 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:20.28595746 +0000 UTC m=+0.334333849 container start 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2) 2024-03-22T02:35:20.367 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:20.366152509 +0000 UTC m=+0.414528910 container attach 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, ceph=True) 2024-03-22T02:35:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:20 smithi103 ceph-mon[115211]: osdmap e61: 6 total, 5 up, 6 in 2024-03-22T02:35:20.624 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:20 smithi103 podman[137901]: 2024-03-22 02:35:20.573667797 +0000 UTC m=+0.622044182 container died 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-22T02:35:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:20 smithi122 ceph-mon[83368]: osdmap e61: 6 total, 5 up, 6 in 2024-03-22T02:35:21.204 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[137927]: 2024-03-22 02:35:21.170543594 +0000 UTC m=+0.591784767 container remove 26a83764d12b05a9d2a8d6671edb7d13f87ee188d397f991d10775cc74f1de41 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T02:35:21.204 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-22T02:35:21.204 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 systemd[1]: Stopped Ceph osd.2 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:35:21.204 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.2.service: Consumed 55.368s CPU time. 2024-03-22T02:35:21.610 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 systemd[1]: Starting Ceph osd.2 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:35:21.862 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:21 smithi103 ceph-mon[115211]: pgmap v92: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:21.862 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[138033]: 2024-03-22 02:35:21.512224869 +0000 UTC m=+0.017854360 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:21.862 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[138033]: 2024-03-22 02:35:21.638857777 +0000 UTC m=+0.144487322 container create 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:35:21.862 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[138033]: 2024-03-22 02:35:21.858389232 +0000 UTC m=+0.364018717 container init 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:35:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:21 smithi122 ceph-mon[83368]: pgmap v92: 65 pgs: 9 stale+active+clean, 56 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-22T02:35:22.124 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[138033]: 2024-03-22 02:35:21.861837727 +0000 UTC m=+0.367467209 container start 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-22T02:35:22.124 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:21 smithi103 podman[138033]: 2024-03-22 02:35:21.94995897 +0000 UTC m=+0.455588456 container attach 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 bash[138033]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-22T02:35:22.625 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T02:35:22.626 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate[138047]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-22T02:35:22.626 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 podman[138033]: 2024-03-22 02:35:22.335089197 +0000 UTC m=+0.840718664 container died 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:35:22.971 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:22 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 27/321 objects degraded (8.411%), 9 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:23.231 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:22 smithi103 podman[138187]: 2024-03-22 02:35:22.971577127 +0000 UTC m=+0.631115701 container remove 0e547726fbfb9d0fa7e28bca6586807a85474a661f9433b76726196d08306707 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-22T02:35:23.231 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 podman[138249]: 2024-03-22 02:35:23.091425953 +0000 UTC m=+0.022910585 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:23.231 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 podman[138249]: 2024-03-22 02:35:23.230996053 +0000 UTC m=+0.162480683 container create 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:35:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:22 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 27/321 objects degraded (8.411%), 9 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:23.517 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 podman[138249]: 2024-03-22 02:35:23.409213593 +0000 UTC m=+0.340698233 container init 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-22T02:35:23.518 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 podman[138249]: 2024-03-22 02:35:23.412416554 +0000 UTC m=+0.343901186 container start 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:35:23.518 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-osd[138268]: -- 172.21.15.103:0/493199416 <== mon.0 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dce794f680 con 0x55dce7918c00 2024-03-22T02:35:23.518 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 bash[138249]: 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 2024-03-22T02:35:23.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 systemd[1]: Started Ceph osd.2 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:35:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:23 smithi122 ceph-mon[83368]: pgmap v93: 65 pgs: 9 active+undersized+degraded, 4 stale+active+clean, 11 active+undersized, 41 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27/321 objects degraded (8.411%) 2024-03-22T02:35:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:24.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-mon[115211]: pgmap v93: 65 pgs: 9 active+undersized+degraded, 4 stale+active+clean, 11 active+undersized, 41 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27/321 objects degraded (8.411%) 2024-03-22T02:35:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:24.374 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:23 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T02:35:23.953+0000 7fa0f36937c0 -1 Falling back to public interface 2024-03-22T02:35:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:25 smithi122 ceph-mon[83368]: pgmap v94: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:25 smithi103 ceph-mon[115211]: pgmap v94: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:28.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:27 smithi103 ceph-mon[115211]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:28.287 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T02:35:28.140+0000 7fa0f36937c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-22T02:35:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:27 smithi122 ceph-mon[83368]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:28.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T02:35:28.540+0000 7fa0f36937c0 -1 osd.2 59 log_to_monitors true 2024-03-22T02:35:29.250 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:28 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:29.250 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:28 smithi122 ceph-mon[83368]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:35:29.250 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:29.250 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:29.374 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T02:35:28.969+0000 7fa0e4aeb700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:35:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-mon[115211]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:35:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:30.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:35:30.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: osdmap e62: 6 total, 5 up, 6 in 2024-03-22T02:35:30.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:35:30.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:30.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: pgmap v96: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: osdmap e62: 6 total, 5 up, 6 in 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: from='osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:30 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:35:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:30 smithi103 ceph-mon[115211]: osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184] boot 2024-03-22T02:35:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:30 smithi103 ceph-mon[115211]: osdmap e63: 6 total, 6 up, 6 in 2024-03-22T02:35:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:30 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:35:31.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:30 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:35:31.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:30 smithi122 ceph-mon[83368]: osd.2 [v2:172.21.15.103:6818/117269184,v1:172.21.15.103:6819/117269184] boot 2024-03-22T02:35:31.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:30 smithi122 ceph-mon[83368]: osdmap e63: 6 total, 6 up, 6 in 2024-03-22T02:35:31.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:30 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:35:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:31 smithi103 ceph-mon[115211]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:31 smithi103 ceph-mon[115211]: osdmap e64: 6 total, 6 up, 6 in 2024-03-22T02:35:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:31 smithi122 ceph-mon[83368]: pgmap v99: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:35:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:31 smithi122 ceph-mon[83368]: osdmap e64: 6 total, 6 up, 6 in 2024-03-22T02:35:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: pgmap v101: 65 pgs: 5 active+undersized+degraded, 7 active+undersized, 53 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17/321 objects degraded (5.296%) 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 17/321 objects degraded (5.296%), 5 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:33 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: pgmap v101: 65 pgs: 5 active+undersized+degraded, 7 active+undersized, 53 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17/321 objects degraded (5.296%) 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 17/321 objects degraded (5.296%), 5 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:34.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:33 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:34 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:34 smithi122 ceph-mon[83368]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-22T02:35:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:34 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:34 smithi103 ceph-mon[115211]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-22T02:35:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:35 smithi122 ceph-mon[83368]: pgmap v102: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:35 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/321 objects degraded (5.296%), 5 pgs degraded) 2024-03-22T02:35:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:35 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:35:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:35 smithi103 ceph-mon[115211]: pgmap v102: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:35 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/321 objects degraded (5.296%), 5 pgs degraded) 2024-03-22T02:35:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:35 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:35:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:36 smithi122 ceph-mon[83368]: pgmap v103: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:36 smithi103 ceph-mon[115211]: pgmap v103: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:39 smithi122 ceph-mon[83368]: pgmap v104: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:39 smithi103 ceph-mon[115211]: pgmap v104: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:41 smithi122 ceph-mon[83368]: pgmap v105: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:41 smithi103 ceph-mon[115211]: pgmap v105: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:43.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:44.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:43 smithi122 ceph-mon[83368]: pgmap v106: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:43 smithi103 ceph-mon[115211]: pgmap v106: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:45 smithi122 ceph-mon[83368]: pgmap v107: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:45 smithi103 ceph-mon[115211]: pgmap v107: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 -- 172.21.15.103:0/3339781150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 msgr2=0x7fd6980fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 --2- 172.21.15.103:0/3339781150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fb910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd680004940 tx=0x7fd680009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 -- 172.21.15.103:0/3339781150 shutdown_connections 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 --2- 172.21.15.103:0/3339781150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fb910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 --2- 172.21.15.103:0/3339781150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 0x7fd698103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 -- 172.21.15.103:0/3339781150 >> 172.21.15.103:0/3339781150 conn(0x7fd6980714c0 msgr2=0x7fd6980718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 -- 172.21.15.103:0/3339781150 shutdown_connections 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.974+0000 7fd69cdca700 1 -- 172.21.15.103:0/3339781150 wait complete. 2024-03-22T02:35:46.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 Processor -- start 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 -- start start 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 0x7fd6980fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980fd150 con 0x7fd6980fb4a0 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69cdca700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980fd290 con 0x7fd698102e70 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54372/0 (socket says 172.21.15.103:54372) 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 -- 172.21.15.103:0/533511891 learned_addr learned my addr 172.21.15.103:0/533511891 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd68dd9b700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 0x7fd6980fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 -- 172.21.15.103:0/533511891 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 msgr2=0x7fd6980fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 0x7fd6980fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:46.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.975+0000 7fd69659c700 1 -- 172.21.15.103:0/533511891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd680005040 con 0x7fd6980fb4a0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd69659c700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd688002a50 tx=0x7fd6880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd688007cc0 con 0x7fd6980fb4a0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd698119e70 con 0x7fd6980fb4a0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd68800f040 con 0x7fd6980fb4a0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68800da30 con 0x7fd6980fb4a0 2024-03-22T02:35:46.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.976+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69811a2f0 con 0x7fd6980fb4a0 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.977+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698100110 con 0x7fd6980fb4a0 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.978+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd68800db90 con 0x7fd6980fb4a0 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.978+0000 7fd68f7fe700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 0x7fd684070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.978+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd68808db40 con 0x7fd6980fb4a0 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.979+0000 7fd68dd9b700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 0x7fd684070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:46.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.979+0000 7fd68dd9b700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 0x7fd684070850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6980fdfc0 tx=0x7fd680009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:46.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:46.984+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd688056d40 con 0x7fd6980fb4a0 2024-03-22T02:35:47.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.200+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd69804bed0 con 0x7fd68406e3a0 2024-03-22T02:35:47.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.202+0000 7fd68f7fe700 1 -- 172.21.15.103:0/533511891 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd69804bed0 con 0x7fd68406e3a0 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 msgr2=0x7fd684070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 0x7fd684070850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6980fdfc0 tx=0x7fd680009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 msgr2=0x7fd6980fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd688002a50 tx=0x7fd6880046b0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 shutdown_connections 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd68406e3a0 0x7fd684070850 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6980fb4a0 0x7fd6980fc680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 --2- 172.21.15.103:0/533511891 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd698102e70 0x7fd6980fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 >> 172.21.15.103:0/533511891 conn(0x7fd6980714c0 msgr2=0x7fd6980fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 shutdown_connections 2024-03-22T02:35:47.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.204+0000 7fd69cdca700 1 -- 172.21.15.103:0/533511891 wait complete. 2024-03-22T02:35:47.215 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:35:47.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 -- 172.21.15.103:0/899185386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fe830 msgr2=0x7f2efc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 --2- 172.21.15.103:0/899185386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fe830 0x7f2efc106f00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8004940 tx=0x7f2ef8009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:47.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 -- 172.21.15.103:0/899185386 shutdown_connections 2024-03-22T02:35:47.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 --2- 172.21.15.103:0/899185386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fe830 0x7f2efc106f00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 --2- 172.21.15.103:0/899185386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2efc0fde90 0x7f2efc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 -- 172.21.15.103:0/899185386 >> 172.21.15.103:0/899185386 conn(0x7f2efc072bc0 msgr2=0x7f2efc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 -- 172.21.15.103:0/899185386 shutdown_connections 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 -- 172.21.15.103:0/899185386 wait complete. 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.308+0000 7f2f036d7700 1 Processor -- start 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 -- start start 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2efc0fe830 0x7f2efc1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc1147b0 con 0x7f2efc0fde90 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc10db10 con 0x7f2efc0fe830 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54394/0 (socket says 172.21.15.103:54394) 2024-03-22T02:35:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 -- 172.21.15.103:0/2051019866 learned_addr learned my addr 172.21.15.103:0/2051019866 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 -- 172.21.15.103:0/2051019866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2efc0fe830 msgr2=0x7f2efc1140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2efc0fe830 0x7f2efc1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 -- 172.21.15.103:0/2051019866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8005040 con 0x7f2efc0fde90 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f01473700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00b790 tx=0x7f2eec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eec00f770 con 0x7f2efc0fde90 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc10ddf0 con 0x7f2efc0fde90 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2eec00fda0 con 0x7f2efc0fde90 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.309+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eec00d310 con 0x7f2efc0fde90 2024-03-22T02:35:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.310+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc10e330 con 0x7f2efc0fde90 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.311+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc065e30 con 0x7f2efc0fde90 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.311+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2eec010420 con 0x7f2efc0fde90 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.311+0000 7f2ef1ffb700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 0x7f2ee8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.312+0000 7f2f00c72700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 0x7f2ee8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.312+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2eec08dd70 con 0x7f2efc0fde90 2024-03-22T02:35:47.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.312+0000 7f2f00c72700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 0x7f2ee8070850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8004f90 tx=0x7f2ef8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.316+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2eec05abc0 con 0x7f2efc0fde90 2024-03-22T02:35:47.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.530+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2efc04bed0 con 0x7f2ee806e3a0 2024-03-22T02:35:47.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.531+0000 7f2ef1ffb700 1 -- 172.21.15.103:0/2051019866 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2efc04bed0 con 0x7f2ee806e3a0 2024-03-22T02:35:47.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.533+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 msgr2=0x7f2ee8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.533+0000 7f2f036d7700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 0x7f2ee8070850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8004f90 tx=0x7f2ef8009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:47.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.533+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 msgr2=0x7f2efc113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.533+0000 7f2f036d7700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00b790 tx=0x7f2eec009250 comp rx=0 tx=0).stop 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 shutdown_connections 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ee806e3a0 0x7f2ee8070850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2efc0fde90 0x7f2efc113b90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 --2- 172.21.15.103:0/2051019866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2efc0fe830 0x7f2efc1140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 >> 172.21.15.103:0/2051019866 conn(0x7f2efc072bc0 msgr2=0x7f2efc103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 shutdown_connections 2024-03-22T02:35:47.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.534+0000 7f2f036d7700 1 -- 172.21.15.103:0/2051019866 wait complete. 2024-03-22T02:35:47.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/3871163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04101d80 msgr2=0x7f3b04102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/3871163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04101d80 0x7f3b04102150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b00004940 tx=0x7f3b00009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/3871163055 shutdown_connections 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/3871163055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04102720 0x7f3b040fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/3871163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04101d80 0x7f3b04102150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/3871163055 >> 172.21.15.103:0/3871163055 conn(0x7f3b040f78f0 msgr2=0x7f3b040f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/3871163055 shutdown_connections 2024-03-22T02:35:47.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/3871163055 wait complete. 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.636+0000 7f3b0b2bc700 1 Processor -- start 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 -- start start 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04101d80 0x7f3b0410eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b040671c0 con 0x7f3b04102720 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0410f5d0 con 0x7f3b04101d80 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54406/0 (socket says 172.21.15.103:54406) 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 -- 172.21.15.103:0/2550780799 learned_addr learned my addr 172.21.15.103:0/2550780799 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:47.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b09058700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04101d80 0x7f3b0410eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 -- 172.21.15.103:0/2550780799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04101d80 msgr2=0x7f3b0410eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04101d80 0x7f3b0410eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 -- 172.21.15.103:0/2550780799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00005040 con 0x7f3b04102720 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b08857700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3af4002a50 tx=0x7f3af4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3af4007cc0 con 0x7f3b04102720 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0410f8b0 con 0x7f3b04102720 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.637+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3af400f040 con 0x7f3b04102720 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.638+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3af400d9c0 con 0x7f3b04102720 2024-03-22T02:35:47.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.638+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b04117f10 con 0x7f3b04102720 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.639+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b040ff090 con 0x7f3b04102720 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.639+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3af4009730 con 0x7f3b04102720 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.639+0000 7f3af9ffb700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 0x7f3af0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.640+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3af408da80 con 0x7f3b04102720 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.640+0000 7f3b09058700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 0x7f3af0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.640+0000 7f3b09058700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 0x7f3af0070850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b00004f90 tx=0x7f3b00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.644+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3af405a850 con 0x7f3b04102720 2024-03-22T02:35:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.850+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b040667c0 con 0x7f3af006e3a0 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.860+0000 7f3af9ffb700 1 -- 172.21.15.103:0/2550780799 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f3b040667c0 con 0x7f3af006e3a0 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (4m) 19s ago 15m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (2m) 19s ago 15m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (118s) 114s ago 14m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (4m) 19s ago 15m 81.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (10m) 19s ago 10m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (9m) 19s ago 9m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (9m) 114s ago 9m 15.3M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (9m) 114s ago 9m 116M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (5m) 19s ago 16m 504M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:35:47.861 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (5m) 114s ago 14m 449M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (3m) 19s ago 17m 44.0M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (2m) 114s ago 14m 35.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (5m) 19s ago 15m 15.3M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 114s ago 14m 16.8M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (97s) 19s ago 13m 87.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (60s) 19s ago 13m 98.1M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (24s) 19s ago 12m 12.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (12m) 114s ago 12m 265M 2581M 17.2.7-726-g9c4e0c34 837124da804f 8eb9eab7c622 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (11m) 114s ago 11m 231M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (11m) 114s ago 11m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:35:47.862 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (4m) 19s ago 15m 64.7M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:35:47.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 msgr2=0x7f3af0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 0x7f3af0070850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b00004f90 tx=0x7f3b00009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:47.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 msgr2=0x7f3b0410f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3af4002a50 tx=0x7f3af4004600 comp rx=0 tx=0).stop 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 shutdown_connections 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3af006e3a0 0x7f3af0070850 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b04101d80 0x7f3b0410eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 --2- 172.21.15.103:0/2550780799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b04102720 0x7f3b0410f090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 >> 172.21.15.103:0/2550780799 conn(0x7f3b040f78f0 msgr2=0x7f3b04106320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 shutdown_connections 2024-03-22T02:35:47.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.862+0000 7f3b0b2bc700 1 -- 172.21.15.103:0/2550780799 wait complete. 2024-03-22T02:35:47.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 -- 172.21.15.103:0/453251344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f4101d20 msgr2=0x7f06f41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 --2- 172.21.15.103:0/453251344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f4101d20 0x7f06f41020f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009ce0 tx=0x7f06ec004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:47.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 -- 172.21.15.103:0/453251344 shutdown_connections 2024-03-22T02:35:47.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 --2- 172.21.15.103:0/453251344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f41026c0 0x7f06f40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 --2- 172.21.15.103:0/453251344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f4101d20 0x7f06f41020f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.962+0000 7f06fc19d700 1 -- 172.21.15.103:0/453251344 >> 172.21.15.103:0/453251344 conn(0x7f06f40f7850 msgr2=0x7f06f40f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:47.964 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:47 smithi103 ceph-mon[115211]: pgmap v108: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:47.964 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:47.964 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:47.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 -- 172.21.15.103:0/453251344 shutdown_connections 2024-03-22T02:35:47.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 -- 172.21.15.103:0/453251344 wait complete. 2024-03-22T02:35:47.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 Processor -- start 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 -- start start 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f4101d20 0x7f06f4113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4114690 con 0x7f06f41026c0 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06fc19d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f410dc70 con 0x7f06f4101d20 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.963+0000 7f06f9738700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54428/0 (socket says 172.21.15.103:54428) 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 -- 172.21.15.103:0/4285481115 learned_addr learned my addr 172.21.15.103:0/4285481115 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9f39700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f4101d20 0x7f06f4113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 -- 172.21.15.103:0/4285481115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f4101d20 msgr2=0x7f06f4113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f4101d20 0x7f06f4113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 -- 172.21.15.103:0/4285481115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec009940 con 0x7f06f41026c0 2024-03-22T02:35:47.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06f9738700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c800 tx=0x7f06e4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e40106f0 con 0x7f06f41026c0 2024-03-22T02:35:47.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06e4010d20 con 0x7f06f41026c0 2024-03-22T02:35:47.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f410df50 con 0x7f06f41026c0 2024-03-22T02:35:47.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e40113a0 con 0x7f06f41026c0 2024-03-22T02:35:47.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.964+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f410e510 con 0x7f06f41026c0 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.965+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f40fd470 con 0x7f06f41026c0 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.966+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f06e4012400 con 0x7f06f41026c0 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.966+0000 7f06ea7fc700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 0x7f06e0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.966+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f06e408dc20 con 0x7f06f41026c0 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.966+0000 7f06f9f39700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 0x7f06e0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:47.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.967+0000 7f06f9f39700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 0x7f06e0070850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00b740 tx=0x7f06ec004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:47.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:47.971+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06e405aa90 con 0x7f06f41026c0 2024-03-22T02:35:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.251+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f06f4108e20 con 0x7f06f41026c0 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.251+0000 7f06ea7fc700 1 -- 172.21.15.103:0/4285481115 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f06e405a620 con 0x7f06f41026c0 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:35:48.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 3 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 7, 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 7 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:35:48.253 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 msgr2=0x7f06e0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 0x7f06e0070850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00b740 tx=0x7f06ec004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 msgr2=0x7f06f4114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f06e400c800 tx=0x7f06e4004470 comp rx=0 tx=0).stop 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 shutdown_connections 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06e006e3a0 0x7f06e0070850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06f4101d20 0x7f06f4113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 --2- 172.21.15.103:0/4285481115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06f41026c0 0x7f06f4114040 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 >> 172.21.15.103:0/4285481115 conn(0x7f06f40f7850 msgr2=0x7f06f40f8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.253+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 shutdown_connections 2024-03-22T02:35:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.254+0000 7f06fc19d700 1 -- 172.21.15.103:0/4285481115 wait complete. 2024-03-22T02:35:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:47 smithi122 ceph-mon[83368]: pgmap v108: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 -- 172.21.15.103:0/3564063459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4101cf0 msgr2=0x7fa9d41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 --2- 172.21.15.103:0/3564063459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4101cf0 0x7fa9d41020c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009ce0 tx=0x7fa9cc004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 -- 172.21.15.103:0/3564063459 shutdown_connections 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 --2- 172.21.15.103:0/3564063459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4102690 0x7fa9d40fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 --2- 172.21.15.103:0/3564063459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4101cf0 0x7fa9d41020c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 -- 172.21.15.103:0/3564063459 >> 172.21.15.103:0/3564063459 conn(0x7fa9d40f7860 msgr2=0x7fa9d40f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 -- 172.21.15.103:0/3564063459 shutdown_connections 2024-03-22T02:35:48.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.354+0000 7fa9d950f700 1 -- 172.21.15.103:0/3564063459 wait complete. 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 Processor -- start 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 -- start start 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4101cf0 0x7fa9d4113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41146e0 con 0x7fa9d4102690 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d950f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d410dad0 con 0x7fa9d4101cf0 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54448/0 (socket says 172.21.15.103:54448) 2024-03-22T02:35:48.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d27fc700 1 -- 172.21.15.103:0/4138544085 learned_addr learned my addr 172.21.15.103:0/4138544085 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.355+0000 7fa9d2ffd700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4101cf0 0x7fa9d4113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d27fc700 1 -- 172.21.15.103:0/4138544085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4101cf0 msgr2=0x7fa9d4113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d27fc700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4101cf0 0x7fa9d4113b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d27fc700 1 -- 172.21.15.103:0/4138544085 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc009940 con 0x7fa9d4102690 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d27fc700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00eb40 tx=0x7fa9bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9bc007cc0 con 0x7fa9d4102690 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9bc00f040 con 0x7fa9d4102690 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d410dd50 con 0x7fa9d4102690 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9bc0095e0 con 0x7fa9d4102690 2024-03-22T02:35:48.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.356+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d410e310 con 0x7fa9d4102690 2024-03-22T02:35:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.357+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d40fd440 con 0x7fa9d4102690 2024-03-22T02:35:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.358+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa9bc009740 con 0x7fa9d4102690 2024-03-22T02:35:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.358+0000 7fa9cb7fe700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 0x7fa9c0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.358+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa9bc08d740 con 0x7fa9d4102690 2024-03-22T02:35:48.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.358+0000 7fa9d2ffd700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 0x7fa9c0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.358+0000 7fa9d2ffd700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 0x7fa9c0070850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00b460 tx=0x7fa9cc004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:48.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.363+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa9bc05a640 con 0x7fa9d4102690 2024-03-22T02:35:48.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.605+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa9d4108df0 con 0x7fa9d4102690 2024-03-22T02:35:48.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.605+0000 7fa9cb7fe700 1 -- 172.21.15.103:0/4138544085 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7fa9bc05a1d0 con 0x7fa9d4102690 2024-03-22T02:35:48.606 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:35:48.607 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:35:48.608 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:48.609 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 msgr2=0x7fa9c0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 0x7fa9c0070850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00b460 tx=0x7fa9cc004d10 comp rx=0 tx=0).stop 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 msgr2=0x7fa9d4114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00eb40 tx=0x7fa9bc00b040 comp rx=0 tx=0).stop 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 shutdown_connections 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa9c006e3a0 0x7fa9c0070850 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9d4101cf0 0x7fa9d4113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 --2- 172.21.15.103:0/4138544085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9d4102690 0x7fa9d4114090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 >> 172.21.15.103:0/4138544085 conn(0x7fa9d40f7860 msgr2=0x7fa9d40f8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:48.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 shutdown_connections 2024-03-22T02:35:48.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.608+0000 7fa9d950f700 1 -- 172.21.15.103:0/4138544085 wait complete. 2024-03-22T02:35:48.611 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 -- 172.21.15.103:0/3667470172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c101d70 msgr2=0x7f494c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 --2- 172.21.15.103:0/3667470172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c101d70 0x7f494c102140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f493c004940 tx=0x7f493c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 -- 172.21.15.103:0/3667470172 shutdown_connections 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 --2- 172.21.15.103:0/3667470172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c102710 0x7f494c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 --2- 172.21.15.103:0/3667470172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c101d70 0x7f494c102140 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 -- 172.21.15.103:0/3667470172 >> 172.21.15.103:0/3667470172 conn(0x7f494c072bc0 msgr2=0x7f494c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 -- 172.21.15.103:0/3667470172 shutdown_connections 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 -- 172.21.15.103:0/3667470172 wait complete. 2024-03-22T02:35:48.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.713+0000 7f4952554700 1 Processor -- start 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4952554700 1 -- start start 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4952554700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c101d70 0x7f494c112a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4952554700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4952554700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c1135d0 con 0x7f494c102710 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4952554700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c10c9c0 con 0x7f494c101d70 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54454/0 (socket says 172.21.15.103:54454) 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 -- 172.21.15.103:0/1953976801 learned_addr learned my addr 172.21.15.103:0/1953976801 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494bfff700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c101d70 0x7f494c112a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 -- 172.21.15.103:0/1953976801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c101d70 msgr2=0x7f494c112a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c101d70 0x7f494c112a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 -- 172.21.15.103:0/1953976801 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c005040 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f494b7fe700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f494000eb40 tx=0x7f494000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.714+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4940007cc0 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.715+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c10cc40 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.715+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f494000f040 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.715+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f494000cad0 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.715+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c10d180 con 0x7f494c102710 2024-03-22T02:35:48.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.715+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c0ff080 con 0x7f494c102710 2024-03-22T02:35:48.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.716+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f494000cc30 con 0x7f494c102710 2024-03-22T02:35:48.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.716+0000 7f4948ff9700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 0x7f4934070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:48.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.717+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f494008d870 con 0x7f494c102710 2024-03-22T02:35:48.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.717+0000 7f494bfff700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 0x7f4934070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:48.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.717+0000 7f494bfff700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 0x7f4934070850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f493c009b30 tx=0x7f493c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:48.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.721+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f494005a640 con 0x7f494c102710 2024-03-22T02:35:48.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.940+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f494c10e090 con 0x7f493406e3a0 2024-03-22T02:35:48.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.942+0000 7f4948ff9700 1 -- 172.21.15.103:0/1953976801 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f494c10e090 con 0x7f493406e3a0 2024-03-22T02:35:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "9/21 daemons upgraded", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:35:48.943 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 msgr2=0x7f4934070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 0x7f4934070850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f493c009b30 tx=0x7f493c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 msgr2=0x7f494c112f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f494000eb40 tx=0x7f494000b040 comp rx=0 tx=0).stop 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 shutdown_connections 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f493406e3a0 0x7f4934070850 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f494c101d70 0x7f494c112a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 --2- 172.21.15.103:0/1953976801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f494c102710 0x7f494c112f80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 >> 172.21.15.103:0/1953976801 conn(0x7f494c072bc0 msgr2=0x7f494c0faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 shutdown_connections 2024-03-22T02:35:48.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:48.944+0000 7f4952554700 1 -- 172.21.15.103:0/1953976801 wait complete. 2024-03-22T02:35:49.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 -- 172.21.15.103:0/2986461022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481037a0 msgr2=0x7f23480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:49.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 --2- 172.21.15.103:0/2986461022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481037a0 0x7f23480fce10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f233c004940 tx=0x7f233c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:49.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 -- 172.21.15.103:0/2986461022 shutdown_connections 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 --2- 172.21.15.103:0/2986461022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481037a0 0x7f23480fce10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 --2- 172.21.15.103:0/2986461022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2348102e00 0x7f23481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 -- 172.21.15.103:0/2986461022 >> 172.21.15.103:0/2986461022 conn(0x7f23480f89b0 msgr2=0x7f23480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 -- 172.21.15.103:0/2986461022 shutdown_connections 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.044+0000 7f234e526700 1 -- 172.21.15.103:0/2986461022 wait complete. 2024-03-22T02:35:49.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 Processor -- start 2024-03-22T02:35:49.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 -- start start 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23481037a0 0x7f2348113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348114490 con 0x7f2348102e00 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f234e526700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234810da90 con 0x7f23481037a0 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f2347fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f2347fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54466/0 (socket says 172.21.15.103:54466) 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f2347fff700 1 -- 172.21.15.103:0/3712563142 learned_addr learned my addr 172.21.15.103:0/3712563142 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.045+0000 7f23477fe700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23481037a0 0x7f2348113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2347fff700 1 -- 172.21.15.103:0/3712563142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23481037a0 msgr2=0x7f2348113e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2347fff700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23481037a0 0x7f2348113e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2347fff700 1 -- 172.21.15.103:0/3712563142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c005040 con 0x7f2348102e00 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2347fff700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f233800eb70 tx=0x7f233800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2338007cc0 con 0x7f2348102e00 2024-03-22T02:35:49.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f233800f040 con 0x7f2348102e00 2024-03-22T02:35:49.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f233800ca90 con 0x7f2348102e00 2024-03-22T02:35:49.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234810dd10 con 0x7f2348102e00 2024-03-22T02:35:49.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.046+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234810e1d0 con 0x7f2348102e00 2024-03-22T02:35:49.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.047+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23481052d0 con 0x7f2348102e00 2024-03-22T02:35:49.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.047+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f233800cbf0 con 0x7f2348102e00 2024-03-22T02:35:49.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.048+0000 7f2344ff9700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 0x7f2330070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:49.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.048+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f233808d8a0 con 0x7f2348102e00 2024-03-22T02:35:49.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.048+0000 7f23477fe700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 0x7f2330070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:49.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.048+0000 7f23477fe700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 0x7f2330070850 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f233c004890 tx=0x7f233c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:49.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.053+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f233805a670 con 0x7f2348102e00 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4285481115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4138544085' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:35:49.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:48 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:49.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.331+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f234810ef50 con 0x7f2348102e00 2024-03-22T02:35:49.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.333 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:49.333 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4285481115' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:35:49.333 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4138544085' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:35:49.333 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:48 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:49.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.332+0000 7f2344ff9700 1 -- 172.21.15.103:0/3712563142 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f233805a200 con 0x7f2348102e00 2024-03-22T02:35:49.333 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 msgr2=0x7f2330070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 0x7f2330070850 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f233c004890 tx=0x7f233c009e30 comp rx=0 tx=0).stop 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 msgr2=0x7f2348113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f233800eb70 tx=0x7f233800b040 comp rx=0 tx=0).stop 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 shutdown_connections 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f233006e3a0 0x7f2330070850 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2348102e00 0x7f2348113900 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 --2- 172.21.15.103:0/3712563142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23481037a0 0x7f2348113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 >> 172.21.15.103:0/3712563142 conn(0x7f23480f89b0 msgr2=0x7f23480fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 shutdown_connections 2024-03-22T02:35:49.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:35:49.334+0000 7f234e526700 1 -- 172.21.15.103:0/3712563142 wait complete. 2024-03-22T02:35:50.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:49 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:50.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:49 smithi122 ceph-mon[83368]: Upgrade: osd.3 is safe to restart 2024-03-22T02:35:50.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:49 smithi122 ceph-mon[83368]: pgmap v109: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:50.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:49 smithi122 ceph-mon[83368]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:50.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:49 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3712563142' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:35:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:49 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T02:35:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:49 smithi103 ceph-mon[115211]: Upgrade: osd.3 is safe to restart 2024-03-22T02:35:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:49 smithi103 ceph-mon[115211]: pgmap v109: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:49 smithi103 ceph-mon[115211]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:35:50.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:49 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3712563142' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:35:51.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: Upgrade: Updating osd.3 2024-03-22T02:35:51.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:51.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:35:51.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:51.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: Deploying daemon osd.3 on smithi122 2024-03-22T02:35:51.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:51 smithi122 ceph-mon[83368]: pgmap v110: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: Upgrade: Updating osd.3 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: Deploying daemon osd.3 on smithi122 2024-03-22T02:35:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:51 smithi103 ceph-mon[115211]: pgmap v110: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:54.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:53 smithi122 ceph-mon[83368]: pgmap v111: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:54.023 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:53 smithi122 systemd[1]: Stopping Ceph osd.3 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:35:54.348 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:54 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[41797]: 2024-03-22T02:35:54.025+0000 7fdcba07e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:35:54.348 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:54 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[41797]: 2024-03-22T02:35:54.025+0000 7fdcba07e700 -1 osd.3 64 *** Got signal Terminated *** 2024-03-22T02:35:54.348 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:54 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[41797]: 2024-03-22T02:35:54.025+0000 7fdcba07e700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:35:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:53 smithi103 ceph-mon[115211]: pgmap v111: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:55.268 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:54 smithi122 ceph-mon[83368]: osd.3 marked itself down and dead 2024-03-22T02:35:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:54 smithi103 ceph-mon[115211]: osd.3 marked itself down and dead 2024-03-22T02:35:55.560 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:55 smithi122 podman[91840]: 2024-03-22 02:35:55.267894786 +0000 UTC m=+1.356225279 container died 8eb9eab7c6226721d8c25e1812bdda6afc784d4a065174ec0306bca3916336c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=quincy-9c4e0c3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-22T02:35:55.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:55 smithi122 podman[91840]: 2024-03-22 02:35:55.642538352 +0000 UTC m=+1.730868844 container cleanup 8eb9eab7c6226721d8c25e1812bdda6afc784d4a065174ec0306bca3916336c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, ceph=True) 2024-03-22T02:35:55.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:55 smithi122 bash[91840]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3 2024-03-22T02:35:56.227 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:55 smithi122 ceph-mon[83368]: pgmap v112: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:56.227 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:55 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:35:56.227 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:55 smithi122 ceph-mon[83368]: osdmap e65: 6 total, 5 up, 6 in 2024-03-22T02:35:56.227 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:55 smithi122 podman[91870]: 2024-03-22 02:35:55.927045558 +0000 UTC m=+0.654998934 container remove 8eb9eab7c6226721d8c25e1812bdda6afc784d4a065174ec0306bca3916336c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, CEPH_POINT_RELEASE=, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:35:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:55 smithi103 ceph-mon[115211]: pgmap v112: 65 pgs: 65 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:55 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:35:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:55 smithi103 ceph-mon[115211]: osdmap e65: 6 total, 5 up, 6 in 2024-03-22T02:35:56.481 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.129477881 +0000 UTC m=+0.017825362 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:56.481 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.227953838 +0000 UTC m=+0.116301317 container create 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:35:56.481 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.480455489 +0000 UTC m=+0.368802987 container init 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-22T02:35:56.763 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.483912341 +0000 UTC m=+0.372259821 container start 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-22T02:35:56.763 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.555463872 +0000 UTC m=+0.443811356 container attach 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-22T02:35:57.060 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:56 smithi122 ceph-mon[83368]: osdmap e66: 6 total, 5 up, 6 in 2024-03-22T02:35:57.060 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:56 smithi122 podman[91961]: 2024-03-22 02:35:56.763318324 +0000 UTC m=+0.651665804 container died 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-22T02:35:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:56 smithi103 ceph-mon[115211]: osdmap e66: 6 total, 5 up, 6 in 2024-03-22T02:35:57.848 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 podman[91988]: 2024-03-22 02:35:57.451344739 +0000 UTC m=+0.677592423 container remove 8824bb568ef42ecb7302d5d1934a59957221b159d426b074ab69939386888cdf (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-22T02:35:57.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-22T02:35:57.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 systemd[1]: Stopped Ceph osd.3 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:35:57.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.3.service: Consumed 1min 15.656s CPU time. 2024-03-22T02:35:57.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 systemd[1]: Starting Ceph osd.3 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:35:58.205 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:57 smithi122 ceph-mon[83368]: pgmap v115: 65 pgs: 16 stale+active+clean, 49 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:58.206 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 podman[92092]: 2024-03-22 02:35:57.793037983 +0000 UTC m=+0.017420764 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:58.206 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:57 smithi122 podman[92092]: 2024-03-22 02:35:57.919647918 +0000 UTC m=+0.144030683 container create 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-22T02:35:58.206 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 podman[92092]: 2024-03-22 02:35:58.121933203 +0000 UTC m=+0.346315979 container init 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:35:58.206 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 podman[92092]: 2024-03-22 02:35:58.125318105 +0000 UTC m=+0.349700876 container start 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-22T02:35:58.206 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 podman[92092]: 2024-03-22 02:35:58.205146379 +0000 UTC m=+0.429529145 container attach 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-22T02:35:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:57 smithi103 ceph-mon[115211]: pgmap v115: 65 pgs: 16 stale+active+clean, 49 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 bash[92092]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-22T02:35:58.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T02:35:58.850 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T02:35:58.850 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-22T02:35:58.850 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T02:35:58.850 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate[92107]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-22T02:35:58.850 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:58 smithi122 podman[92092]: 2024-03-22 02:35:58.593837262 +0000 UTC m=+0.818220027 container died 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:35:59.185 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:58 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 64/321 objects degraded (19.938%), 18 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:58 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 64/321 objects degraded (19.938%), 18 pgs degraded (PG_DEGRADED) 2024-03-22T02:35:59.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 podman[92232]: 2024-03-22 02:35:59.184264322 +0000 UTC m=+0.585090192 container remove 0a58c6798346981beb5b52bfe7f11214af7389225cf15cb11bd02487d97c3c5b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T02:35:59.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 podman[92296]: 2024-03-22 02:35:59.303181715 +0000 UTC m=+0.017430952 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:35:59.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 podman[92296]: 2024-03-22 02:35:59.426982619 +0000 UTC m=+0.141231901 container create 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:35:59.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 podman[92296]: 2024-03-22 02:35:59.646156949 +0000 UTC m=+0.360406200 container init 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-22T02:35:59.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 podman[92296]: 2024-03-22 02:35:59.649319487 +0000 UTC m=+0.363568728 container start 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T02:35:59.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 ceph-osd[92315]: -- 172.21.15.122:0/416971461 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560bc3897680 con 0x560bc3861000 2024-03-22T02:35:59.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 bash[92296]: 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 2024-03-22T02:35:59.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:35:59 smithi122 systemd[1]: Started Ceph osd.3 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:36:00.348 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:36:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T02:36:00.193+0000 7f2dd5c0b7c0 -1 Falling back to public interface 2024-03-22T02:36:00.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:59 smithi122 ceph-mon[83368]: pgmap v116: 65 pgs: 18 active+undersized+degraded, 1 stale+active+clean, 16 active+undersized, 30 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 64/321 objects degraded (19.938%) 2024-03-22T02:36:00.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:59 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:00.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:59 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:00.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:35:59 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:59 smithi103 ceph-mon[115211]: pgmap v116: 65 pgs: 18 active+undersized+degraded, 1 stale+active+clean, 16 active+undersized, 30 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 64/321 objects degraded (19.938%) 2024-03-22T02:36:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:35:59 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:02.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:01 smithi122 ceph-mon[83368]: pgmap v117: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:02.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:02.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:01 smithi103 ceph-mon[115211]: pgmap v117: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:03 smithi122 ceph-mon[83368]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:03 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:03 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:03 smithi103 ceph-mon[115211]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:03 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:03 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:04.969 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:36:04 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T02:36:04.785+0000 7f2dd5c0b7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-22T02:36:05.320 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:05.320 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:05.320 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:05 smithi122 ceph-mon[83368]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:05.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:05 smithi103 ceph-mon[115211]: pgmap v119: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:05.598 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:36:05 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T02:36:05.318+0000 7f2dd5c0b7c0 -1 osd.3 64 log_to_monitors true 2024-03-22T02:36:06.343 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:06 smithi122 ceph-mon[83368]: from='osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:36:06.343 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:06 smithi122 ceph-mon[83368]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:36:06.344 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 02:36:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T02:36:06.103+0000 7f2dc7063700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:36:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:06 smithi103 ceph-mon[115211]: from='osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:36:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:06 smithi103 ceph-mon[115211]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:36:07.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:07 smithi122 ceph-mon[83368]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:36:07.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:07 smithi122 ceph-mon[83368]: from='osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:07.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:07 smithi122 ceph-mon[83368]: osdmap e67: 6 total, 5 up, 6 in 2024-03-22T02:36:07.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:07 smithi122 ceph-mon[83368]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:07.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:07 smithi122 ceph-mon[83368]: pgmap v121: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:07 smithi103 ceph-mon[115211]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:36:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:07 smithi103 ceph-mon[115211]: from='osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:07 smithi103 ceph-mon[115211]: osdmap e67: 6 total, 5 up, 6 in 2024-03-22T02:36:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:07 smithi103 ceph-mon[115211]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:07 smithi103 ceph-mon[115211]: pgmap v121: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 67/321 objects degraded (20.872%) 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991] boot 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: osdmap e68: 6 total, 6 up, 6 in 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:08.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:08.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:08 smithi122 ceph-mon[83368]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-22T02:36:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:36:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: osd.3 [v2:172.21.15.122:6800/2866011991,v1:172.21.15.122:6801/2866011991] boot 2024-03-22T02:36:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: osdmap e68: 6 total, 6 up, 6 in 2024-03-22T02:36:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:08.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:08 smithi103 ceph-mon[115211]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-22T02:36:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:09 smithi103 ceph-mon[115211]: osdmap e69: 6 total, 6 up, 6 in 2024-03-22T02:36:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:09 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 67/321 objects degraded (20.872%), 19 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:09 smithi103 ceph-mon[115211]: pgmap v124: 65 pgs: 8 peering, 12 active+undersized+degraded, 11 active+undersized, 34 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 46/321 objects degraded (14.330%) 2024-03-22T02:36:09.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:09 smithi122 ceph-mon[83368]: osdmap e69: 6 total, 6 up, 6 in 2024-03-22T02:36:09.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:09 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 67/321 objects degraded (20.872%), 19 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:09.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:09 smithi122 ceph-mon[83368]: pgmap v124: 65 pgs: 8 peering, 12 active+undersized+degraded, 11 active+undersized, 34 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 46/321 objects degraded (14.330%) 2024-03-22T02:36:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:11 smithi122 ceph-mon[83368]: pgmap v125: 65 pgs: 8 peering, 11 active+undersized+degraded, 9 active+undersized, 37 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/321 objects degraded (13.396%) 2024-03-22T02:36:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:11 smithi103 ceph-mon[115211]: pgmap v125: 65 pgs: 8 peering, 11 active+undersized+degraded, 9 active+undersized, 37 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/321 objects degraded (13.396%) 2024-03-22T02:36:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:12 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/321 objects degraded (13.396%), 11 pgs degraded) 2024-03-22T02:36:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:12 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:36:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:12 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/321 objects degraded (13.396%), 11 pgs degraded) 2024-03-22T02:36:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:12 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:36:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:13 smithi122 ceph-mon[83368]: pgmap v126: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:13 smithi103 ceph-mon[115211]: pgmap v126: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:15 smithi122 ceph-mon[83368]: pgmap v127: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:15 smithi103 ceph-mon[115211]: pgmap v127: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:17 smithi122 ceph-mon[83368]: pgmap v128: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:17 smithi103 ceph-mon[115211]: pgmap v128: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/1296277792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e441026c0 msgr2=0x7f6e440fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/1296277792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e441026c0 0x7f6e440fbd30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e34004940 tx=0x7f6e34009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/1296277792 shutdown_connections 2024-03-22T02:36:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/1296277792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e441026c0 0x7f6e440fbd30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/1296277792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e44101d20 0x7f6e441020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/1296277792 >> 172.21.15.103:0/1296277792 conn(0x7f6e440f7870 msgr2=0x7f6e440f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/1296277792 shutdown_connections 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.448+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/1296277792 wait complete. 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 Processor -- start 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 -- start start 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e441026c0 0x7f6e4410f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4410f5e0 con 0x7f6e44101d20 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4c2cd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4410f720 con 0x7f6e441026c0 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36922/0 (socket says 172.21.15.103:36922) 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 -- 172.21.15.103:0/4221916541 learned_addr learned my addr 172.21.15.103:0/4221916541 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:19.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 -- 172.21.15.103:0/4221916541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e441026c0 msgr2=0x7f6e4410f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e49868700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e441026c0 0x7f6e4410f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e441026c0 0x7f6e4410f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.449+0000 7f6e4a069700 1 -- 172.21.15.103:0/4221916541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e34005040 con 0x7f6e44101d20 2024-03-22T02:36:19.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e4a069700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e400044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e4000a980 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e44117bd0 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e4000ce60 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e400115e0 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.450+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e44117fa0 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.451+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e440fd470 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.451+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6e40012400 con 0x7f6e44101d20 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.452+0000 7f6e3affd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 0x7f6e30070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.452+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6e4008dd30 con 0x7f6e44101d20 2024-03-22T02:36:19.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.452+0000 7f6e49868700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 0x7f6e30070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.452+0000 7f6e49868700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 0x7f6e30070850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e34000c00 tx=0x7f6e34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:19.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.457+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6e400570f0 con 0x7f6e44101d20 2024-03-22T02:36:19.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.674+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e440668a0 con 0x7f6e3006e3a0 2024-03-22T02:36:19.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.675+0000 7f6e3affd700 1 -- 172.21.15.103:0/4221916541 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f6e440668a0 con 0x7f6e3006e3a0 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 msgr2=0x7f6e30070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 0x7f6e30070850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e34000c00 tx=0x7f6e34009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 msgr2=0x7f6e4410eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e400044c0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 shutdown_connections 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e3006e3a0 0x7f6e30070850 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e44101d20 0x7f6e4410eb60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 --2- 172.21.15.103:0/4221916541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e441026c0 0x7f6e4410f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 >> 172.21.15.103:0/4221916541 conn(0x7f6e440f7870 msgr2=0x7f6e440f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 shutdown_connections 2024-03-22T02:36:19.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.677+0000 7f6e4c2cd700 1 -- 172.21.15.103:0/4221916541 wait complete. 2024-03-22T02:36:19.689 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 -- 172.21.15.103:0/3115386313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c102660 msgr2=0x7f332c0fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 --2- 172.21.15.103:0/3115386313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c102660 0x7f332c0fc5b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3320004940 tx=0x7f3320009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 -- 172.21.15.103:0/3115386313 shutdown_connections 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 --2- 172.21.15.103:0/3115386313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c102660 0x7f332c0fc5b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 --2- 172.21.15.103:0/3115386313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c101d50 0x7f332c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 -- 172.21.15.103:0/3115386313 >> 172.21.15.103:0/3115386313 conn(0x7f332c0714c0 msgr2=0x7f332c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:19.781 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:19 smithi103 ceph-mon[115211]: pgmap v129: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 -- 172.21.15.103:0/3115386313 shutdown_connections 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.779+0000 7f3332d86700 1 -- 172.21.15.103:0/3115386313 wait complete. 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 Processor -- start 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 -- start start 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c102660 0x7f332c10f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c10f7c0 con 0x7f332c101d50 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3332d86700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c10f900 con 0x7f332c102660 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36940/0 (socket says 172.21.15.103:36940) 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 -- 172.21.15.103:0/467909040 learned_addr learned my addr 172.21.15.103:0/467909040 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:19.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f332bfff700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c102660 0x7f332c10f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 -- 172.21.15.103:0/467909040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c102660 msgr2=0x7f332c10f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c102660 0x7f332c10f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.780+0000 7f3330b22700 1 -- 172.21.15.103:0/467909040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320005040 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f3330b22700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f332400bbb0 tx=0x7f3324009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33240092e0 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c117c70 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f332400cbd0 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3324007400 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.781+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c118180 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.782+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c0fd5c0 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.783+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3324012070 con 0x7f332c101d50 2024-03-22T02:36:19.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.783+0000 7f33297fa700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 0x7f3314070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:19.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.783+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f332408d9e0 con 0x7f332c101d50 2024-03-22T02:36:19.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.783+0000 7f332bfff700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 0x7f3314070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:19.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.784+0000 7f332bfff700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 0x7f3314070850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f332000b3b0 tx=0x7f3320009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:19.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:19.787+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3324056d00 con 0x7f332c101d50 2024-03-22T02:36:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:19 smithi122 ceph-mon[83368]: pgmap v129: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:20.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.003+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f332c0fbc50 con 0x7f331406e3a0 2024-03-22T02:36:20.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.004+0000 7f33297fa700 1 -- 172.21.15.103:0/467909040 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f332c0fbc50 con 0x7f331406e3a0 2024-03-22T02:36:20.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 msgr2=0x7f3314070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 0x7f3314070850 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f332000b3b0 tx=0x7f3320009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 msgr2=0x7f332c10eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f332400bbb0 tx=0x7f3324009840 comp rx=0 tx=0).stop 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 shutdown_connections 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f331406e3a0 0x7f3314070850 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f332c101d50 0x7f332c10eba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 --2- 172.21.15.103:0/467909040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f332c102660 0x7f332c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 >> 172.21.15.103:0/467909040 conn(0x7f332c0714c0 msgr2=0x7f332c104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 shutdown_connections 2024-03-22T02:36:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.007+0000 7f3332d86700 1 -- 172.21.15.103:0/467909040 wait complete. 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- 172.21.15.103:0/1348814034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854102710 msgr2=0x7fd8540fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/1348814034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854102710 0x7fd8540fbd80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd844004940 tx=0x7fd844009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- 172.21.15.103:0/1348814034 shutdown_connections 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/1348814034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854102710 0x7fd8540fbd80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/1348814034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854101d70 0x7fd854102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- 172.21.15.103:0/1348814034 >> 172.21.15.103:0/1348814034 conn(0x7fd854072bc0 msgr2=0x7fd854072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- 172.21.15.103:0/1348814034 shutdown_connections 2024-03-22T02:36:20.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- 172.21.15.103:0/1348814034 wait complete. 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 Processor -- start 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.108+0000 7fd85b1cd700 1 -- start start 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd85b1cd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd85b1cd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 0x7fd85410ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd85b1cd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85410d430 con 0x7fd854101d70 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd85b1cd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85410d5a0 con 0x7fd854102710 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36960/0 (socket says 172.21.15.103:36960) 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 -- 172.21.15.103:0/3022954049 learned_addr learned my addr 172.21.15.103:0/3022954049 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd84bfff700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 0x7fd85410ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 -- 172.21.15.103:0/3022954049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 msgr2=0x7fd85410ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 0x7fd85410ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd858f69700 1 -- 172.21.15.103:0/3022954049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd844005040 con 0x7fd854101d70 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.109+0000 7fd84bfff700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 0x7fd85410ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd858f69700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd85000b790 tx=0x7fd850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd85000f770 con 0x7fd854101d70 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd85000fda0 con 0x7fd854101d70 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd85000d310 con 0x7fd854101d70 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd854119f90 con 0x7fd854101d70 2024-03-22T02:36:20.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.110+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85411a360 con 0x7fd854101d70 2024-03-22T02:36:20.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.111+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8540ff080 con 0x7fd854101d70 2024-03-22T02:36:20.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.111+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd850010420 con 0x7fd854101d70 2024-03-22T02:36:20.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.112+0000 7fd8497fa700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 0x7fd83c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.112+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd85008dcc0 con 0x7fd854101d70 2024-03-22T02:36:20.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.112+0000 7fd84bfff700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 0x7fd83c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.112+0000 7fd84bfff700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 0x7fd83c070850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd84400b3b0 tx=0x7fd844009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.116+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd85005ab10 con 0x7fd854101d70 2024-03-22T02:36:20.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.322+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd854108ec0 con 0x7fd83c06e3a0 2024-03-22T02:36:20.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.333+0000 7fd8497fa700 1 -- 172.21.15.103:0/3022954049 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd854108ec0 con 0x7fd83c06e3a0 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (4m) 51s ago 16m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (2m) 51s ago 16m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (2m) 17s ago 15m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (4m) 51s ago 15m 81.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (10m) 51s ago 10m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (10m) 51s ago 10m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (10m) 17s ago 10m 15.9M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (10m) 17s ago 10m 115M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (6m) 51s ago 17m 504M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (5m) 17s ago 15m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (3m) 51s ago 17m 44.0M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:36:20.335 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (3m) 17s ago 14m 38.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (5m) 51s ago 16m 15.3M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 17s ago 15m 18.3M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (2m) 51s ago 14m 87.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (93s) 51s ago 13m 98.1M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (57s) 51s ago 13m 12.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (21s) 17s ago 12m 12.2M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (12m) 17s ago 12m 230M 2581M 17.2.7-726-g9c4e0c34 837124da804f d3b87dd61748 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (12m) 17s ago 12m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:36:20.336 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (5m) 51s ago 15m 64.7M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 msgr2=0x7fd83c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 0x7fd83c070850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd84400b3b0 tx=0x7fd844009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 msgr2=0x7fd85410c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd85000b790 tx=0x7fd850009250 comp rx=0 tx=0).stop 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 shutdown_connections 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd83c06e3a0 0x7fd83c070850 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd854101d70 0x7fd85410c920 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 --2- 172.21.15.103:0/3022954049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd854102710 0x7fd85410ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 >> 172.21.15.103:0/3022954049 conn(0x7fd854072bc0 msgr2=0x7fd8540faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 shutdown_connections 2024-03-22T02:36:20.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.335+0000 7fd85b1cd700 1 -- 172.21.15.103:0/3022954049 wait complete. 2024-03-22T02:36:20.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 -- 172.21.15.103:0/4236693703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 msgr2=0x7f02200fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 --2- 172.21.15.103:0/4236693703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f02200fb990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0214004940 tx=0x7f0214009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 -- 172.21.15.103:0/4236693703 shutdown_connections 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 --2- 172.21.15.103:0/4236693703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f02200fb990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 --2- 172.21.15.103:0/4236693703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0220102fd0 0x7f02201033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 -- 172.21.15.103:0/4236693703 >> 172.21.15.103:0/4236693703 conn(0x7f0220071530 msgr2=0x7f0220071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 -- 172.21.15.103:0/4236693703 shutdown_connections 2024-03-22T02:36:20.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 -- 172.21.15.103:0/4236693703 wait complete. 2024-03-22T02:36:20.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.437+0000 7f0226035700 1 Processor -- start 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0226035700 1 -- start start 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0226035700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0226035700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0220102fd0 0x7f0220113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0226035700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220114520 con 0x7f02200fb540 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0226035700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022010db20 con 0x7f0220102fd0 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36976/0 (socket says 172.21.15.103:36976) 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 -- 172.21.15.103:0/1529523605 learned_addr learned my addr 172.21.15.103:0/1529523605 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 -- 172.21.15.103:0/1529523605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0220102fd0 msgr2=0x7f0220113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0220102fd0 0x7f0220113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 -- 172.21.15.103:0/1529523605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214005040 con 0x7f02200fb540 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f021f7fe700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0210002a50 tx=0x7f02100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.438+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0210007cc0 con 0x7f02200fb540 2024-03-22T02:36:20.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.439+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022010de00 con 0x7f02200fb540 2024-03-22T02:36:20.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.439+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021000f040 con 0x7f02200fb540 2024-03-22T02:36:20.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.439+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021000db70 con 0x7f02200fb540 2024-03-22T02:36:20.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.439+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022010e3c0 con 0x7f02200fb540 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.440+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02201053b0 con 0x7f02200fb540 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.440+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f021000dd10 con 0x7f02200fb540 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.440+0000 7f0207fff700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 0x7f0208070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.440+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f021008d450 con 0x7f02200fb540 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.440+0000 7f021effd700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 0x7f0208070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.441+0000 7f021effd700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 0x7f0208070850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0214004f90 tx=0x7f0214009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.445+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f021005a220 con 0x7f02200fb540 2024-03-22T02:36:20.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.723+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f022010a030 con 0x7f02200fb540 2024-03-22T02:36:20.725 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:20 smithi103 ceph-mon[115211]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.724+0000 7f0207fff700 1 -- 172.21.15.103:0/1529523605 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f0210059db0 con 0x7f02200fb540 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:36:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2, 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6, 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 8 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:36:20.727 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 msgr2=0x7f0208070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 0x7f0208070850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0214004f90 tx=0x7f0214009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 msgr2=0x7f0220113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0210002a50 tx=0x7f02100046b0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 shutdown_connections 2024-03-22T02:36:20.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f020806e3a0 0x7f0208070850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02200fb540 0x7f0220113990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 --2- 172.21.15.103:0/1529523605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0220102fd0 0x7f0220113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 >> 172.21.15.103:0/1529523605 conn(0x7f0220071530 msgr2=0x7f02200fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 shutdown_connections 2024-03-22T02:36:20.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.726+0000 7f0226035700 1 -- 172.21.15.103:0/1529523605 wait complete. 2024-03-22T02:36:20.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 -- 172.21.15.103:0/2859122009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 msgr2=0x7ffafc1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 --2- 172.21.15.103:0/2859122009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc1033a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0004940 tx=0x7ffaf0009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 -- 172.21.15.103:0/2859122009 shutdown_connections 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 --2- 172.21.15.103:0/2859122009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 0x7ffafc0fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 --2- 172.21.15.103:0/2859122009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc1033a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 -- 172.21.15.103:0/2859122009 >> 172.21.15.103:0/2859122009 conn(0x7ffafc071530 msgr2=0x7ffafc071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 -- 172.21.15.103:0/2859122009 shutdown_connections 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.827+0000 7ffb00e61700 1 -- 172.21.15.103:0/2859122009 wait complete. 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 Processor -- start 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 -- start start 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 0x7ffafc113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc1144b0 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffb00e61700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10dab0 con 0x7ffafc0fb540 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffaf9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffaf9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36990/0 (socket says 172.21.15.103:36990) 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffaf9d9b700 1 -- 172.21.15.103:0/3303988908 learned_addr learned my addr 172.21.15.103:0/3303988908 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.828+0000 7ffafa59c700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 0x7ffafc113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaf9d9b700 1 -- 172.21.15.103:0/3303988908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 msgr2=0x7ffafc113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaf9d9b700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 0x7ffafc113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaf9d9b700 1 -- 172.21.15.103:0/3303988908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0005040 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaf9d9b700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00c8c0 tx=0x7ffaec0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaec00a980 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc10dd90 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffaec00ce60 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaec0115e0 con 0x7ffafc102fd0 2024-03-22T02:36:20.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.829+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc10e350 con 0x7ffafc102fd0 2024-03-22T02:36:20.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.830+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc100270 con 0x7ffafc102fd0 2024-03-22T02:36:20.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.830+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ffaec012400 con 0x7ffafc102fd0 2024-03-22T02:36:20.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.831+0000 7ffaeaffd700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 0x7ffae4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:20.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.831+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ffaec08de30 con 0x7ffafc102fd0 2024-03-22T02:36:20.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.831+0000 7ffafa59c700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 0x7ffae4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:20.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.831+0000 7ffafa59c700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 0x7ffae4070850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b020 tx=0x7ffaf0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:20.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:20.835+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffaec05ac80 con 0x7ffafc102fd0 2024-03-22T02:36:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:20 smithi122 ceph-mon[83368]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.079+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffafc10ef70 con 0x7ffafc102fd0 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.080+0000 7ffaeaffd700 1 -- 172.21.15.103:0/3303988908 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7ffaec0126b0 con 0x7ffafc102fd0 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:36:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:36:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:21.084 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:21.085 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:36:21.085 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:21.085 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:21.085 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 msgr2=0x7ffae4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 0x7ffae4070850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b020 tx=0x7ffaf0009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 msgr2=0x7ffafc113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00c8c0 tx=0x7ffaec0044c0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 shutdown_connections 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffae406e3a0 0x7ffae4070850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffafc0fb540 0x7ffafc113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 --2- 172.21.15.103:0/3303988908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffafc102fd0 0x7ffafc113e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.082+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 >> 172.21.15.103:0/3303988908 conn(0x7ffafc071530 msgr2=0x7ffafc0fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.083+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 shutdown_connections 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.083+0000 7ffb00e61700 1 -- 172.21.15.103:0/3303988908 wait complete. 2024-03-22T02:36:21.086 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- 172.21.15.103:0/1710718748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 msgr2=0x7f19ac0fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/1710718748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac0fe1c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f199c004940 tx=0x7f199c009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- 172.21.15.103:0/1710718748 shutdown_connections 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/1710718748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0fe790 0x7f19ac106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/1710718748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac0fe1c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- 172.21.15.103:0/1710718748 >> 172.21.15.103:0/1710718748 conn(0x7f19ac0f78b0 msgr2=0x7f19ac0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- 172.21.15.103:0/1710718748 shutdown_connections 2024-03-22T02:36:21.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- 172.21.15.103:0/1710718748 wait complete. 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 Processor -- start 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- start start 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0fe790 0x7f19ac113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac114530 con 0x7f19ac0fddf0 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.185+0000 7f19b0bf4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac10db30 con 0x7f19ac0fe790 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37012/0 (socket says 172.21.15.103:37012) 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 -- 172.21.15.103:0/454344487 learned_addr learned my addr 172.21.15.103:0/454344487 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 -- 172.21.15.103:0/454344487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0fe790 msgr2=0x7f19ac113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0fe790 0x7f19ac113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 -- 172.21.15.103:0/454344487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c005040 con 0x7f19ac0fddf0 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19aa59c700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f199c009380 tx=0x7f199c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199c00e040 con 0x7f19ac0fddf0 2024-03-22T02:36:21.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f199c009af0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac10ddb0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199c01c630 con 0x7f19ac0fddf0 2024-03-22T02:36:21.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.186+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac10e2f0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.187+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac066ad0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.188+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f199c00bbc0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.188+0000 7f199affd700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 0x7f1994070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.188+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f199c08eff0 con 0x7f19ac0fddf0 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.188+0000 7f19a9d9b700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 0x7f1994070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.188+0000 7f19a9d9b700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 0x7f1994070850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19ac10ecf0 tx=0x7f19a0007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.193+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f199c05be40 con 0x7f19ac0fddf0 2024-03-22T02:36:21.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.410+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19ac109ea0 con 0x7f199406e3a0 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.411+0000 7f199affd700 1 -- 172.21.15.103:0/454344487 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f19ac109ea0 con 0x7f199406e3a0 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:36:21.413 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "10/21 daemons upgraded", 2024-03-22T02:36:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:36:21.414 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:36:21.414 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 msgr2=0x7f1994070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 0x7f1994070850 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19ac10ecf0 tx=0x7f19a0007d40 comp rx=0 tx=0).stop 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 msgr2=0x7f19ac1139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f199c009380 tx=0x7f199c009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 shutdown_connections 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199406e3a0 0x7f1994070850 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19ac0fddf0 0x7f19ac1139a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 --2- 172.21.15.103:0/454344487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19ac0fe790 0x7f19ac113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 >> 172.21.15.103:0/454344487 conn(0x7f19ac0f78b0 msgr2=0x7f19ac0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 shutdown_connections 2024-03-22T02:36:21.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.413+0000 7f19b0bf4700 1 -- 172.21.15.103:0/454344487 wait complete. 2024-03-22T02:36:21.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- 172.21.15.103:0/742742902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 msgr2=0x7f0eb01020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/742742902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb01020f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0004940 tx=0x7f0ea0009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- 172.21.15.103:0/742742902 shutdown_connections 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/742742902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb00fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/742742902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb01020f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- 172.21.15.103:0/742742902 >> 172.21.15.103:0/742742902 conn(0x7f0eb00f78b0 msgr2=0x7f0eb00f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- 172.21.15.103:0/742742902 shutdown_connections 2024-03-22T02:36:21.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- 172.21.15.103:0/742742902 wait complete. 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 Processor -- start 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.516+0000 7f0eb4b77700 1 -- start start 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb01145a0 con 0x7f0eb0101d20 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb010db10 con 0x7f0eb01026c0 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37022/0 (socket says 172.21.15.103:37022) 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 -- 172.21.15.103:0/3604261757 learned_addr learned my addr 172.21.15.103:0/3604261757 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0ea5d9b700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 -- 172.21.15.103:0/3604261757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 msgr2=0x7f0eb0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb0113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 -- 172.21.15.103:0/3604261757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea0005040 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0ea5d9b700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eae59c700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000b8b0 tx=0x7f0ea0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ea000e040 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ea0004290 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb010dd90 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb010e350 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.517+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ea001c3f0 con 0x7f0eb0101d20 2024-03-22T02:36:21.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.518+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb00fd470 con 0x7f0eb0101d20 2024-03-22T02:36:21.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.519+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0ea00137b0 con 0x7f0eb0101d20 2024-03-22T02:36:21.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.519+0000 7f0ea77fe700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 0x7f0e9c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.520+0000 7f0ea5d9b700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 0x7f0e9c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.520+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0ea008f060 con 0x7f0eb0101d20 2024-03-22T02:36:21.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.520+0000 7f0ea5d9b700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 0x7f0e9c070850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0eb010ecd0 tx=0x7f0e98008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.524+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ea005beb0 con 0x7f0eb0101d20 2024-03-22T02:36:21.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.802+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0eb0108e90 con 0x7f0eb0101d20 2024-03-22T02:36:21.804 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:21 smithi103 ceph-mon[115211]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:21.804 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:21 smithi103 ceph-mon[115211]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:21.804 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1529523605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:21.804 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:21 smithi103 ceph-mon[115211]: pgmap v130: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:21.805 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3303988908' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:36:21.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.802+0000 7f0ea77fe700 1 -- 172.21.15.103:0/3604261757 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f0ea005ba40 con 0x7f0eb0101d20 2024-03-22T02:36:21.805 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 msgr2=0x7f0e9c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 0x7f0e9c070850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0eb010ecd0 tx=0x7f0e98008040 comp rx=0 tx=0).stop 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 msgr2=0x7f0eb0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000b8b0 tx=0x7f0ea0009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 shutdown_connections 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0e9c06e3a0 0x7f0e9c070850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0eb0101d20 0x7f0eb0113980 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 --2- 172.21.15.103:0/3604261757 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0eb01026c0 0x7f0eb0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 >> 172.21.15.103:0/3604261757 conn(0x7f0eb00f78b0 msgr2=0x7f0eb00f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 shutdown_connections 2024-03-22T02:36:21.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:21.805+0000 7f0eb4b77700 1 -- 172.21.15.103:0/3604261757 wait complete. 2024-03-22T02:36:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:21 smithi122 ceph-mon[83368]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:21 smithi122 ceph-mon[83368]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1529523605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:21 smithi122 ceph-mon[83368]: pgmap v130: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3303988908' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:36:22.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:22 smithi122 ceph-mon[83368]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:22.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:22 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3604261757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:36:22.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:22 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:22 smithi103 ceph-mon[115211]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:22 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3604261757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:36:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:22 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:23.609 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:23 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:23.609 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:23 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:23.609 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:23 smithi122 ceph-mon[83368]: Upgrade: osd.4 is safe to restart 2024-03-22T02:36:23.609 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:23 smithi122 ceph-mon[83368]: pgmap v131: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:23 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:23 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T02:36:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:23 smithi103 ceph-mon[115211]: Upgrade: osd.4 is safe to restart 2024-03-22T02:36:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:23 smithi103 ceph-mon[115211]: pgmap v131: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: Upgrade: Updating osd.4 2024-03-22T02:36:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:36:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: Deploying daemon osd.4 on smithi122 2024-03-22T02:36:25.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:25 smithi122 ceph-mon[83368]: pgmap v132: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: Upgrade: Updating osd.4 2024-03-22T02:36:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:25.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:36:25.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:25.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: Deploying daemon osd.4 on smithi122 2024-03-22T02:36:25.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:25 smithi103 ceph-mon[115211]: pgmap v132: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:27 smithi122 ceph-mon[83368]: pgmap v133: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:28.099 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:27 smithi122 systemd[1]: Stopping Ceph osd.4 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:36:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:27 smithi103 ceph-mon[115211]: pgmap v133: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:28.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:28 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:36:28.121+0000 7f0367e14700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:36:28.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:28 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:36:28.121+0000 7f0367e14700 -1 osd.4 69 *** Got signal Terminated *** 2024-03-22T02:36:28.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:28 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[48361]: 2024-03-22T02:36:28.121+0000 7f0367e14700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:36:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:28 smithi122 ceph-mon[83368]: osd.4 marked itself down and dead 2024-03-22T02:36:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:28 smithi103 ceph-mon[115211]: osd.4 marked itself down and dead 2024-03-22T02:36:29.707 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:29 smithi122 podman[96696]: 2024-03-22 02:36:29.4276687 +0000 UTC m=+1.426569283 container died d3b87dd61748b403b0af91dedf2b5776df740f8942242cdf01126fe202f70661 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-22T02:36:30.074 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:29 smithi122 ceph-mon[83368]: pgmap v134: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:30.074 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:29 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:36:30.074 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:29 smithi122 ceph-mon[83368]: osdmap e70: 6 total, 5 up, 6 in 2024-03-22T02:36:30.074 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:29 smithi122 podman[96696]: 2024-03-22 02:36:29.814213428 +0000 UTC m=+1.813114008 container cleanup d3b87dd61748b403b0af91dedf2b5776df740f8942242cdf01126fe202f70661 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-22T02:36:30.075 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:29 smithi122 bash[96696]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4 2024-03-22T02:36:30.348 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96726]: 2024-03-22 02:36:30.073872006 +0000 UTC m=+0.640878498 container remove d3b87dd61748b403b0af91dedf2b5776df740f8942242cdf01126fe202f70661 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-22T02:36:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:29 smithi103 ceph-mon[115211]: pgmap v134: 65 pgs: 65 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:29 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:36:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:29 smithi103 ceph-mon[115211]: osdmap e70: 6 total, 5 up, 6 in 2024-03-22T02:36:30.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.290742046 +0000 UTC m=+0.017762530 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:36:30.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.41659821 +0000 UTC m=+0.143618681 container create f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-22T02:36:30.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.685920693 +0000 UTC m=+0.412941178 container init f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS) 2024-03-22T02:36:30.975 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:30 smithi122 ceph-mon[83368]: osdmap e71: 6 total, 5 up, 6 in 2024-03-22T02:36:30.975 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.689050212 +0000 UTC m=+0.416070692 container start f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:36:30.975 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.802406001 +0000 UTC m=+0.529426473 container attach f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T02:36:31.297 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:30 smithi122 podman[96819]: 2024-03-22 02:36:30.974602016 +0000 UTC m=+0.701622489 container died f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T02:36:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:30 smithi103 ceph-mon[115211]: osdmap e71: 6 total, 5 up, 6 in 2024-03-22T02:36:31.848 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 podman[96845]: 2024-03-22 02:36:31.55613361 +0000 UTC m=+0.576509954 container remove f7dea3905d5baee43689543c2f13548f8bd1d43bde0af279569494c318a4250b (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T02:36:31.849 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-22T02:36:31.849 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 systemd[1]: Stopped Ceph osd.4 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:36:31.849 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.4.service: Consumed 1min 2.918s CPU time. 2024-03-22T02:36:31.849 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 systemd[1]: Starting Ceph osd.4 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:36:32.175 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:31 smithi122 ceph-mon[83368]: pgmap v137: 65 pgs: 2 peering, 5 stale+active+clean, 58 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:32.176 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:31 smithi122 ceph-mon[83368]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T02:36:32.176 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:32.176 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:32.176 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:31 smithi122 podman[96951]: 2024-03-22 02:36:31.895700448 +0000 UTC m=+0.018834497 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:36:32.176 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 podman[96951]: 2024-03-22 02:36:32.04099048 +0000 UTC m=+0.164124574 container create 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:36:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:31 smithi103 ceph-mon[115211]: pgmap v137: 65 pgs: 2 peering, 5 stale+active+clean, 58 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:31 smithi103 ceph-mon[115211]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T02:36:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:32.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 podman[96951]: 2024-03-22 02:36:32.293542977 +0000 UTC m=+0.416677029 container init 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T02:36:32.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 podman[96951]: 2024-03-22 02:36:32.296713942 +0000 UTC m=+0.419848010 container start 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T02:36:32.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 podman[96951]: 2024-03-22 02:36:32.376715032 +0000 UTC m=+0.499849099 container attach 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:36:33.071 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 43/321 objects degraded (13.396%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 bash[96951]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-22T02:36:33.072 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T02:36:33.073 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T02:36:33.073 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-22T02:36:33.073 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T02:36:33.073 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate[96966]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-22T02:36:33.073 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:32 smithi122 podman[96951]: 2024-03-22 02:36:32.759478744 +0000 UTC m=+0.882612805 container died 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:36:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:32 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 43/321 objects degraded (13.396%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:33.665 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 podman[97086]: 2024-03-22 02:36:33.389056287 +0000 UTC m=+0.620816740 container remove 0e79b99f50cfef7cd6f2443d8542f364e5f0f4622001d63f821a4a9350889c96 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-22T02:36:33.665 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 podman[97149]: 2024-03-22 02:36:33.506195117 +0000 UTC m=+0.018785133 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:36:33.666 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 podman[97149]: 2024-03-22 02:36:33.665059705 +0000 UTC m=+0.177649707 container create d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T02:36:33.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 podman[97149]: 2024-03-22 02:36:33.867451275 +0000 UTC m=+0.380041288 container init d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:36:33.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 podman[97149]: 2024-03-22 02:36:33.870382139 +0000 UTC m=+0.382972143 container start d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:36:33.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 ceph-osd[97168]: -- 172.21.15.122:0/664739447 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cb9f5d680 con 0x561cb9f26c00 2024-03-22T02:36:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:33 smithi122 ceph-mon[83368]: pgmap v138: 65 pgs: 12 active+undersized+degraded, 2 peering, 1 stale+active+clean, 12 active+undersized, 38 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/321 objects degraded (13.396%) 2024-03-22T02:36:34.349 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 bash[97149]: d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 2024-03-22T02:36:34.349 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:33 smithi122 systemd[1]: Started Ceph osd.4 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:36:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:33 smithi103 ceph-mon[115211]: pgmap v138: 65 pgs: 12 active+undersized+degraded, 2 peering, 1 stale+active+clean, 12 active+undersized, 38 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/321 objects degraded (13.396%) 2024-03-22T02:36:34.848 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:34 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T02:36:34.413+0000 7fa43c6807c0 -1 Falling back to public interface 2024-03-22T02:36:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:35 smithi122 ceph-mon[83368]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:35 smithi103 ceph-mon[115211]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:36 smithi122 ceph-mon[83368]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T02:36:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:36 smithi103 ceph-mon[115211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T02:36:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:37 smithi122 ceph-mon[83368]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:37 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:37 smithi103 ceph-mon[115211]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:37 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:38.981 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:38 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T02:36:38.963+0000 7fa43c6807c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-22T02:36:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:39 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:39 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 53/321 objects degraded (16.511%), 16 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:39 smithi122 ceph-mon[83368]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:39 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:39 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 53/321 objects degraded (16.511%), 16 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:39 smithi103 ceph-mon[115211]: pgmap v141: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:39.757 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:39 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T02:36:39.575+0000 7fa43c6807c0 -1 osd.4 69 log_to_monitors true 2024-03-22T02:36:40.440 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:40 smithi122 ceph-mon[83368]: from='osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:36:40.440 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:40 smithi122 ceph-mon[83368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:36:40.440 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 02:36:40 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T02:36:40.191+0000 7fa42dad8700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:36:40.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:40 smithi103 ceph-mon[115211]: from='osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:36:40.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:40 smithi103 ceph-mon[115211]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:36:41.534 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:41 smithi122 ceph-mon[83368]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:36:41.534 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:41 smithi122 ceph-mon[83368]: osdmap e72: 6 total, 5 up, 6 in 2024-03-22T02:36:41.535 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:41 smithi122 ceph-mon[83368]: from='osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:41.535 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:41 smithi122 ceph-mon[83368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:41.535 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:41 smithi122 ceph-mon[83368]: pgmap v143: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:41.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:41 smithi103 ceph-mon[115211]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:36:41.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:41 smithi103 ceph-mon[115211]: osdmap e72: 6 total, 5 up, 6 in 2024-03-22T02:36:41.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:41 smithi103 ceph-mon[115211]: from='osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:41.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:41 smithi103 ceph-mon[115211]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:36:41.625 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:41 smithi103 ceph-mon[115211]: pgmap v143: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 53/321 objects degraded (16.511%) 2024-03-22T02:36:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471] boot 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: osdmap e73: 6 total, 6 up, 6 in 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: osd.4 [v2:172.21.15.122:6808/1009596471,v1:172.21.15.122:6809/1009596471] boot 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: osdmap e73: 6 total, 6 up, 6 in 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:42.600 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:43.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:43 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:43.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:43 smithi122 ceph-mon[83368]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-22T02:36:43.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:43 smithi122 ceph-mon[83368]: osdmap e74: 6 total, 6 up, 6 in 2024-03-22T02:36:43.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:43 smithi122 ceph-mon[83368]: pgmap v146: 65 pgs: 6 peering, 12 active+undersized+degraded, 12 active+undersized, 35 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:36:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:43 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:43 smithi103 ceph-mon[115211]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-22T02:36:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:43 smithi103 ceph-mon[115211]: osdmap e74: 6 total, 6 up, 6 in 2024-03-22T02:36:43.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:43 smithi103 ceph-mon[115211]: pgmap v146: 65 pgs: 6 peering, 12 active+undersized+degraded, 12 active+undersized, 35 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-22T02:36:44.586 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:44 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:44.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:44 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T02:36:45.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:45 smithi122 ceph-mon[83368]: pgmap v147: 65 pgs: 6 peering, 8 active+undersized+degraded, 8 active+undersized, 43 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/321 objects degraded (9.034%) 2024-03-22T02:36:45.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:45 smithi103 ceph-mon[115211]: pgmap v147: 65 pgs: 6 peering, 8 active+undersized+degraded, 8 active+undersized, 43 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29/321 objects degraded (9.034%) 2024-03-22T02:36:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:47 smithi122 ceph-mon[83368]: pgmap v148: 65 pgs: 6 peering, 59 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:47 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/321 objects degraded (9.034%), 8 pgs degraded) 2024-03-22T02:36:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:47 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:36:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:47 smithi103 ceph-mon[115211]: pgmap v148: 65 pgs: 6 peering, 59 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:48.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:36:48.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:47 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/321 objects degraded (9.034%), 8 pgs degraded) 2024-03-22T02:36:48.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:47 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:36:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:49 smithi122 ceph-mon[83368]: pgmap v149: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:49 smithi103 ceph-mon[115211]: pgmap v149: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:51.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.923+0000 7fdeb2328700 1 -- 172.21.15.103:0/3517556795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 msgr2=0x7fdeac103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.923+0000 7fdeb2328700 1 --2- 172.21.15.103:0/3517556795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac103320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004940 tx=0x7fde9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.923+0000 7fdeb2328700 1 -- 172.21.15.103:0/3517556795 shutdown_connections 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.923+0000 7fdeb2328700 1 --2- 172.21.15.103:0/3517556795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 0x7fdeac0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.923+0000 7fdeb2328700 1 --2- 172.21.15.103:0/3517556795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac103320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.924+0000 7fdeb2328700 1 -- 172.21.15.103:0/3517556795 >> 172.21.15.103:0/3517556795 conn(0x7fdeac0714c0 msgr2=0x7fdeac0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.924+0000 7fdeb2328700 1 -- 172.21.15.103:0/3517556795 shutdown_connections 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.924+0000 7fdeb2328700 1 -- 172.21.15.103:0/3517556795 wait complete. 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.924+0000 7fdeb2328700 1 Processor -- start 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.924+0000 7fdeb2328700 1 -- start start 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 0x7fdeac0fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0fe690 con 0x7fdeac102f50 2024-03-22T02:36:51.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac0fd130 con 0x7fdeac0fb4d0 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55946/0 (socket says 172.21.15.103:55946) 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 -- 172.21.15.103:0/195901217 learned_addr learned my addr 172.21.15.103:0/195901217 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeabfff700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 0x7fdeac0fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 -- 172.21.15.103:0/195901217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 msgr2=0x7fdeac0fc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 0x7fdeac0fc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 -- 172.21.15.103:0/195901217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c005040 con 0x7fdeac102f50 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeab7fe700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdea000b790 tx=0x7fdea0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea000f6f0 con 0x7fdeac102f50 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac0fd410 con 0x7fdeac102f50 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdea000fd20 con 0x7fdeac102f50 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea00103a0 con 0x7fdeac102f50 2024-03-22T02:36:51.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.925+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac11a240 con 0x7fdeac102f50 2024-03-22T02:36:51.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.926+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac100200 con 0x7fdeac102f50 2024-03-22T02:36:51.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.927+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fdea0011400 con 0x7fdeac102f50 2024-03-22T02:36:51.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.927+0000 7fdea8ff9700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 0x7fde94070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:51.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.927+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fdea008db40 con 0x7fdeac102f50 2024-03-22T02:36:51.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.927+0000 7fdeabfff700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 0x7fde94070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:51.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.928+0000 7fdeabfff700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 0x7fde94070850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004890 tx=0x7fde9c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:51.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:51.932+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdea0056e60 con 0x7fdeac102f50 2024-03-22T02:36:52.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.147+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdeac0fdb90 con 0x7fde9406e3a0 2024-03-22T02:36:52.148 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:51 smithi103 ceph-mon[115211]: pgmap v150: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:52.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.148+0000 7fdea8ff9700 1 -- 172.21.15.103:0/195901217 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fdeac0fdb90 con 0x7fde9406e3a0 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 msgr2=0x7fde94070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 0x7fde94070850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004890 tx=0x7fde9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 msgr2=0x7fdeac0fcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdea000b790 tx=0x7fdea0009250 comp rx=0 tx=0).stop 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 shutdown_connections 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde9406e3a0 0x7fde94070850 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdeac0fb4d0 0x7fdeac0fc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 --2- 172.21.15.103:0/195901217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeac102f50 0x7fdeac0fcbf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 >> 172.21.15.103:0/195901217 conn(0x7fdeac0714c0 msgr2=0x7fdeac0fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 shutdown_connections 2024-03-22T02:36:52.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.151+0000 7fdeb2328700 1 -- 172.21.15.103:0/195901217 wait complete. 2024-03-22T02:36:52.162 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.253+0000 7f244b935700 1 -- 172.21.15.103:0/1861348165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2444101d20 msgr2=0x7f24441020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 --2- 172.21.15.103:0/1861348165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2444101d20 0x7f24441020f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2440004940 tx=0x7f2440009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- 172.21.15.103:0/1861348165 shutdown_connections 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 --2- 172.21.15.103:0/1861348165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24441026c0 0x7f24440fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 --2- 172.21.15.103:0/1861348165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2444101d20 0x7f24441020f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- 172.21.15.103:0/1861348165 >> 172.21.15.103:0/1861348165 conn(0x7f24440f7870 msgr2=0x7f24440f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- 172.21.15.103:0/1861348165 shutdown_connections 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- 172.21.15.103:0/1861348165 wait complete. 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 Processor -- start 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- start start 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2444101d20 0x7f2444115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2444116790 con 0x7f24441026c0 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.254+0000 7f244b935700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244410fd70 con 0x7f2444101d20 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55962/0 (socket says 172.21.15.103:55962) 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 -- 172.21.15.103:0/4294730793 learned_addr learned my addr 172.21.15.103:0/4294730793 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f24496d1700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2444101d20 0x7f2444115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 -- 172.21.15.103:0/4294730793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2444101d20 msgr2=0x7f2444115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2444101d20 0x7f2444115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 -- 172.21.15.103:0/4294730793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440005040 con 0x7f24441026c0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2448ed0700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2434002a80 tx=0x7f2434004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2434004e80 con 0x7f24441026c0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2444110050 con 0x7f24441026c0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f243400f040 con 0x7f24441026c0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f243400daa0 con 0x7f24441026c0 2024-03-22T02:36:52.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.255+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2444110610 con 0x7f24441026c0 2024-03-22T02:36:52.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.256+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24440fd470 con 0x7f24441026c0 2024-03-22T02:36:52.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.257+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2434007cc0 con 0x7f24441026c0 2024-03-22T02:36:52.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.257+0000 7f2439ffb700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 0x7f2430070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.257+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f243408d3b0 con 0x7f24441026c0 2024-03-22T02:36:52.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.257+0000 7f24496d1700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 0x7f2430070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.257+0000 7f24496d1700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 0x7f2430070850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f244000b3b0 tx=0x7f2440009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.262+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f243405a180 con 0x7f24441026c0 2024-03-22T02:36:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:51 smithi122 ceph-mon[83368]: pgmap v150: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:52.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.476+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2444108e40 con 0x7f243006e3a0 2024-03-22T02:36:52.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.478+0000 7f2439ffb700 1 -- 172.21.15.103:0/4294730793 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2444108e40 con 0x7f243006e3a0 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 msgr2=0x7f2430070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 0x7f2430070850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f244000b3b0 tx=0x7f2440009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 msgr2=0x7f2444116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2434002a80 tx=0x7f2434004630 comp rx=0 tx=0).stop 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 shutdown_connections 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f243006e3a0 0x7f2430070850 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2444101d20 0x7f2444115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 --2- 172.21.15.103:0/4294730793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24441026c0 0x7f2444116140 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 >> 172.21.15.103:0/4294730793 conn(0x7f24440f7870 msgr2=0x7f24440f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 shutdown_connections 2024-03-22T02:36:52.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.480+0000 7f244b935700 1 -- 172.21.15.103:0/4294730793 wait complete. 2024-03-22T02:36:52.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/3692997765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf0102e20 msgr2=0x7f1bf01031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/3692997765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf0102e20 0x7f1bf01031f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1be4004940 tx=0x7f1be4009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/3692997765 shutdown_connections 2024-03-22T02:36:52.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/3692997765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf01037c0 0x7f1bf00fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/3692997765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf0102e20 0x7f1bf01031f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/3692997765 >> 172.21.15.103:0/3692997765 conn(0x7f1bf00f89b0 msgr2=0x7f1bf00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/3692997765 shutdown_connections 2024-03-22T02:36:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/3692997765 wait complete. 2024-03-22T02:36:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.581+0000 7f1bf6ccc700 1 Processor -- start 2024-03-22T02:36:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1bf6ccc700 1 -- start start 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1bf6ccc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf0102e20 0x7f1bf0113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1bf6ccc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1bf6ccc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf01144a0 con 0x7f1bf01037c0 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1bf6ccc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf010daf0 con 0x7f1bf0102e20 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55972/0 (socket says 172.21.15.103:55972) 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 -- 172.21.15.103:0/725195850 learned_addr learned my addr 172.21.15.103:0/725195850 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 -- 172.21.15.103:0/725195850 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf0102e20 msgr2=0x7f1bf0113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf0102e20 0x7f1bf0113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 -- 172.21.15.103:0/725195850 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be4005040 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.582+0000 7f1beffff700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00eb40 tx=0x7f1bdc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.583+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bdc007cc0 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.583+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf010dd70 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.583+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bdc00f040 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.583+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bdc0095e0 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.583+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf010e2b0 con 0x7f1bf01037c0 2024-03-22T02:36:52.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.584+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf0100130 con 0x7f1bf01037c0 2024-03-22T02:36:52.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.584+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1bdc009740 con 0x7f1bf01037c0 2024-03-22T02:36:52.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.584+0000 7f1bed7fa700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 0x7f1be0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.585+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1bdc08d800 con 0x7f1bf01037c0 2024-03-22T02:36:52.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.585+0000 7f1bf4a68700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 0x7f1be0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.585+0000 7f1bf4a68700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 0x7f1be0070850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009b30 tx=0x7f1be4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.589+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1bdc05a680 con 0x7f1bf01037c0 2024-03-22T02:36:52.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.796+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bf010ef60 con 0x7f1be006e3a0 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.806+0000 7f1bed7fa700 1 -- 172.21.15.103:0/725195850 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1bf010ef60 con 0x7f1be006e3a0 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (5m) 84s ago 16m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (3m) 84s ago 16m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (3m) 15s ago 15m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (5m) 84s ago 16m 81.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (11m) 84s ago 11m 14.1M - 17.2.7-726-g9c4e0c34 837124da804f 61c69cd3b420 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (10m) 84s ago 10m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:36:52.808 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (10m) 15s ago 10m 16.0M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (11m) 15s ago 11m 115M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (6m) 84s ago 18m 504M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (6m) 15s ago 15m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (4m) 84s ago 18m 44.0M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (3m) 15s ago 15m 38.8M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (6m) 84s ago 16m 15.3M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 15s ago 15m 18.6M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (2m) 84s ago 14m 87.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (2m) 84s ago 14m 98.1M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (89s) 84s ago 13m 12.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (53s) 15s ago 13m 95.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (19s) 15s ago 13m 12.6M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (12m) 15s ago 12m 220M 2581M 17.2.7-726-g9c4e0c34 837124da804f be4dcf48beb9 2024-03-22T02:36:52.809 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (5m) 84s ago 16m 64.7M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 msgr2=0x7f1be0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 0x7f1be0070850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009b30 tx=0x7f1be4009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 msgr2=0x7f1bf0113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00eb40 tx=0x7f1bdc00b040 comp rx=0 tx=0).stop 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 shutdown_connections 2024-03-22T02:36:52.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1be006e3a0 0x7f1be0070850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bf0102e20 0x7f1bf0113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 --2- 172.21.15.103:0/725195850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bf01037c0 0x7f1bf0113e50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 >> 172.21.15.103:0/725195850 conn(0x7f1bf00f89b0 msgr2=0x7f1bf00fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 shutdown_connections 2024-03-22T02:36:52.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.809+0000 7f1bf6ccc700 1 -- 172.21.15.103:0/725195850 wait complete. 2024-03-22T02:36:52.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 -- 172.21.15.103:0/3721441579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 msgr2=0x7fde900fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 --2- 172.21.15.103:0/3721441579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde900fbd20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fde78004940 tx=0x7fde78009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 -- 172.21.15.103:0/3721441579 shutdown_connections 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 --2- 172.21.15.103:0/3721441579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde900fbd20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 --2- 172.21.15.103:0/3721441579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 0x7fde901020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 -- 172.21.15.103:0/3721441579 >> 172.21.15.103:0/3721441579 conn(0x7fde900f7880 msgr2=0x7fde900f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 -- 172.21.15.103:0/3721441579 shutdown_connections 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.910+0000 7fde964f7700 1 -- 172.21.15.103:0/3721441579 wait complete. 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 Processor -- start 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 -- start start 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 0x7fde90113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde90114a20 con 0x7fde901026b0 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde964f7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9010ddf0 con 0x7fde90101d10 2024-03-22T02:36:52.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55998/0 (socket says 172.21.15.103:55998) 2024-03-22T02:36:52.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 -- 172.21.15.103:0/40006881 learned_addr learned my addr 172.21.15.103:0/40006881 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde8ffff700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 0x7fde90113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 -- 172.21.15.103:0/40006881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 msgr2=0x7fde90113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 0x7fde90113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 -- 172.21.15.103:0/40006881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78005040 con 0x7fde901026b0 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.911+0000 7fde87fff700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fde78009100 tx=0x7fde78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.912+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde7800e040 con 0x7fde901026b0 2024-03-22T02:36:52.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.912+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde78013420 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.912+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9010e070 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.912+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde7801c610 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.912+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9010e630 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.913+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900fd460 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.913+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fde78013590 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.913+0000 7fde8d7fa700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 0x7fde7c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.913+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fde7808eff0 con 0x7fde901026b0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.914+0000 7fde8ffff700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 0x7fde7c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.914+0000 7fde8ffff700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 0x7fde7c070850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde80007fd0 tx=0x7fde80009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:52.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:52.918+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde7805be40 con 0x7fde901026b0 2024-03-22T02:36:53.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.200+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fde9004bed0 con 0x7fde901026b0 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.201+0000 7fde8d7fa700 1 -- 172.21.15.103:0/40006881 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fde7805b9d0 con 0x7fde901026b0 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:36:53.202 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 5 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 5, 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 9 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:36:53.203 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:36:53.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 msgr2=0x7fde7c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 0x7fde7c070850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde80007fd0 tx=0x7fde80009040 comp rx=0 tx=0).stop 2024-03-22T02:36:53.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 msgr2=0x7fde901143d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fde78009100 tx=0x7fde78009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 shutdown_connections 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde7c06e3a0 0x7fde7c070850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde90101d10 0x7fde90113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 --2- 172.21.15.103:0/40006881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde901026b0 0x7fde901143d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.203+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 >> 172.21.15.103:0/40006881 conn(0x7fde900f7880 msgr2=0x7fde900f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.204+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 shutdown_connections 2024-03-22T02:36:53.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.204+0000 7fde964f7700 1 -- 172.21.15.103:0/40006881 wait complete. 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- 172.21.15.103:0/4117690466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fddc0 msgr2=0x7fc1740fe190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 --2- 172.21.15.103:0/4117690466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fddc0 0x7fc1740fe190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc164004940 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- 172.21.15.103:0/4117690466 shutdown_connections 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 --2- 172.21.15.103:0/4117690466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fe6d0 0x7fc174106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 --2- 172.21.15.103:0/4117690466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fddc0 0x7fc1740fe190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- 172.21.15.103:0/4117690466 >> 172.21.15.103:0/4117690466 conn(0x7fc1740f7820 msgr2=0x7fc1740f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- 172.21.15.103:0/4117690466 shutdown_connections 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- 172.21.15.103:0/4117690466 wait complete. 2024-03-22T02:36:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 Processor -- start 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- start start 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fddc0 0x7fc174113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1741146f0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.305+0000 7fc1788b1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17410dae0 con 0x7fc1740fddc0 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56026/0 (socket says 172.21.15.103:56026) 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 -- 172.21.15.103:0/2516123606 learned_addr learned my addr 172.21.15.103:0/2516123606 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc17259c700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fddc0 0x7fc174113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 -- 172.21.15.103:0/2516123606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fddc0 msgr2=0x7fc174113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fddc0 0x7fc174113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 -- 172.21.15.103:0/2516123606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc1740fe6d0 2024-03-22T02:36:53.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc171d9b700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc168002a50 tx=0x7fc1680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc168007cc0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc16800f040 con 0x7fc1740fe6d0 2024-03-22T02:36:53.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17410ddc0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc16800da30 con 0x7fc1740fe6d0 2024-03-22T02:36:53.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.306+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17410e300 con 0x7fc1740fe6d0 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.307+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1540052b0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.308+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc16800dbb0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.308+0000 7fc162ffd700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 0x7fc15c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.308+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc16808d950 con 0x7fc1740fe6d0 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.308+0000 7fc17259c700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 0x7fc15c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.309+0000 7fc17259c700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 0x7fc15c070850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc164004f90 tx=0x7fc164009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.313+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc16805a7a0 con 0x7fc1740fe6d0 2024-03-22T02:36:53.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.556+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc154005d70 con 0x7fc1740fe6d0 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.556+0000 7fc162ffd700 1 -- 172.21.15.103:0/2516123606 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1909 (secure 0 0 0) 0x7fc16805a330 con 0x7fc1740fe6d0 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout:e21 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:53.557 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 21 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:32:54.641310+0000 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:36:53.558 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 46 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=14472,1=24271} 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:36:53.559 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 14472 members: 24271,14472 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{0:14472} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6828/2193581075,v1:172.21.15.103:6829/2193581075] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{1:24271} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/1237228570,v1:172.21.15.122:6825/1237228570] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/3867073217,v1:172.21.15.103:6827/3867073217] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:53.560 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:36:53.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 msgr2=0x7fc15c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 0x7fc15c070850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc164004f90 tx=0x7fc164009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 msgr2=0x7fc1741140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc168002a50 tx=0x7fc1680046b0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 shutdown_connections 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc15c06e3a0 0x7fc15c070850 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1740fddc0 0x7fc174113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 --2- 172.21.15.103:0/2516123606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1740fe6d0 0x7fc1741140a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 >> 172.21.15.103:0/2516123606 conn(0x7fc1740f7820 msgr2=0x7fc1740f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 shutdown_connections 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.558+0000 7fc1788b1700 1 -- 172.21.15.103:0/2516123606 wait complete. 2024-03-22T02:36:53.562 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 -- 172.21.15.103:0/1400527630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8102680 msgr2=0x7efca80fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 --2- 172.21.15.103:0/1400527630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8102680 0x7efca80fbcf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc94004940 tx=0x7efc94009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 -- 172.21.15.103:0/1400527630 shutdown_connections 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 --2- 172.21.15.103:0/1400527630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8102680 0x7efca80fbcf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 --2- 172.21.15.103:0/1400527630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8101ce0 0x7efca81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 -- 172.21.15.103:0/1400527630 >> 172.21.15.103:0/1400527630 conn(0x7efca80f7870 msgr2=0x7efca80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 -- 172.21.15.103:0/1400527630 shutdown_connections 2024-03-22T02:36:53.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.660+0000 7efcaeaca700 1 -- 172.21.15.103:0/1400527630 wait complete. 2024-03-22T02:36:53.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 Processor -- start 2024-03-22T02:36:53.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 -- start start 2024-03-22T02:36:53.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8102680 0x7efca8103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca8104b20 con 0x7efca8101ce0 2024-03-22T02:36:53.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcaeaca700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81035c0 con 0x7efca8102680 2024-03-22T02:36:53.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56050/0 (socket says 172.21.15.103:56050) 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 -- 172.21.15.103:0/339107576 learned_addr learned my addr 172.21.15.103:0/339107576 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efca7fff700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8102680 0x7efca8103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 -- 172.21.15.103:0/339107576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8102680 msgr2=0x7efca8103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8102680 0x7efca8103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.661+0000 7efcac866700 1 -- 172.21.15.103:0/339107576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94005040 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efcac866700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b790 tx=0x7efc9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c00f770 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc9c00fda0 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca81038a0 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc9c00d310 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.662+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca811a1e0 con 0x7efca8101ce0 2024-03-22T02:36:53.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.663+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca80fd430 con 0x7efca8101ce0 2024-03-22T02:36:53.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.663+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efc9c010420 con 0x7efca8101ce0 2024-03-22T02:36:53.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.664+0000 7efca57fa700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 0x7efc98070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.664+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7efc9c08dd80 con 0x7efca8101ce0 2024-03-22T02:36:53.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.664+0000 7efca7fff700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 0x7efc98070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.664+0000 7efca7fff700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 0x7efc98070850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b3b0 tx=0x7efc94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.668+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc9c0570a0 con 0x7efca8101ce0 2024-03-22T02:36:53.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.883+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efca8104090 con 0x7efc9806e3a0 2024-03-22T02:36:53.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.884+0000 7efca57fa700 1 -- 172.21.15.103:0/339107576 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7efca8104090 con 0x7efc9806e3a0 2024-03-22T02:36:53.884 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "11/21 daemons upgraded", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:36:53.885 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 msgr2=0x7efc98070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 0x7efc98070850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b3b0 tx=0x7efc94009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 msgr2=0x7efca8102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b790 tx=0x7efc9c009250 comp rx=0 tx=0).stop 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 shutdown_connections 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9806e3a0 0x7efc98070850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca8101ce0 0x7efca8102b40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 --2- 172.21.15.103:0/339107576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca8102680 0x7efca8103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 >> 172.21.15.103:0/339107576 conn(0x7efca80f7870 msgr2=0x7efca80f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 shutdown_connections 2024-03-22T02:36:53.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.886+0000 7efcaeaca700 1 -- 172.21.15.103:0/339107576 wait complete. 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 -- 172.21.15.103:0/4235284539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 msgr2=0x7efc080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 --2- 172.21.15.103:0/4235284539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc080fce10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efbf8004940 tx=0x7efbf8009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 -- 172.21.15.103:0/4235284539 shutdown_connections 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 --2- 172.21.15.103:0/4235284539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc080fce10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 --2- 172.21.15.103:0/4235284539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 0x7efc081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 -- 172.21.15.103:0/4235284539 >> 172.21.15.103:0/4235284539 conn(0x7efc080f89b0 msgr2=0x7efc080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 -- 172.21.15.103:0/4235284539 shutdown_connections 2024-03-22T02:36:53.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.988+0000 7efc0fe60700 1 -- 172.21.15.103:0/4235284539 wait complete. 2024-03-22T02:36:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 Processor -- start 2024-03-22T02:36:53.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 -- start start 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 0x7efc08113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08114700 con 0x7efc081037a0 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0fe60700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0810daf0 con 0x7efc08102e00 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50514/0 (socket says 172.21.15.103:50514) 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 -- 172.21.15.103:0/3508763214 learned_addr learned my addr 172.21.15.103:0/3508763214 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0dbfc700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 0x7efc08113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 -- 172.21.15.103:0/3508763214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 msgr2=0x7efc08113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 0x7efc08113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 -- 172.21.15.103:0/3508763214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8005040 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.989+0000 7efc0d3fb700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009100 tx=0x7efbf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.990+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf800e040 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.990+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbf8009af0 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.990+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0810dd70 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.990+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf801c3f0 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.990+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0810e330 con 0x7efc081037a0 2024-03-22T02:36:53.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.991+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc081052d0 con 0x7efc081037a0 2024-03-22T02:36:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.991+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efbf800baf0 con 0x7efc081037a0 2024-03-22T02:36:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.991+0000 7efbfe7fc700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 0x7efbf4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:53.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.992+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6324+0+0 (secure 0 0 0) 0x7efbf808eeb0 con 0x7efc081037a0 2024-03-22T02:36:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.992+0000 7efc0dbfc700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 0x7efbf4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:53.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.992+0000 7efc0dbfc700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 0x7efbf4070850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efc04004eb0 tx=0x7efc0400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:53.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:53.996+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efbf805bda0 con 0x7efc081037a0 2024-03-22T02:36:54.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.285+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efc080fe550 con 0x7efc081037a0 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: pgmap v151: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/40006881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:54.287 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:53 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2516123606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:36:54.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.285+0000 7efbfe7fc700 1 -- 172.21.15.103:0/3508763214 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7efbf805b930 con 0x7efc081037a0 2024-03-22T02:36:54.288 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 msgr2=0x7efbf4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 0x7efbf4070850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efc04004eb0 tx=0x7efc0400b040 comp rx=0 tx=0).stop 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 msgr2=0x7efc081140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009100 tx=0x7efbf8009e30 comp rx=0 tx=0).stop 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 shutdown_connections 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efbf406e3a0 0x7efbf4070850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc08102e00 0x7efc08113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 --2- 172.21.15.103:0/3508763214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc081037a0 0x7efc081140b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 >> 172.21.15.103:0/3508763214 conn(0x7efc080f89b0 msgr2=0x7efc08102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 shutdown_connections 2024-03-22T02:36:54.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:36:54.287+0000 7efc0fe60700 1 -- 172.21.15.103:0/3508763214 wait complete. 2024-03-22T02:36:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:54.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: pgmap v151: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:54.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/40006881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:36:54.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:53 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2516123606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:36:55.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:54 smithi103 ceph-mon[115211]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:55.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:54 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3508763214' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:36:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:54 smithi122 ceph-mon[83368]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:36:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:54 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3508763214' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:36:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:55 smithi122 ceph-mon[83368]: pgmap v152: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:55 smithi103 ceph-mon[115211]: pgmap v152: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:57 smithi122 ceph-mon[83368]: pgmap v153: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:57 smithi103 ceph-mon[115211]: pgmap v153: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:36:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:59.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:58 smithi122 ceph-mon[83368]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:59.238 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:58 smithi122 ceph-mon[83368]: Upgrade: osd.5 is safe to restart 2024-03-22T02:36:59.238 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:59.238 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:36:59.238 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:58 smithi103 ceph-mon[115211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T02:36:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:58 smithi103 ceph-mon[115211]: Upgrade: osd.5 is safe to restart 2024-03-22T02:36:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:36:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:36:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:00.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:59 smithi122 ceph-mon[83368]: Upgrade: Updating osd.5 2024-03-22T02:37:00.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:59 smithi122 ceph-mon[83368]: Deploying daemon osd.5 on smithi122 2024-03-22T02:37:00.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:36:59 smithi122 ceph-mon[83368]: pgmap v154: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:59 smithi103 ceph-mon[115211]: Upgrade: Updating osd.5 2024-03-22T02:37:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:59 smithi103 ceph-mon[115211]: Deploying daemon osd.5 on smithi122 2024-03-22T02:37:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:36:59 smithi103 ceph-mon[115211]: pgmap v154: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:02.125 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:01 smithi122 systemd[1]: Stopping Ceph osd.5 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:37:02.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:01 smithi122 ceph-mon[83368]: pgmap v155: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:02.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:02.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:01 smithi103 ceph-mon[115211]: pgmap v155: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:02.598 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[54950]: 2024-03-22T02:37:02.187+0000 7f981827c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:37:02.598 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[54950]: 2024-03-22T02:37:02.187+0000 7f981827c700 -1 osd.5 74 *** Got signal Terminated *** 2024-03-22T02:37:02.599 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[54950]: 2024-03-22T02:37:02.187+0000 7f981827c700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:37:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:02 smithi122 ceph-mon[83368]: osd.5 marked itself down and dead 2024-03-22T02:37:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:02 smithi103 ceph-mon[115211]: osd.5 marked itself down and dead 2024-03-22T02:37:03.727 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:03 smithi122 podman[101959]: 2024-03-22 02:37:03.449905143 +0000 UTC m=+1.440664925 container died be4dcf48beb911b735d8c68b1e1bb746ff189b4c7bc198b09a9e755fe0188223 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, RELEASE=quincy-9c4e0c3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T02:37:04.095 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:03 smithi122 podman[101959]: 2024-03-22 02:37:03.826091918 +0000 UTC m=+1.816851708 container cleanup be4dcf48beb911b735d8c68b1e1bb746ff189b4c7bc198b09a9e755fe0188223 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T02:37:04.095 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:03 smithi122 bash[101959]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5 2024-03-22T02:37:04.095 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:03 smithi122 ceph-mon[83368]: pgmap v156: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:04.095 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:03 smithi122 ceph-mon[83368]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:37:04.095 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:03 smithi122 ceph-mon[83368]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-22T02:37:04.096 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:03 smithi122 ceph-mon[83368]: osdmap e75: 6 total, 5 up, 6 in 2024-03-22T02:37:04.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[101988]: 2024-03-22 02:37:04.094424964 +0000 UTC m=+0.642550341 container remove be4dcf48beb911b735d8c68b1e1bb746ff189b4c7bc198b09a9e755fe0188223 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, RELEASE=quincy-9c4e0c3, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T02:37:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:03 smithi103 ceph-mon[115211]: pgmap v156: 65 pgs: 65 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:03 smithi103 ceph-mon[115211]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T02:37:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:03 smithi103 ceph-mon[115211]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-22T02:37:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:03 smithi103 ceph-mon[115211]: osdmap e75: 6 total, 5 up, 6 in 2024-03-22T02:37:04.718 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[102082]: 2024-03-22 02:37:04.336224838 +0000 UTC m=+0.018153990 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:37:04.718 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[102082]: 2024-03-22 02:37:04.445275972 +0000 UTC m=+0.127205122 container create 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-22T02:37:04.719 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[102082]: 2024-03-22 02:37:04.718309395 +0000 UTC m=+0.400238550 container init 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-22T02:37:05.006 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:04 smithi122 ceph-mon[83368]: osdmap e76: 6 total, 5 up, 6 in 2024-03-22T02:37:05.006 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[102082]: 2024-03-22 02:37:04.721683713 +0000 UTC m=+0.403612861 container start 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T02:37:05.006 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:04 smithi122 podman[102082]: 2024-03-22 02:37:04.822764088 +0000 UTC m=+0.504693246 container attach 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-22T02:37:05.301 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 podman[102082]: 2024-03-22 02:37:05.006344936 +0000 UTC m=+0.688274086 container died 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:37:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:04 smithi103 ceph-mon[115211]: osdmap e76: 6 total, 5 up, 6 in 2024-03-22T02:37:05.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 podman[102108]: 2024-03-22 02:37:05.576194444 +0000 UTC m=+0.564227995 container remove 5053e5569f90d856ddd49f208346c0dea5e4ce55ed4596e371f9f05df2f62e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-22T02:37:05.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-22T02:37:05.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 systemd[1]: Stopped Ceph osd.5 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:37:05.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 systemd[1]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.5.service: Consumed 1min 5.021s CPU time. 2024-03-22T02:37:05.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:05 smithi122 systemd[1]: Starting Ceph osd.5 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T02:37:06.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:05 smithi122 ceph-mon[83368]: pgmap v159: 65 pgs: 3 active+undersized+degraded, 3 peering, 9 stale+active+clean, 2 active+undersized, 48 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/321 objects degraded (2.181%) 2024-03-22T02:37:06.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:05 smithi122 ceph-mon[83368]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T02:37:06.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:05 smithi122 ceph-mon[83368]: Health check failed: Degraded data redundancy: 7/321 objects degraded (2.181%), 3 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:06.216 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:05.929172348 +0000 UTC m=+0.017739838 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:37:06.216 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:06.036024328 +0000 UTC m=+0.124591806 container create ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T02:37:06.216 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:06.215174012 +0000 UTC m=+0.303741500 container init ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-22T02:37:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:05 smithi103 ceph-mon[115211]: pgmap v159: 65 pgs: 3 active+undersized+degraded, 3 peering, 9 stale+active+clean, 2 active+undersized, 48 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/321 objects degraded (2.181%) 2024-03-22T02:37:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:05 smithi103 ceph-mon[115211]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-22T02:37:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:05 smithi103 ceph-mon[115211]: Health check failed: Degraded data redundancy: 7/321 objects degraded (2.181%), 3 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:06.598 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:06.218662841 +0000 UTC m=+0.307230320 container start ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-22T02:37:06.599 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:06.330321207 +0000 UTC m=+0.418888694 container attach ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T02:37:07.081 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 bash[102214]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate[102230]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-22T02:37:07.082 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:06 smithi122 podman[102214]: 2024-03-22 02:37:06.686872214 +0000 UTC m=+0.775439693 container died ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:37:07.848 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 podman[102364]: 2024-03-22 02:37:07.517653116 +0000 UTC m=+0.822097999 container remove ec0b0f464d2e0b220cb58176dd839711d40c6674c2b9898bb5fa1e23eb4fa652 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T02:37:07.848 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 podman[102425]: 2024-03-22 02:37:07.63519292 +0000 UTC m=+0.017004751 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:37:07.849 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 podman[102425]: 2024-03-22 02:37:07.743571151 +0000 UTC m=+0.125382969 container create e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-22T02:37:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:07 smithi122 ceph-mon[83368]: pgmap v160: 65 pgs: 12 active+undersized+degraded, 3 peering, 2 stale+active+clean, 10 active+undersized, 38 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 44/321 objects degraded (13.707%) 2024-03-22T02:37:08.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 podman[102425]: 2024-03-22 02:37:07.954345941 +0000 UTC m=+0.336157772 container init e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-22T02:37:08.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 podman[102425]: 2024-03-22 02:37:07.957496513 +0000 UTC m=+0.339308340 container start e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T02:37:08.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:07 smithi122 ceph-osd[102444]: -- 172.21.15.122:0/3527516546 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8aad49680 con 0x55c8aad13000 2024-03-22T02:37:08.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:08 smithi122 bash[102425]: e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 2024-03-22T02:37:08.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:08 smithi122 systemd[1]: Started Ceph osd.5 for 67659b4a-e7f2-11ee-95cd-87774f69a715. 2024-03-22T02:37:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:07 smithi103 ceph-mon[115211]: pgmap v160: 65 pgs: 12 active+undersized+degraded, 3 peering, 2 stale+active+clean, 10 active+undersized, 38 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 44/321 objects degraded (13.707%) 2024-03-22T02:37:08.848 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T02:37:08.495+0000 7ff1df6097c0 -1 Falling back to public interface 2024-03-22T02:37:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:09 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:09.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:09 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:09.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:09 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:09.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:09 smithi122 ceph-mon[83368]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:09 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:09 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:09 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:09 smithi103 ceph-mon[115211]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:10 smithi122 ceph-mon[83368]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T02:37:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:10 smithi103 ceph-mon[115211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-22T02:37:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:11 smithi122 ceph-mon[83368]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:11 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:11 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:11 smithi103 ceph-mon[115211]: pgmap v162: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:11 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:11 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:12.848 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:12 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T02:37:12.721+0000 7ff1df6097c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-22T02:37:13.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:13 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T02:37:13.136+0000 7ff1df6097c0 -1 osd.5 74 log_to_monitors true 2024-03-22T02:37:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:12 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:12 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:12 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:12 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:14 smithi122 ceph-mon[83368]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:14 smithi122 ceph-mon[83368]: from='osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:37:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:14 smithi122 ceph-mon[83368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:37:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:14 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 51/321 objects degraded (15.888%), 14 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:14.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 02:37:14 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T02:37:14.015+0000 7ff1d0a61700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:37:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:14 smithi103 ceph-mon[115211]: pgmap v163: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:14 smithi103 ceph-mon[115211]: from='osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:37:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:14 smithi103 ceph-mon[115211]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:37:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:14 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 51/321 objects degraded (15.888%), 14 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:15 smithi122 ceph-mon[83368]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:37:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:15 smithi122 ceph-mon[83368]: from='osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:37:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:15 smithi122 ceph-mon[83368]: osdmap e77: 6 total, 5 up, 6 in 2024-03-22T02:37:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:15 smithi122 ceph-mon[83368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:37:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:15 smithi122 ceph-mon[83368]: pgmap v165: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:15 smithi103 ceph-mon[115211]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:37:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:15 smithi103 ceph-mon[115211]: from='osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:37:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:15 smithi103 ceph-mon[115211]: osdmap e77: 6 total, 5 up, 6 in 2024-03-22T02:37:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:15 smithi103 ceph-mon[115211]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-22T02:37:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:15 smithi103 ceph-mon[115211]: pgmap v165: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093] boot 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: osdmap e78: 6 total, 6 up, 6 in 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all osd 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T02:37:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: Upgrade: Setting require_osd_release to 19 squid 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-22T02:37:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:16 smithi122 ceph-mon[83368]: osdmap e79: 6 total, 6 up, 6 in 2024-03-22T02:37:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T02:37:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: osd.5 [v2:172.21.15.122:6816/1621369093,v1:172.21.15.122:6817/1621369093] boot 2024-03-22T02:37:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: osdmap e78: 6 total, 6 up, 6 in 2024-03-22T02:37:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all osd 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T02:37:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: Upgrade: Setting require_osd_release to 19 squid 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-22T02:37:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-mon[115211]: osdmap e79: 6 total, 6 up, 6 in 2024-03-22T02:37:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:16 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:37:16.023+0000 7f4cf154f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:37:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: osdmap e80: 6 total, 6 up, 6 in 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: pgmap v169: 65 pgs: 3 peering, 7 active+undersized+degraded, 11 active+undersized, 44 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21/321 objects degraded (6.542%) 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: osdmap e80: 6 total, 6 up, 6 in 2024-03-22T02:37:17.360 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-22T02:37:17.361 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: pgmap v169: 65 pgs: 3 peering, 7 active+undersized+degraded, 11 active+undersized, 44 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21/321 objects degraded (6.542%) 2024-03-22T02:37:17.361 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:17.361 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.122:6824/615549338,v1:172.21.15.122:6825/615549338] up:boot 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/1635524194,v1:172.21.15.103:6829/1635524194] up:boot 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: Upgrade: Updating mds.cephfs.smithi103.gtuatp 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:18 smithi122 ceph-mon[83368]: Deploying daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.122:6824/615549338,v1:172.21.15.122:6825/615549338] up:boot 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/1635524194,v1:172.21.15.103:6829/1635524194] up:boot 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: Upgrade: Updating mds.cephfs.smithi103.gtuatp 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.gtuatp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:18.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:18.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:18 smithi103 ceph-mon[115211]: Deploying daemon mds.cephfs.smithi103.gtuatp on smithi103 2024-03-22T02:37:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:19 smithi122 ceph-mon[83368]: Health check update: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:19 smithi122 ceph-mon[83368]: pgmap v170: 65 pgs: 3 peering, 6 active+undersized+degraded, 9 active+undersized, 47 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 17/321 objects degraded (5.296%) 2024-03-22T02:37:19.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:19 smithi103 ceph-mon[115211]: Health check update: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded (PG_DEGRADED) 2024-03-22T02:37:19.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:19 smithi103 ceph-mon[115211]: pgmap v170: 65 pgs: 3 peering, 6 active+undersized+degraded, 9 active+undersized, 47 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 17/321 objects degraded (5.296%) 2024-03-22T02:37:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:21 smithi122 ceph-mon[83368]: osdmap e81: 6 total, 6 up, 6 in 2024-03-22T02:37:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:21 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:21 smithi122 ceph-mon[83368]: pgmap v172: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:21 smithi103 ceph-mon[115211]: osdmap e81: 6 total, 6 up, 6 in 2024-03-22T02:37:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:21 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:21 smithi103 ceph-mon[115211]: pgmap v172: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:22 smithi122 ceph-mon[83368]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/321 objects degraded (5.296%), 6 pgs degraded) 2024-03-22T02:37:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:22 smithi103 ceph-mon[115211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/321 objects degraded (5.296%), 6 pgs degraded) 2024-03-22T02:37:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:23 smithi122 ceph-mon[83368]: pgmap v173: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:23 smithi103 ceph-mon[115211]: pgmap v173: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:24.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 -- 172.21.15.103:0/1790418833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 msgr2=0x7f4c981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1790418833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c981031d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4c94004940 tx=0x7f4c94009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:24.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 -- 172.21.15.103:0/1790418833 shutdown_connections 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1790418833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 0x7f4c980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1790418833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c981031d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.397+0000 7f4ca045f700 1 -- 172.21.15.103:0/1790418833 >> 172.21.15.103:0/1790418833 conn(0x7f4c980f89b0 msgr2=0x7f4c980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 -- 172.21.15.103:0/1790418833 shutdown_connections 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 -- 172.21.15.103:0/1790418833 wait complete. 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 Processor -- start 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 -- start start 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 0x7f4c981140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c981146f0 con 0x7f4c98102e00 2024-03-22T02:37:24.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4ca045f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9810db30 con 0x7f4c981037a0 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4c9e1fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.398+0000 7f4c9e1fb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56348/0 (socket says 172.21.15.103:56348) 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9e1fb700 1 -- 172.21.15.103:0/1305005091 learned_addr learned my addr 172.21.15.103:0/1305005091 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9d9fa700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 0x7f4c981140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9e1fb700 1 -- 172.21.15.103:0/1305005091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 msgr2=0x7f4c981140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9e1fb700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 0x7f4c981140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9e1fb700 1 -- 172.21.15.103:0/1305005091 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c94005040 con 0x7f4c98102e00 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c9e1fb700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009150 tx=0x7f4c94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c9400e040 con 0x7f4c98102e00 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9810ddb0 con 0x7f4c98102e00 2024-03-22T02:37:24.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c94013420 con 0x7f4c98102e00 2024-03-22T02:37:24.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c9401c630 con 0x7f4c98102e00 2024-03-22T02:37:24.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.399+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9810e370 con 0x7f4c98102e00 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.400+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c98100110 con 0x7f4c98102e00 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.401+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4c9400baf0 con 0x7f4c98102e00 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.401+0000 7f4c8effd700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 0x7f4c84070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.401+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4c9408f0b0 con 0x7f4c98102e00 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.401+0000 7f4c9d9fa700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 0x7f4c84070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.402+0000 7f4c9d9fa700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 0x7f4c84070860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c88007fd0 tx=0x7f4c88009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:24.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.406+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c9405be60 con 0x7f4c98102e00 2024-03-22T02:37:24.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.652+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c9810f220 con 0x7f4c8406e3b0 2024-03-22T02:37:24.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.653+0000 7f4c8effd700 1 -- 172.21.15.103:0/1305005091 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c9810f220 con 0x7f4c8406e3b0 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 msgr2=0x7f4c84070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 0x7f4c84070860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c88007fd0 tx=0x7f4c88009040 comp rx=0 tx=0).stop 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 msgr2=0x7f4c98113b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009150 tx=0x7f4c94009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 shutdown_connections 2024-03-22T02:37:24.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c8406e3b0 0x7f4c84070860 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c98102e00 0x7f4c98113b60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 --2- 172.21.15.103:0/1305005091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c981037a0 0x7f4c981140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 >> 172.21.15.103:0/1305005091 conn(0x7f4c980f89b0 msgr2=0x7f4c98107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:24.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 shutdown_connections 2024-03-22T02:37:24.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.656+0000 7f4ca045f700 1 -- 172.21.15.103:0/1305005091 wait complete. 2024-03-22T02:37:24.668 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 -- 172.21.15.103:0/1463621783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd101037a0 msgr2=0x7fdd100fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 --2- 172.21.15.103:0/1463621783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd101037a0 0x7fdd100fce10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd04004940 tx=0x7fdd04009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 -- 172.21.15.103:0/1463621783 shutdown_connections 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 --2- 172.21.15.103:0/1463621783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd101037a0 0x7fdd100fce10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 --2- 172.21.15.103:0/1463621783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd10102e00 0x7fdd101031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 -- 172.21.15.103:0/1463621783 >> 172.21.15.103:0/1463621783 conn(0x7fdd100f89b0 msgr2=0x7fdd100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 -- 172.21.15.103:0/1463621783 shutdown_connections 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.765+0000 7fdd148ee700 1 -- 172.21.15.103:0/1463621783 wait complete. 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 Processor -- start 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 -- start start 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 0x7fdd10113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10114510 con 0x7fdd10102e00 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd148ee700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010db10 con 0x7fdd101037a0 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd0e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd0e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56364/0 (socket says 172.21.15.103:56364) 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.766+0000 7fdd0e59c700 1 -- 172.21.15.103:0/723507638 learned_addr learned my addr 172.21.15.103:0/723507638 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0dd9b700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 0x7fdd10113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0e59c700 1 -- 172.21.15.103:0/723507638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 msgr2=0x7fdd10113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0e59c700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 0x7fdd10113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0e59c700 1 -- 172.21.15.103:0/723507638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04005040 con 0x7fdd10102e00 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0dd9b700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 0x7fdd10113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd0e59c700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd00002a50 tx=0x7fdd000046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd00007cc0 con 0x7fdd10102e00 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd0000f040 con 0x7fdd10102e00 2024-03-22T02:37:24.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1010ddf0 con 0x7fdd10102e00 2024-03-22T02:37:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd00009640 con 0x7fdd10102e00 2024-03-22T02:37:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.767+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1010e3b0 con 0x7fdd10102e00 2024-03-22T02:37:24.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.768+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd101052d0 con 0x7fdd10102e00 2024-03-22T02:37:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.769+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fdd000097a0 con 0x7fdd10102e00 2024-03-22T02:37:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.769+0000 7fdcfeffd700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 0x7fdcf8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.769+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdd0008db00 con 0x7fdd10102e00 2024-03-22T02:37:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.769+0000 7fdd0dd9b700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 0x7fdcf8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:24.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.770+0000 7fdd0dd9b700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 0x7fdcf8070850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400b3b0 tx=0x7fdd04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:24.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.774+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd0005a8b0 con 0x7fdd10102e00 2024-03-22T02:37:24.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.994+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd1010f040 con 0x7fdcf806e3a0 2024-03-22T02:37:24.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.995+0000 7fdcfeffd700 1 -- 172.21.15.103:0/723507638 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd1010f040 con 0x7fdcf806e3a0 2024-03-22T02:37:24.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 msgr2=0x7fdcf8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 0x7fdcf8070850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400b3b0 tx=0x7fdd04009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 msgr2=0x7fdd10113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd00002a50 tx=0x7fdd000046b0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 shutdown_connections 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcf806e3a0 0x7fdcf8070850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd10102e00 0x7fdd10113980 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 --2- 172.21.15.103:0/723507638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd101037a0 0x7fdd10113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 >> 172.21.15.103:0/723507638 conn(0x7fdd100f89b0 msgr2=0x7fdd101021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 shutdown_connections 2024-03-22T02:37:24.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:24.998+0000 7fdd148ee700 1 -- 172.21.15.103:0/723507638 wait complete. 2024-03-22T02:37:25.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- 172.21.15.103:0/845909072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f100708a0 msgr2=0x7f7f10070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 --2- 172.21.15.103:0/845909072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f100708a0 0x7f7f10070d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c004970 tx=0x7f7f0c009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:25.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- 172.21.15.103:0/845909072 shutdown_connections 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 --2- 172.21.15.103:0/845909072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f100708a0 0x7f7f10070d10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 --2- 172.21.15.103:0/845909072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f10071f90 0x7f7f10070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- 172.21.15.103:0/845909072 >> 172.21.15.103:0/845909072 conn(0x7f7f100f8a20 msgr2=0x7f7f100fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- 172.21.15.103:0/845909072 shutdown_connections 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- 172.21.15.103:0/845909072 wait complete. 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 Processor -- start 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.105+0000 7f7f1748c700 1 -- start start 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f1748c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 0x7f7f10119f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f1748c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f1748c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1011ab20 con 0x7f7f10071f90 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f1748c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10114100 con 0x7f7f100708a0 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56378/0 (socket says 172.21.15.103:56378) 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 -- 172.21.15.103:0/2449540628 learned_addr learned my addr 172.21.15.103:0/2449540628 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f15228700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 0x7f7f10119f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 -- 172.21.15.103:0/2449540628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 msgr2=0x7f7f10119f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 0x7f7f10119f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 -- 172.21.15.103:0/2449540628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c005040 con 0x7f7f10071f90 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f15228700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 0x7f7f10119f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f14a27700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005010 tx=0x7f7f0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.106+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c00e040 con 0x7f7f10071f90 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.107+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f10114380 con 0x7f7f10071f90 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.107+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f0c013420 con 0x7f7f10071f90 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.107+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c009920 con 0x7f7f10071f90 2024-03-22T02:37:25.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.107+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f101148c0 con 0x7f7f10071f90 2024-03-22T02:37:25.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.108+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f10071aa0 con 0x7f7f10071f90 2024-03-22T02:37:25.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.108+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7f0c013590 con 0x7f7f10071f90 2024-03-22T02:37:25.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.109+0000 7f7f05ffb700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 0x7f7efc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.109+0000 7f7f15228700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 0x7f7efc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.109+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f0c08ef70 con 0x7f7f10071f90 2024-03-22T02:37:25.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.109+0000 7f7f15228700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 0x7f7efc070850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f00009b80 tx=0x7f7f0000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.113+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f0c0c3740 con 0x7f7f10071f90 2024-03-22T02:37:25.175 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:boot 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:25.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:25 smithi103 ceph-mon[115211]: pgmap v174: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:25.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.328+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f101105c0 con 0x7f7efc06e3a0 2024-03-22T02:37:25.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.338+0000 7f7f05ffb700 1 -- 172.21.15.103:0/2449540628 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7f101105c0 con 0x7f7efc06e3a0 2024-03-22T02:37:25.339 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:37:25.339 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (5m) 116s ago 17m 28.0M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:37:25.339 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (3m) 116s ago 17m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:37:25.339 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (3m) 14s ago 16m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (5m) 116s ago 16m 81.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 starting - - - - 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (11m) 116s ago 11m 126M - 17.2.7-726-g9c4e0c34 837124da804f baf157d716cd 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (11m) 14s ago 11m 16.2M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (11m) 14s ago 11m 115M - 17.2.7-726-g9c4e0c34 837124da804f fbeb0b52313a 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (7m) 116s ago 18m 504M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (6m) 14s ago 16m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (4m) 116s ago 18m 44.0M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (4m) 14s ago 16m 39.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (6m) 116s ago 17m 15.3M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 14s ago 16m 18.7M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (3m) 116s ago 15m 87.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (2m) 116s ago 14m 98.1M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (2m) 116s ago 14m 12.7M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:37:25.340 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (86s) 14s ago 14m 97.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:37:25.341 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (51s) 14s ago 13m 91.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:37:25.341 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (17s) 14s ago 13m 12.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:37:25.341 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (6m) 116s ago 16m 64.7M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:37:25.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.341+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 msgr2=0x7f7efc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.341+0000 7f7f1748c700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 0x7f7efc070850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f00009b80 tx=0x7f7f0000b040 comp rx=0 tx=0).stop 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.341+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 msgr2=0x7f7f1011a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.341+0000 7f7f1748c700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005010 tx=0x7f7f0c009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 shutdown_connections 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7efc06e3a0 0x7f7efc070850 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f100708a0 0x7f7f10119f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 --2- 172.21.15.103:0/2449540628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f10071f90 0x7f7f1011a4d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 >> 172.21.15.103:0/2449540628 conn(0x7f7f100f8a20 msgr2=0x7f7f100f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 shutdown_connections 2024-03-22T02:37:25.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.342+0000 7f7f1748c700 1 -- 172.21.15.103:0/2449540628 wait complete. 2024-03-22T02:37:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:boot 2024-03-22T02:37:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:25.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:25 smithi122 ceph-mon[83368]: pgmap v174: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 -- 172.21.15.103:0/652071921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18101d70 msgr2=0x7f7b18102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 --2- 172.21.15.103:0/652071921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18101d70 0x7f7b18102140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7b14004940 tx=0x7f7b14009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 -- 172.21.15.103:0/652071921 shutdown_connections 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 --2- 172.21.15.103:0/652071921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18102710 0x7f7b180fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 --2- 172.21.15.103:0/652071921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18101d70 0x7f7b18102140 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 -- 172.21.15.103:0/652071921 >> 172.21.15.103:0/652071921 conn(0x7f7b18072bc0 msgr2=0x7f7b18072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:25.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 -- 172.21.15.103:0/652071921 shutdown_connections 2024-03-22T02:37:25.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.467+0000 7f7b203f2700 1 -- 172.21.15.103:0/652071921 wait complete. 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 Processor -- start 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 -- start start 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18101d70 0x7f7b18112a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181135d0 con 0x7f7b18102710 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b203f2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1810c9c0 con 0x7f7b18101d70 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b1d98d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b1d98d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56384/0 (socket says 172.21.15.103:56384) 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b1d98d700 1 -- 172.21.15.103:0/3494942502 learned_addr learned my addr 172.21.15.103:0/3494942502 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.468+0000 7f7b1e18e700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18101d70 0x7f7b18112a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b1d98d700 1 -- 172.21.15.103:0/3494942502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18101d70 msgr2=0x7f7b18112a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b1d98d700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18101d70 0x7f7b18112a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b1d98d700 1 -- 172.21.15.103:0/3494942502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b14005040 con 0x7f7b18102710 2024-03-22T02:37:25.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b1d98d700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7b08004e70 tx=0x7f7b08004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b08004ea0 con 0x7f7b18102710 2024-03-22T02:37:25.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1810cc40 con 0x7f7b18102710 2024-03-22T02:37:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.469+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1810d200 con 0x7f7b18102710 2024-03-22T02:37:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.470+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b080095a0 con 0x7f7b18102710 2024-03-22T02:37:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.470+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b0800dc70 con 0x7f7b18102710 2024-03-22T02:37:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.471+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc0052e0 con 0x7f7b18102710 2024-03-22T02:37:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.472+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7b08009710 con 0x7f7b18102710 2024-03-22T02:37:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.472+0000 7f7b0effd700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 0x7f7b04070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.472+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b080903c0 con 0x7f7b18102710 2024-03-22T02:37:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.473+0000 7f7b1e18e700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 0x7f7b04070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.473+0000 7f7b1e18e700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 0x7f7b04070850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b14004f90 tx=0x7f7b14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.476+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b0805d230 con 0x7f7b18102710 2024-03-22T02:37:25.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.781+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7afc005860 con 0x7f7b18102710 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.782+0000 7f7b0effd700 1 -- 172.21.15.103:0/3494942502 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f7b0805cdc0 con 0x7f7b18102710 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:25.783 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 1 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 3, 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 11 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:37:25.784 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:37:25.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 msgr2=0x7f7b04070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 0x7f7b04070850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b14004f90 tx=0x7f7b14009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 msgr2=0x7f7b18112f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7b08004e70 tx=0x7f7b08004600 comp rx=0 tx=0).stop 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 shutdown_connections 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b0406e3a0 0x7f7b04070850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b18101d70 0x7f7b18112a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 --2- 172.21.15.103:0/3494942502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18102710 0x7f7b18112f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.784+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 >> 172.21.15.103:0/3494942502 conn(0x7f7b18072bc0 msgr2=0x7f7b180fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.785+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 shutdown_connections 2024-03-22T02:37:25.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.785+0000 7f7b203f2700 1 -- 172.21.15.103:0/3494942502 wait complete. 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 -- 172.21.15.103:0/883221211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 msgr2=0x7f31300fbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 --2- 172.21.15.103:0/883221211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f31300fbd00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3120004940 tx=0x7f3120009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 -- 172.21.15.103:0/883221211 shutdown_connections 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 --2- 172.21.15.103:0/883221211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f31300fbd00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 --2- 172.21.15.103:0/883221211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 0x7f31301020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 -- 172.21.15.103:0/883221211 >> 172.21.15.103:0/883221211 conn(0x7f31300f7860 msgr2=0x7f31300f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 -- 172.21.15.103:0/883221211 shutdown_connections 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.896+0000 7f3137e2f700 1 -- 172.21.15.103:0/883221211 wait complete. 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 Processor -- start 2024-03-22T02:37:25.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 -- start start 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 0x7f313010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313010f680 con 0x7f3130102690 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f3137e2f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313010f7c0 con 0x7f3130101cf0 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.897+0000 7f31353ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56402/0 (socket says 172.21.15.103:56402) 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 -- 172.21.15.103:0/324461599 learned_addr learned my addr 172.21.15.103:0/324461599 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f3135bcb700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 0x7f313010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 -- 172.21.15.103:0/324461599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 msgr2=0x7f313010ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 0x7f313010ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 -- 172.21.15.103:0/324461599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120005040 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31353ca700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f3120009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3120021070 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3120013c00 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130117ab0 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312000e070 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.898+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3130117f40 con 0x7f3130102690 2024-03-22T02:37:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.899+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31300fd440 con 0x7f3130102690 2024-03-22T02:37:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.900+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f312001ca60 con 0x7f3130102690 2024-03-22T02:37:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.901+0000 7f31267fc700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 0x7f311c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.901+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f312008f220 con 0x7f3130102690 2024-03-22T02:37:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.901+0000 7f3135bcb700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 0x7f311c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.901+0000 7f3135bcb700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 0x7f311c070850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f312c009b80 tx=0x7f312c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:25.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:25.910+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31200584a0 con 0x7f3130102690 2024-03-22T02:37:26.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.159+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f313004bed0 con 0x7f3130102690 2024-03-22T02:37:26.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.159+0000 7f31267fc700 1 -- 172.21.15.103:0/324461599 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1876 (secure 0 0 0) 0x7f3120004490 con 0x7f3130102690 2024-03-22T02:37:26.160 INFO:teuthology.orchestra.run.smithi103.stdout:e25 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:37:26.161 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:37:26.162 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:37:26.163 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:26.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:37:26.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:37:26.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:44253} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/1635524194,v1:172.21.15.103:6829/1635524194] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:37:26.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44255} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/615549338,v1:172.21.15.122:6825/615549338] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 msgr2=0x7f311c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 0x7f311c070850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f312c009b80 tx=0x7f312c00b040 comp rx=0 tx=0).stop 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 msgr2=0x7f313010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f3120009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 shutdown_connections 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f311c06e3a0 0x7f311c070850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3130101cf0 0x7f313010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 --2- 172.21.15.103:0/324461599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3130102690 0x7f313010f140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 >> 172.21.15.103:0/324461599 conn(0x7f31300f7860 msgr2=0x7f31300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 shutdown_connections 2024-03-22T02:37:26.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.162+0000 7f3137e2f700 1 -- 172.21.15.103:0/324461599 wait complete. 2024-03-22T02:37:26.166 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 25 2024-03-22T02:37:26.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 -- 172.21.15.103:0/4104371699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fde10 msgr2=0x7f1dac0fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 --2- 172.21.15.103:0/4104371699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fe1e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1da0004940 tx=0x7f1da0009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:26.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 -- 172.21.15.103:0/4104371699 shutdown_connections 2024-03-22T02:37:26.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 --2- 172.21.15.103:0/4104371699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 --2- 172.21.15.103:0/4104371699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fe1e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 -- 172.21.15.103:0/4104371699 >> 172.21.15.103:0/4104371699 conn(0x7f1dac0f7860 msgr2=0x7f1dac0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 -- 172.21.15.103:0/4104371699 shutdown_connections 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 -- 172.21.15.103:0/4104371699 wait complete. 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.275+0000 7f1db316e700 1 Processor -- start 2024-03-22T02:37:26.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db316e700 1 -- start start 2024-03-22T02:37:26.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db316e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db316e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db316e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0fdb90 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db316e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0fc6c0 con 0x7f1dac0fde10 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1dabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1dabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56420/0 (socket says 172.21.15.103:56420) 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1dabfff700 1 -- 172.21.15.103:0/1697842750 learned_addr learned my addr 172.21.15.103:0/1697842750 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.276+0000 7f1db0f0a700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1dabfff700 1 -- 172.21.15.103:0/1697842750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fde10 msgr2=0x7f1dac0fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1dabfff700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1dabfff700 1 -- 172.21.15.103:0/1697842750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0005040 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1dabfff700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800eb40 tx=0x7f1d9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d98007cc0 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d9800f040 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d9800ca10 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac0fc940 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.277+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac119f80 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.278+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac066af0 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.279+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1d9800cb70 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.279+0000 7f1da97fa700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 0x7f1d9c070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.279+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1d9808d950 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.279+0000 7f1db0f0a700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 0x7f1d9c070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.280+0000 7f1db0f0a700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 0x7f1d9c070920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009fd0 tx=0x7f1da0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:26.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.285+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d98056c00 con 0x7f1dac0fe7b0 2024-03-22T02:37:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:26 smithi122 ceph-mon[83368]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:26 smithi122 ceph-mon[83368]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:26 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3494942502' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:26 smithi103 ceph-mon[115211]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:26 smithi103 ceph-mon[115211]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:26 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3494942502' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:26.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.511+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dac0f9440 con 0x7f1d9c06e470 2024-03-22T02:37:26.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.512+0000 7f1da97fa700 1 -- 172.21.15.103:0/1697842750 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1dac0f9440 con 0x7f1d9c06e470 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:37:26.513 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "12/21 daemons upgraded", 2024-03-22T02:37:26.514 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:37:26.514 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:37:26.514 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 msgr2=0x7f1d9c070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 0x7f1d9c070920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009fd0 tx=0x7f1da0009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 msgr2=0x7f1dac0fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800eb40 tx=0x7f1d9800b040 comp rx=0 tx=0).stop 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 shutdown_connections 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d9c06e470 0x7f1d9c070920 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dac0fde10 0x7f1dac0fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 --2- 172.21.15.103:0/1697842750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dac0fe7b0 0x7f1dac0fc180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 >> 172.21.15.103:0/1697842750 conn(0x7f1dac0f7860 msgr2=0x7f1dac0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:26.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 shutdown_connections 2024-03-22T02:37:26.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.514+0000 7f1db316e700 1 -- 172.21.15.103:0/1697842750 wait complete. 2024-03-22T02:37:26.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.620+0000 7f1e10545700 1 -- 172.21.15.103:0/3905720200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 msgr2=0x7f1e08103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.620+0000 7f1e10545700 1 --2- 172.21.15.103:0/3905720200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08103340 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004940 tx=0x7f1e04009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:26.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- 172.21.15.103:0/3905720200 shutdown_connections 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 --2- 172.21.15.103:0/3905720200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 0x7f1e080fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 --2- 172.21.15.103:0/3905720200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08103340 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- 172.21.15.103:0/3905720200 >> 172.21.15.103:0/3905720200 conn(0x7f1e08072310 msgr2=0x7f1e08072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- 172.21.15.103:0/3905720200 shutdown_connections 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- 172.21.15.103:0/3905720200 wait complete. 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 Processor -- start 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- start start 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 0x7f1e08113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.621+0000 7f1e10545700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081144b0 con 0x7f1e08102f70 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e10545700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0810dab0 con 0x7f1e080fb3b0 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56444/0 (socket says 172.21.15.103:56444) 2024-03-22T02:37:26.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 -- 172.21.15.103:0/3845219329 learned_addr learned my addr 172.21.15.103:0/3845219329 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0e2e1700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 0x7f1e08113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 -- 172.21.15.103:0/3845219329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 msgr2=0x7f1e08113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 0x7f1e08113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 -- 172.21.15.103:0/3845219329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04005040 con 0x7f1e08102f70 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e0dae0700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1df8002a50 tx=0x7f1df80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df8007cc0 con 0x7f1e08102f70 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1df800f040 con 0x7f1e08102f70 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0810dd90 con 0x7f1e08102f70 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df800da70 con 0x7f1e08102f70 2024-03-22T02:37:26.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.622+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0810e350 con 0x7f1e08102f70 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.623+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e080fe6d0 con 0x7f1e08102f70 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.624+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1df800dc10 con 0x7f1e08102f70 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.624+0000 7f1dfeffd700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 0x7f1df4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.624+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1df808d360 con 0x7f1e08102f70 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.624+0000 7f1e0e2e1700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 0x7f1df4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:26.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.625+0000 7f1e0e2e1700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 0x7f1df4070850 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004f90 tx=0x7f1e04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:26.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.629+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1df805a110 con 0x7f1e08102f70 2024-03-22T02:37:26.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.920+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1e0804c710 con 0x7f1e08102f70 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.921+0000 7f1dfeffd700 1 -- 172.21.15.103:0/3845219329 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1df8059ca0 con 0x7f1e08102f70 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:37:26.922 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 msgr2=0x7f1df4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 0x7f1df4070850 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004f90 tx=0x7f1e04009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 msgr2=0x7f1e08113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1df8002a50 tx=0x7f1df80046b0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 shutdown_connections 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1df406e3a0 0x7f1df4070850 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e080fb3b0 0x7f1e08113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 --2- 172.21.15.103:0/3845219329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e08102f70 0x7f1e08113e60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 >> 172.21.15.103:0/3845219329 conn(0x7f1e08072310 msgr2=0x7f1e080f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 shutdown_connections 2024-03-22T02:37:26.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:26.923+0000 7f1e10545700 1 -- 172.21.15.103:0/3845219329 wait complete. 2024-03-22T02:37:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:27 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/324461599' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:37:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:27 smithi122 ceph-mon[83368]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:27 smithi122 ceph-mon[83368]: pgmap v175: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:27 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3845219329' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:37:27.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:27 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/324461599' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:37:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:27 smithi103 ceph-mon[115211]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:27 smithi103 ceph-mon[115211]: pgmap v175: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:27 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3845219329' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:37:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:29 smithi122 ceph-mon[83368]: pgmap v176: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:29.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:29.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:29 smithi103 ceph-mon[115211]: pgmap v176: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:31.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:31.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:31.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:31 smithi122 ceph-mon[83368]: pgmap v177: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:31.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:31.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:31.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:31 smithi103 ceph-mon[115211]: pgmap v177: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:33.161 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:33.161 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:33.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:33 smithi122 ceph-mon[83368]: pgmap v178: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:33 smithi103 ceph-mon[115211]: pgmap v178: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:34 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:35.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:34 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:36.338 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:35 smithi103 ceph-mon[115211]: pgmap v179: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:36.338 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:36.338 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:36.339 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:35 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:35 smithi122 ceph-mon[83368]: pgmap v179: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.xhmfjp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:35 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:36 smithi122 ceph-mon[83368]: Upgrade: Updating mds.cephfs.smithi103.xhmfjp 2024-03-22T02:37:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:36 smithi122 ceph-mon[83368]: Deploying daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:37:37.362 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:36 smithi103 ceph-mon[115211]: Upgrade: Updating mds.cephfs.smithi103.xhmfjp 2024-03-22T02:37:37.362 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:36 smithi103 ceph-mon[115211]: Deploying daemon mds.cephfs.smithi103.xhmfjp on smithi103 2024-03-22T02:37:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:37 smithi122 ceph-mon[83368]: pgmap v180: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:37 smithi103 ceph-mon[115211]: pgmap v180: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:38 smithi122 ceph-mon[83368]: osdmap e82: 6 total, 6 up, 6 in 2024-03-22T02:37:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:38 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:38 smithi103 ceph-mon[115211]: osdmap e82: 6 total, 6 up, 6 in 2024-03-22T02:37:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:38 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:39 smithi122 ceph-mon[83368]: pgmap v182: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:40.367 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:39 smithi103 ceph-mon[115211]: pgmap v182: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:41 smithi122 ceph-mon[83368]: pgmap v183: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:41 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:41 smithi103 ceph-mon[115211]: pgmap v183: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:42.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:41 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:43.329 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:42 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:boot 2024-03-22T02:37:43.329 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:42 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:43.329 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:42 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:37:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:42 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:boot 2024-03-22T02:37:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:42 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:42 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:37:44.248 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:43 smithi103 ceph-mon[115211]: pgmap v184: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:43 smithi122 ceph-mon[83368]: pgmap v184: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:45 smithi122 ceph-mon[83368]: pgmap v185: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:45 smithi103 ceph-mon[115211]: pgmap v185: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:47.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:47 smithi122 ceph-mon[83368]: pgmap v186: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:47.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:47 smithi103 ceph-mon[115211]: pgmap v186: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:47.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:37:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:48 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:48.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:48 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:48 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:48 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:49.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:49 smithi103 ceph-mon[115211]: pgmap v187: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:49 smithi122 ceph-mon[83368]: pgmap v187: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: pgmap v188: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:51 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: pgmap v188: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:51 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:53.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: Upgrade: Updating mds.cephfs.smithi122.ztrkgp 2024-03-22T02:37:53.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:53.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:53.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:53.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: Deploying daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:37:53.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:53 smithi122 ceph-mon[83368]: pgmap v189: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:54.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: Upgrade: Updating mds.cephfs.smithi122.ztrkgp 2024-03-22T02:37:54.091 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:54.092 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.ztrkgp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:37:54.092 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:54.092 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: Deploying daemon mds.cephfs.smithi122.ztrkgp on smithi122 2024-03-22T02:37:54.092 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:53 smithi103 ceph-mon[115211]: pgmap v189: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:55 smithi122 ceph-mon[83368]: pgmap v190: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:55 smithi122 ceph-mon[83368]: osdmap e83: 6 total, 6 up, 6 in 2024-03-22T02:37:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:55 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:55 smithi103 ceph-mon[115211]: pgmap v190: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:55 smithi103 ceph-mon[115211]: osdmap e83: 6 total, 6 up, 6 in 2024-03-22T02:37:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:55 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 -- 172.21.15.103:0/2214677307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 msgr2=0x7ff674100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 --2- 172.21.15.103:0/2214677307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff674100bd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff664004940 tx=0x7ff664009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 -- 172.21.15.103:0/2214677307 shutdown_connections 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 --2- 172.21.15.103:0/2214677307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff674100bd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 --2- 172.21.15.103:0/2214677307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 0x7ff6740fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 -- 172.21.15.103:0/2214677307 >> 172.21.15.103:0/2214677307 conn(0x7ff6740f7870 msgr2=0x7ff6740f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 -- 172.21.15.103:0/2214677307 shutdown_connections 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.040+0000 7ff67c195700 1 -- 172.21.15.103:0/2214677307 wait complete. 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 Processor -- start 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 -- start start 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 0x7ff674119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67411ab70 con 0x7ff6740fc5b0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff67c195700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674114150 con 0x7ff6740fbc10 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48556/0 (socket says 172.21.15.103:48556) 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 -- 172.21.15.103:0/3821841611 learned_addr learned my addr 172.21.15.103:0/3821841611 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679f31700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 0x7ff674119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 -- 172.21.15.103:0/3821841611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 msgr2=0x7ff674119fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 0x7ff674119fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.041+0000 7ff679730700 1 -- 172.21.15.103:0/3821841611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664005040 con 0x7ff6740fc5b0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff679730700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff664009b30 tx=0x7ff664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664021070 con 0x7ff6740fc5b0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff664013c00 con 0x7ff6740fc5b0 2024-03-22T02:37:57.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6741143d0 con 0x7ff6740fc5b0 2024-03-22T02:37:57.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff66400e070 con 0x7ff6740fc5b0 2024-03-22T02:37:57.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.042+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6741148b0 con 0x7ff6740fc5b0 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.043+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff674066b70 con 0x7ff6740fc5b0 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.043+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff66401cad0 con 0x7ff6740fc5b0 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.044+0000 7ff66a7fc700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 0x7ff660070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.044+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff66408f1d0 con 0x7ff6740fc5b0 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.044+0000 7ff679f31700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 0x7ff660070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.044+0000 7ff679f31700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 0x7ff660070850 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff670009b80 tx=0x7ff67000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.049+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff66405bf80 con 0x7ff6740fc5b0 2024-03-22T02:37:57.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.267+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6741105b0 con 0x7ff66006e3a0 2024-03-22T02:37:57.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.268+0000 7ff66a7fc700 1 -- 172.21.15.103:0/3821841611 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff6741105b0 con 0x7ff66006e3a0 2024-03-22T02:37:57.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 msgr2=0x7ff660070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 0x7ff660070850 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff670009b80 tx=0x7ff67000b040 comp rx=0 tx=0).stop 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 msgr2=0x7ff67411a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff664009b30 tx=0x7ff664009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 shutdown_connections 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff66006e3a0 0x7ff660070850 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6740fbc10 0x7ff674119fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 --2- 172.21.15.103:0/3821841611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6740fc5b0 0x7ff67411a520 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 >> 172.21.15.103:0/3821841611 conn(0x7ff6740f7870 msgr2=0x7ff6740f8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 shutdown_connections 2024-03-22T02:37:57.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.271+0000 7ff67c195700 1 -- 172.21.15.103:0/3821841611 wait complete. 2024-03-22T02:37:57.283 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 -- 172.21.15.103:0/3073777111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e90102e10 msgr2=0x7f4e901031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/3073777111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e90102e10 0x7f4e901031e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009ce0 tx=0x7f4e88004d10 comp rx=0 tx=0).stop 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 -- 172.21.15.103:0/3073777111 shutdown_connections 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/3073777111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e901037b0 0x7f4e900fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/3073777111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e90102e10 0x7f4e901031e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 -- 172.21.15.103:0/3073777111 >> 172.21.15.103:0/3073777111 conn(0x7f4e900f89a0 msgr2=0x7f4e900fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 -- 172.21.15.103:0/3073777111 shutdown_connections 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.373+0000 7f4e97aeb700 1 -- 172.21.15.103:0/3073777111 wait complete. 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 Processor -- start 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 -- start start 2024-03-22T02:37:57.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e90102e10 0x7f4e90113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90114920 con 0x7f4e901037b0 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e97aeb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9010dc60 con 0x7f4e90102e10 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48576/0 (socket says 172.21.15.103:48576) 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 -- 172.21.15.103:0/1625258990 learned_addr learned my addr 172.21.15.103:0/1625258990 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95887700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e90102e10 0x7f4e90113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 -- 172.21.15.103:0/1625258990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e90102e10 msgr2=0x7f4e90113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e90102e10 0x7f4e90113d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.374+0000 7f4e95086700 1 -- 172.21.15.103:0/1625258990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88009940 con 0x7f4e901037b0 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e95086700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000b790 tx=0x7f4e80009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8000f6f0 con 0x7f4e901037b0 2024-03-22T02:37:57.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e8000fd20 con 0x7f4e901037b0 2024-03-22T02:37:57.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9010dee0 con 0x7f4e901037b0 2024-03-22T02:37:57.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e800103a0 con 0x7f4e901037b0 2024-03-22T02:37:57.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.375+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9010e4a0 con 0x7f4e901037b0 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.376+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e900fe560 con 0x7f4e901037b0 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.376+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4e80011400 con 0x7f4e901037b0 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.377+0000 7f4e867fc700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 0x7f4e7c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.377+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4e8008db80 con 0x7f4e901037b0 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.377+0000 7f4e95887700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 0x7f4e7c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.377+0000 7f4e95887700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 0x7f4e7c070850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800b740 tx=0x7f4e88004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.381+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e8005a930 con 0x7f4e901037b0 2024-03-22T02:37:57.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.607+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e90109ff0 con 0x7f4e7c06e3a0 2024-03-22T02:37:57.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.608+0000 7f4e867fc700 1 -- 172.21.15.103:0/1625258990 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4e90109ff0 con 0x7f4e7c06e3a0 2024-03-22T02:37:57.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 msgr2=0x7f4e7c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 0x7f4e7c070850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800b740 tx=0x7f4e88004d10 comp rx=0 tx=0).stop 2024-03-22T02:37:57.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 msgr2=0x7f4e90114240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000b790 tx=0x7f4e80009220 comp rx=0 tx=0).stop 2024-03-22T02:37:57.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 shutdown_connections 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e7c06e3a0 0x7f4e7c070850 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e90102e10 0x7f4e90113d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 --2- 172.21.15.103:0/1625258990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e901037b0 0x7f4e90114240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 >> 172.21.15.103:0/1625258990 conn(0x7f4e900f89a0 msgr2=0x7f4e900f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 shutdown_connections 2024-03-22T02:37:57.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.610+0000 7f4e97aeb700 1 -- 172.21.15.103:0/1625258990 wait complete. 2024-03-22T02:37:57.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.718+0000 7f28f0949700 1 -- 172.21.15.103:0/816750128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fde50 msgr2=0x7f28ec0fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.718+0000 7f28f0949700 1 --2- 172.21.15.103:0/816750128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fde50 0x7f28ec0fe220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f28dc004940 tx=0x7f28dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- 172.21.15.103:0/816750128 shutdown_connections 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 --2- 172.21.15.103:0/816750128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 --2- 172.21.15.103:0/816750128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fde50 0x7f28ec0fe220 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- 172.21.15.103:0/816750128 >> 172.21.15.103:0/816750128 conn(0x7f28ec0f7870 msgr2=0x7f28ec0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- 172.21.15.103:0/816750128 shutdown_connections 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- 172.21.15.103:0/816750128 wait complete. 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 Processor -- start 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- start start 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 0x7f28ec113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec114500 con 0x7f28ec0fe7f0 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.719+0000 7f28f0949700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec10db00 con 0x7f28ec0fde50 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28e9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28ea59c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 0x7f28ec113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28e9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48596/0 (socket says 172.21.15.103:48596) 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28e9d9b700 1 -- 172.21.15.103:0/710897357 learned_addr learned my addr 172.21.15.103:0/710897357 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28ea59c700 1 -- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 msgr2=0x7f28ec113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28ea59c700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28ea59c700 1 -- 172.21.15.103:0/710897357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc005040 con 0x7f28ec0fde50 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28e9d9b700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:57.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28ea59c700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 0x7f28ec113970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28dc00e070 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28dc013c00 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec10dd80 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28dc012600 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.720+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec10e2c0 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.722+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec066a90 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.722+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f28dc012880 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.722+0000 7f28daffd700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 0x7f28d4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.722+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f28dc08f5e0 con 0x7f28ec0fde50 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.722+0000 7f28e9d9b700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 0x7f28d4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:57.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.723+0000 7f28e9d9b700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 0x7f28d4070850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28ec10ecc0 tx=0x7f28e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:57.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.728+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f28dc05c350 con 0x7f28ec0fde50 2024-03-22T02:37:57.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.946+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28ec109f10 con 0x7f28d406e3a0 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.955+0000 7f28daffd700 1 -- 172.21.15.103:0/710897357 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f28ec109f10 con 0x7f28d406e3a0 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (6m) 11s ago 17m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (4m) 11s ago 17m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (4m) 47s ago 16m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:37:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (6m) 11s ago 17m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (34s) 11s ago 12m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (16s) 11s ago 12m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (12m) 47s ago 12m 16.2M - 17.2.7-726-g9c4e0c34 837124da804f 93f573b329af 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 starting - - - - 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (8m) 11s ago 19m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (7m) 47s ago 16m 450M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (5m) 11s ago 19m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (4m) 47s ago 16m 39.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (7m) 11s ago 17m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 47s ago 16m 18.7M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (3m) 11s ago 15m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:37:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (3m) 11s ago 15m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:37:57.959 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (2m) 11s ago 14m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:37:57.959 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (118s) 47s ago 14m 97.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:37:57.959 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (84s) 47s ago 14m 91.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:37:57.959 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (50s) 47s ago 13m 12.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:37:57.959 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (6m) 11s ago 17m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.957+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 msgr2=0x7f28d4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.957+0000 7f28f0949700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 0x7f28d4070850 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f28ec10ecc0 tx=0x7f28e000b040 comp rx=0 tx=0).stop 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.957+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 msgr2=0x7f28ec113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.957+0000 7f28f0949700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 0x7f28ec113970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 shutdown_connections 2024-03-22T02:37:57.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f28d406e3a0 0x7f28d4070850 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28ec0fde50 0x7f28ec113970 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 --2- 172.21.15.103:0/710897357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28ec0fe7f0 0x7f28ec113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:57.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 >> 172.21.15.103:0/710897357 conn(0x7f28ec0f7870 msgr2=0x7f28ec0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:57.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 shutdown_connections 2024-03-22T02:37:57.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:57.958+0000 7f28f0949700 1 -- 172.21.15.103:0/710897357 wait complete. 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 -- 172.21.15.103:0/807687066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 msgr2=0x7f6e181020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/807687066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e181020f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6e08004940 tx=0x7f6e08009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 -- 172.21.15.103:0/807687066 shutdown_connections 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/807687066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 0x7f6e180fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/807687066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e181020f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 -- 172.21.15.103:0/807687066 >> 172.21.15.103:0/807687066 conn(0x7f6e180f78d0 msgr2=0x7f6e180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:58.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 -- 172.21.15.103:0/807687066 shutdown_connections 2024-03-22T02:37:58.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 -- 172.21.15.103:0/807687066 wait complete. 2024-03-22T02:37:58.061 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:57 smithi103 ceph-mon[115211]: pgmap v192: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:58.061 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:58.061 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:58.061 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:57 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.058+0000 7f6e1e54a700 1 Processor -- start 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e1e54a700 1 -- start start 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e1e54a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e1e54a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 0x7f6e181030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e1e54a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18103650 con 0x7f6e18101d20 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e1e54a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18103790 con 0x7f6e181026c0 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48628/0 (socket says 172.21.15.103:48628) 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 -- 172.21.15.103:0/866748539 learned_addr learned my addr 172.21.15.103:0/866748539 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e177fe700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 0x7f6e181030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 -- 172.21.15.103:0/866748539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 msgr2=0x7f6e181030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 0x7f6e181030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 -- 172.21.15.103:0/866748539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08005040 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.059+0000 7f6e17fff700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009b30 tx=0x7f6e08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.060+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e08021070 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.060+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e181039e0 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.060+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e08013c00 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.060+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1811a1a0 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.060+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0800e070 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.061+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e180fd470 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.061+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6e0801cad0 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.062+0000 7f6e14ff9700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 0x7f6e00070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.062+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6e0808f1c0 con 0x7f6e18101d20 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.062+0000 7f6e177fe700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 0x7f6e00070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.062+0000 7f6e177fe700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 0x7f6e00070850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e181044c0 tx=0x7f6e0c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.068+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6e08058440 con 0x7f6e18101d20 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.346+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6e1804bf60 con 0x7f6e18101d20 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.347+0000 7f6e14ff9700 1 -- 172.21.15.103:0/866748539 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f6e08004490 con 0x7f6e18101d20 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:37:58.349 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 3 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 1, 2024-03-22T02:37:58.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 13 2024-03-22T02:37:58.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:37:58.351 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:37:58.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:57 smithi122 ceph-mon[83368]: pgmap v192: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:37:58.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:58.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:37:58.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:57 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 msgr2=0x7f6e00070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 0x7f6e00070850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e181044c0 tx=0x7f6e0c00b040 comp rx=0 tx=0).stop 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 msgr2=0x7f6e18102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009b30 tx=0x7f6e08009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 shutdown_connections 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6e0006e3a0 0x7f6e00070850 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e18101d20 0x7f6e18102b80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 --2- 172.21.15.103:0/866748539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e181026c0 0x7f6e181030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 >> 172.21.15.103:0/866748539 conn(0x7f6e180f78d0 msgr2=0x7f6e18065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:58.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 shutdown_connections 2024-03-22T02:37:58.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.349+0000 7f6e1e54a700 1 -- 172.21.15.103:0/866748539 wait complete. 2024-03-22T02:37:58.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 -- 172.21.15.103:0/173274172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e00 msgr2=0x7f4c4c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 --2- 172.21.15.103:0/173274172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e00 0x7f4c4c1031d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c004940 tx=0x7f4c3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 -- 172.21.15.103:0/173274172 shutdown_connections 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 --2- 172.21.15.103:0/173274172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 --2- 172.21.15.103:0/173274172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c102e00 0x7f4c4c1031d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 -- 172.21.15.103:0/173274172 >> 172.21.15.103:0/173274172 conn(0x7f4c4c0f89b0 msgr2=0x7f4c4c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 -- 172.21.15.103:0/173274172 shutdown_connections 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.450+0000 7f4c546f2700 1 -- 172.21.15.103:0/173274172 wait complete. 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 Processor -- start 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 -- start start 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 0x7f4c4c113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c114710 con 0x7f4c4c1037a0 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c546f2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c10db50 con 0x7f4c4c102e00 2024-03-22T02:37:58.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48642/0 (socket says 172.21.15.103:48642) 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 -- 172.21.15.103:0/1587566849 learned_addr learned my addr 172.21.15.103:0/1587566849 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c5248e700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 0x7f4c4c113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 -- 172.21.15.103:0/1587566849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 msgr2=0x7f4c4c113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 0x7f4c4c113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 -- 172.21.15.103:0/1587566849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c005040 con 0x7f4c4c1037a0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c5248e700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 0x7f4c4c113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.451+0000 7f4c51c8d700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c7e0 tx=0x7f4c48007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.452+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c48012810 con 0x7f4c4c1037a0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.452+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c48012e40 con 0x7f4c4c1037a0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.452+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c10de30 con 0x7f4c4c1037a0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.452+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4800a550 con 0x7f4c4c1037a0 2024-03-22T02:37:58.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.452+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c10e3f0 con 0x7f4c4c1037a0 2024-03-22T02:37:58.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.453+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c100110 con 0x7f4c4c1037a0 2024-03-22T02:37:58.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.453+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4c48011380 con 0x7f4c4c1037a0 2024-03-22T02:37:58.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.453+0000 7f4c42ffd700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 0x7f4c38070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.454+0000 7f4c5248e700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 0x7f4c38070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.454+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4c4808e140 con 0x7f4c4c1037a0 2024-03-22T02:37:58.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.454+0000 7f4c5248e700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 0x7f4c38070850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c004560 tx=0x7f4c3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.458+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c4805aef0 con 0x7f4c4c1037a0 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.702+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4c4c0fe550 con 0x7f4c4c1037a0 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.702+0000 7f4c42ffd700 1 -- 172.21.15.103:0/1587566849 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f4c48011630 con 0x7f4c4c1037a0 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stdout:e29 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:37:58.704 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:37:58.705 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:37:58.706 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:24285} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1723132276,v1:172.21.15.122:6827/1723132276] compat {c=[1],r=[1],i=[7ff]}] 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:37:58.707 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 msgr2=0x7f4c38070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 0x7f4c38070850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c004560 tx=0x7f4c3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 msgr2=0x7f4c4c1140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c7e0 tx=0x7f4c48007f90 comp rx=0 tx=0).stop 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 shutdown_connections 2024-03-22T02:37:58.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4c3806e3a0 0x7f4c38070850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c4c102e00 0x7f4c4c113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 --2- 172.21.15.103:0/1587566849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c4c1037a0 0x7f4c4c1140c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.704+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 >> 172.21.15.103:0/1587566849 conn(0x7f4c4c0f89b0 msgr2=0x7f4c4c1073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.705+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 shutdown_connections 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.705+0000 7f4c546f2700 1 -- 172.21.15.103:0/1587566849 wait complete. 2024-03-22T02:37:58.709 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 29 2024-03-22T02:37:58.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- 172.21.15.103:0/3939669730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e980fb650 msgr2=0x7f4e980fbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/3939669730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e980fb650 0x7f4e980fbac0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4e80004940 tx=0x7f4e80009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- 172.21.15.103:0/3939669730 shutdown_connections 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/3939669730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e980fb650 0x7f4e980fbac0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/3939669730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e981015c0 0x7f4e98101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- 172.21.15.103:0/3939669730 >> 172.21.15.103:0/3939669730 conn(0x7f4e98071530 msgr2=0x7f4e98071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- 172.21.15.103:0/3939669730 shutdown_connections 2024-03-22T02:37:58.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- 172.21.15.103:0/3939669730 wait complete. 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 Processor -- start 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.807+0000 7f4e9cacb700 1 -- start start 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9cacb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e980fb650 0x7f4e980fd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9cacb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9cacb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980ff130 con 0x7f4e981015c0 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9cacb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980fdc60 con 0x7f4e980fb650 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48662/0 (socket says 172.21.15.103:48662) 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 -- 172.21.15.103:0/2887484461 learned_addr learned my addr 172.21.15.103:0/2887484461 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9659c700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e980fb650 0x7f4e980fd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 -- 172.21.15.103:0/2887484461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e980fb650 msgr2=0x7f4e980fd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e980fb650 0x7f4e980fd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 -- 172.21.15.103:0/2887484461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e80005040 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8dd9b700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000b8b0 tx=0x7f4e80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8000e040 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e980fdee0 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e80004290 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.808+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8001c3f0 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.809+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9811a090 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.809+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e980fcd20 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.810+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4e800137b0 con 0x7f4e981015c0 2024-03-22T02:37:58.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.810+0000 7f4e8f7fe700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 0x7f4e84070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:58.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.810+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4e8008f100 con 0x7f4e981015c0 2024-03-22T02:37:58.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.810+0000 7f4e9659c700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 0x7f4e84070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:58.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.811+0000 7f4e9659c700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 0x7f4e84070850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e88004990 tx=0x7f4e88009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:58.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:58.815+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e80058380 con 0x7f4e981015c0 2024-03-22T02:37:59.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.031+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e980fa8b0 con 0x7f4e8406e3a0 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] up:boot 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='client.44295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/866748539' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:59.033 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:58 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1587566849' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.032+0000 7f4e8f7fe700 1 -- 172.21.15.103:0/2887484461 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4e980fa8b0 con 0x7f4e8406e3a0 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:37:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "14/21 daemons upgraded", 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:37:59.035 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 msgr2=0x7f4e84070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 0x7f4e84070850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4e88004990 tx=0x7f4e88009ee0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 msgr2=0x7f4e980fd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000b8b0 tx=0x7f4e80009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 shutdown_connections 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4e8406e3a0 0x7f4e84070850 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e980fb650 0x7f4e980fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 --2- 172.21.15.103:0/2887484461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e981015c0 0x7f4e980fd720 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 >> 172.21.15.103:0/2887484461 conn(0x7f4e98071530 msgr2=0x7f4e980fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:59.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 shutdown_connections 2024-03-22T02:37:59.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.034+0000 7f4e9cacb700 1 -- 172.21.15.103:0/2887484461 wait complete. 2024-03-22T02:37:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 -- 172.21.15.103:0/665878389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 msgr2=0x7fda38106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/665878389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38106ec0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda34004940 tx=0x7fda34009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 -- 172.21.15.103:0/665878389 shutdown_connections 2024-03-22T02:37:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/665878389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38106ec0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/665878389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 0x7fda380fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 -- 172.21.15.103:0/665878389 >> 172.21.15.103:0/665878389 conn(0x7fda380f7850 msgr2=0x7fda380f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 -- 172.21.15.103:0/665878389 shutdown_connections 2024-03-22T02:37:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.135+0000 7fda3fcc9700 1 -- 172.21.15.103:0/665878389 wait complete. 2024-03-22T02:37:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 Processor -- start 2024-03-22T02:37:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 -- start start 2024-03-22T02:37:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 0x7fda38113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:59.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda381145a0 con 0x7fda380fe7f0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3fcc9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3810db30 con 0x7fda380fde50 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48684/0 (socket says 172.21.15.103:48684) 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 -- 172.21.15.103:0/3638827936 learned_addr learned my addr 172.21.15.103:0/3638827936 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3da65700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 0x7fda38113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 -- 172.21.15.103:0/3638827936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 msgr2=0x7fda38113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 0x7fda38113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 -- 172.21.15.103:0/3638827936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda34005040 con 0x7fda380fe7f0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.136+0000 7fda3d264700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b3b0 tx=0x7fda34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.137+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3400e040 con 0x7fda380fe7f0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.137+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda34013420 con 0x7fda380fe7f0 2024-03-22T02:37:59.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.137+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda3401c5f0 con 0x7fda380fe7f0 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.137+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3810ddb0 con 0x7fda380fe7f0 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.137+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3810e2f0 con 0x7fda380fe7f0 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.138+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda38066a90 con 0x7fda380fe7f0 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.138+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fda34013590 con 0x7fda380fe7f0 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.138+0000 7fda2e7fc700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 0x7fda24070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:59.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.139+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fda3408efa0 con 0x7fda380fe7f0 2024-03-22T02:37:59.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.139+0000 7fda3da65700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 0x7fda24070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:59.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.139+0000 7fda3da65700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 0x7fda24070850 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda28009bf0 tx=0x7fda280091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:59.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.143+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda340c3740 con 0x7fda380fe7f0 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] up:boot 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:37:59.312 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='client.44295 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:37:59.313 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/866748539' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:37:59.313 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:58 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1587566849' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:37:59.422 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.421+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fda3810efd0 con 0x7fda380fe7f0 2024-03-22T02:37:59.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.421+0000 7fda2e7fc700 1 -- 172.21.15.103:0/3638827936 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fda34058220 con 0x7fda380fe7f0 2024-03-22T02:37:59.423 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:37:59.423 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:37:59.424 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:37:59.424 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:37:59.424 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:37:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 msgr2=0x7fda24070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 0x7fda24070850 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda28009bf0 tx=0x7fda280091a0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 msgr2=0x7fda38113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:59.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b3b0 tx=0x7fda34009e30 comp rx=0 tx=0).stop 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 shutdown_connections 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fda2406e3a0 0x7fda24070850 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda380fde50 0x7fda38113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 --2- 172.21.15.103:0/3638827936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda380fe7f0 0x7fda38113ec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.423+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 >> 172.21.15.103:0/3638827936 conn(0x7fda380f7850 msgr2=0x7fda38103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.424+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 shutdown_connections 2024-03-22T02:37:59.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:37:59.424+0000 7fda3fcc9700 1 -- 172.21.15.103:0/3638827936 wait complete. 2024-03-22T02:38:00.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:59 smithi122 ceph-mon[83368]: pgmap v193: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:00.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:37:59 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3638827936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:38:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:59 smithi103 ceph-mon[115211]: pgmap v193: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:37:59 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3638827936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:38:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:00 smithi122 ceph-mon[83368]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:00 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:00 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:00 smithi103 ceph-mon[115211]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:00 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:01.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:00 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:01 smithi122 ceph-mon[83368]: pgmap v194: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:01 smithi103 ceph-mon[115211]: pgmap v194: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:03 smithi122 ceph-mon[83368]: pgmap v195: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:03 smithi103 ceph-mon[115211]: pgmap v195: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: pgmap v196: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:06.189 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:05 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: pgmap v196: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:06.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:05 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:07.999 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:07 smithi122 ceph-mon[83368]: pgmap v197: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:07.999 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:07.999 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:38:08.000 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:07 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:07 smithi103 ceph-mon[115211]: pgmap v197: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lvsegg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:38:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:07 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:09.183 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:08 smithi122 ceph-mon[83368]: Upgrade: Updating mds.cephfs.smithi122.lvsegg 2024-03-22T02:38:09.184 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:08 smithi122 ceph-mon[83368]: Deploying daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:38:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:08 smithi103 ceph-mon[115211]: Upgrade: Updating mds.cephfs.smithi122.lvsegg 2024-03-22T02:38:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:08 smithi103 ceph-mon[115211]: Deploying daemon mds.cephfs.smithi122.lvsegg on smithi122 2024-03-22T02:38:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:09 smithi122 ceph-mon[83368]: pgmap v198: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:09 smithi103 ceph-mon[115211]: pgmap v198: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:11.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:10 smithi122 ceph-mon[83368]: osdmap e84: 6 total, 6 up, 6 in 2024-03-22T02:38:11.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:10 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:38:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:10 smithi103 ceph-mon[115211]: osdmap e84: 6 total, 6 up, 6 in 2024-03-22T02:38:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:10 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-22T02:38:12.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:11 smithi122 ceph-mon[83368]: pgmap v200: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:12.276 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:11 smithi103 ceph-mon[115211]: pgmap v200: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:13 smithi122 ceph-mon[83368]: pgmap v201: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:13 smithi103 ceph-mon[115211]: pgmap v201: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] up:boot 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] up:boot 2024-03-22T02:38:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:38:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:38:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:16.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:15 smithi122 ceph-mon[83368]: pgmap v202: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:15 smithi103 ceph-mon[115211]: pgmap v202: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:17 smithi103 ceph-mon[115211]: pgmap v203: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:17 smithi122 ceph-mon[83368]: pgmap v203: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:17.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:18 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:18 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:38:19.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:19 smithi122 ceph-mon[83368]: pgmap v204: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:19 smithi103 ceph-mon[115211]: pgmap v204: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:21 smithi122 ceph-mon[83368]: pgmap v205: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:21 smithi103 ceph-mon[115211]: pgmap v205: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:23 smithi122 ceph-mon[83368]: pgmap v206: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:23 smithi103 ceph-mon[115211]: pgmap v206: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:25 smithi122 ceph-mon[83368]: pgmap v207: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:25 smithi103 ceph-mon[115211]: pgmap v207: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:27 smithi122 ceph-mon[83368]: pgmap v208: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:27 smithi103 ceph-mon[115211]: pgmap v208: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 -- 172.21.15.103:0/1503449747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 msgr2=0x7f6d541031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/1503449747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d541031b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004940 tx=0x7f6d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 -- 172.21.15.103:0/1503449747 shutdown_connections 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/1503449747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 0x7f6d540fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/1503449747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d541031b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 -- 172.21.15.103:0/1503449747 >> 172.21.15.103:0/1503449747 conn(0x7f6d540f8950 msgr2=0x7f6d540fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 -- 172.21.15.103:0/1503449747 shutdown_connections 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 -- 172.21.15.103:0/1503449747 wait complete. 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.541+0000 7f6d59bb7700 1 Processor -- start 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d59bb7700 1 -- start start 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d59bb7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d59bb7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 0x7f6d54070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d59bb7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54072210 con 0x7f6d54102de0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d59bb7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54070d40 con 0x7f6d54103780 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59586/0 (socket says 172.21.15.103:59586) 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 -- 172.21.15.103:0/3619644443 learned_addr learned my addr 172.21.15.103:0/3619644443 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d52ffd700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 0x7f6d54070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 -- 172.21.15.103:0/3619644443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 msgr2=0x7f6d54070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 0x7f6d54070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 -- 172.21.15.103:0/3619644443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44005040 con 0x7f6d54102de0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.542+0000 7f6d537fe700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400b3b0 tx=0x7f6d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.543+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4400e040 con 0x7f6d54102de0 2024-03-22T02:38:29.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.543+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d54070fc0 con 0x7f6d54102de0 2024-03-22T02:38:29.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.543+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d44013420 con 0x7f6d54102de0 2024-03-22T02:38:29.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.543+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4401c630 con 0x7f6d54102de0 2024-03-22T02:38:29.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.543+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5411a0d0 con 0x7f6d54102de0 2024-03-22T02:38:29.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.544+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d541000f0 con 0x7f6d54102de0 2024-03-22T02:38:29.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.544+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6d4400baf0 con 0x7f6d54102de0 2024-03-22T02:38:29.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.545+0000 7f6d3bfff700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 0x7f6d3c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.545+0000 7f6d52ffd700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 0x7f6d3c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.545+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d4408f040 con 0x7f6d54102de0 2024-03-22T02:38:29.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.545+0000 7f6d52ffd700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 0x7f6d3c070850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d540719f0 tx=0x7f6d4800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:29.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.550+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d440582c0 con 0x7f6d54102de0 2024-03-22T02:38:29.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.766+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d540717a0 con 0x7f6d3c06e3a0 2024-03-22T02:38:29.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.767+0000 7f6d3bfff700 1 -- 172.21.15.103:0/3619644443 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6d540717a0 con 0x7f6d3c06e3a0 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 msgr2=0x7f6d3c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 0x7f6d3c070850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d540719f0 tx=0x7f6d4800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 msgr2=0x7f6d540702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400b3b0 tx=0x7f6d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 shutdown_connections 2024-03-22T02:38:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d3c06e3a0 0x7f6d3c070850 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d54102de0 0x7f6d540702c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 --2- 172.21.15.103:0/3619644443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6d54103780 0x7f6d54070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 >> 172.21.15.103:0/3619644443 conn(0x7f6d540f8950 msgr2=0x7f6d54107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:29.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.769+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 shutdown_connections 2024-03-22T02:38:29.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.770+0000 7f6d59bb7700 1 -- 172.21.15.103:0/3619644443 wait complete. 2024-03-22T02:38:29.781 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:38:29.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 -- 172.21.15.103:0/142009146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c071f90 msgr2=0x7f9f1c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 --2- 172.21.15.103:0/142009146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c071f90 0x7f9f1c070360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9f10004970 tx=0x7f9f10009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 -- 172.21.15.103:0/142009146 shutdown_connections 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 --2- 172.21.15.103:0/142009146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 --2- 172.21.15.103:0/142009146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c071f90 0x7f9f1c070360 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 -- 172.21.15.103:0/142009146 >> 172.21.15.103:0/142009146 conn(0x7f9f1c0f89b0 msgr2=0x7f9f1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.871+0000 7f9f22f28700 1 -- 172.21.15.103:0/142009146 shutdown_connections 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 -- 172.21.15.103:0/142009146 wait complete. 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 Processor -- start 2024-03-22T02:38:29.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 -- start start 2024-03-22T02:38:29.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 0x7f9f1c11a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c11ab50 con 0x7f9f1c0708a0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f22f28700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c114150 con 0x7f9f1c071f90 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.872+0000 7f9f20cc4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59616/0 (socket says 172.21.15.103:59616) 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 -- 172.21.15.103:0/4165038114 learned_addr learned my addr 172.21.15.103:0/4165038114 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f1bfff700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 0x7f9f1c11a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 -- 172.21.15.103:0/4165038114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 msgr2=0x7f9f1c11a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 0x7f9f1c11a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 -- 172.21.15.103:0/4165038114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10005040 con 0x7f9f1c0708a0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f1bfff700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 0x7f9f1c11a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f20cc4700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f10004f50 tx=0x7f9f10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1000e040 con 0x7f9f1c0708a0 2024-03-22T02:38:29.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f1000bc10 con 0x7f9f1c0708a0 2024-03-22T02:38:29.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.873+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c1143d0 con 0x7f9f1c0708a0 2024-03-22T02:38:29.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.874+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1001c630 con 0x7f9f1c0708a0 2024-03-22T02:38:29.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.874+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c114910 con 0x7f9f1c0708a0 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.874+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c071aa0 con 0x7f9f1c0708a0 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.875+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9f10013420 con 0x7f9f1c0708a0 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.875+0000 7f9f197fa700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 0x7f9f0c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.875+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9f1008efd0 con 0x7f9f1c0708a0 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.875+0000 7f9f1bfff700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 0x7f9f0c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:29.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.876+0000 7f9f1bfff700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 0x7f9f0c070850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c10cec0 tx=0x7f9f08008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:29.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:29.880+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f10058310 con 0x7f9f1c0708a0 2024-03-22T02:38:30.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.095+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f1c1105a0 con 0x7f9f0c06e3a0 2024-03-22T02:38:30.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.096+0000 7f9f197fa700 1 -- 172.21.15.103:0/4165038114 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9f1c1105a0 con 0x7f9f0c06e3a0 2024-03-22T02:38:30.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 msgr2=0x7f9f0c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 0x7f9f0c070850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c10cec0 tx=0x7f9f08008040 comp rx=0 tx=0).stop 2024-03-22T02:38:30.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 msgr2=0x7f9f1c119f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f10004f50 tx=0x7f9f10009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 shutdown_connections 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9f0c06e3a0 0x7f9f0c070850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f1c0708a0 0x7f9f1c119f30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 --2- 172.21.15.103:0/4165038114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1c071f90 0x7f9f1c11a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 >> 172.21.15.103:0/4165038114 conn(0x7f9f1c0f89b0 msgr2=0x7f9f1c0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 shutdown_connections 2024-03-22T02:38:30.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.098+0000 7f9f22f28700 1 -- 172.21.15.103:0/4165038114 wait complete. 2024-03-22T02:38:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:29 smithi103 ceph-mon[115211]: pgmap v209: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- 172.21.15.103:0/1196054964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 msgr2=0x7f51d80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 --2- 172.21.15.103:0/1196054964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d80fce10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51c0004940 tx=0x7f51c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- 172.21.15.103:0/1196054964 shutdown_connections 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 --2- 172.21.15.103:0/1196054964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d80fce10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 --2- 172.21.15.103:0/1196054964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 0x7f51d81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- 172.21.15.103:0/1196054964 >> 172.21.15.103:0/1196054964 conn(0x7f51d80f89b0 msgr2=0x7f51d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- 172.21.15.103:0/1196054964 shutdown_connections 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- 172.21.15.103:0/1196054964 wait complete. 2024-03-22T02:38:30.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 Processor -- start 2024-03-22T02:38:30.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- start start 2024-03-22T02:38:30.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 0x7f51d8113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8114510 con 0x7f51d81037a0 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.200+0000 7f51de28a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d810db10 con 0x7f51d8102e00 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59638/0 (socket says 172.21.15.103:59638) 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 -- 172.21.15.103:0/1986375610 learned_addr learned my addr 172.21.15.103:0/1986375610 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51d7fff700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 0x7f51d8113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 -- 172.21.15.103:0/1986375610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 msgr2=0x7f51d8113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 0x7f51d8113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 -- 172.21.15.103:0/1986375610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0005040 con 0x7f51d81037a0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51cffff700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009180 tx=0x7f51c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c000e040 con 0x7f51d81037a0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51c0013420 con 0x7f51d81037a0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c001c610 con 0x7f51d81037a0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d810dd90 con 0x7f51d81037a0 2024-03-22T02:38:30.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.201+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d810e350 con 0x7f51d81037a0 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.202+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d81052d0 con 0x7f51d81037a0 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.203+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f51c0013a00 con 0x7f51d81037a0 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.203+0000 7f51d57fa700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 0x7f51c4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.203+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f51c008edc0 con 0x7f51d81037a0 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.203+0000 7f51d7fff700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 0x7f51c4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.204+0000 7f51d7fff700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 0x7f51c4070850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004f40 tx=0x7f51c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.208+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f51c005bc30 con 0x7f51d81037a0 2024-03-22T02:38:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:29 smithi122 ceph-mon[83368]: pgmap v209: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:30.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.417+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d804c7b0 con 0x7f51c406e3a0 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.427+0000 7f51d57fa700 1 -- 172.21.15.103:0/1986375610 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f51d804c7b0 con 0x7f51c406e3a0 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (7m) 44s ago 18m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (4m) 44s ago 18m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (4m) 14s ago 17m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (6m) 44s ago 17m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:38:30.428 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (66s) 44s ago 12m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (49s) 44s ago 12m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (16s) 14s ago 12m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (33s) 14s ago 12m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (8m) 44s ago 19m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (7m) 14s ago 17m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (5m) 44s ago 19m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (5m) 14s ago 17m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (7m) 44s ago 18m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 14s ago 17m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (4m) 44s ago 16m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (3m) 44s ago 15m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (3m) 44s ago 15m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (2m) 14s ago 15m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:38:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (116s) 14s ago 14m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:38:30.430 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (82s) 14s ago 14m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:38:30.430 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (7m) 44s ago 17m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:38:30.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 msgr2=0x7f51c4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 0x7f51c4070850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f51c8004f40 tx=0x7f51c800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 msgr2=0x7f51d8113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009180 tx=0x7f51c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 shutdown_connections 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f51c406e3a0 0x7f51c4070850 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51d8102e00 0x7f51d8113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 --2- 172.21.15.103:0/1986375610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51d81037a0 0x7f51d8113ec0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 >> 172.21.15.103:0/1986375610 conn(0x7f51d80f89b0 msgr2=0x7f51d81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.429+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 shutdown_connections 2024-03-22T02:38:30.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.430+0000 7f51de28a700 1 -- 172.21.15.103:0/1986375610 wait complete. 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- 172.21.15.103:0/1976531493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c4101d40 msgr2=0x7f26c4102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1976531493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c4101d40 0x7f26c4102110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26b4004940 tx=0x7f26b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- 172.21.15.103:0/1976531493 shutdown_connections 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1976531493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c40fc420 0x7f26c40fc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1976531493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c4101d40 0x7f26c4102110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- 172.21.15.103:0/1976531493 >> 172.21.15.103:0/1976531493 conn(0x7f26c4071300 msgr2=0x7f26c4071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- 172.21.15.103:0/1976531493 shutdown_connections 2024-03-22T02:38:30.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- 172.21.15.103:0/1976531493 wait complete. 2024-03-22T02:38:30.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 Processor -- start 2024-03-22T02:38:30.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.530+0000 7f26c9da1700 1 -- start start 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c9da1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c9da1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4101d40 0x7f26c410ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c9da1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c410f570 con 0x7f26c40fc420 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c9da1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c410f6b0 con 0x7f26c4101d40 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59668/0 (socket says 172.21.15.103:59668) 2024-03-22T02:38:30.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 -- 172.21.15.103:0/1710357259 learned_addr learned my addr 172.21.15.103:0/1710357259 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c3fff700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4101d40 0x7f26c410ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 -- 172.21.15.103:0/1710357259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4101d40 msgr2=0x7f26c410ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4101d40 0x7f26c410ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 -- 172.21.15.103:0/1710357259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4005040 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c8d9f700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009b30 tx=0x7f26b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.531+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b400e070 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.532+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4117a50 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.532+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26b4013c00 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.532+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b400b650 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.532+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c4117eb0 con 0x7f26c40fc420 2024-03-22T02:38:30.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.532+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c4104180 con 0x7f26c40fc420 2024-03-22T02:38:30.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.533+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f26b401ca40 con 0x7f26c40fc420 2024-03-22T02:38:30.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.533+0000 7f26c17fa700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 0x7f26ac070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.534+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f26b408f6e0 con 0x7f26c40fc420 2024-03-22T02:38:30.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.534+0000 7f26c3fff700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 0x7f26ac070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.534+0000 7f26c3fff700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 0x7f26ac070850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f26c40fbd10 tx=0x7f26b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.538+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26b4058960 con 0x7f26c40fc420 2024-03-22T02:38:30.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.816+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f26c40fd430 con 0x7f26c40fc420 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.816+0000 7f26c17fa700 1 -- 172.21.15.103:0/1710357259 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f26b4004490 con 0x7f26c40fc420 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:38:30.817 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:38:30.818 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 msgr2=0x7f26ac070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 0x7f26ac070850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f26c40fbd10 tx=0x7f26b800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 msgr2=0x7f26c410e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009b30 tx=0x7f26b4009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 shutdown_connections 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26ac06e3a0 0x7f26ac070850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c40fc420 0x7f26c410e9e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 --2- 172.21.15.103:0/1710357259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4101d40 0x7f26c410ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 >> 172.21.15.103:0/1710357259 conn(0x7f26c4071300 msgr2=0x7f26c40f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 shutdown_connections 2024-03-22T02:38:30.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.819+0000 7f26c9da1700 1 -- 172.21.15.103:0/1710357259 wait complete. 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 -- 172.21.15.103:0/165625862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fe830 msgr2=0x7f11fc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 --2- 172.21.15.103:0/165625862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fe830 0x7f11fc106f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f11ec004940 tx=0x7f11ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 -- 172.21.15.103:0/165625862 shutdown_connections 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 --2- 172.21.15.103:0/165625862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fe830 0x7f11fc106f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 --2- 172.21.15.103:0/165625862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 -- 172.21.15.103:0/165625862 >> 172.21.15.103:0/165625862 conn(0x7f11fc072bc0 msgr2=0x7f11fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:30.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 -- 172.21.15.103:0/165625862 shutdown_connections 2024-03-22T02:38:30.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.921+0000 7f1203775700 1 -- 172.21.15.103:0/165625862 wait complete. 2024-03-22T02:38:30.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 Processor -- start 2024-03-22T02:38:30.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 -- start start 2024-03-22T02:38:30.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 0x7f11fc0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0fc820 con 0x7f11fc0fde90 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1203775700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0fc960 con 0x7f11fc0fe830 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59686/0 (socket says 172.21.15.103:59686) 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 -- 172.21.15.103:0/2763184776 learned_addr learned my addr 172.21.15.103:0/2763184776 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1200d10700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 0x7f11fc0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 -- 172.21.15.103:0/2763184776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 msgr2=0x7f11fc0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 0x7f11fc0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1201511700 1 -- 172.21.15.103:0/2763184776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec005040 con 0x7f11fc0fde90 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.922+0000 7f1200d10700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 0x7f11fc0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f1201511700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f11f800eb40 tx=0x7f11f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f8007cc0 con 0x7f11fc0fde90 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11f800f040 con 0x7f11fc0fde90 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc0fcbb0 con 0x7f11fc0fde90 2024-03-22T02:38:30.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f800ca10 con 0x7f11fc0fde90 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.923+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc11a1d0 con 0x7f11fc0fde90 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.924+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc065e30 con 0x7f11fc0fde90 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.924+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f11f800cb70 con 0x7f11fc0fde90 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.925+0000 7f11f1ffb700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 0x7f11e8070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.925+0000 7f1200d10700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 0x7f11e8070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.925+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11f808d8c0 con 0x7f11fc0fde90 2024-03-22T02:38:30.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.925+0000 7f1200d10700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 0x7f11e8070860 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0fd690 tx=0x7f11ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:30.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:30.929+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f11f8056bf0 con 0x7f11fc0fde90 2024-03-22T02:38:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:30 smithi103 ceph-mon[115211]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:30 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1710357259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.175+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f11fc04bf60 con 0x7f11fc0fde90 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.175+0000 7f11f1ffb700 1 -- 172.21.15.103:0/2763184776 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f11f805a2b0 con 0x7f11fc0fde90 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:38:31.176 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:38:31.177 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:38:31.178 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:38:31.179 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 msgr2=0x7f11e8070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 0x7f11e8070860 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0fd690 tx=0x7f11ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 msgr2=0x7f11fc0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f11f800eb40 tx=0x7f11f800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 shutdown_connections 2024-03-22T02:38:31.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f11e806e3b0 0x7f11e8070860 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f11fc0fde90 0x7f11fc0fbd50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 --2- 172.21.15.103:0/2763184776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11fc0fe830 0x7f11fc0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 >> 172.21.15.103:0/2763184776 conn(0x7f11fc072bc0 msgr2=0x7f11fc103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 shutdown_connections 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.178+0000 7f1203775700 1 -- 172.21.15.103:0/2763184776 wait complete. 2024-03-22T02:38:31.181 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- 172.21.15.103:0/2978785692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8102680 msgr2=0x7fb7e80fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 --2- 172.21.15.103:0/2978785692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8102680 0x7fb7e80fbcf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc004940 tx=0x7fb7dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- 172.21.15.103:0/2978785692 shutdown_connections 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 --2- 172.21.15.103:0/2978785692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8102680 0x7fb7e80fbcf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 --2- 172.21.15.103:0/2978785692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- 172.21.15.103:0/2978785692 >> 172.21.15.103:0/2978785692 conn(0x7fb7e80f7870 msgr2=0x7fb7e80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- 172.21.15.103:0/2978785692 shutdown_connections 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- 172.21.15.103:0/2978785692 wait complete. 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 Processor -- start 2024-03-22T02:38:31.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.280+0000 7fb7ed512700 1 -- start start 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8102680 0x7fb7e8112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8113370 con 0x7fb7e8101ce0 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e810c970 con 0x7fb7e8102680 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59706/0 (socket says 172.21.15.103:59706) 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 -- 172.21.15.103:0/615896270 learned_addr learned my addr 172.21.15.103:0/615896270 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e67fc700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8102680 0x7fb7e8112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 -- 172.21.15.103:0/615896270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8102680 msgr2=0x7fb7e8112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8102680 0x7fb7e8112d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 -- 172.21.15.103:0/615896270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc005040 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7e6ffd700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800eb40 tx=0x7fb7d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7d8007cc0 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e810cbf0 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7d800f040 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7d800ca10 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.281+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e810d100 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.282+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e80fd430 con 0x7fb7e8101ce0 2024-03-22T02:38:31.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.283+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb7d800cc80 con 0x7fb7e8101ce0 2024-03-22T02:38:31.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.283+0000 7fb7cf7fe700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 0x7fb7d0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.283+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb7d808d9e0 con 0x7fb7e8101ce0 2024-03-22T02:38:31.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.283+0000 7fb7e67fc700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 0x7fb7d0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.283+0000 7fb7e67fc700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 0x7fb7d0070850 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc004890 tx=0x7fb7dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:31.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.288+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7d805a7c0 con 0x7fb7e8101ce0 2024-03-22T02:38:31.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:30 smithi122 ceph-mon[83368]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:31.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:30 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1710357259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:38:31.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.500+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7e8108e10 con 0x7fb7d006e3a0 2024-03-22T02:38:31.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.502+0000 7fb7cf7fe700 1 -- 172.21.15.103:0/615896270 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb7e8108e10 con 0x7fb7d006e3a0 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:38:31.503 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:38:31.504 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:38:31.504 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:38:31.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 msgr2=0x7fb7d0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 0x7fb7d0070850 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc004890 tx=0x7fb7dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 msgr2=0x7fb7e81127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800eb40 tx=0x7fb7d800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 shutdown_connections 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb7d006e3a0 0x7fb7d0070850 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e8101ce0 0x7fb7e81127e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 --2- 172.21.15.103:0/615896270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7e8102680 0x7fb7e8112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 >> 172.21.15.103:0/615896270 conn(0x7fb7e80f7870 msgr2=0x7fb7e80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 shutdown_connections 2024-03-22T02:38:31.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.504+0000 7fb7ed512700 1 -- 172.21.15.103:0/615896270 wait complete. 2024-03-22T02:38:31.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 -- 172.21.15.103:0/1618096037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 msgr2=0x7f19981020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 --2- 172.21.15.103:0/1618096037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981020f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1994004940 tx=0x7f1994009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 -- 172.21.15.103:0/1618096037 shutdown_connections 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 --2- 172.21.15.103:0/1618096037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 0x7f19980fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 --2- 172.21.15.103:0/1618096037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981020f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 -- 172.21.15.103:0/1618096037 >> 172.21.15.103:0/1618096037 conn(0x7f19980f7870 msgr2=0x7f19980f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 -- 172.21.15.103:0/1618096037 shutdown_connections 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.606+0000 7f19a0293700 1 -- 172.21.15.103:0/1618096037 wait complete. 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 Processor -- start 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 -- start start 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 0x7f1998112e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1998113460 con 0x7f1998101d20 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f19a0293700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199810ca60 con 0x7f19981026c0 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59724/0 (socket says 172.21.15.103:59724) 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 -- 172.21.15.103:0/2390311307 learned_addr learned my addr 172.21.15.103:0/2390311307 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199d82e700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 0x7f1998112e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 -- 172.21.15.103:0/2390311307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 msgr2=0x7f1998112e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 0x7f1998112e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 -- 172.21.15.103:0/2390311307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994005040 con 0x7f1998101d20 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.607+0000 7f199e02f700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f199400b3b0 tx=0x7f1994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.608+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f199400e040 con 0x7f1998101d20 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.608+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f199400bd00 con 0x7f1998101d20 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.608+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199810cce0 con 0x7f1998101d20 2024-03-22T02:38:31.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.608+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1994013660 con 0x7f1998101d20 2024-03-22T02:38:31.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.608+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199810d240 con 0x7f1998101d20 2024-03-22T02:38:31.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.609+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19980fd470 con 0x7f1998101d20 2024-03-22T02:38:31.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.609+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f199401e420 con 0x7f1998101d20 2024-03-22T02:38:31.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.609+0000 7f198effd700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 0x7f1984070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:31.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.610+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1994091bb0 con 0x7f1998101d20 2024-03-22T02:38:31.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.610+0000 7f199d82e700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 0x7f1984070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:31.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.610+0000 7f199d82e700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 0x7f1984070850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f199810dc20 tx=0x7f198800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.614+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f199405e960 con 0x7f1998101d20 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.893+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1998108e20 con 0x7f1998101d20 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.893+0000 7f198effd700 1 -- 172.21.15.103:0/2390311307 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f199405e4f0 con 0x7f1998101d20 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:38:31.894 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 msgr2=0x7f1984070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 0x7f1984070850 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f199810dc20 tx=0x7f198800b040 comp rx=0 tx=0).stop 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 msgr2=0x7f19981128d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f199400b3b0 tx=0x7f1994009e30 comp rx=0 tx=0).stop 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 shutdown_connections 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f198406e3a0 0x7f1984070850 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1998101d20 0x7f19981128d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 --2- 172.21.15.103:0/2390311307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19981026c0 0x7f1998112e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:31.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 >> 172.21.15.103:0/2390311307 conn(0x7f19980f7870 msgr2=0x7f19980f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:31.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 shutdown_connections 2024-03-22T02:38:31.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:38:31.895+0000 7f19a0293700 1 -- 172.21.15.103:0/2390311307 wait complete. 2024-03-22T02:38:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: pgmap v210: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2763184776' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:38:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:31 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2390311307' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: pgmap v210: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2763184776' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:31 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2390311307' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:38:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:32 smithi122 ceph-mon[83368]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:32 smithi103 ceph-mon[115211]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:38:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:33 smithi122 ceph-mon[83368]: pgmap v211: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:33 smithi103 ceph-mon[115211]: pgmap v211: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:35 smithi122 ceph-mon[83368]: pgmap v212: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:35 smithi103 ceph-mon[115211]: pgmap v212: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:37 smithi122 ceph-mon[83368]: pgmap v213: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:37 smithi103 ceph-mon[115211]: pgmap v213: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:39 smithi122 ceph-mon[83368]: pgmap v214: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:39 smithi103 ceph-mon[115211]: pgmap v214: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:41 smithi122 ceph-mon[83368]: pgmap v215: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:41 smithi103 ceph-mon[115211]: pgmap v215: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:43 smithi122 ceph-mon[83368]: pgmap v216: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:43 smithi103 ceph-mon[115211]: pgmap v216: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:45 smithi122 ceph-mon[83368]: pgmap v217: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:45 smithi103 ceph-mon[115211]: pgmap v217: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:46 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:46 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:38:48.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:47 smithi122 ceph-mon[83368]: pgmap v218: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:47 smithi103 ceph-mon[115211]: pgmap v218: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:49 smithi122 ceph-mon[83368]: pgmap v219: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:49 smithi103 ceph-mon[115211]: pgmap v219: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:51 smithi122 ceph-mon[83368]: pgmap v220: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:51 smithi103 ceph-mon[115211]: pgmap v220: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:54.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:53 smithi122 ceph-mon[83368]: pgmap v221: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:53 smithi103 ceph-mon[115211]: pgmap v221: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:55 smithi122 ceph-mon[83368]: pgmap v222: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:55 smithi103 ceph-mon[115211]: pgmap v222: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:57 smithi122 ceph-mon[83368]: pgmap v223: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:38:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:57 smithi103 ceph-mon[115211]: pgmap v223: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:38:59 smithi122 ceph-mon[83368]: pgmap v224: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:38:59 smithi103 ceph-mon[115211]: pgmap v224: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 -- 172.21.15.103:0/1574681566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00101d70 msgr2=0x7f0a00102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 --2- 172.21.15.103:0/1574681566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00101d70 0x7f0a00102140 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f09f4004940 tx=0x7f09f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 -- 172.21.15.103:0/1574681566 shutdown_connections 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 --2- 172.21.15.103:0/1574681566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00102710 0x7f0a000fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 --2- 172.21.15.103:0/1574681566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00101d70 0x7f0a00102140 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 -- 172.21.15.103:0/1574681566 >> 172.21.15.103:0/1574681566 conn(0x7f0a00072bc0 msgr2=0x7f0a00072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 -- 172.21.15.103:0/1574681566 shutdown_connections 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.008+0000 7f0a07204700 1 -- 172.21.15.103:0/1574681566 wait complete. 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 Processor -- start 2024-03-22T02:39:02.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 -- start start 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00101d70 0x7f0a00115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00116840 con 0x7f0a00102710 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f0a07204700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0010fcc0 con 0x7f0a00101d70 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f09fffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f09fffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35218/0 (socket says 172.21.15.103:35218) 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.009+0000 7f09fffff700 1 -- 172.21.15.103:0/2644758368 learned_addr learned my addr 172.21.15.103:0/2644758368 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:02.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f0a04fa0700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00101d70 0x7f0a00115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fffff700 1 -- 172.21.15.103:0/2644758368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00101d70 msgr2=0x7f0a00115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fffff700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00101d70 0x7f0a00115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fffff700 1 -- 172.21.15.103:0/2644758368 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4005040 con 0x7f0a00102710 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fffff700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09ec002a50 tx=0x7f09ec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09ec007cc0 con 0x7f0a00102710 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09ec00f040 con 0x7f0a00102710 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0010ffa0 con 0x7f0a00102710 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09ec00d9c0 con 0x7f0a00102710 2024-03-22T02:39:02.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.010+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00110560 con 0x7f0a00102710 2024-03-22T02:39:02.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.011+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a000fd4c0 con 0x7f0a00102710 2024-03-22T02:39:02.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.012+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f09ec009730 con 0x7f0a00102710 2024-03-22T02:39:02.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.012+0000 7f09fd7fa700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 0x7f09f0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.012+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f09ec08da80 con 0x7f0a00102710 2024-03-22T02:39:02.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.012+0000 7f0a04fa0700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 0x7f09f0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.012+0000 7f0a04fa0700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 0x7f09f0070850 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f09f4004f90 tx=0x7f09f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.018+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09ec056d00 con 0x7f0a00102710 2024-03-22T02:39:02.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.233+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a00108e70 con 0x7f09f006e3a0 2024-03-22T02:39:02.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.234+0000 7f09fd7fa700 1 -- 172.21.15.103:0/2644758368 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0a00108e70 con 0x7f09f006e3a0 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 msgr2=0x7f09f0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 0x7f09f0070850 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f09f4004f90 tx=0x7f09f4009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 msgr2=0x7f0a001161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09ec002a50 tx=0x7f09ec004600 comp rx=0 tx=0).stop 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 shutdown_connections 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f09f006e3a0 0x7f09f0070850 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a00101d70 0x7f0a00115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 --2- 172.21.15.103:0/2644758368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a00102710 0x7f0a001161f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 >> 172.21.15.103:0/2644758368 conn(0x7f0a00072bc0 msgr2=0x7f0a00065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 shutdown_connections 2024-03-22T02:39:02.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.236+0000 7f0a07204700 1 -- 172.21.15.103:0/2644758368 wait complete. 2024-03-22T02:39:02.248 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:39:02.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 -- 172.21.15.103:0/120312024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 msgr2=0x7ff4080fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/120312024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff4080fc050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8004940 tx=0x7ff3f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.340 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:02 smithi103 ceph-mon[115211]: pgmap v225: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:02.341 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:02.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 -- 172.21.15.103:0/120312024 shutdown_connections 2024-03-22T02:39:02.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/120312024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 0x7ff4081092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/120312024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff4080fc050 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 -- 172.21.15.103:0/120312024 >> 172.21.15.103:0/120312024 conn(0x7ff408072bc0 msgr2=0x7ff408072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 -- 172.21.15.103:0/120312024 shutdown_connections 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.338+0000 7ff40dd4b700 1 -- 172.21.15.103:0/120312024 wait complete. 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 Processor -- start 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 -- start start 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 0x7ff4081151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4081157a0 con 0x7ff4080fbc80 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff40dd4b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810fd70 con 0x7ff4080fc590 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff4077fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff4077fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35234/0 (socket says 172.21.15.103:35234) 2024-03-22T02:39:02.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.339+0000 7ff4077fe700 1 -- 172.21.15.103:0/2367393178 learned_addr learned my addr 172.21.15.103:0/2367393178 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff3fffff700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 0x7ff4081151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff4077fe700 1 -- 172.21.15.103:0/2367393178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 msgr2=0x7ff4081151a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff4077fe700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 0x7ff4081151a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff4077fe700 1 -- 172.21.15.103:0/2367393178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8005040 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff4077fe700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800b3b0 tx=0x7ff3f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f800e040 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40810fff0 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3f800bbc0 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f801c3f0 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.340+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4081105b0 con 0x7ff4080fbc80 2024-03-22T02:39:02.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.342+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408065e30 con 0x7ff4080fbc80 2024-03-22T02:39:02.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.343+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff3f800bd30 con 0x7ff4080fbc80 2024-03-22T02:39:02.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.343+0000 7ff404ff9700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 0x7ff3f4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.343+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff3f808f060 con 0x7ff4080fbc80 2024-03-22T02:39:02.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.343+0000 7ff3fffff700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 0x7ff3f4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.343+0000 7ff3fffff700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 0x7ff3f4070850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0007fd0 tx=0x7ff3f0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:02 smithi122 ceph-mon[83368]: pgmap v225: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:02.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.347+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3f8058260 con 0x7ff4080fbc80 2024-03-22T02:39:02.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.562+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff40804bed0 con 0x7ff3f406e3a0 2024-03-22T02:39:02.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.564+0000 7ff404ff9700 1 -- 172.21.15.103:0/2367393178 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff40804bed0 con 0x7ff3f406e3a0 2024-03-22T02:39:02.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 msgr2=0x7ff3f4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 0x7ff3f4070850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0007fd0 tx=0x7ff3f0009040 comp rx=0 tx=0).stop 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 msgr2=0x7ff408114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800b3b0 tx=0x7ff3f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 shutdown_connections 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff3f406e3a0 0x7ff3f4070850 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4080fbc80 0x7ff408114c60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 --2- 172.21.15.103:0/2367393178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4080fc590 0x7ff4081151a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 >> 172.21.15.103:0/2367393178 conn(0x7ff408072bc0 msgr2=0x7ff408105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 shutdown_connections 2024-03-22T02:39:02.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.566+0000 7ff40dd4b700 1 -- 172.21.15.103:0/2367393178 wait complete. 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- 172.21.15.103:0/2134205458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 msgr2=0x7f19f0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/2134205458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0103320 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f19e4004940 tx=0x7f19e4009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- 172.21.15.103:0/2134205458 shutdown_connections 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/2134205458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 0x7f19f00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/2134205458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0103320 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- 172.21.15.103:0/2134205458 >> 172.21.15.103:0/2134205458 conn(0x7f19f00714c0 msgr2=0x7f19f00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- 172.21.15.103:0/2134205458 shutdown_connections 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- 172.21.15.103:0/2134205458 wait complete. 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 Processor -- start 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 -- start start 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 0x7f19f0113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.668+0000 7f19f7ec9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f7ec9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f01145b0 con 0x7f19f0102f50 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f7ec9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f010db90 con 0x7f19f00fb4d0 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35252/0 (socket says 172.21.15.103:35252) 2024-03-22T02:39:02.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 -- 172.21.15.103:0/834101171 learned_addr learned my addr 172.21.15.103:0/834101171 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5c65700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 0x7f19f0113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 -- 172.21.15.103:0/834101171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 msgr2=0x7f19f0113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 0x7f19f0113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 -- 172.21.15.103:0/834101171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4005040 con 0x7f19f0102f50 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f5464700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c830 tx=0x7f19ec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19ec0106f0 con 0x7f19f0102f50 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19ec010d20 con 0x7f19f0102f50 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19ec0113a0 con 0x7f19f0102f50 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f010de70 con 0x7f19f0102f50 2024-03-22T02:39:02.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.669+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f010e3b0 con 0x7f19f0102f50 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.670+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0100200 con 0x7f19f0102f50 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.671+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f19ec012400 con 0x7f19f0102f50 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.671+0000 7f19e27fc700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 0x7f19dc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.671+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19ec08dd70 con 0x7f19f0102f50 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.671+0000 7f19f5c65700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 0x7f19dc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.672+0000 7f19f5c65700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 0x7f19dc070850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009fd0 tx=0x7f19e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.676+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19ec05ab20 con 0x7f19f0102f50 2024-03-22T02:39:02.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.880+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f19f010f070 con 0x7f19dc06e3a0 2024-03-22T02:39:02.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.890+0000 7f19e27fc700 1 -- 172.21.15.103:0/834101171 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f19f010f070 con 0x7f19dc06e3a0 2024-03-22T02:39:02.891 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:39:02.891 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (7m) 76s ago 18m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:39:02.891 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (5m) 76s ago 18m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:39:02.891 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (5m) 46s ago 17m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (7m) 76s ago 18m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (99s) 76s ago 13m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (81s) 76s ago 13m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (49s) 46s ago 13m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (65s) 46s ago 13m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (9m) 76s ago 20m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (8m) 46s ago 17m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (6m) 76s ago 20m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (6m) 46s ago 17m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (8m) 76s ago 18m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 46s ago 17m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (4m) 76s ago 16m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (4m) 76s ago 16m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:39:02.892 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (3m) 76s ago 16m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:39:02.893 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (3m) 46s ago 15m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:39:02.893 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (2m) 46s ago 15m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:39:02.893 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (115s) 46s ago 14m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:39:02.893 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (7m) 76s ago 18m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:39:02.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.892+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 msgr2=0x7f19dc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.892+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 0x7f19dc070850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009fd0 tx=0x7f19e4009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.892+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 msgr2=0x7f19f0113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c830 tx=0x7f19ec004470 comp rx=0 tx=0).stop 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 shutdown_connections 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f19dc06e3a0 0x7f19dc070850 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19f00fb4d0 0x7f19f0113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 --2- 172.21.15.103:0/834101171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f0102f50 0x7f19f0113f60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 >> 172.21.15.103:0/834101171 conn(0x7f19f00714c0 msgr2=0x7f19f00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 shutdown_connections 2024-03-22T02:39:02.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.893+0000 7f19f7ec9700 1 -- 172.21.15.103:0/834101171 wait complete. 2024-03-22T02:39:02.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.993+0000 7f0dc08c0700 1 -- 172.21.15.103:0/1650864543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 msgr2=0x7f0dbc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.993+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/1650864543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fe260 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0db0004940 tx=0x7f0db0009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- 172.21.15.103:0/1650864543 shutdown_connections 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/1650864543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 0x7f0dbc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/1650864543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fe260 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- 172.21.15.103:0/1650864543 >> 172.21.15.103:0/1650864543 conn(0x7f0dbc072bc0 msgr2=0x7f0dbc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- 172.21.15.103:0/1650864543 shutdown_connections 2024-03-22T02:39:02.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- 172.21.15.103:0/1650864543 wait complete. 2024-03-22T02:39:02.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 Processor -- start 2024-03-22T02:39:02.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- start start 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 0x7f0dbc0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0fc820 con 0x7f0dbc0fde90 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.994+0000 7f0dc08c0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0fc960 con 0x7f0dbc0fe830 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35272/0 (socket says 172.21.15.103:35272) 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 -- 172.21.15.103:0/3012190799 learned_addr learned my addr 172.21.15.103:0/3012190799 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0db9d9b700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 0x7f0dbc0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 -- 172.21.15.103:0/3012190799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 msgr2=0x7f0dbc0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 0x7f0dbc0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 -- 172.21.15.103:0/3012190799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0005040 con 0x7f0dbc0fde90 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dba59c700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0db0000c00 tx=0x7f0db0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:02.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db0021070 con 0x7f0dbc0fde90 2024-03-22T02:39:02.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc0fcbb0 con 0x7f0dbc0fde90 2024-03-22T02:39:02.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0db0013c00 con 0x7f0dbc0fde90 2024-03-22T02:39:02.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db000e070 con 0x7f0dbc0fde90 2024-03-22T02:39:02.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.995+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc11a1e0 con 0x7f0dbc0fde90 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.996+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc065e30 con 0x7f0dbc0fde90 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.997+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0db001c400 con 0x7f0dbc0fde90 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.997+0000 7f0daaffd700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 0x7f0da4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.997+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0db008f1f0 con 0x7f0dbc0fde90 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.997+0000 7f0db9d9b700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 0x7f0da4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:02.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:02.998+0000 7f0db9d9b700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 0x7f0da4070850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0fd690 tx=0x7f0dac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:03.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.002+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0db0058470 con 0x7f0dbc0fde90 2024-03-22T02:39:03.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.280+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0dbc0fd440 con 0x7f0dbc0fde90 2024-03-22T02:39:03.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.281+0000 7f0daaffd700 1 -- 172.21.15.103:0/3012190799 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0db005bb30 con 0x7f0dbc0fde90 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:03.282 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:39:03.283 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 msgr2=0x7f0da4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 0x7f0da4070850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0fd690 tx=0x7f0dac00b040 comp rx=0 tx=0).stop 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 msgr2=0x7f0dbc0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0db0000c00 tx=0x7f0db0009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 shutdown_connections 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0da406e3a0 0x7f0da4070850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0dbc0fde90 0x7f0dbc0fbd50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 --2- 172.21.15.103:0/3012190799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dbc0fe830 0x7f0dbc0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 >> 172.21.15.103:0/3012190799 conn(0x7f0dbc072bc0 msgr2=0x7f0dbc1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 shutdown_connections 2024-03-22T02:39:03.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.283+0000 7f0dc08c0700 1 -- 172.21.15.103:0/3012190799 wait complete. 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 -- 172.21.15.103:0/2982712020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fe7c0 msgr2=0x7ffacc106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 --2- 172.21.15.103:0/2982712020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc106e00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffac0004940 tx=0x7ffac0009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 -- 172.21.15.103:0/2982712020 shutdown_connections 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 --2- 172.21.15.103:0/2982712020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc106e00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 --2- 172.21.15.103:0/2982712020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 -- 172.21.15.103:0/2982712020 >> 172.21.15.103:0/2982712020 conn(0x7ffacc0f7870 msgr2=0x7ffacc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 -- 172.21.15.103:0/2982712020 shutdown_connections 2024-03-22T02:39:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 -- 172.21.15.103:0/2982712020 wait complete. 2024-03-22T02:39:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.384+0000 7ffad1f15700 1 Processor -- start 2024-03-22T02:39:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffad1f15700 1 -- start start 2024-03-22T02:39:03.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffad1f15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffad1f15700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc0fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffad1f15700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0fdba0 con 0x7ffacc0fde20 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffad1f15700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0fc6d0 con 0x7ffacc0fe7c0 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35298/0 (socket says 172.21.15.103:35298) 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 -- 172.21.15.103:0/662764742 learned_addr learned my addr 172.21.15.103:0/662764742 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacaffd700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc0fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 -- 172.21.15.103:0/662764742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fe7c0 msgr2=0x7ffacc0fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc0fc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 -- 172.21.15.103:0/662764742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0005040 con 0x7ffacc0fde20 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffacb7fe700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00eb40 tx=0x7ffabc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.385+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffabc007cc0 con 0x7ffacc0fde20 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.386+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc0fc950 con 0x7ffacc0fde20 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.386+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffabc00f040 con 0x7ffacc0fde20 2024-03-22T02:39:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.386+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffabc00ca10 con 0x7ffacc0fde20 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.386+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc119fd0 con 0x7ffacc0fde20 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.386+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc066b50 con 0x7ffacc0fde20 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.387+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ffabc00cc80 con 0x7ffacc0fde20 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.387+0000 7ffab3fff700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 0x7ffab4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.387+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffabc08d9a0 con 0x7ffacc0fde20 2024-03-22T02:39:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.387+0000 7ffacaffd700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 0x7ffab4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.388+0000 7ffacaffd700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 0x7ffab4070850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0fd380 tx=0x7ffac0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:03.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.392+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffabc056c50 con 0x7ffacc0fde20 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.634+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffacc04c710 con 0x7ffacc0fde20 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.634+0000 7ffab3fff700 1 -- 172.21.15.103:0/662764742 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7ffabc05a310 con 0x7ffacc0fde20 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:39:03.635 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:39:03.636 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:39:03.637 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:03.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 msgr2=0x7ffab4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 0x7ffab4070850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0fd380 tx=0x7ffac0009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:03.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 msgr2=0x7ffacc0fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00eb40 tx=0x7ffabc00b040 comp rx=0 tx=0).stop 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 shutdown_connections 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffab406e3a0 0x7ffab4070850 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffacc0fde20 0x7ffacc0fbc50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 --2- 172.21.15.103:0/662764742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffacc0fe7c0 0x7ffacc0fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 >> 172.21.15.103:0/662764742 conn(0x7ffacc0f7870 msgr2=0x7ffacc0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 shutdown_connections 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.637+0000 7ffad1f15700 1 -- 172.21.15.103:0/662764742 wait complete. 2024-03-22T02:39:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:39:03.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 -- 172.21.15.103:0/86997252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd550102e00 msgr2=0x7fd5501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 --2- 172.21.15.103:0/86997252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd550102e00 0x7fd5501031d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd544004940 tx=0x7fd544009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 -- 172.21.15.103:0/86997252 shutdown_connections 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 --2- 172.21.15.103:0/86997252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5501037a0 0x7fd5500fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 --2- 172.21.15.103:0/86997252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd550102e00 0x7fd5501031d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 -- 172.21.15.103:0/86997252 >> 172.21.15.103:0/86997252 conn(0x7fd5500f89b0 msgr2=0x7fd5500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 -- 172.21.15.103:0/86997252 shutdown_connections 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.740+0000 7fd5565e1700 1 -- 172.21.15.103:0/86997252 wait complete. 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 Processor -- start 2024-03-22T02:39:03.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 -- start start 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 0x7fd550113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550114520 con 0x7fd5501037a0 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd5565e1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55010db70 con 0x7fd550102e00 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39624/0 (socket says 172.21.15.103:39624) 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 -- 172.21.15.103:0/1776172132 learned_addr learned my addr 172.21.15.103:0/1776172132 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54ffff700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 0x7fd550113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 -- 172.21.15.103:0/1776172132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 msgr2=0x7fd550113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 0x7fd550113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54f7fe700 1 -- 172.21.15.103:0/1776172132 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544005040 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.741+0000 7fd54ffff700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 0x7fd550113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd54f7fe700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c800 tx=0x7fd540004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54000a760 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55010de50 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd54000ad90 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5400113a0 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.742+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55010e410 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.743+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550100110 con 0x7fd5501037a0 2024-03-22T02:39:03.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.743+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd540012400 con 0x7fd5501037a0 2024-03-22T02:39:03.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.744+0000 7fd54cff9700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 0x7fd538070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:03.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.744+0000 7fd54ffff700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 0x7fd538070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:03.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.744+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd54008de20 con 0x7fd5501037a0 2024-03-22T02:39:03.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.744+0000 7fd54ffff700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 0x7fd538070850 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd544004560 tx=0x7fd544009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:03.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.748+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd54005abd0 con 0x7fd5501037a0 2024-03-22T02:39:03.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.963+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd55010f050 con 0x7fd53806e3a0 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.964+0000 7fd54cff9700 1 -- 172.21.15.103:0/1776172132 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd55010f050 con 0x7fd53806e3a0 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:39:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:39:03.966 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:39:03.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 msgr2=0x7fd538070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 0x7fd538070850 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd544004560 tx=0x7fd544009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 msgr2=0x7fd550113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c800 tx=0x7fd540004470 comp rx=0 tx=0).stop 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 shutdown_connections 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd53806e3a0 0x7fd538070850 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd550102e00 0x7fd550113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 --2- 172.21.15.103:0/1776172132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5501037a0 0x7fd550113ed0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 >> 172.21.15.103:0/1776172132 conn(0x7fd5500f89b0 msgr2=0x7fd5501073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 shutdown_connections 2024-03-22T02:39:03.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:03.967+0000 7fd5565e1700 1 -- 172.21.15.103:0/1776172132 wait complete. 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- 172.21.15.103:0/431784398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 msgr2=0x7f38600fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 --2- 172.21.15.103:0/431784398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fb820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3850004940 tx=0x7f3850009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- 172.21.15.103:0/431784398 shutdown_connections 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 --2- 172.21.15.103:0/431784398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fb820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 --2- 172.21.15.103:0/431784398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 0x7f3860103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- 172.21.15.103:0/431784398 >> 172.21.15.103:0/431784398 conn(0x7f3860072310 msgr2=0x7f3860072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- 172.21.15.103:0/431784398 shutdown_connections 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- 172.21.15.103:0/431784398 wait complete. 2024-03-22T02:39:04.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 Processor -- start 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- start start 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 0x7f38600fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600fe4e0 con 0x7f38600fb3b0 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.070+0000 7f386844d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600fd010 con 0x7f3860102f70 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39648/0 (socket says 172.21.15.103:39648) 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 -- 172.21.15.103:0/744552505 learned_addr learned my addr 172.21.15.103:0/744552505 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38659e8700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 0x7f38600fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 -- 172.21.15.103:0/744552505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 msgr2=0x7f38600fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 0x7f38600fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 -- 172.21.15.103:0/744552505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850005040 con 0x7f38600fb3b0 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f38661e9700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f385c002a50 tx=0x7f385c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:04.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c007cc0 con 0x7f38600fb3b0 2024-03-22T02:39:04.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f385c00f040 con 0x7f38600fb3b0 2024-03-22T02:39:04.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38600fd2f0 con 0x7f38600fb3b0 2024-03-22T02:39:04.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385c00da30 con 0x7f38600fb3b0 2024-03-22T02:39:04.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.071+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386011a110 con 0x7f38600fb3b0 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.072+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860105350 con 0x7f38600fb3b0 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.073+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f385c00db90 con 0x7f38600fb3b0 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.073+0000 7f3856ffd700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 0x7f384c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.073+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f385c08db40 con 0x7f38600fb3b0 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.073+0000 7f38659e8700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 0x7f384c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:04.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.074+0000 7f38659e8700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 0x7f384c070850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3850004f90 tx=0x7f3850009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:04.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.078+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f385c056d40 con 0x7f38600fb3b0 2024-03-22T02:39:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: pgmap v226: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:04.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3012190799' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:39:04.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:04 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/662764742' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:39:04.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.357+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f38600fda70 con 0x7f38600fb3b0 2024-03-22T02:39:04.358 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.359 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.359 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:04.359 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: pgmap v226: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:04.359 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3012190799' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:39:04.359 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:04 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/662764742' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:39:04.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.358+0000 7f3856ffd700 1 -- 172.21.15.103:0/744552505 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f385c05a400 con 0x7f38600fb3b0 2024-03-22T02:39:04.359 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:39:04.359 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:39:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:39:04.360 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:39:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:39:04.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 msgr2=0x7f384c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:04.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 0x7f384c070850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3850004f90 tx=0x7f3850009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:04.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 msgr2=0x7f38600fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:04.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f385c002a50 tx=0x7f385c0046b0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 shutdown_connections 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f384c06e3a0 0x7f384c070850 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38600fb3b0 0x7f38600fc590 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 --2- 172.21.15.103:0/744552505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3860102f70 0x7f38600fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 >> 172.21.15.103:0/744552505 conn(0x7f3860072310 msgr2=0x7f38600f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 shutdown_connections 2024-03-22T02:39:04.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:04.361+0000 7f386844d700 1 -- 172.21.15.103:0/744552505 wait complete. 2024-03-22T02:39:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:05 smithi122 ceph-mon[83368]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:05.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:05 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/744552505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:39:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:05 smithi103 ceph-mon[115211]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:05 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/744552505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:39:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:06 smithi122 ceph-mon[83368]: pgmap v227: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:06 smithi103 ceph-mon[115211]: pgmap v227: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:07 smithi122 ceph-mon[83368]: pgmap v228: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:07 smithi103 ceph-mon[115211]: pgmap v228: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:09 smithi122 ceph-mon[83368]: pgmap v229: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:09 smithi103 ceph-mon[115211]: pgmap v229: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:11 smithi122 ceph-mon[83368]: pgmap v230: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:11 smithi103 ceph-mon[115211]: pgmap v230: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:13 smithi122 ceph-mon[83368]: pgmap v231: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:13 smithi103 ceph-mon[115211]: pgmap v231: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:15 smithi122 ceph-mon[83368]: pgmap v232: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:15 smithi103 ceph-mon[115211]: pgmap v232: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:17 smithi122 ceph-mon[83368]: pgmap v233: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:17 smithi103 ceph-mon[115211]: pgmap v233: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:19 smithi122 ceph-mon[83368]: pgmap v234: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:19 smithi103 ceph-mon[115211]: pgmap v234: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:21 smithi122 ceph-mon[83368]: pgmap v235: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:21 smithi103 ceph-mon[115211]: pgmap v235: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:23 smithi122 ceph-mon[83368]: pgmap v236: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:23 smithi103 ceph-mon[115211]: pgmap v236: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:25 smithi122 ceph-mon[83368]: pgmap v237: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:25 smithi103 ceph-mon[115211]: pgmap v237: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:27 smithi122 ceph-mon[83368]: pgmap v238: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:27 smithi103 ceph-mon[115211]: pgmap v238: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:29 smithi122 ceph-mon[83368]: pgmap v239: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:29 smithi103 ceph-mon[115211]: pgmap v239: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:31 smithi122 ceph-mon[83368]: pgmap v240: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:31 smithi103 ceph-mon[115211]: pgmap v240: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:33 smithi122 ceph-mon[83368]: pgmap v241: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:33 smithi103 ceph-mon[115211]: pgmap v241: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:34.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- 172.21.15.103:0/3937761289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 msgr2=0x7f27ac1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 --2- 172.21.15.103:0/3937761289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac1020b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f279c004940 tx=0x7f279c009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:34.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- 172.21.15.103:0/3937761289 shutdown_connections 2024-03-22T02:39:34.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 --2- 172.21.15.103:0/3937761289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 0x7f27ac0fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 --2- 172.21.15.103:0/3937761289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac1020b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- 172.21.15.103:0/3937761289 >> 172.21.15.103:0/3937761289 conn(0x7f27ac0f7890 msgr2=0x7f27ac0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- 172.21.15.103:0/3937761289 shutdown_connections 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- 172.21.15.103:0/3937761289 wait complete. 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 Processor -- start 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.476+0000 7f27b0d30700 1 -- start start 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 0x7f27ac10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac10f5c0 con 0x7f27ac101ce0 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac10f700 con 0x7f27ac102680 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48726/0 (socket says 172.21.15.103:48726) 2024-03-22T02:39:34.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 -- 172.21.15.103:0/504998793 learned_addr learned my addr 172.21.15.103:0/504998793 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27a9d9b700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 0x7f27ac10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 -- 172.21.15.103:0/504998793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 msgr2=0x7f27ac10f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 0x7f27ac10f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 -- 172.21.15.103:0/504998793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c005040 con 0x7f27ac101ce0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27aa59c700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f279c00b3b0 tx=0x7f279c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279c00e040 con 0x7f27ac101ce0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac10f980 con 0x7f27ac101ce0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f279c013420 con 0x7f27ac101ce0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f279c01c630 con 0x7f27ac101ce0 2024-03-22T02:39:34.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.477+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac117eb0 con 0x7f27ac101ce0 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.478+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac0fd430 con 0x7f27ac101ce0 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.479+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f279c013590 con 0x7f27ac101ce0 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.479+0000 7f279affd700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 0x7f2794070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.479+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f279c08efb0 con 0x7f27ac101ce0 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.479+0000 7f27a9d9b700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 0x7f2794070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.480+0000 7f27a9d9b700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 0x7f2794070850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f27ac066ae0 tx=0x7f27a000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:34.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.484+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f279c0c3740 con 0x7f27ac101ce0 2024-03-22T02:39:34.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.698+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27ac066950 con 0x7f279406e3a0 2024-03-22T02:39:34.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.700+0000 7f279affd700 1 -- 172.21.15.103:0/504998793 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f27ac066950 con 0x7f279406e3a0 2024-03-22T02:39:34.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.701+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 msgr2=0x7f2794070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.701+0000 7f27b0d30700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 0x7f2794070850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f27ac066ae0 tx=0x7f27a000b040 comp rx=0 tx=0).stop 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 msgr2=0x7f27ac10eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f279c00b3b0 tx=0x7f279c009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 shutdown_connections 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f279406e3a0 0x7f2794070850 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27ac101ce0 0x7f27ac10eb40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 --2- 172.21.15.103:0/504998793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27ac102680 0x7f27ac10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 >> 172.21.15.103:0/504998793 conn(0x7f27ac0f7890 msgr2=0x7f27ac0f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 shutdown_connections 2024-03-22T02:39:34.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.702+0000 7f27b0d30700 1 -- 172.21.15.103:0/504998793 wait complete. 2024-03-22T02:39:34.714 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:39:34.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/3292095220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 msgr2=0x7fdcec0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/3292095220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec0fce10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc004940 tx=0x7fdcdc009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/3292095220 shutdown_connections 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/3292095220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec0fce10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/3292095220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/3292095220 >> 172.21.15.103:0/3292095220 conn(0x7fdcec0f89b0 msgr2=0x7fdcec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/3292095220 shutdown_connections 2024-03-22T02:39:34.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.804+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/3292095220 wait complete. 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 Processor -- start 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 -- start start 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec114510 con 0x7fdcec1037a0 2024-03-22T02:39:34.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcf2e7d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec10db10 con 0x7fdcec102e00 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48744/0 (socket says 172.21.15.103:48744) 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.805+0000 7fdcebfff700 1 -- 172.21.15.103:0/2968697523 learned_addr learned my addr 172.21.15.103:0/2968697523 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcf0c19700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcebfff700 1 -- 172.21.15.103:0/2968697523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 msgr2=0x7fdcec113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcebfff700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcebfff700 1 -- 172.21.15.103:0/2968697523 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc005040 con 0x7fdcec1037a0 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcf0c19700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcebfff700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009100 tx=0x7fdcdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:34.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcdc00e040 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcdc013420 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec10dd90 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcdc01c610 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.806+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec10e350 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.807+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec1052d0 con 0x7fdcec1037a0 2024-03-22T02:39:34.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.808+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fdcdc013590 con 0x7fdcec1037a0 2024-03-22T02:39:34.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.808+0000 7fdce97fa700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 0x7fdcd8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:34.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.809+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdcdc08f090 con 0x7fdcec1037a0 2024-03-22T02:39:34.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.809+0000 7fdcf0c19700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 0x7fdcd8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:34.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.809+0000 7fdcf0c19700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 0x7fdcd8070850 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdce4004f60 tx=0x7fdce400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:34.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:34.813+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdcdc05be40 con 0x7fdcec1037a0 2024-03-22T02:39:35.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.031+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcec10efd0 con 0x7fdcd806e3a0 2024-03-22T02:39:35.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.032+0000 7fdce97fa700 1 -- 172.21.15.103:0/2968697523 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdcec10efd0 con 0x7fdcd806e3a0 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 msgr2=0x7fdcd8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 0x7fdcd8070850 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdce4004f60 tx=0x7fdce400b040 comp rx=0 tx=0).stop 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 msgr2=0x7fdcec113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009100 tx=0x7fdcdc009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 shutdown_connections 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdcd806e3a0 0x7fdcd8070850 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdcec102e00 0x7fdcec113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 --2- 172.21.15.103:0/2968697523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdcec1037a0 0x7fdcec113ec0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 >> 172.21.15.103:0/2968697523 conn(0x7fdcec0f89b0 msgr2=0x7fdcec1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 shutdown_connections 2024-03-22T02:39:35.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.034+0000 7fdcf2e7d700 1 -- 172.21.15.103:0/2968697523 wait complete. 2024-03-22T02:39:35.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 -- 172.21.15.103:0/664147698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 msgr2=0x7f08fc0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 --2- 172.21.15.103:0/664147698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc0fcdf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08ec004940 tx=0x7f08ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 -- 172.21.15.103:0/664147698 shutdown_connections 2024-03-22T02:39:35.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 --2- 172.21.15.103:0/664147698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc0fcdf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 --2- 172.21.15.103:0/664147698 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 0x7f08fc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 -- 172.21.15.103:0/664147698 >> 172.21.15.103:0/664147698 conn(0x7f08fc0f8950 msgr2=0x7f08fc0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 -- 172.21.15.103:0/664147698 shutdown_connections 2024-03-22T02:39:35.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 -- 172.21.15.103:0/664147698 wait complete. 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.137+0000 7f09010f0700 1 Processor -- start 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 -- start start 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 0x7f08fc115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1166c0 con 0x7f08fc103780 2024-03-22T02:39:35.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc10fca0 con 0x7f08fc102de0 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48762/0 (socket says 172.21.15.103:48762) 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 -- 172.21.15.103:0/3241648981 learned_addr learned my addr 172.21.15.103:0/3241648981 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fad9d700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 0x7f08fc115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 -- 172.21.15.103:0/3241648981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 msgr2=0x7f08fc115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 0x7f08fc115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 -- 172.21.15.103:0/3241648981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec005040 con 0x7f08fc103780 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08fa59c700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009180 tx=0x7f08ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08ec00e040 con 0x7f08fc103780 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc10ff20 con 0x7f08fc103780 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08ec00bbc0 con 0x7f08fc103780 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.138+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08ec01c3f0 con 0x7f08fc103780 2024-03-22T02:39:35.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.139+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc1104e0 con 0x7f08fc103780 2024-03-22T02:39:35.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.139+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc0fe530 con 0x7f08fc103780 2024-03-22T02:39:35.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.140+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f08ec00bd30 con 0x7f08fc103780 2024-03-22T02:39:35.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.140+0000 7f08e37fe700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 0x7f08e4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.140+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f08ec08f110 con 0x7f08fc103780 2024-03-22T02:39:35.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.140+0000 7f08fad9d700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 0x7f08e4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.141+0000 7f08fad9d700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 0x7f08e4070850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009fd0 tx=0x7f08f0009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.145+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08ec05bec0 con 0x7f08fc103780 2024-03-22T02:39:35.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.351+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f08fc109f30 con 0x7f08e406e3a0 2024-03-22T02:39:35.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.361+0000 7f08e37fe700 1 -- 172.21.15.103:0/3241648981 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f08fc109f30 con 0x7f08e406e3a0 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (8m) 109s ago 19m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (5m) 109s ago 19m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (5m) 79s ago 18m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (7m) 109s ago 19m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (2m) 109s ago 13m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (114s) 109s ago 13m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (81s) 79s ago 13m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (98s) 79s ago 13m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (9m) 109s ago 20m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (9m) 79s ago 18m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (6m) 109s ago 20m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (6m) 79s ago 18m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:39:35.363 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (8m) 109s ago 19m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 79s ago 18m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (5m) 109s ago 17m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (4m) 109s ago 17m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (4m) 109s ago 16m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (3m) 79s ago 16m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (3m) 79s ago 15m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (2m) 79s ago 15m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:39:35.364 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (8m) 109s ago 18m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:39:35.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 msgr2=0x7f08e4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 0x7f08e4070850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009fd0 tx=0x7f08f0009f60 comp rx=0 tx=0).stop 2024-03-22T02:39:35.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 msgr2=0x7f08fc116070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009180 tx=0x7f08ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 shutdown_connections 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f08e406e3a0 0x7f08e4070850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08fc102de0 0x7f08fc115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 --2- 172.21.15.103:0/3241648981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08fc103780 0x7f08fc116070 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 >> 172.21.15.103:0/3241648981 conn(0x7f08fc0f8950 msgr2=0x7f08fc071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 shutdown_connections 2024-03-22T02:39:35.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.363+0000 7f09010f0700 1 -- 172.21.15.103:0/3241648981 wait complete. 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.463+0000 7f7b80c3d700 1 -- 172.21.15.103:0/2873474468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0fb540 msgr2=0x7f7b7c0fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.463+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/2873474468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c0fb990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b70004940 tx=0x7f7b70009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- 172.21.15.103:0/2873474468 shutdown_connections 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/2873474468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c0fb990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/2873474468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c1033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- 172.21.15.103:0/2873474468 >> 172.21.15.103:0/2873474468 conn(0x7f7b7c071530 msgr2=0x7f7b7c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- 172.21.15.103:0/2873474468 shutdown_connections 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- 172.21.15.103:0/2873474468 wait complete. 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 Processor -- start 2024-03-22T02:39:35.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- start start 2024-03-22T02:39:35.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c1145b0 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b80c3d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c10db40 con 0x7f7b7c0fb540 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.464+0000 7f7b79d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48776/0 (socket says 172.21.15.103:48776) 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 -- 172.21.15.103:0/1361219418 learned_addr learned my addr 172.21.15.103:0/1361219418 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b7a59c700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 -- 172.21.15.103:0/1361219418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c0fb540 msgr2=0x7f7b7c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 -- 172.21.15.103:0/1361219418 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70005040 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b79d9b700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b3b0 tx=0x7f7b70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7000e040 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b70013420 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c10ddc0 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7001c630 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.465+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c10e300 con 0x7f7b7c102fd0 2024-03-22T02:39:35.469 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.467+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c100270 con 0x7f7b7c102fd0 2024-03-22T02:39:35.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.467+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7b70013590 con 0x7f7b7c102fd0 2024-03-22T02:39:35.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.467+0000 7f7b6affd700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 0x7f7b64070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.467+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b7008ef00 con 0x7f7b7c102fd0 2024-03-22T02:39:35.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.467+0000 7f7b7a59c700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 0x7f7b64070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.468+0000 7f7b7a59c700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 0x7f7b64070850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009bf0 tx=0x7f7b6c0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.473+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b70058240 con 0x7f7b7c102fd0 2024-03-22T02:39:35.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.753+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b7c10efe0 con 0x7f7b7c102fd0 2024-03-22T02:39:35.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.753+0000 7f7b6affd700 1 -- 172.21.15.103:0/1361219418 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7b7005b900 con 0x7f7b7c102fd0 2024-03-22T02:39:35.755 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:39:35.756 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:39:35.757 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:39:35.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 msgr2=0x7f7b64070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 0x7f7b64070850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009bf0 tx=0x7f7b6c0091a0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 msgr2=0x7f7b7c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b3b0 tx=0x7f7b70009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 shutdown_connections 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b6406e3a0 0x7f7b64070850 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b7c0fb540 0x7f7b7c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 --2- 172.21.15.103:0/1361219418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b7c102fd0 0x7f7b7c113ed0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 >> 172.21.15.103:0/1361219418 conn(0x7f7b7c071530 msgr2=0x7f7b7c0fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 shutdown_connections 2024-03-22T02:39:35.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.755+0000 7f7b80c3d700 1 -- 172.21.15.103:0/1361219418 wait complete. 2024-03-22T02:39:35.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 -- 172.21.15.103:0/4184210523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fe810 msgr2=0x7f3458106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 --2- 172.21.15.103:0/4184210523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fe810 0x7f3458106e50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3440004940 tx=0x7f3440009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 -- 172.21.15.103:0/4184210523 shutdown_connections 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 --2- 172.21.15.103:0/4184210523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fe810 0x7f3458106e50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 --2- 172.21.15.103:0/4184210523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fde70 0x7f34580fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 -- 172.21.15.103:0/4184210523 >> 172.21.15.103:0/4184210523 conn(0x7f34580f7880 msgr2=0x7f34580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 -- 172.21.15.103:0/4184210523 shutdown_connections 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 -- 172.21.15.103:0/4184210523 wait complete. 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.857+0000 7f345d2d7700 1 Processor -- start 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 -- start start 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fe810 0x7f3458114370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34581149c0 con 0x7f34580fde70 2024-03-22T02:39:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345810dd90 con 0x7f34580fe810 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48804/0 (socket says 172.21.15.103:48804) 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 -- 172.21.15.103:0/731953058 learned_addr learned my addr 172.21.15.103:0/731953058 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f344e7fc700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fe810 0x7f3458114370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 -- 172.21.15.103:0/731953058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fe810 msgr2=0x7f3458114370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fe810 0x7f3458114370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 -- 172.21.15.103:0/731953058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440005040 con 0x7f34580fde70 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f3456ffd700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f344800c830 tx=0x7f3448004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f344800a760 con 0x7f34580fde70 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f344800ad90 con 0x7f34580fde70 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.858+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345810e070 con 0x7f34580fde70 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.859+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34480113a0 con 0x7f34580fde70 2024-03-22T02:39:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.859+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345810e5b0 con 0x7f34580fde70 2024-03-22T02:39:35.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.859+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3458066710 con 0x7f34580fde70 2024-03-22T02:39:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.860+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3448012400 con 0x7f34580fde70 2024-03-22T02:39:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.860+0000 7f344ffff700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 0x7f34440708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.860+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f344808dea0 con 0x7f34580fde70 2024-03-22T02:39:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.861+0000 7f344e7fc700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 0x7f34440708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:35.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.861+0000 7f344e7fc700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 0x7f34440708b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f345810f6d0 tx=0x7f3440009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:35.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:35.865+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f344805abd0 con 0x7f34580fde70 2024-03-22T02:39:36.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.109+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f345805f5f0 con 0x7f34580fde70 2024-03-22T02:39:36.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:35 smithi103 ceph-mon[115211]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:36.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:35 smithi103 ceph-mon[115211]: pgmap v242: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:36.111 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:35 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1361219418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:39:36.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.109+0000 7f344ffff700 1 -- 172.21.15.103:0/731953058 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f344805a760 con 0x7f34580fde70 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:39:36.112 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:39:36.113 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:39:36.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 msgr2=0x7f34440708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 0x7f34440708b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f345810f6d0 tx=0x7f3440009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:36.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 msgr2=0x7f3458113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f344800c830 tx=0x7f3448004470 comp rx=0 tx=0).stop 2024-03-22T02:39:36.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 shutdown_connections 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f344406e400 0x7f34440708b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34580fde70 0x7f3458113e30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 --2- 172.21.15.103:0/731953058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f34580fe810 0x7f3458114370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 >> 172.21.15.103:0/731953058 conn(0x7f34580f7880 msgr2=0x7f34580f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 shutdown_connections 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.112+0000 7f345d2d7700 1 -- 172.21.15.103:0/731953058 wait complete. 2024-03-22T02:39:36.116 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 -- 172.21.15.103:0/3638718752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924102710 msgr2=0x7fe9240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/3638718752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924102710 0x7fe9240fbd80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe920004940 tx=0x7fe920009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 -- 172.21.15.103:0/3638718752 shutdown_connections 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/3638718752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924102710 0x7fe9240fbd80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/3638718752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924101d70 0x7fe924102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 -- 172.21.15.103:0/3638718752 >> 172.21.15.103:0/3638718752 conn(0x7fe924072bc0 msgr2=0x7fe924072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 -- 172.21.15.103:0/3638718752 shutdown_connections 2024-03-22T02:39:36.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.215+0000 7fe92b6bd700 1 -- 172.21.15.103:0/3638718752 wait complete. 2024-03-22T02:39:36.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 Processor -- start 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 -- start start 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924102710 0x7fe9241162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe924116910 con 0x7fe924101d70 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92410fce0 con 0x7fe924102710 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48826/0 (socket says 172.21.15.103:48826) 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 -- 172.21.15.103:0/1439695937 learned_addr learned my addr 172.21.15.103:0/1439695937 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe928c58700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924102710 0x7fe9241162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 -- 172.21.15.103:0/1439695937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924102710 msgr2=0x7fe9241162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924102710 0x7fe9241162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 -- 172.21.15.103:0/1439695937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920005040 con 0x7fe924101d70 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe929459700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c8c0 tx=0x7fe9140044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91400a980 con 0x7fe924101d70 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe91400ce60 con 0x7fe924101d70 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92410ffc0 con 0x7fe924101d70 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.216+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9140115e0 con 0x7fe924101d70 2024-03-22T02:39:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.217+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe924110580 con 0x7fe924101d70 2024-03-22T02:39:36.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.218+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240fd4c0 con 0x7fe924101d70 2024-03-22T02:39:36.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.218+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe914012400 con 0x7fe924101d70 2024-03-22T02:39:36.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.218+0000 7fe919ffb700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 0x7fe910070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.219+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe91408ded0 con 0x7fe924101d70 2024-03-22T02:39:36.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.219+0000 7fe928c58700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 0x7fe910070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.219+0000 7fe928c58700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 0x7fe910070850 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b3b0 tx=0x7fe920009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:36.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.223+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe91405ac80 con 0x7fe924101d70 2024-03-22T02:39:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:35 smithi122 ceph-mon[83368]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:35 smithi122 ceph-mon[83368]: pgmap v242: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:36.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:35 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1361219418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:39:36.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.443+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe92404bed0 con 0x7fe91006e3a0 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.444+0000 7fe919ffb700 1 -- 172.21.15.103:0/1439695937 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe92404bed0 con 0x7fe91006e3a0 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:39:36.446 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:39:36.447 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:39:36.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 msgr2=0x7fe910070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 0x7fe910070850 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe92000b3b0 tx=0x7fe920009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:36.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 msgr2=0x7fe924115d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c8c0 tx=0x7fe9140044c0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 shutdown_connections 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe91006e3a0 0x7fe910070850 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe924101d70 0x7fe924115d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 --2- 172.21.15.103:0/1439695937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe924102710 0x7fe9241162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 >> 172.21.15.103:0/1439695937 conn(0x7fe924072bc0 msgr2=0x7fe9240659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 shutdown_connections 2024-03-22T02:39:36.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.446+0000 7fe92b6bd700 1 -- 172.21.15.103:0/1439695937 wait complete. 2024-03-22T02:39:36.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- 172.21.15.103:0/4024998974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd41037a0 msgr2=0x7f6dd40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/4024998974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd41037a0 0x7f6dd40fce10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc004940 tx=0x7f6dbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:36.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- 172.21.15.103:0/4024998974 shutdown_connections 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/4024998974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd41037a0 0x7f6dd40fce10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/4024998974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd4102e00 0x7f6dd41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- 172.21.15.103:0/4024998974 >> 172.21.15.103:0/4024998974 conn(0x7f6dd40f89b0 msgr2=0x7f6dd40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- 172.21.15.103:0/4024998974 shutdown_connections 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- 172.21.15.103:0/4024998974 wait complete. 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 Processor -- start 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 -- start start 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.549+0000 7f6dd9f77700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd9f77700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 0x7f6dd4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd9f77700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4072210 con 0x7f6dd4102e00 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd9f77700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4070d40 con 0x7f6dd41037a0 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48848/0 (socket says 172.21.15.103:48848) 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 -- 172.21.15.103:0/59903940 learned_addr learned my addr 172.21.15.103:0/59903940 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd2ffd700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 0x7f6dd4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 -- 172.21.15.103:0/59903940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 msgr2=0x7f6dd4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 0x7f6dd4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 -- 172.21.15.103:0/59903940 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc005040 con 0x7f6dd4102e00 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd2ffd700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 0x7f6dd4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd37fe700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4002a50 tx=0x7f6dc40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc4007cc0 con 0x7f6dd4102e00 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd4071020 con 0x7f6dd4102e00 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc400f040 con 0x7f6dd4102e00 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc4009640 con 0x7f6dd4102e00 2024-03-22T02:39:36.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.550+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd411a130 con 0x7f6dd4102e00 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.551+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd41052d0 con 0x7f6dd4102e00 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.552+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6dc40097a0 con 0x7f6dd4102e00 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.552+0000 7f6dcbfff700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 0x7f6dc0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.552+0000 7f6dd2ffd700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 0x7f6dc0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.552+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6dc408dae0 con 0x7f6dd4102e00 2024-03-22T02:39:36.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.553+0000 7f6dd2ffd700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 0x7f6dc0070850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00b3b0 tx=0x7f6dbc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:36.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.557+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6dc4056d60 con 0x7f6dd4102e00 2024-03-22T02:39:36.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.836+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6dd40717a0 con 0x7f6dd4102e00 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.837+0000 7f6dcbfff700 1 -- 172.21.15.103:0/59903940 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6dc405a420 con 0x7f6dd4102e00 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:39:36.839 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:39:36.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 msgr2=0x7f6dc0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 0x7f6dc0070850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00b3b0 tx=0x7f6dbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 msgr2=0x7f6dd40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4002a50 tx=0x7f6dc40046b0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 shutdown_connections 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6dc006e3a0 0x7f6dc0070850 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6dd4102e00 0x7f6dd40702c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 --2- 172.21.15.103:0/59903940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6dd41037a0 0x7f6dd4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 >> 172.21.15.103:0/59903940 conn(0x7f6dd40f89b0 msgr2=0x7f6dd41021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 shutdown_connections 2024-03-22T02:39:36.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:39:36.839+0000 7f6dd9f77700 1 -- 172.21.15.103:0/59903940 wait complete. 2024-03-22T02:39:36.989 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:36 smithi103 ceph-mon[115211]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:36.990 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:36 smithi103 ceph-mon[115211]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:36.990 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:36 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/731953058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:39:36.990 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:36 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/59903940' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:39:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:36 smithi122 ceph-mon[83368]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:36 smithi122 ceph-mon[83368]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:36 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/731953058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:39:37.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:36 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/59903940' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:39:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:37 smithi122 ceph-mon[83368]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:38.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:37 smithi122 ceph-mon[83368]: pgmap v243: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:37 smithi103 ceph-mon[115211]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:39:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:37 smithi103 ceph-mon[115211]: pgmap v243: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:39 smithi122 ceph-mon[83368]: pgmap v244: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:39 smithi103 ceph-mon[115211]: pgmap v244: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:42.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:41 smithi122 ceph-mon[83368]: pgmap v245: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:41 smithi103 ceph-mon[115211]: pgmap v245: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:43 smithi122 ceph-mon[83368]: pgmap v246: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:43 smithi103 ceph-mon[115211]: pgmap v246: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:46 smithi122 ceph-mon[83368]: pgmap v247: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:46 smithi103 ceph-mon[115211]: pgmap v247: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:39:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:48 smithi122 ceph-mon[83368]: pgmap v248: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:48 smithi103 ceph-mon[115211]: pgmap v248: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:50 smithi122 ceph-mon[83368]: pgmap v249: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:50 smithi103 ceph-mon[115211]: pgmap v249: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:52 smithi122 ceph-mon[83368]: pgmap v250: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:52 smithi103 ceph-mon[115211]: pgmap v250: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:53.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:53 smithi122 ceph-mon[83368]: pgmap v251: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:53.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:53 smithi103 ceph-mon[115211]: pgmap v251: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:55 smithi122 ceph-mon[83368]: pgmap v252: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:55 smithi103 ceph-mon[115211]: pgmap v252: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:57 smithi122 ceph-mon[83368]: pgmap v253: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:39:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:57 smithi103 ceph-mon[115211]: pgmap v253: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:39:59 smithi122 ceph-mon[83368]: pgmap v254: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:39:59 smithi103 ceph-mon[115211]: pgmap v254: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:39:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:40:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:39:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:40:00.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:39:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-22T02:40:00.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:39:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:40:00.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:39:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-22T02:40:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:00 smithi122 ceph-mon[83368]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:40:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:00 smithi122 ceph-mon[83368]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:40:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:00 smithi122 ceph-mon[83368]: fs cephfs is degraded 2024-03-22T02:40:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:00 smithi122 ceph-mon[83368]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:40:01.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:00 smithi122 ceph-mon[83368]: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:40:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-mon[115211]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:40:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-mon[115211]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:40:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-mon[115211]: fs cephfs is degraded 2024-03-22T02:40:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-mon[115211]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:40:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:00 smithi103 ceph-mon[115211]: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:40:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:01 smithi122 ceph-mon[83368]: pgmap v255: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:01 smithi103 ceph-mon[115211]: pgmap v255: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:03 smithi122 ceph-mon[83368]: pgmap v256: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:03 smithi103 ceph-mon[115211]: pgmap v256: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:05 smithi122 ceph-mon[83368]: pgmap v257: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:05 smithi103 ceph-mon[115211]: pgmap v257: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 -- 172.21.15.103:0/1175322886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c101d20 msgr2=0x7f794c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 --2- 172.21.15.103:0/1175322886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c101d20 0x7f794c1020f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7948004940 tx=0x7f7948009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 -- 172.21.15.103:0/1175322886 shutdown_connections 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 --2- 172.21.15.103:0/1175322886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c1026c0 0x7f794c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 --2- 172.21.15.103:0/1175322886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c101d20 0x7f794c1020f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 -- 172.21.15.103:0/1175322886 >> 172.21.15.103:0/1175322886 conn(0x7f794c0f7870 msgr2=0x7f794c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 -- 172.21.15.103:0/1175322886 shutdown_connections 2024-03-22T02:40:06.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 -- 172.21.15.103:0/1175322886 wait complete. 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.956+0000 7f79545e2700 1 Processor -- start 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f79545e2700 1 -- start start 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f79545e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c101d20 0x7f794c1128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f79545e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f79545e2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c1134d0 con 0x7f794c1026c0 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f79545e2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c10ca40 con 0x7f794c101d20 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:06.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35102/0 (socket says 172.21.15.103:35102) 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 -- 172.21.15.103:0/913956965 learned_addr learned my addr 172.21.15.103:0/913956965 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f795237e700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c101d20 0x7f794c1128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 -- 172.21.15.103:0/913956965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c101d20 msgr2=0x7f794c1128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c101d20 0x7f794c1128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 -- 172.21.15.103:0/913956965 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948005040 con 0x7f794c1026c0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7951b7d700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f793c00b790 tx=0x7f793c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.957+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f793c00f6f0 con 0x7f794c1026c0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.958+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c10cd20 con 0x7f794c1026c0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.958+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f793c00fd20 con 0x7f794c1026c0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.958+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f793c0103a0 con 0x7f794c1026c0 2024-03-22T02:40:06.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.958+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c10d2e0 con 0x7f794c1026c0 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.959+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c0fd470 con 0x7f794c1026c0 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.959+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f793c011400 con 0x7f794c1026c0 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.960+0000 7f7942ffd700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 0x7f7938070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.960+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f793c08dbb0 con 0x7f794c1026c0 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.960+0000 7f795237e700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 0x7f7938070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:06.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.960+0000 7f795237e700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 0x7f7938070850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7948004f90 tx=0x7f7948009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:06.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:06.965+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f793c05a960 con 0x7f794c1026c0 2024-03-22T02:40:07.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.182+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f794c112580 con 0x7f793806e3a0 2024-03-22T02:40:07.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.184+0000 7f7942ffd700 1 -- 172.21.15.103:0/913956965 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f794c112580 con 0x7f793806e3a0 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 msgr2=0x7f7938070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 0x7f7938070850 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7948004f90 tx=0x7f7948009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 msgr2=0x7f794c112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f793c00b790 tx=0x7f793c009220 comp rx=0 tx=0).stop 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 shutdown_connections 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f793806e3a0 0x7f7938070850 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f794c101d20 0x7f794c1128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 --2- 172.21.15.103:0/913956965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794c1026c0 0x7f794c112df0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 >> 172.21.15.103:0/913956965 conn(0x7f794c0f7870 msgr2=0x7f794c0f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 shutdown_connections 2024-03-22T02:40:07.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.186+0000 7f79545e2700 1 -- 172.21.15.103:0/913956965 wait complete. 2024-03-22T02:40:07.197 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:40:07.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- 172.21.15.103:0/4184402726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 msgr2=0x7f56881031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 --2- 172.21.15.103:0/4184402726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56881031b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5684004940 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- 172.21.15.103:0/4184402726 shutdown_connections 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 --2- 172.21.15.103:0/4184402726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 0x7f56880fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 --2- 172.21.15.103:0/4184402726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56881031b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- 172.21.15.103:0/4184402726 >> 172.21.15.103:0/4184402726 conn(0x7f56880f8950 msgr2=0x7f56880fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- 172.21.15.103:0/4184402726 shutdown_connections 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- 172.21.15.103:0/4184402726 wait complete. 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 Processor -- start 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.288+0000 7f568fba1700 1 -- start start 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 0x7f5688070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688072210 con 0x7f5688102de0 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688070d40 con 0x7f5688103780 2024-03-22T02:40:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35130/0 (socket says 172.21.15.103:35130) 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 -- 172.21.15.103:0/3390459405 learned_addr learned my addr 172.21.15.103:0/3390459405 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d13c700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 0x7f5688070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 -- 172.21.15.103:0/3390459405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 msgr2=0x7f5688070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 0x7f5688070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 -- 172.21.15.103:0/3390459405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f5688102de0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568d93d700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f568400b8b0 tx=0x7f5684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568400e040 con 0x7f5688102de0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5688070fc0 con 0x7f5688102de0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5684004290 con 0x7f5688102de0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568401c3f0 con 0x7f5688102de0 2024-03-22T02:40:07.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.289+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568811a010 con 0x7f5688102de0 2024-03-22T02:40:07.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.290+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56881000f0 con 0x7f5688102de0 2024-03-22T02:40:07.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.291+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5684013420 con 0x7f5688102de0 2024-03-22T02:40:07.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.291+0000 7f567e7fc700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 0x7f5674070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.291+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f568408f030 con 0x7f5688102de0 2024-03-22T02:40:07.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.291+0000 7f568d13c700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 0x7f5674070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.291+0000 7f568d13c700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 0x7f5674070850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56880719f0 tx=0x7f567800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.296+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56840582b0 con 0x7f5688102de0 2024-03-22T02:40:07.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.509+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5688071860 con 0x7f567406e3a0 2024-03-22T02:40:07.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.510+0000 7f567e7fc700 1 -- 172.21.15.103:0/3390459405 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5688071860 con 0x7f567406e3a0 2024-03-22T02:40:07.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 msgr2=0x7f5674070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 0x7f5674070850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56880719f0 tx=0x7f567800b040 comp rx=0 tx=0).stop 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 msgr2=0x7f56880702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f568400b8b0 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 shutdown_connections 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f567406e3a0 0x7f5674070850 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5688102de0 0x7f56880702c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 --2- 172.21.15.103:0/3390459405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5688103780 0x7f5688070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 >> 172.21.15.103:0/3390459405 conn(0x7f56880f8950 msgr2=0x7f56880fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 shutdown_connections 2024-03-22T02:40:07.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.512+0000 7f568fba1700 1 -- 172.21.15.103:0/3390459405 wait complete. 2024-03-22T02:40:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 -- 172.21.15.103:0/2724662165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 msgr2=0x7f60000fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 --2- 172.21.15.103:0/2724662165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60000fce30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc004940 tx=0x7f5ffc009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 -- 172.21.15.103:0/2724662165 shutdown_connections 2024-03-22T02:40:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 --2- 172.21.15.103:0/2724662165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60000fce30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 --2- 172.21.15.103:0/2724662165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f60001031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 -- 172.21.15.103:0/2724662165 >> 172.21.15.103:0/2724662165 conn(0x7f60000f89d0 msgr2=0x7f60000fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 -- 172.21.15.103:0/2724662165 shutdown_connections 2024-03-22T02:40:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.612+0000 7f600830b700 1 -- 172.21.15.103:0/2724662165 wait complete. 2024-03-22T02:40:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 Processor -- start 2024-03-22T02:40:07.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 -- start start 2024-03-22T02:40:07.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f6000113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000114740 con 0x7f60001037c0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f600830b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600010db50 con 0x7f6000102e20 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35160/0 (socket says 172.21.15.103:35160) 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 -- 172.21.15.103:0/1716138632 learned_addr learned my addr 172.21.15.103:0/1716138632 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60060a7700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f6000113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 -- 172.21.15.103:0/1716138632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 msgr2=0x7f6000113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f6000113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60058a6700 1 -- 172.21.15.103:0/1716138632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc005040 con 0x7f60001037c0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.613+0000 7f60060a7700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f6000113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f60058a6700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009100 tx=0x7f5ffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc00e040 con 0x7f60001037c0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600010ddd0 con 0x7f60001037c0 2024-03-22T02:40:07.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ffc013420 con 0x7f60001037c0 2024-03-22T02:40:07.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc01c610 con 0x7f60001037c0 2024-03-22T02:40:07.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.614+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600010e390 con 0x7f60001037c0 2024-03-22T02:40:07.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.615+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60001052f0 con 0x7f60001037c0 2024-03-22T02:40:07.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.615+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5ffc013590 con 0x7f60001037c0 2024-03-22T02:40:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.616+0000 7f5ff6ffd700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 0x7f5fec070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.616+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5ffc08f090 con 0x7f60001037c0 2024-03-22T02:40:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.616+0000 7f60060a7700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 0x7f5fec070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.616+0000 7f60060a7700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 0x7f5fec070850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00041d0 tx=0x7f5ff0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.620+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ffc058310 con 0x7f60001037c0 2024-03-22T02:40:07.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.826+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f600005f690 con 0x7f5fec06e3a0 2024-03-22T02:40:07.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.836+0000 7f5ff6ffd700 1 -- 172.21.15.103:0/1716138632 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f600005f690 con 0x7f5fec06e3a0 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (8m) 2m ago 19m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (6m) 2m ago 20m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (6m) 111s ago 19m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (8m) 2m ago 19m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (2m) 2m ago 14m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (2m) 2m ago 14m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (114s) 111s ago 14m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (2m) 111s ago 14m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (10m) 2m ago 21m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (9m) 111s ago 18m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (7m) 2m ago 21m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (7m) 111s ago 18m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (9m) 2m ago 19m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:40:07.837 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (9m) 111s ago 18m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (5m) 2m ago 18m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (5m) 2m ago 17m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (4m) 2m ago 17m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (4m) 111s ago 16m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (3m) 111s ago 16m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (3m) 111s ago 15m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:40:07.838 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (8m) 2m ago 19m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:40:07.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 msgr2=0x7f5fec070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 0x7f5fec070850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00041d0 tx=0x7f5ff0009040 comp rx=0 tx=0).stop 2024-03-22T02:40:07.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 msgr2=0x7f60001140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009100 tx=0x7f5ffc009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 shutdown_connections 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5fec06e3a0 0x7f5fec070850 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6000102e20 0x7f6000113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 --2- 172.21.15.103:0/1716138632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60001037c0 0x7f60001140f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.838+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 >> 172.21.15.103:0/1716138632 conn(0x7f60000f89d0 msgr2=0x7f6000102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.839+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 shutdown_connections 2024-03-22T02:40:07.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.839+0000 7f600830b700 1 -- 172.21.15.103:0/1716138632 wait complete. 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.939+0000 7fbcd70e5700 1 -- 172.21.15.103:0/832743008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 msgr2=0x7fbcd01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.939+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/832743008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd01031d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4004940 tx=0x7fbcc4009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- 172.21.15.103:0/832743008 shutdown_connections 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/832743008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 0x7fbcd00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/832743008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd01031d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- 172.21.15.103:0/832743008 >> 172.21.15.103:0/832743008 conn(0x7fbcd00f89b0 msgr2=0x7fbcd00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- 172.21.15.103:0/832743008 shutdown_connections 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- 172.21.15.103:0/832743008 wait complete. 2024-03-22T02:40:07.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 Processor -- start 2024-03-22T02:40:07.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- start start 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 0x7fbcd0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0114510 con 0x7fbcd0102e00 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.940+0000 7fbcd70e5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd010db30 con 0x7fbcd01037a0 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35176/0 (socket says 172.21.15.103:35176) 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 -- 172.21.15.103:0/2821133582 learned_addr learned my addr 172.21.15.103:0/2821133582 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:07.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbccffff700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 0x7fbcd0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 -- 172.21.15.103:0/2821133582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 msgr2=0x7fbcd0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 0x7fbcd0113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 -- 172.21.15.103:0/2821133582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4005040 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd4e81700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400b3b0 tx=0x7fbcc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc400e040 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd010ddb0 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcc4013420 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcc401c630 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.941+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd010e370 con 0x7fbcd0102e00 2024-03-22T02:40:07.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.942+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd0100110 con 0x7fbcd0102e00 2024-03-22T02:40:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.943+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fbcc4013590 con 0x7fbcd0102e00 2024-03-22T02:40:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.943+0000 7fbccd7fa700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 0x7fbcc0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.943+0000 7fbccffff700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 0x7fbcc0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.943+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbcc408efe0 con 0x7fbcd0102e00 2024-03-22T02:40:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.944+0000 7fbccffff700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 0x7fbcc0070850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbcd010f220 tx=0x7fbcbc0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:07.948+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbcc4058260 con 0x7fbcd0102e00 2024-03-22T02:40:08.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.231+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbcd0109f20 con 0x7fbcd0102e00 2024-03-22T02:40:08.232 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:07 smithi103 ceph-mon[115211]: pgmap v258: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:08.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.231+0000 7fbccd7fa700 1 -- 172.21.15.103:0/2821133582 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fbcc405b920 con 0x7fbcd0102e00 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:08.233 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:40:08.234 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:40:08.234 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:40:08.234 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.233+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 msgr2=0x7fbcc0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.233+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 0x7fbcc0070850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbcd010f220 tx=0x7fbcbc0043c0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.233+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 msgr2=0x7fbcd0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.233+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400b3b0 tx=0x7fbcc4009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 shutdown_connections 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbcc006e3a0 0x7fbcc0070850 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbcd0102e00 0x7fbcd0113980 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 --2- 172.21.15.103:0/2821133582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcd01037a0 0x7fbcd0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 >> 172.21.15.103:0/2821133582 conn(0x7fbcd00f89b0 msgr2=0x7fbcd01073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 shutdown_connections 2024-03-22T02:40:08.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.234+0000 7fbcd70e5700 1 -- 172.21.15.103:0/2821133582 wait complete. 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.333+0000 7f921ac4c700 1 -- 172.21.15.103:0/1569562179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9214101d20 msgr2=0x7f92141020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.333+0000 7f921ac4c700 1 --2- 172.21.15.103:0/1569562179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9214101d20 0x7f92141020f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f92080049c0 tx=0x7f9208009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 -- 172.21.15.103:0/1569562179 shutdown_connections 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 --2- 172.21.15.103:0/1569562179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92141026c0 0x7f92140fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 --2- 172.21.15.103:0/1569562179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9214101d20 0x7f92141020f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 -- 172.21.15.103:0/1569562179 >> 172.21.15.103:0/1569562179 conn(0x7f92140f7870 msgr2=0x7f92140f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 -- 172.21.15.103:0/1569562179 shutdown_connections 2024-03-22T02:40:08.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 -- 172.21.15.103:0/1569562179 wait complete. 2024-03-22T02:40:08.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 Processor -- start 2024-03-22T02:40:08.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 -- start start 2024-03-22T02:40:08.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.334+0000 7f921ac4c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9214101d20 0x7f9214111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f921ac4c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f921ac4c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92141122e0 con 0x7f92141026c0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f921ac4c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921410b8e0 con 0x7f9214101d20 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35204/0 (socket says 172.21.15.103:35204) 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 -- 172.21.15.103:0/312596806 learned_addr learned my addr 172.21.15.103:0/312596806 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f92189e8700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9214101d20 0x7f9214111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 -- 172.21.15.103:0/312596806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9214101d20 msgr2=0x7f9214111750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9214101d20 0x7f9214111750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 -- 172.21.15.103:0/312596806 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9208005040 con 0x7f92141026c0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f9213fff700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9200002a50 tx=0x7f9200004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9200007cc0 con 0x7f92141026c0 2024-03-22T02:40:08.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f920000f040 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921410bbc0 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920000d9c0 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.335+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921410c180 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.336+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92140fd470 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.337+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9200009730 con 0x7f92141026c0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.337+0000 7f92117fa700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 0x7f9204070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.337+0000 7f92189e8700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 0x7f9204070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.337+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f920008da40 con 0x7f92141026c0 2024-03-22T02:40:08.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.338+0000 7f92189e8700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 0x7f9204070850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9208009fd0 tx=0x7f9208009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.342+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f920005a7f0 con 0x7f92141026c0 2024-03-22T02:40:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:07 smithi122 ceph-mon[83368]: pgmap v258: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:08.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.579+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9214107ca0 con 0x7f92141026c0 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.580+0000 7f92117fa700 1 -- 172.21.15.103:0/312596806 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f920005a380 con 0x7f92141026c0 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:40:08.581 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:40:08.582 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:08.583 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:08.584 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 msgr2=0x7f9204070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 0x7f9204070850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9208009fd0 tx=0x7f9208009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 msgr2=0x7f9214111c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9200002a50 tx=0x7f9200004600 comp rx=0 tx=0).stop 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 shutdown_connections 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f920406e3a0 0x7f9204070850 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9214101d20 0x7f9214111750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 --2- 172.21.15.103:0/312596806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92141026c0 0x7f9214111c90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 >> 172.21.15.103:0/312596806 conn(0x7f92140f7870 msgr2=0x7f92140f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 shutdown_connections 2024-03-22T02:40:08.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.583+0000 7f921ac4c700 1 -- 172.21.15.103:0/312596806 wait complete. 2024-03-22T02:40:08.586 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 -- 172.21.15.103:0/3554754419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 msgr2=0x7f89981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 --2- 172.21.15.103:0/3554754419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f89981031d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8988004940 tx=0x7f8988009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 -- 172.21.15.103:0/3554754419 shutdown_connections 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 --2- 172.21.15.103:0/3554754419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 --2- 172.21.15.103:0/3554754419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f89981031d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 -- 172.21.15.103:0/3554754419 >> 172.21.15.103:0/3554754419 conn(0x7f89980f89b0 msgr2=0x7f89980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 -- 172.21.15.103:0/3554754419 shutdown_connections 2024-03-22T02:40:08.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.692+0000 7f899e05d700 1 -- 172.21.15.103:0/3554754419 wait complete. 2024-03-22T02:40:08.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 Processor -- start 2024-03-22T02:40:08.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 -- start start 2024-03-22T02:40:08.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89981140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89981147c0 con 0x7f8998102e00 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f899e05d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899810db40 con 0x7f89981037a0 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f89977fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f89977fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35224/0 (socket says 172.21.15.103:35224) 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f89977fe700 1 -- 172.21.15.103:0/206424328 learned_addr learned my addr 172.21.15.103:0/206424328 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.693+0000 7f8996ffd700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89981140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f89977fe700 1 -- 172.21.15.103:0/206424328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 msgr2=0x7f89981140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f89977fe700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89981140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f89977fe700 1 -- 172.21.15.103:0/206424328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988005040 con 0x7f8998102e00 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f8996ffd700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89981140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f89977fe700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f898800b3b0 tx=0x7f8988009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898800e040 con 0x7f8998102e00 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f898800bbc0 con 0x7f8998102e00 2024-03-22T02:40:08.696 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899810ddc0 con 0x7f8998102e00 2024-03-22T02:40:08.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898801c3f0 con 0x7f8998102e00 2024-03-22T02:40:08.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.694+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899810e300 con 0x7f8998102e00 2024-03-22T02:40:08.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.695+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89980fe550 con 0x7f8998102e00 2024-03-22T02:40:08.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.695+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f898800bd30 con 0x7f8998102e00 2024-03-22T02:40:08.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.696+0000 7f897ffff700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 0x7f8980070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.696+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f898808f0a0 con 0x7f8998102e00 2024-03-22T02:40:08.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.696+0000 7f8996ffd700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 0x7f8980070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.696+0000 7f8996ffd700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 0x7f8980070850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f899810f440 tx=0x7f898c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.701+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8988058260 con 0x7f8998102e00 2024-03-22T02:40:08.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.916+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f899804bed0 con 0x7f898006e3a0 2024-03-22T02:40:08.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.917+0000 7f897ffff700 1 -- 172.21.15.103:0/206424328 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f899804bed0 con 0x7f898006e3a0 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:40:08.919 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:40:08.920 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:40:08.920 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:40:08.920 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:40:08.920 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:40:08.920 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 msgr2=0x7f8980070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 0x7f8980070850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f899810f440 tx=0x7f898c008040 comp rx=0 tx=0).stop 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 msgr2=0x7f8998113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f898800b3b0 tx=0x7f8988009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 shutdown_connections 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f898006e3a0 0x7f8980070850 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8998102e00 0x7f8998113ba0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 --2- 172.21.15.103:0/206424328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89981037a0 0x7f89981140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 >> 172.21.15.103:0/206424328 conn(0x7f89980f89b0 msgr2=0x7f89980f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 shutdown_connections 2024-03-22T02:40:08.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:08.920+0000 7f899e05d700 1 -- 172.21.15.103:0/206424328 wait complete. 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- 172.21.15.103:0/2857530356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7701026d0 msgr2=0x7ff7700fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 --2- 172.21.15.103:0/2857530356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7701026d0 0x7ff7700fbd40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff758004940 tx=0x7ff758009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- 172.21.15.103:0/2857530356 shutdown_connections 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 --2- 172.21.15.103:0/2857530356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7701026d0 0x7ff7700fbd40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 --2- 172.21.15.103:0/2857530356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff770101d30 0x7ff770102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- 172.21.15.103:0/2857530356 >> 172.21.15.103:0/2857530356 conn(0x7ff7700f7860 msgr2=0x7ff7700f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- 172.21.15.103:0/2857530356 shutdown_connections 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- 172.21.15.103:0/2857530356 wait complete. 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 Processor -- start 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.023+0000 7ff77614a700 1 -- start start 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff77614a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff77614a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7701026d0 0x7ff770113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff77614a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff770114590 con 0x7ff770101d30 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff77614a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77010db90 con 0x7ff7701026d0 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35234/0 (socket says 172.21.15.103:35234) 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 -- 172.21.15.103:0/2802566484 learned_addr learned my addr 172.21.15.103:0/2802566484 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff767fff700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7701026d0 0x7ff770113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 -- 172.21.15.103:0/2802566484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7701026d0 msgr2=0x7ff770113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7701026d0 0x7ff770113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 -- 172.21.15.103:0/2802566484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758005040 con 0x7ff770101d30 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76f7fe700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff760002a80 tx=0x7ff7600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff760007cc0 con 0x7ff770101d30 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff76000f040 con 0x7ff770101d30 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77010de70 con 0x7ff770101d30 2024-03-22T02:40:09.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.024+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff76000dac0 con 0x7ff770101d30 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.025+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77010e380 con 0x7ff770101d30 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.025+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff770104200 con 0x7ff770101d30 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.026+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff76000dc20 con 0x7ff770101d30 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.026+0000 7ff76cff9700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 0x7ff75c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.027+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff76008d910 con 0x7ff770101d30 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.027+0000 7ff767fff700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 0x7ff75c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.027+0000 7ff767fff700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 0x7ff75c070850 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff77010ee70 tx=0x7ff758009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.031+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff76005a6c0 con 0x7ff770101d30 2024-03-22T02:40:09.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.312+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff770109f00 con 0x7ff770101d30 2024-03-22T02:40:09.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:08 smithi103 ceph-mon[115211]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:08 smithi103 ceph-mon[115211]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:08 smithi103 ceph-mon[115211]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:08 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2821133582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:40:09.313 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:08 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/312596806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.312+0000 7ff76cff9700 1 -- 172.21.15.103:0/2802566484 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff76005a250 con 0x7ff770101d30 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:40:09.314 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 msgr2=0x7ff75c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 0x7ff75c070850 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff77010ee70 tx=0x7ff758009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 msgr2=0x7ff770113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff760002a80 tx=0x7ff7600046b0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 shutdown_connections 2024-03-22T02:40:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff75c06e3a0 0x7ff75c070850 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff770101d30 0x7ff770113a00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 --2- 172.21.15.103:0/2802566484 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7701026d0 0x7ff770113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 >> 172.21.15.103:0/2802566484 conn(0x7ff7700f7860 msgr2=0x7ff7701010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 shutdown_connections 2024-03-22T02:40:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:09.315+0000 7ff77614a700 1 -- 172.21.15.103:0/2802566484 wait complete. 2024-03-22T02:40:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:08 smithi122 ceph-mon[83368]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:08 smithi122 ceph-mon[83368]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:08 smithi122 ceph-mon[83368]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:09.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:08 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2821133582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:40:09.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:08 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/312596806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:40:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:09 smithi122 ceph-mon[83368]: pgmap v259: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:09 smithi122 ceph-mon[83368]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:10.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:09 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2802566484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:40:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:09 smithi103 ceph-mon[115211]: pgmap v259: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:09 smithi103 ceph-mon[115211]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:09 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2802566484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:40:12.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:11 smithi122 ceph-mon[83368]: pgmap v260: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:11 smithi103 ceph-mon[115211]: pgmap v260: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:13 smithi122 ceph-mon[83368]: pgmap v261: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:13 smithi103 ceph-mon[115211]: pgmap v261: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:15 smithi122 ceph-mon[83368]: pgmap v262: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:15 smithi103 ceph-mon[115211]: pgmap v262: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:18 smithi122 ceph-mon[83368]: pgmap v263: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:18 smithi103 ceph-mon[115211]: pgmap v263: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:20 smithi122 ceph-mon[83368]: pgmap v264: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:20 smithi103 ceph-mon[115211]: pgmap v264: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:22 smithi122 ceph-mon[83368]: pgmap v265: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:22 smithi103 ceph-mon[115211]: pgmap v265: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:24 smithi122 ceph-mon[83368]: pgmap v266: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:24 smithi103 ceph-mon[115211]: pgmap v266: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:26 smithi122 ceph-mon[83368]: pgmap v267: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:26 smithi103 ceph-mon[115211]: pgmap v267: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:28 smithi122 ceph-mon[83368]: pgmap v268: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:28 smithi103 ceph-mon[115211]: pgmap v268: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:30 smithi122 ceph-mon[83368]: pgmap v269: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:30 smithi103 ceph-mon[115211]: pgmap v269: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:32 smithi122 ceph-mon[83368]: pgmap v270: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:32 smithi103 ceph-mon[115211]: pgmap v270: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:34 smithi122 ceph-mon[83368]: pgmap v271: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:34 smithi103 ceph-mon[115211]: pgmap v271: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:35 smithi122 ceph-mon[83368]: pgmap v272: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:35 smithi103 ceph-mon[115211]: pgmap v272: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:37 smithi122 ceph-mon[83368]: pgmap v273: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:37 smithi103 ceph-mon[115211]: pgmap v273: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:39.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 -- 172.21.15.103:0/3246480589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 msgr2=0x7f5160100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 --2- 172.21.15.103:0/3246480589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f5160100bd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5154004940 tx=0x7f5154009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 -- 172.21.15.103:0/3246480589 shutdown_connections 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 --2- 172.21.15.103:0/3246480589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f5160100bd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 --2- 172.21.15.103:0/3246480589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 0x7f51600fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 -- 172.21.15.103:0/3246480589 >> 172.21.15.103:0/3246480589 conn(0x7f51600f7870 msgr2=0x7f51600f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 -- 172.21.15.103:0/3246480589 shutdown_connections 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 -- 172.21.15.103:0/3246480589 wait complete. 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.430+0000 7f5165b4f700 1 Processor -- start 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f5165b4f700 1 -- start start 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f5165b4f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 0x7f516011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f5165b4f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f5165b4f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516011ac70 con 0x7f51600fc5b0 2024-03-22T02:40:39.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f5165b4f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5160114250 con 0x7f51600fbc10 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57162/0 (socket says 172.21.15.103:57162) 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 -- 172.21.15.103:0/2495438261 learned_addr learned my addr 172.21.15.103:0/2495438261 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515f7fe700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 0x7f516011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 -- 172.21.15.103:0/2495438261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 msgr2=0x7f516011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 0x7f516011a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 -- 172.21.15.103:0/2495438261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154005040 con 0x7f51600fc5b0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.431+0000 7f515effd700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f515400b3b0 tx=0x7f5154009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.432+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f515400e040 con 0x7f51600fc5b0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.432+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51601144d0 con 0x7f51600fc5b0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.432+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f515400bbc0 con 0x7f51600fc5b0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.432+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f515401c3f0 con 0x7f51600fc5b0 2024-03-22T02:40:39.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.432+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5160114a10 con 0x7f51600fc5b0 2024-03-22T02:40:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.433+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5160066b50 con 0x7f51600fc5b0 2024-03-22T02:40:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.433+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f515400bd30 con 0x7f51600fc5b0 2024-03-22T02:40:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.434+0000 7f5147fff700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 0x7f5148070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.434+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f515408f060 con 0x7f51600fc5b0 2024-03-22T02:40:39.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.434+0000 7f515f7fe700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 0x7f5148070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.434+0000 7f515f7fe700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 0x7f5148070850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5150004c40 tx=0x7f5150009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:39.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.438+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f515405bd90 con 0x7f51600fc5b0 2024-03-22T02:40:39.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.655+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5160115730 con 0x7f514806e3a0 2024-03-22T02:40:39.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.656+0000 7f5147fff700 1 -- 172.21.15.103:0/2495438261 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5160115730 con 0x7f514806e3a0 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 msgr2=0x7f5148070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 0x7f5148070850 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5150004c40 tx=0x7f5150009040 comp rx=0 tx=0).stop 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 msgr2=0x7f516011a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f515400b3b0 tx=0x7f5154009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 shutdown_connections 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f514806e3a0 0x7f5148070850 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51600fbc10 0x7f516011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 --2- 172.21.15.103:0/2495438261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51600fc5b0 0x7f516011a620 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 >> 172.21.15.103:0/2495438261 conn(0x7f51600f7870 msgr2=0x7f51600f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 shutdown_connections 2024-03-22T02:40:39.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.658+0000 7f5165b4f700 1 -- 172.21.15.103:0/2495438261 wait complete. 2024-03-22T02:40:39.670 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- 172.21.15.103:0/2088142812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fc450 msgr2=0x7efc3c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 --2- 172.21.15.103:0/2088142812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fc450 0x7efc3c1091b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efc30004940 tx=0x7efc30009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- 172.21.15.103:0/2088142812 shutdown_connections 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 --2- 172.21.15.103:0/2088142812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fc450 0x7efc3c1091b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 --2- 172.21.15.103:0/2088142812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fbb40 0x7efc3c0fbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- 172.21.15.103:0/2088142812 >> 172.21.15.103:0/2088142812 conn(0x7efc3c0f7890 msgr2=0x7efc3c0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- 172.21.15.103:0/2088142812 shutdown_connections 2024-03-22T02:40:39.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- 172.21.15.103:0/2088142812 wait complete. 2024-03-22T02:40:39.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 Processor -- start 2024-03-22T02:40:39.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.762+0000 7efc42168700 1 -- start start 2024-03-22T02:40:39.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc42168700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc42168700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fc450 0x7efc3c116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc42168700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c1166e0 con 0x7efc3c0fbb40 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc42168700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c10fcc0 con 0x7efc3c0fc450 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57176/0 (socket says 172.21.15.103:57176) 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 -- 172.21.15.103:0/4163636120 learned_addr learned my addr 172.21.15.103:0/4163636120 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3affd700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fc450 0x7efc3c116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 -- 172.21.15.103:0/4163636120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fc450 msgr2=0x7efc3c116090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fc450 0x7efc3c116090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 -- 172.21.15.103:0/4163636120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30005040 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc3b7fe700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efc2c002a50 tx=0x7efc2c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.763+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc2c007cc0 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.764+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c10ffa0 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.764+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc2c00f040 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.764+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c110560 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.764+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc2c00da30 con 0x7efc3c0fbb40 2024-03-22T02:40:39.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.765+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c066bd0 con 0x7efc3c0fbb40 2024-03-22T02:40:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.765+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efc2c00dc70 con 0x7efc3c0fbb40 2024-03-22T02:40:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.766+0000 7efc23fff700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 0x7efc24070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.766+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc2c08db90 con 0x7efc3c0fbb40 2024-03-22T02:40:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.766+0000 7efc3affd700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 0x7efc24070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:39.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.766+0000 7efc3affd700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 0x7efc24070850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc3c1113f0 tx=0x7efc30009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:39.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.771+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc2c05a8c0 con 0x7efc3c0fbb40 2024-03-22T02:40:39.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.986+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc3c1111a0 con 0x7efc2406e3a0 2024-03-22T02:40:39.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.987+0000 7efc23fff700 1 -- 172.21.15.103:0/4163636120 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efc3c1111a0 con 0x7efc2406e3a0 2024-03-22T02:40:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 msgr2=0x7efc24070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 0x7efc24070850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc3c1113f0 tx=0x7efc30009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:39.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 msgr2=0x7efc3c115b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efc2c002a50 tx=0x7efc2c0046b0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 shutdown_connections 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2406e3a0 0x7efc24070850 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc3c0fbb40 0x7efc3c115b50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 --2- 172.21.15.103:0/4163636120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc3c0fc450 0x7efc3c116090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.989+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 >> 172.21.15.103:0/4163636120 conn(0x7efc3c0f7890 msgr2=0x7efc3c0f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.990+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 shutdown_connections 2024-03-22T02:40:39.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:39.990+0000 7efc42168700 1 -- 172.21.15.103:0/4163636120 wait complete. 2024-03-22T02:40:40.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 -- 172.21.15.103:0/659176673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 msgr2=0x7f167c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 --2- 172.21.15.103:0/659176673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c0fbd30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1668004940 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:40.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 -- 172.21.15.103:0/659176673 shutdown_connections 2024-03-22T02:40:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 --2- 172.21.15.103:0/659176673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c0fbd30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 --2- 172.21.15.103:0/659176673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 0x7f167c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 -- 172.21.15.103:0/659176673 >> 172.21.15.103:0/659176673 conn(0x7f167c0f7870 msgr2=0x7f167c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:40.094 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:39 smithi103 ceph-mon[115211]: pgmap v274: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 -- 172.21.15.103:0/659176673 shutdown_connections 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.091+0000 7f1682b66700 1 -- 172.21.15.103:0/659176673 wait complete. 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 Processor -- start 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 -- start start 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 0x7f167c10c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c067120 con 0x7f167c1026c0 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f1682b66700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c10d3d0 con 0x7f167c101d20 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f167bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f167bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57198/0 (socket says 172.21.15.103:57198) 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.092+0000 7f167bfff700 1 -- 172.21.15.103:0/1733321144 learned_addr learned my addr 172.21.15.103:0/1733321144 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f1680902700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 0x7f167c10c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f167bfff700 1 -- 172.21.15.103:0/1733321144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 msgr2=0x7f167c10c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f167bfff700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 0x7f167c10c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f167bfff700 1 -- 172.21.15.103:0/1733321144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668005040 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f167bfff700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1668009100 tx=0x7f1668009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f166800e040 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c10d650 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1668004290 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f166801c3f0 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.093+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c11a180 con 0x7f167c1026c0 2024-03-22T02:40:40.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.094+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c0fd470 con 0x7f167c1026c0 2024-03-22T02:40:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.095+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1668013790 con 0x7f167c1026c0 2024-03-22T02:40:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.095+0000 7f16797fa700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 0x7f166c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.095+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f166808efe0 con 0x7f167c1026c0 2024-03-22T02:40:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.095+0000 7f1680902700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 0x7f166c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.096+0000 7f1680902700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 0x7f166c070850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1670004990 tx=0x7f1670009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.100+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f166805bdf0 con 0x7f167c1026c0 2024-03-22T02:40:40.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.311+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f167c108e70 con 0x7f166c06e3a0 2024-03-22T02:40:40.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.321+0000 7f16797fa700 1 -- 172.21.15.103:0/1733321144 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f167c108e70 con 0x7f166c06e3a0 2024-03-22T02:40:40.323 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (9m) 2m ago 20m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (6m) 2m ago 20m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (6m) 2m ago 19m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (8m) 2m ago 20m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (3m) 2m ago 14m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (2m) 2m ago 14m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (2m) 2m ago 14m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (2m) 2m ago 14m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (10m) 2m ago 21m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (10m) 2m ago 19m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (8m) 2m ago 21m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (7m) 2m ago 19m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (9m) 2m ago 20m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:40:40.324 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (9m) 2m ago 19m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (6m) 2m ago 18m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (5m) 2m ago 18m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (5m) 2m ago 17m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (4m) 2m ago 17m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (4m) 2m ago 16m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (3m) 2m ago 16m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:40:40.325 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (9m) 2m ago 19m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 msgr2=0x7f166c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 0x7f166c070850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1670004990 tx=0x7f1670009ee0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 msgr2=0x7f167c10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1668009100 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 shutdown_connections 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f166c06e3a0 0x7f166c070850 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f167c101d20 0x7f167c10c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 --2- 172.21.15.103:0/1733321144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f167c1026c0 0x7f167c10ce90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 >> 172.21.15.103:0/1733321144 conn(0x7f167c0f7870 msgr2=0x7f167c0f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 shutdown_connections 2024-03-22T02:40:40.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.324+0000 7f1682b66700 1 -- 172.21.15.103:0/1733321144 wait complete. 2024-03-22T02:40:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:39 smithi122 ceph-mon[83368]: pgmap v274: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- 172.21.15.103:0/4069905054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 msgr2=0x7faff8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 --2- 172.21.15.103:0/4069905054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff8102100 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faff4004940 tx=0x7faff4009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- 172.21.15.103:0/4069905054 shutdown_connections 2024-03-22T02:40:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 --2- 172.21.15.103:0/4069905054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff80fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 --2- 172.21.15.103:0/4069905054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff8102100 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- 172.21.15.103:0/4069905054 >> 172.21.15.103:0/4069905054 conn(0x7faff80f78e0 msgr2=0x7faff80f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- 172.21.15.103:0/4069905054 shutdown_connections 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- 172.21.15.103:0/4069905054 wait complete. 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 Processor -- start 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.424+0000 7fafff3d0700 1 -- start start 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafff3d0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafff3d0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff810f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafff3d0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff810f5b0 con 0x7faff8101d30 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafff3d0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff810f6f0 con 0x7faff81026d0 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57218/0 (socket says 172.21.15.103:57218) 2024-03-22T02:40:40.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 -- 172.21.15.103:0/2259989455 learned_addr learned my addr 172.21.15.103:0/2259989455 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffc96b700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff810f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 -- 172.21.15.103:0/2259989455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 msgr2=0x7faff810f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff810f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 -- 172.21.15.103:0/2259989455 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff4005040 con 0x7faff8101d30 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffc96b700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff810f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7faffd16c700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faff400b3b0 tx=0x7faff4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff400e040 con 0x7faff8101d30 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faff400bbc0 con 0x7faff8101d30 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff810f970 con 0x7faff8101d30 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.425+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff401c3f0 con 0x7faff8101d30 2024-03-22T02:40:40.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.426+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff8117ea0 con 0x7faff8101d30 2024-03-22T02:40:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.426+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff80fd480 con 0x7faff8101d30 2024-03-22T02:40:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.427+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7faff400bd30 con 0x7faff8101d30 2024-03-22T02:40:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.427+0000 7fafedffb700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 0x7fafe4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.428+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faff408f060 con 0x7faff8101d30 2024-03-22T02:40:40.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.428+0000 7faffc96b700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 0x7fafe4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.428+0000 7faffc96b700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 0x7fafe4070850 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faff8066a60 tx=0x7fafe8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.432+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faff4058260 con 0x7faff8101d30 2024-03-22T02:40:40.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.714+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faff81180c0 con 0x7faff8101d30 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.714+0000 7fafedffb700 1 -- 172.21.15.103:0/2259989455 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7faff405b920 con 0x7faff8101d30 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:40:40.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:40:40.717 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:40:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 msgr2=0x7fafe4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 0x7fafe4070850 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faff8066a60 tx=0x7fafe8009040 comp rx=0 tx=0).stop 2024-03-22T02:40:40.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 msgr2=0x7faff810eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faff400b3b0 tx=0x7faff4009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 shutdown_connections 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fafe406e3a0 0x7fafe4070850 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faff8101d30 0x7faff810eb30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 --2- 172.21.15.103:0/2259989455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faff81026d0 0x7faff810f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 >> 172.21.15.103:0/2259989455 conn(0x7faff80f78e0 msgr2=0x7faff80f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.716+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 shutdown_connections 2024-03-22T02:40:40.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.717+0000 7fafff3d0700 1 -- 172.21.15.103:0/2259989455 wait complete. 2024-03-22T02:40:40.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/2330081521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fe7b0 msgr2=0x7f6b98106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/2330081521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fe7b0 0x7f6b98106df0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c004940 tx=0x7f6b8c009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/2330081521 shutdown_connections 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/2330081521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fe7b0 0x7f6b98106df0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/2330081521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fde10 0x7f6b980fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/2330081521 >> 172.21.15.103:0/2330081521 conn(0x7f6b980f7860 msgr2=0x7f6b980f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/2330081521 shutdown_connections 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.817+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/2330081521 wait complete. 2024-03-22T02:40:40.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 Processor -- start 2024-03-22T02:40:40.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 -- start start 2024-03-22T02:40:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fe7b0 0x7f6b980fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980fdb90 con 0x7f6b980fde10 2024-03-22T02:40:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b9d6f7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980fc6c0 con 0x7f6b980fe7b0 2024-03-22T02:40:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57228/0 (socket says 172.21.15.103:57228) 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.818+0000 7f6b96ffd700 1 -- 172.21.15.103:0/1894652127 learned_addr learned my addr 172.21.15.103:0/1894652127 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b967fc700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fe7b0 0x7f6b980fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b96ffd700 1 -- 172.21.15.103:0/1894652127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fe7b0 msgr2=0x7f6b980fc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b96ffd700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fe7b0 0x7f6b980fc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b96ffd700 1 -- 172.21.15.103:0/1894652127 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c005040 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b96ffd700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c830 tx=0x7f6b88004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b880106f0 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b980fc9a0 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b88010d20 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b880113a0 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.819+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9811a130 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.820+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b98066af0 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.821+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6b88011500 con 0x7f6b980fde10 2024-03-22T02:40:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.821+0000 7f6b7f7fe700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 0x7f6b80070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:40.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.821+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6b8808dd70 con 0x7f6b980fde10 2024-03-22T02:40:40.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.821+0000 7f6b967fc700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 0x7f6b80070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:40.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.821+0000 7f6b967fc700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 0x7f6b80070850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b980fd370 tx=0x7f6b8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:40.826+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b88056ff0 con 0x7f6b980fde10 2024-03-22T02:40:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.066+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6b9811a380 con 0x7f6b980fde10 2024-03-22T02:40:41.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.067+0000 7f6b7f7fe700 1 -- 172.21.15.103:0/1894652127 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f6b8805a6b0 con 0x7f6b980fde10 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:40:41.069 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:40:41.070 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:40:41.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:40:41.072 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:41.072 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:41.072 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:41.072 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 msgr2=0x7f6b80070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 0x7f6b80070850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b980fd370 tx=0x7f6b8c009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 msgr2=0x7f6b980fbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c830 tx=0x7f6b88004470 comp rx=0 tx=0).stop 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 shutdown_connections 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6b8006e3a0 0x7f6b80070850 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b980fde10 0x7f6b980fbc40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 --2- 172.21.15.103:0/1894652127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b980fe7b0 0x7f6b980fc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 >> 172.21.15.103:0/1894652127 conn(0x7f6b980f7860 msgr2=0x7f6b980f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 shutdown_connections 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.069+0000 7f6b9d6f7700 1 -- 172.21.15.103:0/1894652127 wait complete. 2024-03-22T02:40:41.073 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 -- 172.21.15.103:0/1336507191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 msgr2=0x7febb80fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 --2- 172.21.15.103:0/1336507191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb80fbfe0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feba8004940 tx=0x7feba8009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 -- 172.21.15.103:0/1336507191 shutdown_connections 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 --2- 172.21.15.103:0/1336507191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb80fbfe0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 --2- 172.21.15.103:0/1336507191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 0x7febb8101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 -- 172.21.15.103:0/1336507191 >> 172.21.15.103:0/1336507191 conn(0x7febb8071b70 msgr2=0x7febb8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 -- 172.21.15.103:0/1336507191 shutdown_connections 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.171+0000 7febbe9a0700 1 -- 172.21.15.103:0/1336507191 wait complete. 2024-03-22T02:40:41.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 Processor -- start 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 -- start start 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 0x7febb8112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8113490 con 0x7febb80fbb70 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbe9a0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb810c880 con 0x7febb8101ba0 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57244/0 (socket says 172.21.15.103:57244) 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 -- 172.21.15.103:0/2347326980 learned_addr learned my addr 172.21.15.103:0/2347326980 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd19d700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 0x7febb8112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 -- 172.21.15.103:0/2347326980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 msgr2=0x7febb8112e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 0x7febb8112e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 -- 172.21.15.103:0/2347326980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8005040 con 0x7febb80fbb70 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.172+0000 7febbd99e700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7febb4002a50 tx=0x7febb40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.173+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febb4007cc0 con 0x7febb80fbb70 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.173+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febb400f040 con 0x7febb80fbb70 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.173+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb810cb00 con 0x7febb80fbb70 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.173+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febb400da30 con 0x7febb80fbb70 2024-03-22T02:40:41.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.173+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb810d090 con 0x7febb80fbb70 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.174+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb80fd200 con 0x7febb80fbb70 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.174+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7febb400dbb0 con 0x7febb80fbb70 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.175+0000 7febae7fc700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 0x7feba4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.175+0000 7febbd19d700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 0x7feba4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.175+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7febb408da90 con 0x7febb80fbb70 2024-03-22T02:40:41.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.175+0000 7febbd19d700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 0x7feba4070850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7febb810db60 tx=0x7feba8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:41.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.179+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7febb405a840 con 0x7febb80fbb70 2024-03-22T02:40:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:40 smithi122 ceph-mon[83368]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:41.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:40 smithi122 ceph-mon[83368]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:41.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:40 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2259989455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:40:41.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:40 smithi103 ceph-mon[115211]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:41.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:40 smithi103 ceph-mon[115211]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:41.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:40 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2259989455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:40:41.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.393+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7febb8108c20 con 0x7feba406e3a0 2024-03-22T02:40:41.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.394+0000 7febae7fc700 1 -- 172.21.15.103:0/2347326980 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7febb8108c20 con 0x7feba406e3a0 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:40:41.396 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:40:41.397 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:40:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 msgr2=0x7feba4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 0x7feba4070850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7febb810db60 tx=0x7feba8009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 msgr2=0x7febb8112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7febb4002a50 tx=0x7febb40046b0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 shutdown_connections 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feba406e3a0 0x7feba4070850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febb80fbb70 0x7febb8112900 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 --2- 172.21.15.103:0/2347326980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb8101ba0 0x7febb8112e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.396+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 >> 172.21.15.103:0/2347326980 conn(0x7febb8071b70 msgr2=0x7febb80fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.397+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 shutdown_connections 2024-03-22T02:40:41.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.397+0000 7febbe9a0700 1 -- 172.21.15.103:0/2347326980 wait complete. 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 -- 172.21.15.103:0/245974032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c1026b0 msgr2=0x7f894c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 --2- 172.21.15.103:0/245974032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c1026b0 0x7f894c0fbd20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8938004940 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 -- 172.21.15.103:0/245974032 shutdown_connections 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 --2- 172.21.15.103:0/245974032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c1026b0 0x7f894c0fbd20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 --2- 172.21.15.103:0/245974032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c101d10 0x7f894c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 -- 172.21.15.103:0/245974032 >> 172.21.15.103:0/245974032 conn(0x7f894c0f7880 msgr2=0x7f894c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:41.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 -- 172.21.15.103:0/245974032 shutdown_connections 2024-03-22T02:40:41.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 -- 172.21.15.103:0/245974032 wait complete. 2024-03-22T02:40:41.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.497+0000 7f8952ccf700 1 Processor -- start 2024-03-22T02:40:41.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8952ccf700 1 -- start start 2024-03-22T02:40:41.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8952ccf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8952ccf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c1026b0 0x7f894c10f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8952ccf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c10f9d0 con 0x7f894c101d10 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8952ccf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c10fb10 con 0x7f894c1026b0 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57252/0 (socket says 172.21.15.103:57252) 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 -- 172.21.15.103:0/1885710178 learned_addr learned my addr 172.21.15.103:0/1885710178 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f894bfff700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c1026b0 0x7f894c10f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 -- 172.21.15.103:0/1885710178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c1026b0 msgr2=0x7f894c10f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c1026b0 0x7f894c10f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 -- 172.21.15.103:0/1885710178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938005040 con 0x7f894c101d10 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.498+0000 7f8950a6b700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8940002a50 tx=0x7f89400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.499+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8940007cc0 con 0x7f894c101d10 2024-03-22T02:40:41.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.499+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f894000f040 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.499+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c117eb0 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.499+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894000da30 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.499+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c118390 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.500+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c0fd460 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.500+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f894000dbb0 con 0x7f894c101d10 2024-03-22T02:40:41.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.500+0000 7f89497fa700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 0x7f893c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:41.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.501+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f894008da90 con 0x7f894c101d10 2024-03-22T02:40:41.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.501+0000 7f894bfff700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 0x7f893c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:41.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.501+0000 7f894bfff700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 0x7f893c070850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8938004890 tx=0x7f8938009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:41.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.505+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8940056d10 con 0x7f894c101d10 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.784+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f894c04bed0 con 0x7f894c101d10 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.784+0000 7f89497fa700 1 -- 172.21.15.103:0/1885710178 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f894005a3d0 con 0x7f894c101d10 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:40:41.786 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 msgr2=0x7f893c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 0x7f893c070850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8938004890 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 msgr2=0x7f894c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8940002a50 tx=0x7f89400046b0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 shutdown_connections 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f893c06e3a0 0x7f893c070850 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f894c101d10 0x7f894c10ee40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 --2- 172.21.15.103:0/1885710178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f894c1026b0 0x7f894c10f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:41.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 >> 172.21.15.103:0/1885710178 conn(0x7f894c0f7880 msgr2=0x7f894c0f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:41.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 shutdown_connections 2024-03-22T02:40:41.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:40:41.786+0000 7f8952ccf700 1 -- 172.21.15.103:0/1885710178 wait complete. 2024-03-22T02:40:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:41 smithi103 ceph-mon[115211]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:41 smithi103 ceph-mon[115211]: pgmap v275: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:41 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1894652127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:40:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:41 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1885710178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:40:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:41 smithi122 ceph-mon[83368]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:41 smithi122 ceph-mon[83368]: pgmap v275: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:41 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1894652127' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:40:42.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:41 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1885710178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:40:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:42 smithi122 ceph-mon[83368]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:43.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:42 smithi103 ceph-mon[115211]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:40:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:43 smithi122 ceph-mon[83368]: pgmap v276: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:43 smithi103 ceph-mon[115211]: pgmap v276: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:45 smithi122 ceph-mon[83368]: pgmap v277: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:45 smithi103 ceph-mon[115211]: pgmap v277: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:46 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:46 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:40:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:47 smithi122 ceph-mon[83368]: pgmap v278: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:47 smithi103 ceph-mon[115211]: pgmap v278: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:50 smithi122 ceph-mon[83368]: pgmap v279: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:50 smithi103 ceph-mon[115211]: pgmap v279: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:52 smithi122 ceph-mon[83368]: pgmap v280: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:52.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:52 smithi103 ceph-mon[115211]: pgmap v280: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:54 smithi122 ceph-mon[83368]: pgmap v281: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:54 smithi103 ceph-mon[115211]: pgmap v281: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:56.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:56 smithi122 ceph-mon[83368]: pgmap v282: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:56 smithi103 ceph-mon[115211]: pgmap v282: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:40:58 smithi122 ceph-mon[83368]: pgmap v283: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:40:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:40:58 smithi103 ceph-mon[115211]: pgmap v283: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:00 smithi122 ceph-mon[83368]: pgmap v284: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:00 smithi103 ceph-mon[115211]: pgmap v284: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:02 smithi122 ceph-mon[83368]: pgmap v285: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:02 smithi103 ceph-mon[115211]: pgmap v285: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:02.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:04 smithi122 ceph-mon[83368]: pgmap v286: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:04 smithi103 ceph-mon[115211]: pgmap v286: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:06 smithi122 ceph-mon[83368]: pgmap v287: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:06 smithi103 ceph-mon[115211]: pgmap v287: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:08 smithi122 ceph-mon[83368]: pgmap v288: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:08 smithi103 ceph-mon[115211]: pgmap v288: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:10 smithi122 ceph-mon[83368]: pgmap v289: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:10 smithi103 ceph-mon[115211]: pgmap v289: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:11.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.901+0000 7f64d97cc700 1 -- 172.21.15.103:0/1191809669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fe780 msgr2=0x7f64d4106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:11.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.901+0000 7f64d97cc700 1 --2- 172.21.15.103:0/1191809669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fe780 0x7f64d4106dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f64c4004940 tx=0x7f64c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:11.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 -- 172.21.15.103:0/1191809669 shutdown_connections 2024-03-22T02:41:11.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 --2- 172.21.15.103:0/1191809669 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fe780 0x7f64d4106dc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 --2- 172.21.15.103:0/1191809669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fdde0 0x7f64d40fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 -- 172.21.15.103:0/1191809669 >> 172.21.15.103:0/1191809669 conn(0x7f64d40f7890 msgr2=0x7f64d40f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 -- 172.21.15.103:0/1191809669 shutdown_connections 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 -- 172.21.15.103:0/1191809669 wait complete. 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 Processor -- start 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.902+0000 7f64d97cc700 1 -- start start 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fe780 0x7f64d40fc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40fdb60 con 0x7f64d40fdde0 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40fc690 con 0x7f64d40fe780 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46732/0 (socket says 172.21.15.103:46732) 2024-03-22T02:41:11.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 -- 172.21.15.103:0/2543255401 learned_addr learned my addr 172.21.15.103:0/2543255401 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d27fc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fe780 0x7f64d40fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 -- 172.21.15.103:0/2543255401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fe780 msgr2=0x7f64d40fc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fe780 0x7f64d40fc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 -- 172.21.15.103:0/2543255401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4005040 con 0x7f64d40fdde0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d2ffd700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c830 tx=0x7f64bc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64bc00a760 con 0x7f64d40fdde0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64bc00ad90 con 0x7f64d40fdde0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d40fc970 con 0x7f64d40fdde0 2024-03-22T02:41:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64bc0113a0 con 0x7f64d40fdde0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.903+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d411a050 con 0x7f64d40fdde0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.904+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d4066bd0 con 0x7f64d40fdde0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.905+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f64bc012400 con 0x7f64d40fdde0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.905+0000 7f64cb7fe700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 0x7f64c0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.906+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f64bc08ddd0 con 0x7f64d40fdde0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.906+0000 7f64d27fc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 0x7f64c0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.906+0000 7f64d27fc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 0x7f64c0070850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f64c400b3b0 tx=0x7f64c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:11.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:11.910+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64bc057050 con 0x7f64d40fdde0 2024-03-22T02:41:12.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.128+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f64d40f9520 con 0x7f64c006e3a0 2024-03-22T02:41:12.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.129+0000 7f64cb7fe700 1 -- 172.21.15.103:0/2543255401 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f64d40f9520 con 0x7f64c006e3a0 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.131+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 msgr2=0x7f64c0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.131+0000 7f64d97cc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 0x7f64c0070850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f64c400b3b0 tx=0x7f64c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.131+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 msgr2=0x7f64d40fbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.131+0000 7f64d97cc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c830 tx=0x7f64bc004470 comp rx=0 tx=0).stop 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 shutdown_connections 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f64c006e3a0 0x7f64c0070850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64d40fdde0 0x7f64d40fbc10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 --2- 172.21.15.103:0/2543255401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64d40fe780 0x7f64d40fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 >> 172.21.15.103:0/2543255401 conn(0x7f64d40f7890 msgr2=0x7f64d40f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 shutdown_connections 2024-03-22T02:41:12.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.132+0000 7f64d97cc700 1 -- 172.21.15.103:0/2543255401 wait complete. 2024-03-22T02:41:12.143 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:41:12.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 -- 172.21.15.103:0/1199213824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d500fbca0 msgr2=0x7f2d500fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/1199213824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d500fbca0 0x7f2d500fc110 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2d44004940 tx=0x7f2d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.235 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:12 smithi103 ceph-mon[115211]: pgmap v290: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 -- 172.21.15.103:0/1199213824 shutdown_connections 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/1199213824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d500fbca0 0x7f2d500fc110 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/1199213824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d50101cf0 0x7f2d501020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 -- 172.21.15.103:0/1199213824 >> 172.21.15.103:0/1199213824 conn(0x7f2d50072bc0 msgr2=0x7f2d50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 -- 172.21.15.103:0/1199213824 shutdown_connections 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.234+0000 7f2d54ea1700 1 -- 172.21.15.103:0/1199213824 wait complete. 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 Processor -- start 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 -- start start 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d500fbca0 0x7f2d50113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50114700 con 0x7f2d50101cf0 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d54ea1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5010daf0 con 0x7f2d500fbca0 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46754/0 (socket says 172.21.15.103:46754) 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 -- 172.21.15.103:0/2809348103 learned_addr learned my addr 172.21.15.103:0/2809348103 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4e59c700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d500fbca0 0x7f2d50113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 -- 172.21.15.103:0/2809348103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d500fbca0 msgr2=0x7f2d50113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d500fbca0 0x7f2d50113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.235+0000 7f2d4dd9b700 1 -- 172.21.15.103:0/2809348103 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44005040 con 0x7f2d50101cf0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d4dd9b700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b3b0 tx=0x7f2d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4400e040 con 0x7f2d50101cf0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5010dd70 con 0x7f2d50101cf0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d44013420 con 0x7f2d50101cf0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4401c630 con 0x7f2d50101cf0 2024-03-22T02:41:12.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.236+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5010e2b0 con 0x7f2d50101cf0 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.237+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d500fd390 con 0x7f2d50101cf0 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.237+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2d44013590 con 0x7f2d50101cf0 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.238+0000 7f2d3effd700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 0x7f2d38070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.238+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d4408f070 con 0x7f2d50101cf0 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.238+0000 7f2d4e59c700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 0x7f2d38070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.238+0000 7f2d4e59c700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 0x7f2d38070850 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2d48007a40 tx=0x7f2d48009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.243+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d4405bde0 con 0x7f2d50101cf0 2024-03-22T02:41:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:12 smithi122 ceph-mon[83368]: pgmap v290: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:12.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.459+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d50108d50 con 0x7f2d3806e3a0 2024-03-22T02:41:12.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.460+0000 7f2d3effd700 1 -- 172.21.15.103:0/2809348103 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2d50108d50 con 0x7f2d3806e3a0 2024-03-22T02:41:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 msgr2=0x7f2d38070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 0x7f2d38070850 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2d48007a40 tx=0x7f2d48009da0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 msgr2=0x7f2d501140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b3b0 tx=0x7f2d44009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 shutdown_connections 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d3806e3a0 0x7f2d38070850 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d500fbca0 0x7f2d50113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 --2- 172.21.15.103:0/2809348103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d50101cf0 0x7f2d501140b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.462+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 >> 172.21.15.103:0/2809348103 conn(0x7f2d50072bc0 msgr2=0x7f2d50065600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.463+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 shutdown_connections 2024-03-22T02:41:12.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.463+0000 7f2d54ea1700 1 -- 172.21.15.103:0/2809348103 wait complete. 2024-03-22T02:41:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 -- 172.21.15.103:0/1085515272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fe830 msgr2=0x7f913c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 --2- 172.21.15.103:0/1085515272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fe830 0x7f913c106f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9130004940 tx=0x7f9130009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 -- 172.21.15.103:0/1085515272 shutdown_connections 2024-03-22T02:41:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 --2- 172.21.15.103:0/1085515272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fe830 0x7f913c106f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 --2- 172.21.15.103:0/1085515272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fde90 0x7f913c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 -- 172.21.15.103:0/1085515272 >> 172.21.15.103:0/1085515272 conn(0x7f913c072bc0 msgr2=0x7f913c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.563+0000 7f91426ed700 1 -- 172.21.15.103:0/1085515272 shutdown_connections 2024-03-22T02:41:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 -- 172.21.15.103:0/1085515272 wait complete. 2024-03-22T02:41:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 Processor -- start 2024-03-22T02:41:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 -- start start 2024-03-22T02:41:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fe830 0x7f913c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0fc820 con 0x7f913c0fde90 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.564+0000 7f91426ed700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0fc960 con 0x7f913c0fe830 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46764/0 (socket says 172.21.15.103:46764) 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 -- 172.21.15.103:0/2820842604 learned_addr learned my addr 172.21.15.103:0/2820842604 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913b7fe700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fe830 0x7f913c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 -- 172.21.15.103:0/2820842604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fe830 msgr2=0x7f913c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fe830 0x7f913c0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 -- 172.21.15.103:0/2820842604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9130005040 con 0x7f913c0fde90 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f913bfff700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c830 tx=0x7f912c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912c0106f0 con 0x7f913c0fde90 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f912c010d20 con 0x7f913c0fde90 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c119ea0 con 0x7f913c0fde90 2024-03-22T02:41:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912c0113a0 con 0x7f913c0fde90 2024-03-22T02:41:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.565+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c11a2a0 con 0x7f913c0fde90 2024-03-22T02:41:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.566+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c065e30 con 0x7f913c0fde90 2024-03-22T02:41:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.567+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f912c012400 con 0x7f913c0fde90 2024-03-22T02:41:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.567+0000 7f9138ff9700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 0x7f9124070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.567+0000 7f913b7fe700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 0x7f9124070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.567+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f912c08dd70 con 0x7f913c0fde90 2024-03-22T02:41:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.568+0000 7f913b7fe700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 0x7f9124070850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f913c0fd690 tx=0x7f9130009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.572+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f912c056ff0 con 0x7f913c0fde90 2024-03-22T02:41:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.777+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f913c0fd440 con 0x7f912406e3a0 2024-03-22T02:41:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.787+0000 7f9138ff9700 1 -- 172.21.15.103:0/2820842604 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f913c0fd440 con 0x7f912406e3a0 2024-03-22T02:41:12.788 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:41:12.788 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (9m) 3m ago 20m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (7m) 3m ago 21m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (7m) 2m ago 20m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (9m) 3m ago 20m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (3m) 3m ago 15m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (3m) 3m ago 15m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (2m) 2m ago 15m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (3m) 2m ago 15m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (11m) 3m ago 22m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (10m) 2m ago 19m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (8m) 3m ago 22m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (8m) 2m ago 19m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:41:12.789 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (10m) 3m ago 21m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (10m) 2m ago 20m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (7m) 3m ago 19m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (6m) 3m ago 18m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (5m) 3m ago 18m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (5m) 2m ago 17m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (4m) 2m ago 17m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (4m) 2m ago 16m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:41:12.790 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (10m) 3m ago 20m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 msgr2=0x7f9124070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 0x7f9124070850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f913c0fd690 tx=0x7f9130009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 msgr2=0x7f913c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f912c00c830 tx=0x7f912c004470 comp rx=0 tx=0).stop 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 shutdown_connections 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f912406e3a0 0x7f9124070850 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f913c0fde90 0x7f913c0fbd50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 --2- 172.21.15.103:0/2820842604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f913c0fe830 0x7f913c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 >> 172.21.15.103:0/2820842604 conn(0x7f913c072bc0 msgr2=0x7f913c1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 shutdown_connections 2024-03-22T02:41:12.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.790+0000 7f91426ed700 1 -- 172.21.15.103:0/2820842604 wait complete. 2024-03-22T02:41:12.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- 172.21.15.103:0/1375224499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fe7d0 msgr2=0x7f1d7c106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 --2- 172.21.15.103:0/1375224499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c106e10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c004940 tx=0x7f1d6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- 172.21.15.103:0/1375224499 shutdown_connections 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 --2- 172.21.15.103:0/1375224499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c106e10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 --2- 172.21.15.103:0/1375224499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c0fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- 172.21.15.103:0/1375224499 >> 172.21.15.103:0/1375224499 conn(0x7f1d7c0f78c0 msgr2=0x7f1d7c0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- 172.21.15.103:0/1375224499 shutdown_connections 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- 172.21.15.103:0/1375224499 wait complete. 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 Processor -- start 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.892+0000 7f1d83276700 1 -- start start 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d83276700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d83276700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d83276700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c114540 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d83276700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10dab0 con 0x7f1d7c0fe7d0 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46794/0 (socket says 172.21.15.103:46794) 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 -- 172.21.15.103:0/694320810 learned_addr learned my addr 172.21.15.103:0/694320810 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d80811700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 -- 172.21.15.103:0/694320810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fe7d0 msgr2=0x7f1d7c113e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c113e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 -- 172.21.15.103:0/694320810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c005040 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d81012700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800b790 tx=0x7f1d78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7800f770 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d7800fda0 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c10dd90 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.893+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7800d310 con 0x7f1d7c0fde30 2024-03-22T02:41:12.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.894+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c10e2d0 con 0x7f1d7c0fde30 2024-03-22T02:41:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.895+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c066b90 con 0x7f1d7c0fde30 2024-03-22T02:41:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.895+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1d78010420 con 0x7f1d7c0fde30 2024-03-22T02:41:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.895+0000 7f1d71ffb700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 0x7f1d68070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:12.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.895+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1d7808dd10 con 0x7f1d7c0fde30 2024-03-22T02:41:12.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.896+0000 7f1d80811700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 0x7f1d68070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:12.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.896+0000 7f1d80811700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 0x7f1d68070850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c10ec70 tx=0x7f1d6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:12.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:12.900+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d7805ab80 con 0x7f1d7c0fde30 2024-03-22T02:41:13.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.178+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d7c109f40 con 0x7f1d7c0fde30 2024-03-22T02:41:13.180 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:13 smithi103 ceph-mon[115211]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.180 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:13 smithi103 ceph-mon[115211]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.180 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:13 smithi103 ceph-mon[115211]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.180 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:13 smithi103 ceph-mon[115211]: pgmap v291: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:13.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.179+0000 7f1d71ffb700 1 -- 172.21.15.103:0/694320810 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1d7805a710 con 0x7f1d7c0fde30 2024-03-22T02:41:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:41:13.180 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:41:13.180 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:41:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 msgr2=0x7f1d68070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 0x7f1d68070850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c10ec70 tx=0x7f1d6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 msgr2=0x7f1d7c113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800b790 tx=0x7f1d78009250 comp rx=0 tx=0).stop 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 shutdown_connections 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1d6806e3a0 0x7f1d68070850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d7c0fde30 0x7f1d7c113920 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 --2- 172.21.15.103:0/694320810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0fe7d0 0x7f1d7c113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 >> 172.21.15.103:0/694320810 conn(0x7f1d7c0f78c0 msgr2=0x7f1d7c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 shutdown_connections 2024-03-22T02:41:13.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.182+0000 7f1d83276700 1 -- 172.21.15.103:0/694320810 wait complete. 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 -- 172.21.15.103:0/321168208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fbbf0 msgr2=0x7f35580fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 --2- 172.21.15.103:0/321168208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fbbf0 0x7f35580fbfc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3548004940 tx=0x7f3548009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 -- 172.21.15.103:0/321168208 shutdown_connections 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 --2- 172.21.15.103:0/321168208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fc500 0x7f3558109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 --2- 172.21.15.103:0/321168208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fbbf0 0x7f35580fbfc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 -- 172.21.15.103:0/321168208 >> 172.21.15.103:0/321168208 conn(0x7f35580f7860 msgr2=0x7f35580f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.283+0000 7f355d6ac700 1 -- 172.21.15.103:0/321168208 shutdown_connections 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 -- 172.21.15.103:0/321168208 wait complete. 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 Processor -- start 2024-03-22T02:41:13.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 -- start start 2024-03-22T02:41:13.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 0x7f3558115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3558116630 con 0x7f35580fc500 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f355d6ac700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355810fc10 con 0x7f35580fbbf0 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.284+0000 7f35567fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46804/0 (socket says 172.21.15.103:46804) 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 -- 172.21.15.103:0/3187281390 learned_addr learned my addr 172.21.15.103:0/3187281390 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f3556ffd700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 0x7f3558115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 -- 172.21.15.103:0/3187281390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 msgr2=0x7f3558115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 0x7f3558115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 -- 172.21.15.103:0/3187281390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548005040 con 0x7f35580fc500 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f3556ffd700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 0x7f3558115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f35567fc700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f354c002a50 tx=0x7f354c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f354c007cc0 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355810fef0 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f354c00f040 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f354c009640 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.285+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35581104b0 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.286+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3558066b50 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.287+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f354c0097c0 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.287+0000 7f353f7fe700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 0x7f3540070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.287+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f354c08dba0 con 0x7f35580fc500 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.287+0000 7f3556ffd700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 0x7f3540070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.288+0000 7f3556ffd700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 0x7f3540070850 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3548009fd0 tx=0x7f3548009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.292+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f354c05a8d0 con 0x7f35580fc500 2024-03-22T02:41:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:13 smithi122 ceph-mon[83368]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:13 smithi122 ceph-mon[83368]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:13 smithi122 ceph-mon[83368]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:13.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:13 smithi122 ceph-mon[83368]: pgmap v291: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.532+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f355804c710 con 0x7f35580fc500 2024-03-22T02:41:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.533+0000 7f353f7fe700 1 -- 172.21.15.103:0/3187281390 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f354c05a460 con 0x7f35580fc500 2024-03-22T02:41:13.533 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:41:13.534 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:41:13.535 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:13.536 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 msgr2=0x7f3540070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 0x7f3540070850 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3548009fd0 tx=0x7f3548009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 msgr2=0x7f3558115fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f354c002a50 tx=0x7f354c0046b0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 shutdown_connections 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f354006e3a0 0x7f3540070850 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35580fbbf0 0x7f3558115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 --2- 172.21.15.103:0/3187281390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35580fc500 0x7f3558115fe0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 >> 172.21.15.103:0/3187281390 conn(0x7f35580f7860 msgr2=0x7f35580f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 shutdown_connections 2024-03-22T02:41:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.535+0000 7f355d6ac700 1 -- 172.21.15.103:0/3187281390 wait complete. 2024-03-22T02:41:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 -- 172.21.15.103:0/1304229311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 msgr2=0x7f5bac0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 --2- 172.21.15.103:0/1304229311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac0fcdf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c004940 tx=0x7f5b9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 -- 172.21.15.103:0/1304229311 shutdown_connections 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 --2- 172.21.15.103:0/1304229311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac0fcdf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 --2- 172.21.15.103:0/1304229311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 -- 172.21.15.103:0/1304229311 >> 172.21.15.103:0/1304229311 conn(0x7f5bac0f8950 msgr2=0x7f5bac0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 -- 172.21.15.103:0/1304229311 shutdown_connections 2024-03-22T02:41:13.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.639+0000 7f5bb1276700 1 -- 172.21.15.103:0/1304229311 wait complete. 2024-03-22T02:41:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 Processor -- start 2024-03-22T02:41:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 -- start start 2024-03-22T02:41:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac114530 con 0x7f5bac103780 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5bb1276700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac10db30 con 0x7f5bac102de0 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46826/0 (socket says 172.21.15.103:46826) 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 -- 172.21.15.103:0/3146329710 learned_addr learned my addr 172.21.15.103:0/3146329710 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baaffd700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 -- 172.21.15.103:0/3146329710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 msgr2=0x7f5bac1139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baa7fc700 1 -- 172.21.15.103:0/3146329710 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c005040 con 0x7f5bac103780 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.640+0000 7f5baaffd700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5baa7fc700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009180 tx=0x7f5b9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c00e040 con 0x7f5bac103780 2024-03-22T02:41:13.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac10ddb0 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b9c00bbc0 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c01c3f0 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.641+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac10e370 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.642+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bac1052b0 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.642+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5b9c00bd30 con 0x7f5bac103780 2024-03-22T02:41:13.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.643+0000 7f5b937fe700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 0x7f5b94070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.643+0000 7f5baaffd700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 0x7f5b94070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.643+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b9c08f020 con 0x7f5bac103780 2024-03-22T02:41:13.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.643+0000 7f5baaffd700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 0x7f5b94070850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0007fd0 tx=0x7f5ba0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.648+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b9c05be90 con 0x7f5bac103780 2024-03-22T02:41:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.862+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bac10eff0 con 0x7f5b9406e3a0 2024-03-22T02:41:13.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.863+0000 7f5b937fe700 1 -- 172.21.15.103:0/3146329710 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5bac10eff0 con 0x7f5b9406e3a0 2024-03-22T02:41:13.864 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:41:13.864 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:41:13.864 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:41:13.864 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:41:13.865 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 msgr2=0x7f5b94070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 0x7f5b94070850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0007fd0 tx=0x7f5ba0009040 comp rx=0 tx=0).stop 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 msgr2=0x7f5bac113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009180 tx=0x7f5b9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 shutdown_connections 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5b9406e3a0 0x7f5b94070850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5bac102de0 0x7f5bac1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 --2- 172.21.15.103:0/3146329710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac103780 0x7f5bac113ee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 >> 172.21.15.103:0/3146329710 conn(0x7f5bac0f8950 msgr2=0x7f5bac1021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 shutdown_connections 2024-03-22T02:41:13.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.866+0000 7f5bb1276700 1 -- 172.21.15.103:0/3146329710 wait complete. 2024-03-22T02:41:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 -- 172.21.15.103:0/3799676817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3481015c0 msgr2=0x7fd348101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 --2- 172.21.15.103:0/3799676817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3481015c0 0x7fd348101990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd33c004940 tx=0x7fd33c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:13.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 -- 172.21.15.103:0/3799676817 shutdown_connections 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 --2- 172.21.15.103:0/3799676817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3480fb650 0x7fd3480fbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 --2- 172.21.15.103:0/3799676817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3481015c0 0x7fd348101990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 -- 172.21.15.103:0/3799676817 >> 172.21.15.103:0/3799676817 conn(0x7fd348071530 msgr2=0x7fd348071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 -- 172.21.15.103:0/3799676817 shutdown_connections 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.967+0000 7fd34e817700 1 -- 172.21.15.103:0/3799676817 wait complete. 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 Processor -- start 2024-03-22T02:41:13.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 -- start start 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 0x7fd348112de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348113430 con 0x7fd3480fb650 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd34e817700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34810ca30 con 0x7fd3481015c0 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46554/0 (socket says 172.21.15.103:46554) 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 -- 172.21.15.103:0/2699485844 learned_addr learned my addr 172.21.15.103:0/2699485844 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd3477fe700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 0x7fd348112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 -- 172.21.15.103:0/2699485844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 msgr2=0x7fd348112de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 0x7fd348112de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:13.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.968+0000 7fd347fff700 1 -- 172.21.15.103:0/2699485844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c005040 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd3477fe700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 0x7fd348112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd347fff700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00b8b0 tx=0x7fd33c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd33c00e040 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34810ccb0 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd33c004290 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd33c01c3f0 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.969+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34810d270 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.970+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480fcd20 con 0x7fd3480fb650 2024-03-22T02:41:13.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.970+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd33c0137b0 con 0x7fd3480fb650 2024-03-22T02:41:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.971+0000 7fd344ff9700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 0x7fd330070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.971+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd33c08f130 con 0x7fd3480fb650 2024-03-22T02:41:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.971+0000 7fd3477fe700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 0x7fd330070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:13.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.971+0000 7fd3477fe700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 0x7fd330070850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd338009fd0 tx=0x7fd338009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:13.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:13.975+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd33c05bee0 con 0x7fd3480fb650 2024-03-22T02:41:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:14 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/694320810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:41:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:14 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3187281390' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:41:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:14 smithi103 ceph-mon[115211]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:14.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.256+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd348108f90 con 0x7fd3480fb650 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.257+0000 7fd344ff9700 1 -- 172.21.15.103:0/2699485844 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fd33c05ba70 con 0x7fd3480fb650 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:41:14.258 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 msgr2=0x7fd330070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 0x7fd330070850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd338009fd0 tx=0x7fd338009ee0 comp rx=0 tx=0).stop 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 msgr2=0x7fd3481128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00b8b0 tx=0x7fd33c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 shutdown_connections 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd33006e3a0 0x7fd330070850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3480fb650 0x7fd3481128a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:14.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 --2- 172.21.15.103:0/2699485844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3481015c0 0x7fd348112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 >> 172.21.15.103:0/2699485844 conn(0x7fd348071530 msgr2=0x7fd3480fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 shutdown_connections 2024-03-22T02:41:14.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:14.259+0000 7fd34e817700 1 -- 172.21.15.103:0/2699485844 wait complete. 2024-03-22T02:41:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:14 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/694320810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:41:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:14 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3187281390' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:41:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:14 smithi122 ceph-mon[83368]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:15 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2699485844' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:41:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:15 smithi122 ceph-mon[83368]: pgmap v292: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:15 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2699485844' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:41:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:15 smithi103 ceph-mon[115211]: pgmap v292: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:17 smithi122 ceph-mon[83368]: pgmap v293: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:17 smithi103 ceph-mon[115211]: pgmap v293: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:19 smithi122 ceph-mon[83368]: pgmap v294: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:19 smithi103 ceph-mon[115211]: pgmap v294: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:21 smithi122 ceph-mon[83368]: pgmap v295: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:21 smithi103 ceph-mon[115211]: pgmap v295: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:23 smithi122 ceph-mon[83368]: pgmap v296: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:23 smithi103 ceph-mon[115211]: pgmap v296: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:25 smithi122 ceph-mon[83368]: pgmap v297: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:25 smithi103 ceph-mon[115211]: pgmap v297: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:27 smithi122 ceph-mon[83368]: pgmap v298: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:27 smithi103 ceph-mon[115211]: pgmap v298: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:29 smithi122 ceph-mon[83368]: pgmap v299: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:29 smithi103 ceph-mon[115211]: pgmap v299: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:32 smithi122 ceph-mon[83368]: pgmap v300: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:32 smithi103 ceph-mon[115211]: pgmap v300: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:34 smithi122 ceph-mon[83368]: pgmap v301: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:34 smithi103 ceph-mon[115211]: pgmap v301: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:36 smithi122 ceph-mon[83368]: pgmap v302: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:36 smithi103 ceph-mon[115211]: pgmap v302: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:38 smithi122 ceph-mon[83368]: pgmap v303: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:38 smithi103 ceph-mon[115211]: pgmap v303: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:40 smithi122 ceph-mon[83368]: pgmap v304: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:40 smithi103 ceph-mon[115211]: pgmap v304: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:42 smithi122 ceph-mon[83368]: pgmap v305: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:42 smithi103 ceph-mon[115211]: pgmap v305: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:44 smithi122 ceph-mon[83368]: pgmap v306: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:44.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 -- 172.21.15.103:0/368357044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 msgr2=0x7ff2f80fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 --2- 172.21.15.103:0/368357044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f80fb990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8004940 tx=0x7ff2e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:44 smithi103 ceph-mon[115211]: pgmap v306: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 -- 172.21.15.103:0/368357044 shutdown_connections 2024-03-22T02:41:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 --2- 172.21.15.103:0/368357044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f80fb990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 --2- 172.21.15.103:0/368357044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 0x7ff2f81033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 -- 172.21.15.103:0/368357044 >> 172.21.15.103:0/368357044 conn(0x7ff2f8071530 msgr2=0x7ff2f8071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:44.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 -- 172.21.15.103:0/368357044 shutdown_connections 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.373+0000 7ff300675700 1 -- 172.21.15.103:0/368357044 wait complete. 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 Processor -- start 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 -- start start 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 0x7ff2f81160e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f8116730 con 0x7ff2f80fb540 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff300675700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f810fd10 con 0x7ff2f8102fd0 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36540/0 (socket says 172.21.15.103:36540) 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 -- 172.21.15.103:0/2917091990 learned_addr learned my addr 172.21.15.103:0/2917091990 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fdc10700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 0x7ff2f81160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 -- 172.21.15.103:0/2917091990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 msgr2=0x7ff2f81160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 0x7ff2f81160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.374+0000 7ff2fe411700 1 -- 172.21.15.103:0/2917091990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8005040 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff2fe411700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4002a50 tx=0x7ff2f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f4007cc0 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f810fff0 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f400f040 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f400db70 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.375+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f81105b0 con 0x7ff2f80fb540 2024-03-22T02:41:44.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.376+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f81018d0 con 0x7ff2f80fb540 2024-03-22T02:41:44.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.376+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff2f400dd10 con 0x7ff2f80fb540 2024-03-22T02:41:44.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.377+0000 7ff2eeffd700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 0x7ff2e4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.377+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff2f408d4f0 con 0x7ff2f80fb540 2024-03-22T02:41:44.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.377+0000 7ff2fdc10700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 0x7ff2e4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.377+0000 7ff2fdc10700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 0x7ff2e4070850 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8004f90 tx=0x7ff2e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:44.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.382+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff2f405a220 con 0x7ff2f80fb540 2024-03-22T02:41:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.596+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2f810a0a0 con 0x7ff2e406e3a0 2024-03-22T02:41:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.597+0000 7ff2eeffd700 1 -- 172.21.15.103:0/2917091990 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff2f810a0a0 con 0x7ff2e406e3a0 2024-03-22T02:41:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 msgr2=0x7ff2e4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 0x7ff2e4070850 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8004f90 tx=0x7ff2e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 msgr2=0x7ff2f8115ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4002a50 tx=0x7ff2f40046b0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 shutdown_connections 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff2e406e3a0 0x7ff2e4070850 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff2f80fb540 0x7ff2f8115ba0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 --2- 172.21.15.103:0/2917091990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2f8102fd0 0x7ff2f81160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 >> 172.21.15.103:0/2917091990 conn(0x7ff2f8071530 msgr2=0x7ff2f80fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 shutdown_connections 2024-03-22T02:41:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.599+0000 7ff300675700 1 -- 172.21.15.103:0/2917091990 wait complete. 2024-03-22T02:41:44.610 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:41:44.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 -- 172.21.15.103:0/145043501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 msgr2=0x7fc6100fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 --2- 172.21.15.103:0/145043501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc6100fbd80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc604004940 tx=0x7fc604009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 -- 172.21.15.103:0/145043501 shutdown_connections 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 --2- 172.21.15.103:0/145043501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc6100fbd80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 --2- 172.21.15.103:0/145043501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc610102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 -- 172.21.15.103:0/145043501 >> 172.21.15.103:0/145043501 conn(0x7fc610072bc0 msgr2=0x7fc610072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 -- 172.21.15.103:0/145043501 shutdown_connections 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.701+0000 7fc614c92700 1 -- 172.21.15.103:0/145043501 wait complete. 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 Processor -- start 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 -- start start 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc61010c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61010d2f0 con 0x7fc610102710 2024-03-22T02:41:44.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc614c92700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61010d460 con 0x7fc610101d70 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36560/0 (socket says 172.21.15.103:36560) 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 -- 172.21.15.103:0/1036228377 learned_addr learned my addr 172.21.15.103:0/1036228377 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60e59c700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc61010c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 -- 172.21.15.103:0/1036228377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 msgr2=0x7fc61010c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc61010c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 -- 172.21.15.103:0/1036228377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604005040 con 0x7fc610102710 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60e59c700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc61010c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.702+0000 7fc60dd9b700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc604000c00 tx=0x7fc604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.703+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc604021070 con 0x7fc610102710 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.703+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc604013cd0 con 0x7fc610102710 2024-03-22T02:41:44.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.703+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61010d6e0 con 0x7fc610102710 2024-03-22T02:41:44.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.703+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc60400e070 con 0x7fc610102710 2024-03-22T02:41:44.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.703+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61011a1b0 con 0x7fc610102710 2024-03-22T02:41:44.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.704+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc610104240 con 0x7fc610102710 2024-03-22T02:41:44.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.704+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc60401caf0 con 0x7fc610102710 2024-03-22T02:41:44.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.705+0000 7fc5feffd700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 0x7fc5f8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:44.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.705+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc60408f110 con 0x7fc610102710 2024-03-22T02:41:44.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.705+0000 7fc60e59c700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 0x7fc5f8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:44.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.705+0000 7fc60e59c700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 0x7fc5f8070850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc600004600 tx=0x7fc60000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:44.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.709+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc60405bec0 con 0x7fc610102710 2024-03-22T02:41:44.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.923+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc610065e20 con 0x7fc5f806e3a0 2024-03-22T02:41:44.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.924+0000 7fc5feffd700 1 -- 172.21.15.103:0/1036228377 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc610065e20 con 0x7fc5f806e3a0 2024-03-22T02:41:44.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 msgr2=0x7fc5f8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 0x7fc5f8070850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc600004600 tx=0x7fc60000b040 comp rx=0 tx=0).stop 2024-03-22T02:41:44.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 msgr2=0x7fc61010cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:44.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc604000c00 tx=0x7fc604009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:44.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 shutdown_connections 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc5f806e3a0 0x7fc5f8070850 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc610101d70 0x7fc61010c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 --2- 172.21.15.103:0/1036228377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc610102710 0x7fc61010cdb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 >> 172.21.15.103:0/1036228377 conn(0x7fc610072bc0 msgr2=0x7fc6100faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 shutdown_connections 2024-03-22T02:41:44.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:44.926+0000 7fc614c92700 1 -- 172.21.15.103:0/1036228377 wait complete. 2024-03-22T02:41:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 -- 172.21.15.103:0/2978310618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 msgr2=0x7efc38102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/2978310618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38102150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7efc30009ce0 tx=0x7efc30004d10 comp rx=0 tx=0).stop 2024-03-22T02:41:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 -- 172.21.15.103:0/2978310618 shutdown_connections 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/2978310618 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 0x7efc380fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/2978310618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38102150 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 -- 172.21.15.103:0/2978310618 >> 172.21.15.103:0/2978310618 conn(0x7efc380f78f0 msgr2=0x7efc380f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 -- 172.21.15.103:0/2978310618 shutdown_connections 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.027+0000 7efc3cf3f700 1 -- 172.21.15.103:0/2978310618 wait complete. 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 Processor -- start 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 -- start start 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 0x7efc38114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc381146d0 con 0x7efc38101d80 2024-03-22T02:41:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3cf3f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3810dcb0 con 0x7efc38102720 2024-03-22T02:41:45.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36586/0 (socket says 172.21.15.103:36586) 2024-03-22T02:41:45.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 -- 172.21.15.103:0/1143114185 learned_addr learned my addr 172.21.15.103:0/1143114185 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:45.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc35d9b700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 0x7efc38114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 -- 172.21.15.103:0/1143114185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 msgr2=0x7efc38114080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 0x7efc38114080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 -- 172.21.15.103:0/1143114185 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009940 con 0x7efc38101d80 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.028+0000 7efc3659c700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efc30005010 tx=0x7efc30004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.029+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc3000e040 con 0x7efc38101d80 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.029+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc30007d10 con 0x7efc38101d80 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.029+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3810df30 con 0x7efc38101d80 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.029+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc30012650 con 0x7efc38101d80 2024-03-22T02:41:45.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.029+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3810e4f0 con 0x7efc38101d80 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.030+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc380fd4d0 con 0x7efc38101d80 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.030+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efc3001ca10 con 0x7efc38101d80 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.031+0000 7efc26ffd700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 0x7efc20070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.031+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efc3008f9b0 con 0x7efc38101d80 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.031+0000 7efc35d9b700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 0x7efc20070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.031+0000 7efc35d9b700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 0x7efc20070850 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc3810f3e0 tx=0x7efc2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.035+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc3005c760 con 0x7efc38101d80 2024-03-22T02:41:45.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.249+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc38108f20 con 0x7efc2006e3a0 2024-03-22T02:41:45.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.259+0000 7efc26ffd700 1 -- 172.21.15.103:0/1143114185 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efc2c00f040 con 0x7efc2006e3a0 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (10m) 3m ago 21m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (8m) 3m ago 21m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (7m) 3m ago 20m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (10m) 3m ago 21m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (4m) 3m ago 15m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (4m) 3m ago 15m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (3m) 3m ago 15m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (3m) 3m ago 15m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (11m) 3m ago 22m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:41:45.260 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (11m) 3m ago 20m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (9m) 3m ago 23m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (8m) 3m ago 20m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (10m) 3m ago 21m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (10m) 3m ago 20m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (7m) 3m ago 19m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (6m) 3m ago 19m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:41:45.261 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (6m) 3m ago 18m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:41:45.262 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (5m) 3m ago 18m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:41:45.262 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (5m) 3m ago 17m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:41:45.262 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (4m) 3m ago 17m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:41:45.262 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (10m) 3m ago 21m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 msgr2=0x7efc20070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 0x7efc20070850 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc3810f3e0 tx=0x7efc2c00b040 comp rx=0 tx=0).stop 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 msgr2=0x7efc38113b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efc30005010 tx=0x7efc30004d10 comp rx=0 tx=0).stop 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 shutdown_connections 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc2006e3a0 0x7efc20070850 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101d80 0x7efc38113b40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 --2- 172.21.15.103:0/1143114185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc38102720 0x7efc38114080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 >> 172.21.15.103:0/1143114185 conn(0x7efc380f78f0 msgr2=0x7efc38106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 shutdown_connections 2024-03-22T02:41:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.261+0000 7efc3cf3f700 1 -- 172.21.15.103:0/1143114185 wait complete. 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 -- 172.21.15.103:0/4270739367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 msgr2=0x7f78601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 --2- 172.21.15.103:0/4270739367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78601031d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f785c004940 tx=0x7f785c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 -- 172.21.15.103:0/4270739367 shutdown_connections 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 --2- 172.21.15.103:0/4270739367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f78600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 --2- 172.21.15.103:0/4270739367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78601031d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.360+0000 7f7868174700 1 -- 172.21.15.103:0/4270739367 >> 172.21.15.103:0/4270739367 conn(0x7f78600f89b0 msgr2=0x7f78600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 -- 172.21.15.103:0/4270739367 shutdown_connections 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 -- 172.21.15.103:0/4270739367 wait complete. 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 Processor -- start 2024-03-22T02:41:45.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 -- start start 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f7860070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7860072210 con 0x7f7860102e00 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.361+0000 7f7868174700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7860070d40 con 0x7f78601037a0 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36606/0 (socket says 172.21.15.103:36606) 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 -- 172.21.15.103:0/2365214257 learned_addr learned my addr 172.21.15.103:0/2365214257 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f786570f700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f7860070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 -- 172.21.15.103:0/2365214257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 msgr2=0x7f7860070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f7860070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 -- 172.21.15.103:0/2365214257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c005040 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f786570f700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f7860070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7865f10700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f785c00b8b0 tx=0x7f785c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785c00e040 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7860070fc0 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786011a110 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f785c004290 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.362+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785c01c3f0 con 0x7f7860102e00 2024-03-22T02:41:45.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.363+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78440052e0 con 0x7f7860102e00 2024-03-22T02:41:45.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.364+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f785c0137b0 con 0x7f7860102e00 2024-03-22T02:41:45.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.364+0000 7f78567fc700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.364+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f785c08f020 con 0x7f7860102e00 2024-03-22T02:41:45.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.364+0000 7f786570f700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.365+0000 7f786570f700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7850009fd0 tx=0x7f785000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.369+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f785c058360 con 0x7f7860102e00 2024-03-22T02:41:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.650+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7844005da0 con 0x7f7860102e00 2024-03-22T02:41:45.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.651+0000 7f78567fc700 1 -- 172.21.15.103:0/2365214257 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f785c05ba20 con 0x7f7860102e00 2024-03-22T02:41:45.651 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:41:45.651 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:41:45.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:41:45.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:41:45.653 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 msgr2=0x7f784c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7850009fd0 tx=0x7f785000b040 comp rx=0 tx=0).stop 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 msgr2=0x7f78600702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f785c00b8b0 tx=0x7f785c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 shutdown_connections 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7860102e00 0x7f78600702c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 --2- 172.21.15.103:0/2365214257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78601037a0 0x7f7860070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 >> 172.21.15.103:0/2365214257 conn(0x7f78600f89b0 msgr2=0x7f78601073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:45.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 shutdown_connections 2024-03-22T02:41:45.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.653+0000 7f7868174700 1 -- 172.21.15.103:0/2365214257 wait complete. 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 -- 172.21.15.103:0/713573590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 msgr2=0x7fccd00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 --2- 172.21.15.103:0/713573590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd00fce30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fccc0004940 tx=0x7fccc0009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 -- 172.21.15.103:0/713573590 shutdown_connections 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 --2- 172.21.15.103:0/713573590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd00fce30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 --2- 172.21.15.103:0/713573590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 0x7fccd01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 -- 172.21.15.103:0/713573590 >> 172.21.15.103:0/713573590 conn(0x7fccd00f89d0 msgr2=0x7fccd00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 -- 172.21.15.103:0/713573590 shutdown_connections 2024-03-22T02:41:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.754+0000 7fccd62f8700 1 -- 172.21.15.103:0/713573590 wait complete. 2024-03-22T02:41:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 Processor -- start 2024-03-22T02:41:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 -- start start 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 0x7fccd0113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0114740 con 0x7fccd01037c0 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fccd62f8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd010db30 con 0x7fccd0102e20 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36616/0 (socket says 172.21.15.103:36616) 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 -- 172.21.15.103:0/1666109247 learned_addr learned my addr 172.21.15.103:0/1666109247 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccffff700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 0x7fccd0113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 -- 172.21.15.103:0/1666109247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 msgr2=0x7fccd0113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 0x7fccd0113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.755+0000 7fcccf7fe700 1 -- 172.21.15.103:0/1666109247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc0005040 con 0x7fccd01037c0 2024-03-22T02:41:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fcccf7fe700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000c00 tx=0x7fccc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccc0021070 con 0x7fccd01037c0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccc0013c00 con 0x7fccd01037c0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd010ddb0 con 0x7fccd01037c0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccc000e070 con 0x7fccd01037c0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.756+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd010e290 con 0x7fccd01037c0 2024-03-22T02:41:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.757+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd01052f0 con 0x7fccd01037c0 2024-03-22T02:41:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.757+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fccc001ca60 con 0x7fccd01037c0 2024-03-22T02:41:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.758+0000 7fccccff9700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 0x7fccb8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.758+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fccc008f160 con 0x7fccd01037c0 2024-03-22T02:41:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.758+0000 7fcccffff700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 0x7fccb8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.758+0000 7fcccffff700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 0x7fccb8070850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009b80 tx=0x7fccc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:45.762+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fccc005bf10 con 0x7fccd01037c0 2024-03-22T02:41:46.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.001+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fccd010f200 con 0x7fccd01037c0 2024-03-22T02:41:46.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.001+0000 7fccccff9700 1 -- 172.21.15.103:0/1666109247 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fccc0004490 con 0x7fccd01037c0 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:41:46.003 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:41:46.004 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:46.005 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:41:46.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 msgr2=0x7fccb8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 0x7fccb8070850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009b80 tx=0x7fccc400b040 comp rx=0 tx=0).stop 2024-03-22T02:41:46.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 msgr2=0x7fccd01140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000c00 tx=0x7fccc0009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 shutdown_connections 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fccb806e3a0 0x7fccb8070850 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccd0102e20 0x7fccd0113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 --2- 172.21.15.103:0/1666109247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccd01037c0 0x7fccd01140f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 >> 172.21.15.103:0/1666109247 conn(0x7fccd00f89d0 msgr2=0x7fccd01021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 shutdown_connections 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.004+0000 7fccd62f8700 1 -- 172.21.15.103:0/1666109247 wait complete. 2024-03-22T02:41:46.007 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.105+0000 7f902c2dd700 1 -- 172.21.15.103:0/525538584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90241037a0 msgr2=0x7f90240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.105+0000 7f902c2dd700 1 --2- 172.21.15.103:0/525538584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90241037a0 0x7f90240fce10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9014004940 tx=0x7f9014009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- 172.21.15.103:0/525538584 shutdown_connections 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 --2- 172.21.15.103:0/525538584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90241037a0 0x7f90240fce10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 --2- 172.21.15.103:0/525538584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9024102e00 0x7f90241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- 172.21.15.103:0/525538584 >> 172.21.15.103:0/525538584 conn(0x7f90240f89b0 msgr2=0x7f90240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- 172.21.15.103:0/525538584 shutdown_connections 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- 172.21.15.103:0/525538584 wait complete. 2024-03-22T02:41:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 Processor -- start 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- start start 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 0x7f9024113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024114510 con 0x7f9024102e00 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902c2dd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902410db10 con 0x7f90241037a0 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902a079700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902a079700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36628/0 (socket says 172.21.15.103:36628) 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.106+0000 7f902a079700 1 -- 172.21.15.103:0/3388795541 learned_addr learned my addr 172.21.15.103:0/3388795541 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f9029878700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 0x7f9024113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902a079700 1 -- 172.21.15.103:0/3388795541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 msgr2=0x7f9024113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902a079700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 0x7f9024113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902a079700 1 -- 172.21.15.103:0/3388795541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014005040 con 0x7f9024102e00 2024-03-22T02:41:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f9029878700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 0x7f9024113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902a079700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f902000c8c0 tx=0x7f90200044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f902000a980 con 0x7f9024102e00 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902410ddf0 con 0x7f9024102e00 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f902000ce60 con 0x7f9024102e00 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90200115e0 con 0x7f9024102e00 2024-03-22T02:41:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.107+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902410e3b0 con 0x7f9024102e00 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.108+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90241052d0 con 0x7f9024102e00 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.109+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9020012400 con 0x7f9024102e00 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.109+0000 7f901affd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 0x7f9010070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.109+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f902008ded0 con 0x7f9024102e00 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.109+0000 7f9029878700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 0x7f9010070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.110+0000 7f9029878700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 0x7f9010070850 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f901400b3b0 tx=0x7f9014009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:46.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.114+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f902005ac80 con 0x7f9024102e00 2024-03-22T02:41:46.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.331+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f902410efd0 con 0x7f901006e3a0 2024-03-22T02:41:46.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:46 smithi103 ceph-mon[115211]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:46.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:46 smithi103 ceph-mon[115211]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:46.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:46 smithi103 ceph-mon[115211]: pgmap v307: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:46.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:46 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2365214257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:41:46.332 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:46 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1666109247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.332+0000 7f901affd700 1 -- 172.21.15.103:0/3388795541 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f902410efd0 con 0x7f901006e3a0 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:41:46.333 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:41:46.334 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:41:46.334 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:41:46.334 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 msgr2=0x7f9010070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 0x7f9010070850 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f901400b3b0 tx=0x7f9014009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 msgr2=0x7f9024113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f902000c8c0 tx=0x7f90200044c0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 shutdown_connections 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f901006e3a0 0x7f9010070850 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9024102e00 0x7f9024113980 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 --2- 172.21.15.103:0/3388795541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90241037a0 0x7f9024113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 >> 172.21.15.103:0/3388795541 conn(0x7f90240f89b0 msgr2=0x7f90241021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 shutdown_connections 2024-03-22T02:41:46.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.335+0000 7f902c2dd700 1 -- 172.21.15.103:0/3388795541 wait complete. 2024-03-22T02:41:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:46 smithi122 ceph-mon[83368]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:46 smithi122 ceph-mon[83368]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:46 smithi122 ceph-mon[83368]: pgmap v307: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:46 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2365214257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:41:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:46 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1666109247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:41:46.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 -- 172.21.15.103:0/2320635328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 msgr2=0x7f36440fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2320635328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36440fbd80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f362c004940 tx=0x7f362c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 -- 172.21.15.103:0/2320635328 shutdown_connections 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2320635328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36440fbd80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2320635328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 0x7f3644102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 -- 172.21.15.103:0/2320635328 >> 172.21.15.103:0/2320635328 conn(0x7f3644072bc0 msgr2=0x7f3644072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 -- 172.21.15.103:0/2320635328 shutdown_connections 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 -- 172.21.15.103:0/2320635328 wait complete. 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.436+0000 7f3649cc0700 1 Processor -- start 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3649cc0700 1 -- start start 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3649cc0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 0x7f3644113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3649cc0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3649cc0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644114710 con 0x7f3644102710 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3649cc0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364410db20 con 0x7f3644101d70 2024-03-22T02:41:46.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36650/0 (socket says 172.21.15.103:36650) 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 -- 172.21.15.103:0/2527555359 learned_addr learned my addr 172.21.15.103:0/2527555359 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f36437fe700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 0x7f3644113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 -- 172.21.15.103:0/2527555359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 msgr2=0x7f3644113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 0x7f3644113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 -- 172.21.15.103:0/2527555359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c005040 con 0x7f3644102710 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f3642ffd700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f362c009100 tx=0x7f362c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.437+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f362c00e040 con 0x7f3644102710 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.438+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f362c013420 con 0x7f3644102710 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.438+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f362c01c630 con 0x7f3644102710 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.438+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364410dda0 con 0x7f3644102710 2024-03-22T02:41:46.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.438+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364410e360 con 0x7f3644102710 2024-03-22T02:41:46.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.439+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36440fd4c0 con 0x7f3644102710 2024-03-22T02:41:46.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.439+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f362c013a00 con 0x7f3644102710 2024-03-22T02:41:46.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.439+0000 7f363bfff700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 0x7f3630070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:41:46.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.440+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f362c08ed90 con 0x7f3644102710 2024-03-22T02:41:46.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.440+0000 7f36437fe700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 0x7f3630070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:41:46.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.440+0000 7f36437fe700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 0x7f3630070860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f363c001450 tx=0x7f363c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:41:46.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.444+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f362c0580d0 con 0x7f3644102710 2024-03-22T02:41:46.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.724+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f364410f200 con 0x7f3644102710 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.725+0000 7f363bfff700 1 -- 172.21.15.103:0/2527555359 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f362c08e090 con 0x7f3644102710 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:41:46.726 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 msgr2=0x7f3630070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 0x7f3630070860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f363c001450 tx=0x7f363c00b040 comp rx=0 tx=0).stop 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 msgr2=0x7f36441140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f362c009100 tx=0x7f362c009e30 comp rx=0 tx=0).stop 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 shutdown_connections 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f363006e3b0 0x7f3630070860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3644101d70 0x7f3644113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 --2- 172.21.15.103:0/2527555359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3644102710 0x7f36441140c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 >> 172.21.15.103:0/2527555359 conn(0x7f3644072bc0 msgr2=0x7f36440659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:41:46.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 shutdown_connections 2024-03-22T02:41:46.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:41:46.727+0000 7f3649cc0700 1 -- 172.21.15.103:0/2527555359 wait complete. 2024-03-22T02:41:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:47 smithi122 ceph-mon[83368]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:47 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2527555359' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:41:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:47 smithi103 ceph-mon[115211]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:47 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2527555359' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:41:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:41:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:48 smithi122 ceph-mon[83368]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:48 smithi103 ceph-mon[115211]: from='client.34640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:41:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:49 smithi122 ceph-mon[83368]: pgmap v308: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:49.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:49 smithi103 ceph-mon[115211]: pgmap v308: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:51 smithi122 ceph-mon[83368]: pgmap v309: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:51 smithi103 ceph-mon[115211]: pgmap v309: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:53.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:53 smithi122 ceph-mon[83368]: pgmap v310: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:53.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:53 smithi103 ceph-mon[115211]: pgmap v310: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:55.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:55 smithi122 ceph-mon[83368]: pgmap v311: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:55 smithi103 ceph-mon[115211]: pgmap v311: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:57.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:57 smithi122 ceph-mon[83368]: pgmap v312: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:57 smithi103 ceph-mon[115211]: pgmap v312: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:41:58 smithi122 ceph-mon[83368]: pgmap v313: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:41:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:41:58 smithi103 ceph-mon[115211]: pgmap v313: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:00.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:00 smithi103 ceph-mon[115211]: pgmap v314: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:00.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:00 smithi122 ceph-mon[83368]: pgmap v314: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:02.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:02 smithi122 ceph-mon[83368]: pgmap v315: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:02.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:02 smithi103 ceph-mon[115211]: pgmap v315: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:04.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:04 smithi122 ceph-mon[83368]: pgmap v316: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:04.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:04 smithi103 ceph-mon[115211]: pgmap v316: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:06.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:06 smithi122 ceph-mon[83368]: pgmap v317: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:06 smithi103 ceph-mon[115211]: pgmap v317: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:08.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:08 smithi122 ceph-mon[83368]: pgmap v318: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:08 smithi103 ceph-mon[115211]: pgmap v318: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:10.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:10 smithi103 ceph-mon[115211]: pgmap v319: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:10.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:10 smithi122 ceph-mon[83368]: pgmap v319: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:12.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:12 smithi122 ceph-mon[83368]: pgmap v320: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:12.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:12 smithi103 ceph-mon[115211]: pgmap v320: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:14.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:14 smithi122 ceph-mon[83368]: pgmap v321: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:14.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:14 smithi103 ceph-mon[115211]: pgmap v321: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:16.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:16 smithi122 ceph-mon[83368]: pgmap v322: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:16.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:16 smithi103 ceph-mon[115211]: pgmap v322: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:16.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 -- 172.21.15.103:0/2750545089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 msgr2=0x7f34880fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 --2- 172.21.15.103:0/2750545089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f34880fbd30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f347c004940 tx=0x7f347c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 -- 172.21.15.103:0/2750545089 shutdown_connections 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 --2- 172.21.15.103:0/2750545089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f34880fbd30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 --2- 172.21.15.103:0/2750545089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f34881020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.841+0000 7f348e2e4700 1 -- 172.21.15.103:0/2750545089 >> 172.21.15.103:0/2750545089 conn(0x7f34880f7870 msgr2=0x7f34880f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 -- 172.21.15.103:0/2750545089 shutdown_connections 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 -- 172.21.15.103:0/2750545089 wait complete. 2024-03-22T02:42:16.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 Processor -- start 2024-03-22T02:42:16.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 -- start start 2024-03-22T02:42:16.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f348810c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:16.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.842+0000 7f348e2e4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:16.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f348e2e4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3488067120 con 0x7f34881026c0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f348e2e4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348810d360 con 0x7f3488101d20 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51310/0 (socket says 172.21.15.103:51310) 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 -- 172.21.15.103:0/3119500377 learned_addr learned my addr 172.21.15.103:0/3119500377 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f3487fff700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f348810c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 -- 172.21.15.103:0/3119500377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 msgr2=0x7f348810c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f348810c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 -- 172.21.15.103:0/3119500377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c005040 con 0x7f34881026c0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f3487fff700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f348810c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f34877fe700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f347c009180 tx=0x7f347c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347c00e040 con 0x7f34881026c0 2024-03-22T02:42:16.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348810d5e0 con 0x7f34881026c0 2024-03-22T02:42:16.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f347c013420 con 0x7f34881026c0 2024-03-22T02:42:16.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347c01c610 con 0x7f34881026c0 2024-03-22T02:42:16.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.843+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348811a110 con 0x7f34881026c0 2024-03-22T02:42:16.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.844+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34880fd470 con 0x7f34881026c0 2024-03-22T02:42:16.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.845+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f347c013a00 con 0x7f34881026c0 2024-03-22T02:42:16.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.845+0000 7f3484ff9700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 0x7f3470070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:16.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.845+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f347c08eef0 con 0x7f34881026c0 2024-03-22T02:42:16.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.845+0000 7f3487fff700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 0x7f3470070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:16.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.846+0000 7f3487fff700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 0x7f3470070850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3478009fd0 tx=0x7f347800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:16.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:16.850+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f347c05bd50 con 0x7f34881026c0 2024-03-22T02:42:17.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.067+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3488108e70 con 0x7f347006e3a0 2024-03-22T02:42:17.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.068+0000 7f3484ff9700 1 -- 172.21.15.103:0/3119500377 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3488108e70 con 0x7f347006e3a0 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 msgr2=0x7f3470070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 0x7f3470070850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3478009fd0 tx=0x7f347800b040 comp rx=0 tx=0).stop 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 msgr2=0x7f348810ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f347c009180 tx=0x7f347c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 shutdown_connections 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f347006e3a0 0x7f3470070850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3488101d20 0x7f348810c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 --2- 172.21.15.103:0/3119500377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34881026c0 0x7f348810ce20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.070+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 >> 172.21.15.103:0/3119500377 conn(0x7f34880f7870 msgr2=0x7f34880f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.071+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 shutdown_connections 2024-03-22T02:42:17.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.071+0000 7f348e2e4700 1 -- 172.21.15.103:0/3119500377 wait complete. 2024-03-22T02:42:17.082 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 -- 172.21.15.103:0/2437008247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c102e10 msgr2=0x7ffa4c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 --2- 172.21.15.103:0/2437008247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c102e10 0x7ffa4c1031e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c004940 tx=0x7ffa3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 -- 172.21.15.103:0/2437008247 shutdown_connections 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 --2- 172.21.15.103:0/2437008247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 --2- 172.21.15.103:0/2437008247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c102e10 0x7ffa4c1031e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 -- 172.21.15.103:0/2437008247 >> 172.21.15.103:0/2437008247 conn(0x7ffa4c0f89a0 msgr2=0x7ffa4c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 -- 172.21.15.103:0/2437008247 shutdown_connections 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.174+0000 7ffa53421700 1 -- 172.21.15.103:0/2437008247 wait complete. 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 Processor -- start 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 -- start start 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c102e10 0x7ffa4c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c070d40 con 0x7ffa4c1037b0 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa53421700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c070e80 con 0x7ffa4c102e10 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51332/0 (socket says 172.21.15.103:51332) 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 -- 172.21.15.103:0/285091739 learned_addr learned my addr 172.21.15.103:0/285091739 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa511bd700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c102e10 0x7ffa4c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 -- 172.21.15.103:0/285091739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c102e10 msgr2=0x7ffa4c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c102e10 0x7ffa4c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.175+0000 7ffa509bc700 1 -- 172.21.15.103:0/285091739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c005040 con 0x7ffa4c1037b0 2024-03-22T02:42:17.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa509bc700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c800 tx=0x7ffa48004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa480106f0 con 0x7ffa4c1037b0 2024-03-22T02:42:17.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa48010d20 con 0x7ffa4c1037b0 2024-03-22T02:42:17.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c071130 con 0x7ffa4c1037b0 2024-03-22T02:42:17.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa480113a0 con 0x7ffa4c1037b0 2024-03-22T02:42:17.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.176+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c11a320 con 0x7ffa4c1037b0 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.177+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c100120 con 0x7ffa4c1037b0 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.177+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ffa48012400 con 0x7ffa4c1037b0 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.178+0000 7ffa41ffb700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 0x7ffa38070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.178+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa4808db80 con 0x7ffa4c1037b0 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.178+0000 7ffa511bd700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 0x7ffa38070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.178+0000 7ffa511bd700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 0x7ffa38070860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b3b0 tx=0x7ffa3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.183+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa48056e00 con 0x7ffa4c1037b0 2024-03-22T02:42:17.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.399+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa4c071130 con 0x7ffa3806e3b0 2024-03-22T02:42:17.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.400+0000 7ffa41ffb700 1 -- 172.21.15.103:0/285091739 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffa4c071130 con 0x7ffa3806e3b0 2024-03-22T02:42:17.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 msgr2=0x7ffa38070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 0x7ffa38070860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00b3b0 tx=0x7ffa3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 msgr2=0x7ffa4c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c800 tx=0x7ffa48004470 comp rx=0 tx=0).stop 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 shutdown_connections 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ffa3806e3b0 0x7ffa38070860 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa4c102e10 0x7ffa4c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 --2- 172.21.15.103:0/285091739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa4c1037b0 0x7ffa4c070800 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.402+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 >> 172.21.15.103:0/285091739 conn(0x7ffa4c0f89a0 msgr2=0x7ffa4c0facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.403+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 shutdown_connections 2024-03-22T02:42:17.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.403+0000 7ffa53421700 1 -- 172.21.15.103:0/285091739 wait complete. 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 -- 172.21.15.103:0/1102618920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 msgr2=0x7f591c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 --2- 172.21.15.103:0/1102618920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c0fbd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5918004940 tx=0x7f5918009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 -- 172.21.15.103:0/1102618920 shutdown_connections 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 --2- 172.21.15.103:0/1102618920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c0fbd20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 --2- 172.21.15.103:0/1102618920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 0x7f591c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 -- 172.21.15.103:0/1102618920 >> 172.21.15.103:0/1102618920 conn(0x7f591c0f7880 msgr2=0x7f591c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 -- 172.21.15.103:0/1102618920 shutdown_connections 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.504+0000 7f5923fd1700 1 -- 172.21.15.103:0/1102618920 wait complete. 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 Processor -- start 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 -- start start 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 0x7f591c113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c114a40 con 0x7f591c1026b0 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5923fd1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c10de10 con 0x7f591c101d10 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51338/0 (socket says 172.21.15.103:51338) 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 -- 172.21.15.103:0/3182721910 learned_addr learned my addr 172.21.15.103:0/3182721910 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f5921d6d700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 0x7f591c113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 -- 172.21.15.103:0/3182721910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 msgr2=0x7f591c113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 0x7f591c113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 -- 172.21.15.103:0/3182721910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918005040 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.505+0000 7f592156c700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5918009180 tx=0x7f5918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.506+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591800e040 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.506+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5918013420 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.506+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c10e090 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.506+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591801c610 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.506+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c10e650 con 0x7f591c1026b0 2024-03-22T02:42:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.507+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c0fd460 con 0x7f591c1026b0 2024-03-22T02:42:17.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.507+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5918013a00 con 0x7f591c1026b0 2024-03-22T02:42:17.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.508+0000 7f59127fc700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 0x7f5908070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.508+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f591808ef00 con 0x7f591c1026b0 2024-03-22T02:42:17.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.508+0000 7f5921d6d700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 0x7f5908070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.508+0000 7f5921d6d700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 0x7f5908070850 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f590c004f40 tx=0x7f590c011040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.512+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f591805bd60 con 0x7f591c1026b0 2024-03-22T02:42:17.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:17.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.717+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f591c108e60 con 0x7f590806e3a0 2024-03-22T02:42:17.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.727+0000 7f59127fc700 1 -- 172.21.15.103:0/3182721910 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f591c108e60 con 0x7f590806e3a0 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (10m) 4m ago 21m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (8m) 4m ago 22m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (8m) 4m ago 21m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (10m) 4m ago 21m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (4m) 4m ago 16m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (4m) 4m ago 16m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:42:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (4m) 4m ago 16m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (4m) 4m ago 16m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (12m) 4m ago 23m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (11m) 4m ago 21m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (9m) 4m ago 23m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (9m) 4m ago 20m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (11m) 4m ago 22m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (11m) 4m ago 21m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (8m) 4m ago 20m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (7m) 4m ago 19m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (6m) 4m ago 19m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:42:17.731 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (6m) 4m ago 18m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:42:17.732 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (5m) 4m ago 18m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:42:17.732 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (5m) 4m ago 18m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:42:17.732 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (11m) 4m ago 21m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 msgr2=0x7f5908070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 0x7f5908070850 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f590c004f40 tx=0x7f590c011040 comp rx=0 tx=0).stop 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 msgr2=0x7f591c1143f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5918009180 tx=0x7f5918009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 shutdown_connections 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f590806e3a0 0x7f5908070850 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f591c101d10 0x7f591c113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 --2- 172.21.15.103:0/3182721910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f591c1026b0 0x7f591c1143f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 >> 172.21.15.103:0/3182721910 conn(0x7f591c0f7880 msgr2=0x7f591c065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 shutdown_connections 2024-03-22T02:42:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.730+0000 7f5923fd1700 1 -- 172.21.15.103:0/3182721910 wait complete. 2024-03-22T02:42:17.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/2259654282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e01026c0 msgr2=0x7fb6e00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/2259654282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e01026c0 0x7fb6e00fbd30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004940 tx=0x7fb6d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/2259654282 shutdown_connections 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/2259654282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e01026c0 0x7fb6e00fbd30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/2259654282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e0101d20 0x7fb6e01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/2259654282 >> 172.21.15.103:0/2259654282 conn(0x7fb6e00f7870 msgr2=0x7fb6e00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:17.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/2259654282 shutdown_connections 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/2259654282 wait complete. 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 Processor -- start 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.829+0000 7fb6e5ba7700 1 -- start start 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6e5ba7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6e5ba7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e01026c0 0x7fb6e01030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6e5ba7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0104b60 con 0x7fb6e0101d20 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6e5ba7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0103600 con 0x7fb6e01026c0 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51358/0 (socket says 172.21.15.103:51358) 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 -- 172.21.15.103:0/3196181361 learned_addr learned my addr 172.21.15.103:0/3196181361 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:17.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6deffd700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e01026c0 0x7fb6e01030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 -- 172.21.15.103:0/3196181361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e01026c0 msgr2=0x7fb6e01030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:17.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e01026c0 0x7fb6e01030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:17.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 -- 172.21.15.103:0/3196181361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d4005040 con 0x7fb6e0101d20 2024-03-22T02:42:17.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6df7fe700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000b790 tx=0x7fb6d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.830+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d000f6f0 con 0x7fb6e0101d20 2024-03-22T02:42:17.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.831+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e01038e0 con 0x7fb6e0101d20 2024-03-22T02:42:17.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.831+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6d000fd20 con 0x7fb6e0101d20 2024-03-22T02:42:17.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.831+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e011a190 con 0x7fb6e0101d20 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.831+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d00103a0 con 0x7fb6e0101d20 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.832+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e00fd470 con 0x7fb6e0101d20 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.832+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb6d0010500 con 0x7fb6e0101d20 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.833+0000 7fb6c7fff700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 0x7fb6c8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.833+0000 7fb6deffd700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 0x7fb6c8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.833+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6d008dc20 con 0x7fb6e0101d20 2024-03-22T02:42:17.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.833+0000 7fb6deffd700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 0x7fb6c8070850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004890 tx=0x7fb6d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:17.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:17.837+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb6d0056ea0 con 0x7fb6e0101d20 2024-03-22T02:42:18.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.117+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb6e0104120 con 0x7fb6e0101d20 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.118+0000 7fb6c7fff700 1 -- 172.21.15.103:0/3196181361 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb6d005a560 con 0x7fb6e0101d20 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:18.120 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:42:18.121 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 msgr2=0x7fb6c8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 0x7fb6c8070850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4004890 tx=0x7fb6d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 msgr2=0x7fb6e0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000b790 tx=0x7fb6d0009250 comp rx=0 tx=0).stop 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 shutdown_connections 2024-03-22T02:42:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb6c806e3a0 0x7fb6c8070850 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6e0101d20 0x7fb6e0102b80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 --2- 172.21.15.103:0/3196181361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e01026c0 0x7fb6e01030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 >> 172.21.15.103:0/3196181361 conn(0x7fb6e00f7870 msgr2=0x7fb6e00f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 shutdown_connections 2024-03-22T02:42:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.120+0000 7fb6e5ba7700 1 -- 172.21.15.103:0/3196181361 wait complete. 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- 172.21.15.103:0/3550720934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4102710 msgr2=0x7f4fa40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 --2- 172.21.15.103:0/3550720934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4102710 0x7f4fa40fbd80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4f90004940 tx=0x7f4f90009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- 172.21.15.103:0/3550720934 shutdown_connections 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 --2- 172.21.15.103:0/3550720934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4102710 0x7f4fa40fbd80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 --2- 172.21.15.103:0/3550720934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4101d70 0x7f4fa4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- 172.21.15.103:0/3550720934 >> 172.21.15.103:0/3550720934 conn(0x7f4fa4072bc0 msgr2=0x7f4fa4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- 172.21.15.103:0/3550720934 shutdown_connections 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- 172.21.15.103:0/3550720934 wait complete. 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 Processor -- start 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.224+0000 7f4faae71700 1 -- start start 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4102710 0x7f4fa4116170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa41167c0 con 0x7f4fa4101d70 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa410fc40 con 0x7f4fa4102710 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51368/0 (socket says 172.21.15.103:51368) 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 -- 172.21.15.103:0/2250693360 learned_addr learned my addr 172.21.15.103:0/2250693360 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa3fff700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4102710 0x7f4fa4116170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 -- 172.21.15.103:0/2250693360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4102710 msgr2=0x7f4fa4116170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4102710 0x7f4fa4116170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 -- 172.21.15.103:0/2250693360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90005040 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa8c0d700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4f98002a80 tx=0x7f4f980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f98007cc0 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa410ff20 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f9800f040 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f9800dac0 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.225+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa41104e0 con 0x7f4fa4101d70 2024-03-22T02:42:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.227+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa40fd4c0 con 0x7f4fa4101d70 2024-03-22T02:42:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.227+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4f9800dc20 con 0x7f4fa4101d70 2024-03-22T02:42:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.227+0000 7f4fa17fa700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 0x7f4f94070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.228+0000 7f4fa3fff700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 0x7f4f94070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.228+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f9808da20 con 0x7f4fa4101d70 2024-03-22T02:42:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.228+0000 7f4fa3fff700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 0x7f4f94070860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40fa580 tx=0x7f4f90009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.232+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f98056c20 con 0x7f4fa4101d70 2024-03-22T02:42:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:18 smithi103 ceph-mon[115211]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:18 smithi103 ceph-mon[115211]: pgmap v323: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:18 smithi103 ceph-mon[115211]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:18 smithi103 ceph-mon[115211]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:18 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3196181361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:42:18.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.476+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4fa4108e50 con 0x7f4fa4101d70 2024-03-22T02:42:18.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.477+0000 7f4fa17fa700 1 -- 172.21.15.103:0/2250693360 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f4f9805a2e0 con 0x7f4fa4101d70 2024-03-22T02:42:18.478 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:42:18.479 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:42:18.480 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:18.482 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.478+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 msgr2=0x7f4f94070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.478+0000 7f4faae71700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 0x7f4f94070860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40fa580 tx=0x7f4f90009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 msgr2=0x7f4fa4115c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4f98002a80 tx=0x7f4f980046b0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 shutdown_connections 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f9406e3b0 0x7f4f94070860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fa4101d70 0x7f4fa4115c30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 --2- 172.21.15.103:0/2250693360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fa4102710 0x7f4fa4116170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 >> 172.21.15.103:0/2250693360 conn(0x7f4fa4072bc0 msgr2=0x7f4fa4065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 shutdown_connections 2024-03-22T02:42:18.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.479+0000 7f4faae71700 1 -- 172.21.15.103:0/2250693360 wait complete. 2024-03-22T02:42:18.484 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:42:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 -- 172.21.15.103:0/2968863554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 msgr2=0x7f66801020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 --2- 172.21.15.103:0/2968863554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f66801020e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6670004940 tx=0x7f6670009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 -- 172.21.15.103:0/2968863554 shutdown_connections 2024-03-22T02:42:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 --2- 172.21.15.103:0/2968863554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 0x7f66800fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 --2- 172.21.15.103:0/2968863554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f66801020e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 -- 172.21.15.103:0/2968863554 >> 172.21.15.103:0/2968863554 conn(0x7f66800f7880 msgr2=0x7f66800f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 -- 172.21.15.103:0/2968863554 shutdown_connections 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 -- 172.21.15.103:0/2968863554 wait complete. 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.581+0000 7f66861a9700 1 Processor -- start 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f66861a9700 1 -- start start 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f66861a9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f66861a9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 0x7f668010f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f66861a9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668010fa50 con 0x7f6680101d10 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f66861a9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668010fb90 con 0x7f66801026b0 2024-03-22T02:42:18.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51392/0 (socket says 172.21.15.103:51392) 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 -- 172.21.15.103:0/1636491603 learned_addr learned my addr 172.21.15.103:0/1636491603 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f6676ffd700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 0x7f668010f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 -- 172.21.15.103:0/1636491603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 msgr2=0x7f668010f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 0x7f668010f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 -- 172.21.15.103:0/1636491603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6670005040 con 0x7f6680101d10 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.582+0000 7f667f7fe700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f667000b8b0 tx=0x7f6670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.583+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667000e040 con 0x7f6680101d10 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.583+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6670004290 con 0x7f6680101d10 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.583+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680117ea0 con 0x7f6680101d10 2024-03-22T02:42:18.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.583+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667001c3f0 con 0x7f6680101d10 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.583+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6680118430 con 0x7f6680101d10 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.584+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66800fd460 con 0x7f6680101d10 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.584+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f66700137b0 con 0x7f6680101d10 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.584+0000 7f667cff9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 0x7f666c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.584+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f667008f110 con 0x7f6680101d10 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.584+0000 7f6676ffd700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 0x7f666c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.585+0000 7f6676ffd700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 0x7f666c070850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6668006fd0 tx=0x7f6668008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.589+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6670058390 con 0x7f6680101d10 2024-03-22T02:42:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:18 smithi122 ceph-mon[83368]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:18 smithi122 ceph-mon[83368]: pgmap v323: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:18 smithi122 ceph-mon[83368]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:18 smithi122 ceph-mon[83368]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:18 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3196181361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:42:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.810+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6680066640 con 0x7f666c06e3a0 2024-03-22T02:42:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.812+0000 7f667cff9700 1 -- 172.21.15.103:0/1636491603 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6680066640 con 0x7f666c06e3a0 2024-03-22T02:42:18.813 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:42:18.813 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:42:18.813 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:42:18.814 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:42:18.815 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:42:18.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 msgr2=0x7f666c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 0x7f666c070850 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6668006fd0 tx=0x7f6668008040 comp rx=0 tx=0).stop 2024-03-22T02:42:18.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 msgr2=0x7f668010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f667000b8b0 tx=0x7f6670009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 shutdown_connections 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f666c06e3a0 0x7f666c070850 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6680101d10 0x7f668010ee30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 --2- 172.21.15.103:0/1636491603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66801026b0 0x7f668010f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 >> 172.21.15.103:0/1636491603 conn(0x7f66800f7880 msgr2=0x7f66800f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 shutdown_connections 2024-03-22T02:42:18.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.814+0000 7f66861a9700 1 -- 172.21.15.103:0/1636491603 wait complete. 2024-03-22T02:42:18.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 -- 172.21.15.103:0/3102835970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 msgr2=0x7f55c41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 --2- 172.21.15.103:0/3102835970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c41031d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f55b8004940 tx=0x7f55b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 -- 172.21.15.103:0/3102835970 shutdown_connections 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 --2- 172.21.15.103:0/3102835970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 0x7f55c40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 --2- 172.21.15.103:0/3102835970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c41031d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 -- 172.21.15.103:0/3102835970 >> 172.21.15.103:0/3102835970 conn(0x7f55c40f89b0 msgr2=0x7f55c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 -- 172.21.15.103:0/3102835970 shutdown_connections 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.915+0000 7f55cae6c700 1 -- 172.21.15.103:0/3102835970 wait complete. 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.916+0000 7f55cae6c700 1 Processor -- start 2024-03-22T02:42:18.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.916+0000 7f55cae6c700 1 -- start start 2024-03-22T02:42:18.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.916+0000 7f55cae6c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.916+0000 7f55cae6c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 0x7f55c4113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.916+0000 7f55cae6c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c4114520 con 0x7f55c4102e00 2024-03-22T02:42:18.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55cae6c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c410db70 con 0x7f55c41037a0 2024-03-22T02:42:18.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51408/0 (socket says 172.21.15.103:51408) 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 -- 172.21.15.103:0/1749069384 learned_addr learned my addr 172.21.15.103:0/1749069384 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c3fff700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 0x7f55c4113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 -- 172.21.15.103:0/1749069384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 msgr2=0x7f55c4113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 0x7f55c4113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 -- 172.21.15.103:0/1749069384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8005040 con 0x7f55c4102e00 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c8c08700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f55b800b3b0 tx=0x7f55b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55b800e040 con 0x7f55c4102e00 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55b8013420 con 0x7f55c4102e00 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c410ddf0 con 0x7f55c4102e00 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55b801c630 con 0x7f55c4102e00 2024-03-22T02:42:18.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.917+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c410e3b0 con 0x7f55c4102e00 2024-03-22T02:42:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.918+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c4100110 con 0x7f55c4102e00 2024-03-22T02:42:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.919+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f55b8013590 con 0x7f55c4102e00 2024-03-22T02:42:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.919+0000 7f55c17fa700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 0x7f55b4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.919+0000 7f55c3fff700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 0x7f55b4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:18.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.919+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f55b808efe0 con 0x7f55c4102e00 2024-03-22T02:42:18.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.920+0000 7f55c3fff700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 0x7f55b4070850 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55c410ece0 tx=0x7f55b0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:18.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:18.924+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55b805bd90 con 0x7f55c4102e00 2024-03-22T02:42:19.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.208+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f55c410efe0 con 0x7f55c4102e00 2024-03-22T02:42:19.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.208+0000 7f55c17fa700 1 -- 172.21.15.103:0/1749069384 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f55b805b920 con 0x7f55c4102e00 2024-03-22T02:42:19.210 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:42:19.210 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:42:19.210 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:42:19.211 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:42:19.211 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:42:19.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.210+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 msgr2=0x7f55b4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:19.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.210+0000 7f55cae6c700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 0x7f55b4070850 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55c410ece0 tx=0x7f55b0009040 comp rx=0 tx=0).stop 2024-03-22T02:42:19.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.210+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 msgr2=0x7f55c4113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:19.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.210+0000 7f55cae6c700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f55b800b3b0 tx=0x7f55b8009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:19.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 shutdown_connections 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55b406e3a0 0x7f55b4070850 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55c4102e00 0x7f55c4113990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 --2- 172.21.15.103:0/1749069384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c41037a0 0x7f55c4113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 >> 172.21.15.103:0/1749069384 conn(0x7f55c40f89b0 msgr2=0x7f55c41073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 shutdown_connections 2024-03-22T02:42:19.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:19.211+0000 7f55cae6c700 1 -- 172.21.15.103:0/1749069384 wait complete. 2024-03-22T02:42:19.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:19 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2250693360' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:42:19.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:19 smithi122 ceph-mon[83368]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:19.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:19 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1749069384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:42:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:19 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2250693360' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:42:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:19 smithi103 ceph-mon[115211]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:19.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:19 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1749069384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:42:20.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:20 smithi103 ceph-mon[115211]: pgmap v324: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:20.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:20 smithi122 ceph-mon[83368]: pgmap v324: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:22.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:22 smithi122 ceph-mon[83368]: pgmap v325: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:22.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:22 smithi103 ceph-mon[115211]: pgmap v325: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:24.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:24 smithi122 ceph-mon[83368]: pgmap v326: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:24.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:24 smithi103 ceph-mon[115211]: pgmap v326: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:26.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:26 smithi122 ceph-mon[83368]: pgmap v327: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:26.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:26 smithi103 ceph-mon[115211]: pgmap v327: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:28.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:28 smithi122 ceph-mon[83368]: pgmap v328: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:28.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:28 smithi103 ceph-mon[115211]: pgmap v328: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:30.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:30 smithi103 ceph-mon[115211]: pgmap v329: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:30.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:30 smithi122 ceph-mon[83368]: pgmap v329: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:32.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:32 smithi122 ceph-mon[83368]: pgmap v330: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:32.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:32 smithi103 ceph-mon[115211]: pgmap v330: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:34.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:34 smithi122 ceph-mon[83368]: pgmap v331: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:34.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:34 smithi103 ceph-mon[115211]: pgmap v331: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:36 smithi122 ceph-mon[83368]: pgmap v332: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:36.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:36 smithi103 ceph-mon[115211]: pgmap v332: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:38.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:38 smithi122 ceph-mon[83368]: pgmap v333: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:38 smithi103 ceph-mon[115211]: pgmap v333: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:40.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:40 smithi103 ceph-mon[115211]: pgmap v334: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:40.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:40 smithi122 ceph-mon[83368]: pgmap v334: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:42.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:42 smithi122 ceph-mon[83368]: pgmap v335: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:42.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:42 smithi103 ceph-mon[115211]: pgmap v335: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:44 smithi122 ceph-mon[83368]: pgmap v336: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:44.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:44 smithi103 ceph-mon[115211]: pgmap v336: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:46 smithi122 ceph-mon[83368]: pgmap v337: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:46 smithi103 ceph-mon[115211]: pgmap v337: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:42:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:48 smithi122 ceph-mon[83368]: pgmap v338: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:48 smithi103 ceph-mon[115211]: pgmap v338: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:49.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 -- 172.21.15.103:0/229760837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770103780 msgr2=0x7f27700fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 --2- 172.21.15.103:0/229760837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770103780 0x7f27700fcdf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2760004940 tx=0x7f2760009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:49.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 -- 172.21.15.103:0/229760837 shutdown_connections 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 --2- 172.21.15.103:0/229760837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770103780 0x7f27700fcdf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 --2- 172.21.15.103:0/229760837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770102de0 0x7f27701031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 -- 172.21.15.103:0/229760837 >> 172.21.15.103:0/229760837 conn(0x7f27700f8950 msgr2=0x7f27700fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 -- 172.21.15.103:0/229760837 shutdown_connections 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.325+0000 7f27761f6700 1 -- 172.21.15.103:0/229760837 wait complete. 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 Processor -- start 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 -- start start 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770103780 0x7f2770113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2770114530 con 0x7f2770102de0 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.326+0000 7f27761f6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277010db30 con 0x7f2770103780 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48184/0 (socket says 172.21.15.103:48184) 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 -- 172.21.15.103:0/2104422862 learned_addr learned my addr 172.21.15.103:0/2104422862 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:49.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276effd700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770103780 0x7f2770113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 -- 172.21.15.103:0/2104422862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770103780 msgr2=0x7f2770113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770103780 0x7f2770113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 -- 172.21.15.103:0/2104422862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760005040 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f276f7fe700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2764002a50 tx=0x7f27640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2764007cc0 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f276400f040 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f276400da70 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277010de10 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.327+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277010e3a0 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.328+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27701052b0 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.329+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f276400dc10 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.329+0000 7f2757fff700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 0x7f2758070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.329+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f276408d360 con 0x7f2770102de0 2024-03-22T02:42:49.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.329+0000 7f276effd700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 0x7f2758070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.330+0000 7f276effd700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 0x7f2758070850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f277010ee10 tx=0x7f2760009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.334+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f276405a110 con 0x7f2770102de0 2024-03-22T02:42:49.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.549+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f277010eff0 con 0x7f275806e3a0 2024-03-22T02:42:49.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.551+0000 7f2757fff700 1 -- 172.21.15.103:0/2104422862 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f277010eff0 con 0x7f275806e3a0 2024-03-22T02:42:49.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 msgr2=0x7f2758070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 0x7f2758070850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f277010ee10 tx=0x7f2760009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:49.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 msgr2=0x7f27701139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2764002a50 tx=0x7f27640046b0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 shutdown_connections 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f275806e3a0 0x7f2758070850 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2770102de0 0x7f27701139a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 --2- 172.21.15.103:0/2104422862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2770103780 0x7f2770113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 >> 172.21.15.103:0/2104422862 conn(0x7f27700f8950 msgr2=0x7f27701021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 shutdown_connections 2024-03-22T02:42:49.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.553+0000 7f27761f6700 1 -- 172.21.15.103:0/2104422862 wait complete. 2024-03-22T02:42:49.564 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:42:49.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- 172.21.15.103:0/2776184705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fbc80 msgr2=0x7fb54c0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 --2- 172.21.15.103:0/2776184705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fbc80 0x7fb54c0fc050 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb53c004940 tx=0x7fb53c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- 172.21.15.103:0/2776184705 shutdown_connections 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 --2- 172.21.15.103:0/2776184705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fc590 0x7fb54c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 --2- 172.21.15.103:0/2776184705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fbc80 0x7fb54c0fc050 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- 172.21.15.103:0/2776184705 >> 172.21.15.103:0/2776184705 conn(0x7fb54c072bc0 msgr2=0x7fb54c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- 172.21.15.103:0/2776184705 shutdown_connections 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- 172.21.15.103:0/2776184705 wait complete. 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 Processor -- start 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.656+0000 7fb5517e4700 1 -- start start 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5517e4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fbc80 0x7fb54c114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5517e4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5517e4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c1157c0 con 0x7fb54c0fc590 2024-03-22T02:42:49.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5517e4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c10fd90 con 0x7fb54c0fbc80 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48198/0 (socket says 172.21.15.103:48198) 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 -- 172.21.15.103:0/4210821038 learned_addr learned my addr 172.21.15.103:0/4210821038 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb54affd700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fbc80 0x7fb54c114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 -- 172.21.15.103:0/4210821038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fbc80 msgr2=0x7fb54c114c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fbc80 0x7fb54c114c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 -- 172.21.15.103:0/4210821038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c005040 con 0x7fb54c0fc590 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5427fc700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c810 tx=0x7fb534004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb534012810 con 0x7fb54c0fc590 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c110070 con 0x7fb54c0fc590 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb534012e40 con 0x7fb54c0fc590 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.657+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5340102e0 con 0x7fb54c0fc590 2024-03-22T02:42:49.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.658+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c110630 con 0x7fb54c0fc590 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.659+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c065e30 con 0x7fb54c0fc590 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.659+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb5340105e0 con 0x7fb54c0fc590 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.659+0000 7fb543fff700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 0x7fb538070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.659+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb53408e140 con 0x7fb54c0fc590 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.659+0000 7fb54affd700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 0x7fb538070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.660+0000 7fb54affd700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 0x7fb538070850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb53c004f90 tx=0x7fb53c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.664+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb534057340 con 0x7fb54c0fc590 2024-03-22T02:42:49.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.879+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb54c10c180 con 0x7fb53806e3a0 2024-03-22T02:42:49.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.880+0000 7fb543fff700 1 -- 172.21.15.103:0/4210821038 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb54c10c180 con 0x7fb53806e3a0 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 msgr2=0x7fb538070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 0x7fb538070850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb53c004f90 tx=0x7fb53c009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 msgr2=0x7fb54c1151c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c810 tx=0x7fb534004470 comp rx=0 tx=0).stop 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 shutdown_connections 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb53806e3a0 0x7fb538070850 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb54c0fbc80 0x7fb54c114c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 --2- 172.21.15.103:0/4210821038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb54c0fc590 0x7fb54c1151c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.882+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 >> 172.21.15.103:0/4210821038 conn(0x7fb54c072bc0 msgr2=0x7fb54c105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:49.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.883+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 shutdown_connections 2024-03-22T02:42:49.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.883+0000 7fb5517e4700 1 -- 172.21.15.103:0/4210821038 wait complete. 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 -- 172.21.15.103:0/3784128824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fe7a0 msgr2=0x7ff458106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 --2- 172.21.15.103:0/3784128824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fe7a0 0x7ff458106de0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff454004940 tx=0x7ff454009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 -- 172.21.15.103:0/3784128824 shutdown_connections 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 --2- 172.21.15.103:0/3784128824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fe7a0 0x7ff458106de0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 --2- 172.21.15.103:0/3784128824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fde00 0x7ff4580fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 -- 172.21.15.103:0/3784128824 >> 172.21.15.103:0/3784128824 conn(0x7ff4580f78d0 msgr2=0x7ff4580f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 -- 172.21.15.103:0/3784128824 shutdown_connections 2024-03-22T02:42:49.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.984+0000 7ff45fd08700 1 -- 172.21.15.103:0/3784128824 wait complete. 2024-03-22T02:42:49.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 Processor -- start 2024-03-22T02:42:49.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 -- start start 2024-03-22T02:42:49.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fe7a0 0x7ff4580fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580fc700 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45fd08700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580fc840 con 0x7ff4580fe7a0 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48226/0 (socket says 172.21.15.103:48226) 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 -- 172.21.15.103:0/517950739 learned_addr learned my addr 172.21.15.103:0/517950739 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45d2a3700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fe7a0 0x7ff4580fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 -- 172.21.15.103:0/517950739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fe7a0 msgr2=0x7ff4580fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fe7a0 0x7ff4580fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.985+0000 7ff45daa4700 1 -- 172.21.15.103:0/517950739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff454005040 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff45daa4700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff448002a50 tx=0x7ff4480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff448007cc0 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff44800f040 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458119e70 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44800da30 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.986+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45811a270 con 0x7ff4580fde00 2024-03-22T02:42:49.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.987+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff458066770 con 0x7ff4580fde00 2024-03-22T02:42:49.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.987+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff44800db90 con 0x7ff4580fde00 2024-03-22T02:42:49.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.988+0000 7ff44e7fc700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 0x7ff444070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:49.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.988+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff44808db40 con 0x7ff4580fde00 2024-03-22T02:42:49.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.988+0000 7ff45d2a3700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 0x7ff444070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:49.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.988+0000 7ff45d2a3700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 0x7ff444070850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff4580fd570 tx=0x7ff454009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:49.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:49.992+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff448056d40 con 0x7ff4580fde00 2024-03-22T02:42:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.198+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4580fd320 con 0x7ff44406e3a0 2024-03-22T02:42:50.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.209+0000 7ff44e7fc700 1 -- 172.21.15.103:0/517950739 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff4580fd320 con 0x7ff44406e3a0 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (11m) 5m ago 22m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (9m) 5m ago 22m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (9m) 4m ago 21m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (11m) 5m ago 22m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (5m) 5m ago 17m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (5m) 5m ago 16m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (4m) 4m ago 16m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (4m) 4m ago 17m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (12m) 5m ago 24m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (12m) 4m ago 21m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:42:50.210 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (10m) 5m ago 24m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (9m) 4m ago 21m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (12m) 5m ago 22m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (11m) 4m ago 21m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (8m) 5m ago 20m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (8m) 5m ago 20m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (7m) 5m ago 19m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (6m) 4m ago 19m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (6m) 4m ago 19m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (5m) 4m ago 18m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:42:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (11m) 5m ago 22m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:42:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 msgr2=0x7ff444070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 0x7ff444070850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff4580fd570 tx=0x7ff454009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 msgr2=0x7ff4580fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff448002a50 tx=0x7ff4480046b0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 shutdown_connections 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff44406e3a0 0x7ff444070850 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4580fde00 0x7ff4580fbc30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 --2- 172.21.15.103:0/517950739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4580fe7a0 0x7ff4580fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 >> 172.21.15.103:0/517950739 conn(0x7ff4580f78d0 msgr2=0x7ff4580f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 shutdown_connections 2024-03-22T02:42:50.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.212+0000 7ff45fd08700 1 -- 172.21.15.103:0/517950739 wait complete. 2024-03-22T02:42:50.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 -- 172.21.15.103:0/2052479300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9080fb4d0 msgr2=0x7fa9080fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 --2- 172.21.15.103:0/2052479300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9080fb4d0 0x7fa9080fb940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc004940 tx=0x7fa8fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:50.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 -- 172.21.15.103:0/2052479300 shutdown_connections 2024-03-22T02:42:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 --2- 172.21.15.103:0/2052479300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9080fb4d0 0x7fa9080fb940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 --2- 172.21.15.103:0/2052479300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908102f50 0x7fa908103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 -- 172.21.15.103:0/2052479300 >> 172.21.15.103:0/2052479300 conn(0x7fa9080714c0 msgr2=0x7fa9080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 -- 172.21.15.103:0/2052479300 shutdown_connections 2024-03-22T02:42:50.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.312+0000 7fa90d8db700 1 -- 172.21.15.103:0/2052479300 wait complete. 2024-03-22T02:42:50.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 Processor -- start 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 -- start start 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 0x7fa908113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908114680 con 0x7fa908102f50 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.313+0000 7fa90d8db700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90810dbd0 con 0x7fa9080fb4d0 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48242/0 (socket says 172.21.15.103:48242) 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 -- 172.21.15.103:0/3190943820 learned_addr learned my addr 172.21.15.103:0/3190943820 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa906ffd700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 0x7fa908113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 -- 172.21.15.103:0/3190943820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 msgr2=0x7fa908113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 0x7fa908113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 -- 172.21.15.103:0/3190943820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc005040 con 0x7fa908102f50 2024-03-22T02:42:50.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa906ffd700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 0x7fa908113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa9067fc700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b3b0 tx=0x7fa8fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8fc00e040 con 0x7fa908102f50 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90810ddf0 con 0x7fa908102f50 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8fc00bbc0 con 0x7fa908102f50 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8fc01c3f0 con 0x7fa908102f50 2024-03-22T02:42:50.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.314+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90810e330 con 0x7fa908102f50 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.316+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa908100200 con 0x7fa908102f50 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.316+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa8fc00bd30 con 0x7fa908102f50 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.316+0000 7fa8ef7fe700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 0x7fa8f0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.316+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa8fc08ef90 con 0x7fa908102f50 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.317+0000 7fa906ffd700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 0x7fa8f0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.317+0000 7fa906ffd700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 0x7fa8f0070850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa908065bf0 tx=0x7fa8f8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:50.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.321+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8fc0c3740 con 0x7fa908102f50 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.605+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa90810f0b0 con 0x7fa908102f50 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.605+0000 7fa8ef7fe700 1 -- 172.21.15.103:0/3190943820 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa8fc05bd40 con 0x7fa908102f50 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:42:50.607 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:42:50.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 msgr2=0x7fa8f0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 0x7fa8f0070850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa908065bf0 tx=0x7fa8f8008040 comp rx=0 tx=0).stop 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 msgr2=0x7fa908113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b3b0 tx=0x7fa8fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 shutdown_connections 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa8f006e3a0 0x7fa8f0070850 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa9080fb4d0 0x7fa908113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 --2- 172.21.15.103:0/3190943820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa908102f50 0x7fa908113fa0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.607+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 >> 172.21.15.103:0/3190943820 conn(0x7fa9080714c0 msgr2=0x7fa9080fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.608+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 shutdown_connections 2024-03-22T02:42:50.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.608+0000 7fa90d8db700 1 -- 172.21.15.103:0/3190943820 wait complete. 2024-03-22T02:42:50.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 -- 172.21.15.103:0/3739273940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 msgr2=0x7f253c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 --2- 172.21.15.103:0/3739273940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c103340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2530004940 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 -- 172.21.15.103:0/3739273940 shutdown_connections 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 --2- 172.21.15.103:0/3739273940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 0x7f253c0fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 --2- 172.21.15.103:0/3739273940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c103340 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 -- 172.21.15.103:0/3739273940 >> 172.21.15.103:0/3739273940 conn(0x7f253c072310 msgr2=0x7f253c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 -- 172.21.15.103:0/3739273940 shutdown_connections 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.708+0000 7f2543226700 1 -- 172.21.15.103:0/3739273940 wait complete. 2024-03-22T02:42:50.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 Processor -- start 2024-03-22T02:42:50.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 -- start start 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 0x7f253c115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c1166e0 con 0x7f253c102f70 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2543226700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c10fcc0 con 0x7f253c0fb3b0 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f253bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f253bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48258/0 (socket says 172.21.15.103:48258) 2024-03-22T02:42:50.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f253bfff700 1 -- 172.21.15.103:0/4186753114 learned_addr learned my addr 172.21.15.103:0/4186753114 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.709+0000 7f2540fc2700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 0x7f253c115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f253bfff700 1 -- 172.21.15.103:0/4186753114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 msgr2=0x7f253c115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f253bfff700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 0x7f253c115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f253bfff700 1 -- 172.21.15.103:0/4186753114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530005040 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f253bfff700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2528002a50 tx=0x7f25280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2528007cc0 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c10ffa0 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f252800f040 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252800da70 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.710+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c110560 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.711+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c101160 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.711+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f252800dc10 con 0x7f253c102f70 2024-03-22T02:42:50.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.712+0000 7f25397fa700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 0x7f252c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.712+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f252808d420 con 0x7f253c102f70 2024-03-22T02:42:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.712+0000 7f2540fc2700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 0x7f252c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:50.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.712+0000 7f2540fc2700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 0x7f252c070850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f253000b3b0 tx=0x7f2530009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:50.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.716+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f252805a150 con 0x7f253c102f70 2024-03-22T02:42:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:50 smithi122 ceph-mon[83368]: pgmap v339: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:50 smithi122 ceph-mon[83368]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:50.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:50 smithi122 ceph-mon[83368]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:50 smithi103 ceph-mon[115211]: pgmap v339: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:50 smithi103 ceph-mon[115211]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:50 smithi103 ceph-mon[115211]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.955+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f253c1111a0 con 0x7f253c102f70 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.955+0000 7f25397fa700 1 -- 172.21.15.103:0/4186753114 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f2528059ce0 con 0x7f253c102f70 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:42:50.956 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:42:50.957 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:42:50.958 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:50.959 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.957+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 msgr2=0x7f252c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.957+0000 7f2543226700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 0x7f252c070850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f253000b3b0 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.957+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 msgr2=0x7f253c116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.957+0000 7f2543226700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2528002a50 tx=0x7f25280046b0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 shutdown_connections 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f252c06e3a0 0x7f252c070850 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f253c0fb3b0 0x7f253c115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 --2- 172.21.15.103:0/4186753114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f253c102f70 0x7f253c116090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 >> 172.21.15.103:0/4186753114 conn(0x7f253c072310 msgr2=0x7f253c0f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 shutdown_connections 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:50.958+0000 7f2543226700 1 -- 172.21.15.103:0/4186753114 wait complete. 2024-03-22T02:42:50.960 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.059+0000 7fa853952700 1 -- 172.21.15.103:0/937892793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fde30 msgr2=0x7fa84c0fe200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.059+0000 7fa853952700 1 --2- 172.21.15.103:0/937892793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fde30 0x7fa84c0fe200 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa848004970 tx=0x7fa848009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- 172.21.15.103:0/937892793 shutdown_connections 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 --2- 172.21.15.103:0/937892793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 --2- 172.21.15.103:0/937892793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fde30 0x7fa84c0fe200 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- 172.21.15.103:0/937892793 >> 172.21.15.103:0/937892793 conn(0x7fa84c0f7870 msgr2=0x7fa84c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- 172.21.15.103:0/937892793 shutdown_connections 2024-03-22T02:42:51.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- 172.21.15.103:0/937892793 wait complete. 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 Processor -- start 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- start start 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fde30 0x7fa84c113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c114590 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.060+0000 7fa853952700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c10db20 con 0x7fa84c0fde30 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48276/0 (socket says 172.21.15.103:48276) 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 -- 172.21.15.103:0/1539942863 learned_addr learned my addr 172.21.15.103:0/1539942863 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa8516ee700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fde30 0x7fa84c113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 -- 172.21.15.103:0/1539942863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fde30 msgr2=0x7fa84c113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fde30 0x7fa84c113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 -- 172.21.15.103:0/1539942863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848005040 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa850eed700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00b790 tx=0x7fa83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa83c00f6f0 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c10de00 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa83c00fd20 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa83c0103a0 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.061+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c10e340 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.062+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c066ab0 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.063+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa83c011400 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.063+0000 7fa841ffb700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 0x7fa838070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.063+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa83c08dbe0 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.063+0000 7fa8516ee700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 0x7fa838070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.064+0000 7fa8516ee700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 0x7fa838070850 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa848004f50 tx=0x7fa848009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:51.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.068+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa83c056e60 con 0x7fa84c0fe7d0 2024-03-22T02:42:51.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.281+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa84c109f90 con 0x7fa83806e3a0 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.282+0000 7fa841ffb700 1 -- 172.21.15.103:0/1539942863 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa84c109f90 con 0x7fa83806e3a0 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:42:51.283 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:42:51.284 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:42:51.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.284+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 msgr2=0x7fa838070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.284+0000 7fa853952700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 0x7fa838070850 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa848004f50 tx=0x7fa848009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.284+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 msgr2=0x7fa84c113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.284+0000 7fa853952700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00b790 tx=0x7fa83c009250 comp rx=0 tx=0).stop 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 shutdown_connections 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa83806e3a0 0x7fa838070850 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84c0fde30 0x7fa84c113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 --2- 172.21.15.103:0/1539942863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa84c0fe7d0 0x7fa84c113eb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 >> 172.21.15.103:0/1539942863 conn(0x7fa84c0f7870 msgr2=0x7fa84c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 shutdown_connections 2024-03-22T02:42:51.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.285+0000 7fa853952700 1 -- 172.21.15.103:0/1539942863 wait complete. 2024-03-22T02:42:51.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- 172.21.15.103:0/1528304184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fe820 msgr2=0x7efca8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1528304184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fe820 0x7efca8106e60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efc98004940 tx=0x7efc98009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- 172.21.15.103:0/1528304184 shutdown_connections 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1528304184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fe820 0x7efca8106e60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1528304184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fde80 0x7efca80fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- 172.21.15.103:0/1528304184 >> 172.21.15.103:0/1528304184 conn(0x7efca80f7870 msgr2=0x7efca80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- 172.21.15.103:0/1528304184 shutdown_connections 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- 172.21.15.103:0/1528304184 wait complete. 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 Processor -- start 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- start start 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fe820 0x7efca8113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81144a0 con 0x7efca80fde80 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.386+0000 7efcaf48d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca810daa0 con 0x7efca80fe820 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48288/0 (socket says 172.21.15.103:48288) 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 -- 172.21.15.103:0/1237805784 learned_addr learned my addr 172.21.15.103:0/1237805784 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcaca28700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fe820 0x7efca8113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 -- 172.21.15.103:0/1237805784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fe820 msgr2=0x7efca8113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fe820 0x7efca8113e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 -- 172.21.15.103:0/1237805784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98005040 con 0x7efca80fde80 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcad229700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efca400c830 tx=0x7efca4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:51.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efca40106f0 con 0x7efca80fde80 2024-03-22T02:42:51.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efca4010d20 con 0x7efca80fde80 2024-03-22T02:42:51.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca810dd80 con 0x7efca80fde80 2024-03-22T02:42:51.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.387+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efca40113a0 con 0x7efca80fde80 2024-03-22T02:42:51.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.388+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca810e2c0 con 0x7efca80fde80 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.389+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca8066b10 con 0x7efca80fde80 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.389+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efca4012400 con 0x7efca80fde80 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.389+0000 7efc9dffb700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 0x7efc94070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.389+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efca408dd70 con 0x7efca80fde80 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.389+0000 7efcaca28700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 0x7efc94070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:42:51.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.390+0000 7efcaca28700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 0x7efc94070850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efca810ec60 tx=0x7efc98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:42:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.394+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efca405ab20 con 0x7efca80fde80 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.676+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efca8109e40 con 0x7efca80fde80 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.676+0000 7efc9dffb700 1 -- 172.21.15.103:0/1237805784 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7efca405a6b0 con 0x7efca80fde80 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:42:51.677 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:42:51.678 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:42:51.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.678+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 msgr2=0x7efc94070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.678+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 0x7efc94070850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efca810ec60 tx=0x7efc98009e30 comp rx=0 tx=0).stop 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.678+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 msgr2=0x7efca8113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.678+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efca400c830 tx=0x7efca4004470 comp rx=0 tx=0).stop 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 shutdown_connections 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efc9406e3a0 0x7efc94070850 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efca80fde80 0x7efca8113910 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 --2- 172.21.15.103:0/1237805784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efca80fe820 0x7efca8113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 >> 172.21.15.103:0/1237805784 conn(0x7efca80f7870 msgr2=0x7efca80f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 shutdown_connections 2024-03-22T02:42:51.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:42:51.679+0000 7efcaf48d700 1 -- 172.21.15.103:0/1237805784 wait complete. 2024-03-22T02:42:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:51 smithi122 ceph-mon[83368]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:51.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:51 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3190943820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:42:51.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:51 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4186753114' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:42:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:51 smithi103 ceph-mon[115211]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:51 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3190943820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:42:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:51 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4186753114' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:42:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:52 smithi122 ceph-mon[83368]: pgmap v340: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:52.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:52 smithi122 ceph-mon[83368]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:52.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:52 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1237805784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:42:52.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:52 smithi103 ceph-mon[115211]: pgmap v340: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:52.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:52 smithi103 ceph-mon[115211]: from='client.34696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:42:52.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:52 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1237805784' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:42:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:54 smithi122 ceph-mon[83368]: pgmap v341: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:54 smithi103 ceph-mon[115211]: pgmap v341: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:56 smithi122 ceph-mon[83368]: pgmap v342: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:56 smithi103 ceph-mon[115211]: pgmap v342: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:58.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:42:58 smithi122 ceph-mon[83368]: pgmap v343: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:42:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:42:58 smithi103 ceph-mon[115211]: pgmap v343: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:00 smithi122 ceph-mon[83368]: pgmap v344: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:00 smithi103 ceph-mon[115211]: pgmap v344: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:02 smithi122 ceph-mon[83368]: pgmap v345: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:02 smithi103 ceph-mon[115211]: pgmap v345: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:04.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:04 smithi122 ceph-mon[83368]: pgmap v346: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:04.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:04 smithi103 ceph-mon[115211]: pgmap v346: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:06 smithi122 ceph-mon[83368]: pgmap v347: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:06 smithi103 ceph-mon[115211]: pgmap v347: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:08.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:08 smithi122 ceph-mon[83368]: pgmap v348: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:08.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:08 smithi103 ceph-mon[115211]: pgmap v348: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:10 smithi103 ceph-mon[115211]: pgmap v349: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:10 smithi122 ceph-mon[83368]: pgmap v349: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:12.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:12 smithi103 ceph-mon[115211]: pgmap v350: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:12 smithi122 ceph-mon[83368]: pgmap v350: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:14.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:14 smithi103 ceph-mon[115211]: pgmap v351: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:15.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:14 smithi122 ceph-mon[83368]: pgmap v351: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:16 smithi103 ceph-mon[115211]: pgmap v352: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:16 smithi122 ceph-mon[83368]: pgmap v352: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:18 smithi103 ceph-mon[115211]: pgmap v353: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:18 smithi122 ceph-mon[83368]: pgmap v353: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:20 smithi103 ceph-mon[115211]: pgmap v354: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:20 smithi122 ceph-mon[83368]: pgmap v354: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 -- 172.21.15.103:0/1660989881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 msgr2=0x7f14fc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 --2- 172.21.15.103:0/1660989881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fe260 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f14ec004940 tx=0x7f14ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 -- 172.21.15.103:0/1660989881 shutdown_connections 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 --2- 172.21.15.103:0/1660989881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 --2- 172.21.15.103:0/1660989881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fe260 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:21.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 -- 172.21.15.103:0/1660989881 >> 172.21.15.103:0/1660989881 conn(0x7f14fc072bc0 msgr2=0x7f14fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:21.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 -- 172.21.15.103:0/1660989881 shutdown_connections 2024-03-22T02:43:21.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.792+0000 7f1501382700 1 -- 172.21.15.103:0/1660989881 wait complete. 2024-03-22T02:43:21.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 Processor -- start 2024-03-22T02:43:21.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 -- start start 2024-03-22T02:43:21.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:21.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:21.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0fc820 con 0x7f14fc0fde90 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f1501382700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0fc960 con 0x7f14fc0fe830 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f14faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f14faffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40938/0 (socket says 172.21.15.103:40938) 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f14faffd700 1 -- 172.21.15.103:0/3092568823 learned_addr learned my addr 172.21.15.103:0/3092568823 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.793+0000 7f14f27fc700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14faffd700 1 -- 172.21.15.103:0/3092568823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 msgr2=0x7f14fc0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14faffd700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14faffd700 1 -- 172.21.15.103:0/3092568823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec005040 con 0x7f14fc0fde90 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14f27fc700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14faffd700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00b3b0 tx=0x7f14ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ec00e040 con 0x7f14fc0fde90 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc0fcbb0 con 0x7f14fc0fde90 2024-03-22T02:43:21.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14ec013420 con 0x7f14fc0fde90 2024-03-22T02:43:21.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ec01c610 con 0x7f14fc0fde90 2024-03-22T02:43:21.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.794+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc11a280 con 0x7f14fc0fde90 2024-03-22T02:43:21.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.795+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc065e30 con 0x7f14fc0fde90 2024-03-22T02:43:21.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.795+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f14ec00ba90 con 0x7f14fc0fde90 2024-03-22T02:43:21.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.796+0000 7f14f3fff700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 0x7f14e8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:21.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.796+0000 7f14f27fc700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 0x7f14e8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:21.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.796+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f14ec08f050 con 0x7f14fc0fde90 2024-03-22T02:43:21.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.796+0000 7f14f27fc700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 0x7f14e8070850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14e4007fd0 tx=0x7f14e4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:21.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:21.801+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f14ec0582d0 con 0x7f14fc0fde90 2024-03-22T02:43:22.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.015+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14fc0fd440 con 0x7f14e806e3a0 2024-03-22T02:43:22.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.017+0000 7f14f3fff700 1 -- 172.21.15.103:0/3092568823 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f14fc0fd440 con 0x7f14e806e3a0 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 msgr2=0x7f14e8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 0x7f14e8070850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14e4007fd0 tx=0x7f14e4009040 comp rx=0 tx=0).stop 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 msgr2=0x7f14fc0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00b3b0 tx=0x7f14ec009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 shutdown_connections 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f14e806e3a0 0x7f14e8070850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14fc0fde90 0x7f14fc0fbd50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 --2- 172.21.15.103:0/3092568823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14fc0fe830 0x7f14fc0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 >> 172.21.15.103:0/3092568823 conn(0x7f14fc072bc0 msgr2=0x7f14fc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 shutdown_connections 2024-03-22T02:43:22.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.020+0000 7f1501382700 1 -- 172.21.15.103:0/3092568823 wait complete. 2024-03-22T02:43:22.031 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 -- 172.21.15.103:0/682773545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 msgr2=0x7f896c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 --2- 172.21.15.103:0/682773545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c1020e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f895c004940 tx=0x7f895c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 -- 172.21.15.103:0/682773545 shutdown_connections 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 --2- 172.21.15.103:0/682773545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 0x7f896c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 --2- 172.21.15.103:0/682773545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c1020e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.120+0000 7f8972471700 1 -- 172.21.15.103:0/682773545 >> 172.21.15.103:0/682773545 conn(0x7f896c0f7880 msgr2=0x7f896c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 -- 172.21.15.103:0/682773545 shutdown_connections 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 -- 172.21.15.103:0/682773545 wait complete. 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 Processor -- start 2024-03-22T02:43:22.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 -- start start 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 0x7f896c113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c1144e0 con 0x7f896c101d10 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.121+0000 7f8972471700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c10dae0 con 0x7f896c1026b0 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40954/0 (socket says 172.21.15.103:40954) 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 -- 172.21.15.103:0/3555206687 learned_addr learned my addr 172.21.15.103:0/3555206687 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896b7fe700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 0x7f896c113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 -- 172.21.15.103:0/3555206687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 msgr2=0x7f896c113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 0x7f896c113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 -- 172.21.15.103:0/3555206687 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c005040 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f896bfff700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f895c00b3b0 tx=0x7f895c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895c00e040 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f895c00bbc0 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896c10dd60 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895c01c3f0 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.122+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896c10e320 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.123+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896c0fd460 con 0x7f896c101d10 2024-03-22T02:43:22.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.124+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f895c013420 con 0x7f896c101d10 2024-03-22T02:43:22.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.124+0000 7f8968ff9700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 0x7f8954070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.124+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f895c08f090 con 0x7f896c101d10 2024-03-22T02:43:22.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.124+0000 7f896b7fe700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 0x7f8954070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.125+0000 7f896b7fe700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 0x7f8954070850 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f896c10eca0 tx=0x7f8960009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.129+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f895c05bdc0 con 0x7f896c101d10 2024-03-22T02:43:22.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.351+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f896c108e80 con 0x7f895406e3a0 2024-03-22T02:43:22.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.353+0000 7f8968ff9700 1 -- 172.21.15.103:0/3555206687 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f896c108e80 con 0x7f895406e3a0 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 msgr2=0x7f8954070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 0x7f8954070850 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f896c10eca0 tx=0x7f8960009ee0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 msgr2=0x7f896c113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f895c00b3b0 tx=0x7f895c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 shutdown_connections 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f895406e3a0 0x7f8954070850 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f896c101d10 0x7f896c113950 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 --2- 172.21.15.103:0/3555206687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f896c1026b0 0x7f896c113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 >> 172.21.15.103:0/3555206687 conn(0x7f896c0f7880 msgr2=0x7f896c0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 shutdown_connections 2024-03-22T02:43:22.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.355+0000 7f8972471700 1 -- 172.21.15.103:0/3555206687 wait complete. 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 -- 172.21.15.103:0/383265584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fde20 msgr2=0x7fb3d00fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/383265584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fde20 0x7fb3d00fe1f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0004940 tx=0x7fb3c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 -- 172.21.15.103:0/383265584 shutdown_connections 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/383265584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/383265584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fde20 0x7fb3d00fe1f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 -- 172.21.15.103:0/383265584 >> 172.21.15.103:0/383265584 conn(0x7fb3d00f7870 msgr2=0x7fb3d00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 -- 172.21.15.103:0/383265584 shutdown_connections 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 -- 172.21.15.103:0/383265584 wait complete. 2024-03-22T02:43:22.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.457+0000 7fb3d5c53700 1 Processor -- start 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 -- start start 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fde20 0x7fb3d0113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01145c0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d010dba0 con 0x7fb3d00fde20 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40984/0 (socket says 172.21.15.103:40984) 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 -- 172.21.15.103:0/562035950 learned_addr learned my addr 172.21.15.103:0/562035950 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3cf7fe700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fde20 0x7fb3d0113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 -- 172.21.15.103:0/562035950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fde20 msgr2=0x7fb3d0113a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fde20 0x7fb3d0113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 -- 172.21.15.103:0/562035950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c0005040 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3ceffd700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4002a50 tx=0x7fb3c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3c4007cc0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d010de80 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3c400f040 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3c4009640 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.458+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d010e3c0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.459+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d0066b70 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.460+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb3c40097c0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.460+0000 7fb3b7fff700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 0x7fb3b8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.460+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3c408dbe0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.460+0000 7fb3cf7fe700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 0x7fb3b8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.461+0000 7fb3cf7fe700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 0x7fb3b8070850 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0004560 tx=0x7fb3c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.465+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3c405a8d0 con 0x7fb3d00fe7c0 2024-03-22T02:43:22.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.671+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3d010f0f0 con 0x7fb3b806e3a0 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.681+0000 7fb3b7fff700 1 -- 172.21.15.103:0/562035950 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb3d010f0f0 con 0x7fb3b806e3a0 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (11m) 5m ago 23m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (9m) 5m ago 23m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (9m) 5m ago 22m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (11m) 5m ago 22m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (5m) 5m ago 17m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (5m) 5m ago 17m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:43:22.683 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (5m) 5m ago 17m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (5m) 5m ago 17m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (13m) 5m ago 24m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (12m) 5m ago 22m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (10m) 5m ago 24m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (10m) 5m ago 22m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (12m) 5m ago 23m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (12m) 5m ago 22m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (9m) 5m ago 21m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (8m) 5m ago 20m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (7m) 5m ago 20m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (7m) 5m ago 19m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (6m) 5m ago 19m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (6m) 5m ago 19m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:43:22.684 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (12m) 5m ago 22m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:43:22.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 msgr2=0x7fb3b8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 0x7fb3b8070850 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0004560 tx=0x7fb3c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:22.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 msgr2=0x7fb3d0113f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4002a50 tx=0x7fb3c40046b0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 shutdown_connections 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3b806e3a0 0x7fb3b8070850 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3d00fde20 0x7fb3d0113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 --2- 172.21.15.103:0/562035950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d00fe7c0 0x7fb3d0113f70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.683+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 >> 172.21.15.103:0/562035950 conn(0x7fb3d00f7870 msgr2=0x7fb3d0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.684+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 shutdown_connections 2024-03-22T02:43:22.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.684+0000 7fb3d5c53700 1 -- 172.21.15.103:0/562035950 wait complete. 2024-03-22T02:43:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 -- 172.21.15.103:0/3262388783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12601037a0 msgr2=0x7f12600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 --2- 172.21.15.103:0/3262388783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12601037a0 0x7f12600fce10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1254009ce0 tx=0x7f1254004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:22.785 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:22 smithi103 ceph-mon[115211]: pgmap v355: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:22.785 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:22 smithi103 ceph-mon[115211]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 -- 172.21.15.103:0/3262388783 shutdown_connections 2024-03-22T02:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 --2- 172.21.15.103:0/3262388783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12601037a0 0x7f12600fce10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 --2- 172.21.15.103:0/3262388783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1260102e00 0x7f12601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 -- 172.21.15.103:0/3262388783 >> 172.21.15.103:0/3262388783 conn(0x7f12600f89b0 msgr2=0x7f12600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:22.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 -- 172.21.15.103:0/3262388783 shutdown_connections 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 -- 172.21.15.103:0/3262388783 wait complete. 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.784+0000 7f126872a700 1 Processor -- start 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f126872a700 1 -- start start 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f126872a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f126872a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12601037a0 0x7f12600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f126872a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260070800 con 0x7f1260102e00 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f126872a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260070970 con 0x7f12601037a0 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41002/0 (socket says 172.21.15.103:41002) 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 -- 172.21.15.103:0/2084230401 learned_addr learned my addr 172.21.15.103:0/2084230401 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f1265cc5700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12601037a0 0x7f12600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 -- 172.21.15.103:0/2084230401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12601037a0 msgr2=0x7f12600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12601037a0 0x7f12600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:22.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 -- 172.21.15.103:0/2084230401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1254009940 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.785+0000 7f12664c6700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1258002a50 tx=0x7f12580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.786+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1258007cc0 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.786+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260070c50 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.786+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f125800f040 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.786+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f125800da30 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.786+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126011e7e0 con 0x7f1260102e00 2024-03-22T02:43:22.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.787+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12601052d0 con 0x7f1260102e00 2024-03-22T02:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.787+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f125800db90 con 0x7f1260102e00 2024-03-22T02:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.788+0000 7f125effd700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 0x7f1248070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.788+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f125808dab0 con 0x7f1260102e00 2024-03-22T02:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.788+0000 7f1265cc5700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 0x7f1248070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:22.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.788+0000 7f1265cc5700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 0x7f1248070850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f125400b740 tx=0x7f1254004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:22.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:22.795+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f125805a860 con 0x7f1260102e00 2024-03-22T02:43:23.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.074+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1260109f10 con 0x7f1260102e00 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.075+0000 7f125effd700 1 -- 172.21.15.103:0/2084230401 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f125805a3f0 con 0x7f1260102e00 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:43:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:43:23.077 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 msgr2=0x7f1248070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 0x7f1248070850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f125400b740 tx=0x7f1254004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 msgr2=0x7f1260071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1258002a50 tx=0x7f12580046b0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 shutdown_connections 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f124806e3a0 0x7f1248070850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1260102e00 0x7f1260071d50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 --2- 172.21.15.103:0/2084230401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12601037a0 0x7f12600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 >> 172.21.15.103:0/2084230401 conn(0x7f12600f89b0 msgr2=0x7f12601021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 shutdown_connections 2024-03-22T02:43:23.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.077+0000 7f126872a700 1 -- 172.21.15.103:0/2084230401 wait complete. 2024-03-22T02:43:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:22 smithi122 ceph-mon[83368]: pgmap v355: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:22 smithi122 ceph-mon[83368]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 -- 172.21.15.103:0/54706549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860102e70 msgr2=0x7f2860103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 --2- 172.21.15.103:0/54706549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860102e70 0x7f2860103240 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ce0 tx=0x7f2858004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 -- 172.21.15.103:0/54706549 shutdown_connections 2024-03-22T02:43:23.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 --2- 172.21.15.103:0/54706549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28600fb4a0 0x7f28600fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 --2- 172.21.15.103:0/54706549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2860102e70 0x7f2860103240 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 -- 172.21.15.103:0/54706549 >> 172.21.15.103:0/54706549 conn(0x7f28600714c0 msgr2=0x7f28600718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 -- 172.21.15.103:0/54706549 shutdown_connections 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.178+0000 7f286743b700 1 -- 172.21.15.103:0/54706549 wait complete. 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 Processor -- start 2024-03-22T02:43:23.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 -- start start 2024-03-22T02:43:23.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2860102e70 0x7f28601163e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860116a30 con 0x7f28600fb4a0 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f286743b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286010fe00 con 0x7f2860102e70 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41024/0 (socket says 172.21.15.103:41024) 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 -- 172.21.15.103:0/2543003605 learned_addr learned my addr 172.21.15.103:0/2543003605 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28649d6700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2860102e70 0x7f28601163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 -- 172.21.15.103:0/2543003605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2860102e70 msgr2=0x7f28601163e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2860102e70 0x7f28601163e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.179+0000 7f28651d7700 1 -- 172.21.15.103:0/2543003605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009940 con 0x7f28600fb4a0 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f28651d7700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2858003400 tx=0x7f2858004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285800e040 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2858004610 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2860110080 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285801c6d0 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.180+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860110640 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.181+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28601069c0 con 0x7f28600fb4a0 2024-03-22T02:43:23.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.181+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2858012b00 con 0x7f28600fb4a0 2024-03-22T02:43:23.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.182+0000 7f2855ffb700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 0x7f284c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.182+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f285808f890 con 0x7f28600fb4a0 2024-03-22T02:43:23.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.182+0000 7f28649d6700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 0x7f284c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.182+0000 7f28649d6700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 0x7f284c070850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2860111740 tx=0x7f285000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.187+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f285805c5c0 con 0x7f28600fb4a0 2024-03-22T02:43:23.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.429+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2860065380 con 0x7f28600fb4a0 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.429+0000 7f2855ffb700 1 -- 172.21.15.103:0/2543003605 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f2858004e10 con 0x7f28600fb4a0 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:43:23.430 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:43:23.431 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:23.432 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:43:23.433 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:23.433 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:23.433 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:23.433 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:23.433 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 msgr2=0x7f284c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 0x7f284c070850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2860111740 tx=0x7f285000b040 comp rx=0 tx=0).stop 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 msgr2=0x7f2860115ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2858003400 tx=0x7f2858004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 shutdown_connections 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f284c06e3a0 0x7f284c070850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28600fb4a0 0x7f2860115ea0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 --2- 172.21.15.103:0/2543003605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2860102e70 0x7f28601163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 >> 172.21.15.103:0/2543003605 conn(0x7f28600714c0 msgr2=0x7f28600fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 shutdown_connections 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.431+0000 7f286743b700 1 -- 172.21.15.103:0/2543003605 wait complete. 2024-03-22T02:43:23.434 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- 172.21.15.103:0/73399377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 msgr2=0x7fbe7c0fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 --2- 172.21.15.103:0/73399377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c0fc650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c004940 tx=0x7fbe6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- 172.21.15.103:0/73399377 shutdown_connections 2024-03-22T02:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 --2- 172.21.15.103:0/73399377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c0fc650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 --2- 172.21.15.103:0/73399377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 0x7fbe7c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- 172.21.15.103:0/73399377 >> 172.21.15.103:0/73399377 conn(0x7fbe7c071530 msgr2=0x7fbe7c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- 172.21.15.103:0/73399377 shutdown_connections 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- 172.21.15.103:0/73399377 wait complete. 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 Processor -- start 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.536+0000 7fbe83097700 1 -- start start 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe83097700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 0x7fbe7c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe83097700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe83097700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c114580 con 0x7fbe7c102800 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe83097700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c10db40 con 0x7fbe7c101e60 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41052/0 (socket says 172.21.15.103:41052) 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 -- 172.21.15.103:0/2424451282 learned_addr learned my addr 172.21.15.103:0/2424451282 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe80e33700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 0x7fbe7c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 -- 172.21.15.103:0/2424451282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 msgr2=0x7fbe7c113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 0x7fbe7c113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 -- 172.21.15.103:0/2424451282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c005040 con 0x7fbe7c102800 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe73fff700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009100 tx=0x7fbe6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6c00e040 con 0x7fbe7c102800 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe6c004290 con 0x7fbe7c102800 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6c01c3f0 con 0x7fbe7c102800 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.537+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7c10ddc0 con 0x7fbe7c102800 2024-03-22T02:43:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.538+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7c10e2b0 con 0x7fbe7c102800 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.539+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7c1001d0 con 0x7fbe7c102800 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.539+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fbe6c013790 con 0x7fbe7c102800 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.540+0000 7fbe717fa700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 0x7fbe64070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.540+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe6c08f080 con 0x7fbe7c102800 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.540+0000 7fbe80e33700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 0x7fbe64070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.540+0000 7fbe80e33700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 0x7fbe64070850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe78006fd0 tx=0x7fbe78008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.544+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe6c058300 con 0x7fbe7c102800 2024-03-22T02:43:23.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.761+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7c10efe0 con 0x7fbe6406e3a0 2024-03-22T02:43:23.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:23 smithi103 ceph-mon[115211]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:23.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:23 smithi103 ceph-mon[115211]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:23.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:23 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2084230401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:43:23.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:23 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2543003605' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:43:23.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.763+0000 7fbe717fa700 1 -- 172.21.15.103:0/2424451282 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbe7c10efe0 con 0x7fbe6406e3a0 2024-03-22T02:43:23.763 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:43:23.763 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:43:23.763 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:43:23.764 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:43:23.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 msgr2=0x7fbe64070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 0x7fbe64070850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe78006fd0 tx=0x7fbe78008040 comp rx=0 tx=0).stop 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 msgr2=0x7fbe7c113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009100 tx=0x7fbe6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 shutdown_connections 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbe6406e3a0 0x7fbe64070850 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe7c101e60 0x7fbe7c113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 --2- 172.21.15.103:0/2424451282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe7c102800 0x7fbe7c113f40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 >> 172.21.15.103:0/2424451282 conn(0x7fbe7c071530 msgr2=0x7fbe7c1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 shutdown_connections 2024-03-22T02:43:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.765+0000 7fbe83097700 1 -- 172.21.15.103:0/2424451282 wait complete. 2024-03-22T02:43:23.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 -- 172.21.15.103:0/3361957622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 msgr2=0x7f2b140fbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/3361957622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b140fbb60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2b10004940 tx=0x7f2b10009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 -- 172.21.15.103:0/3361957622 shutdown_connections 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/3361957622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b140fc0a0 0x7f2b1410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/3361957622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b140fbb60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 -- 172.21.15.103:0/3361957622 >> 172.21.15.103:0/3361957622 conn(0x7f2b14071530 msgr2=0x7f2b14071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 -- 172.21.15.103:0/3361957622 shutdown_connections 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.865+0000 7f2b1b52a700 1 -- 172.21.15.103:0/3361957622 wait complete. 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 Processor -- start 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 -- start start 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b140fc0a0 0x7f2b1411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1411abc0 con 0x7f2b140fb790 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141141f0 con 0x7f2b140fc0a0 2024-03-22T02:43:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58600/0 (socket says 172.21.15.103:58600) 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 -- 172.21.15.103:0/1907224200 learned_addr learned my addr 172.21.15.103:0/1907224200 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 -- 172.21.15.103:0/1907224200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b140fc0a0 msgr2=0x7f2b1411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b140fc0a0 0x7f2b1411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 -- 172.21.15.103:0/1907224200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b10005040 con 0x7f2b140fb790 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b192c6700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000b3b0 tx=0x7f2b10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1000e040 con 0x7f2b140fb790 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b1000bbc0 con 0x7f2b140fb790 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b14114470 con 0x7f2b140fb790 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.866+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1001c3f0 con 0x7f2b140fb790 2024-03-22T02:43:23.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.867+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b141149b0 con 0x7f2b140fb790 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.868+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b140fe110 con 0x7f2b140fb790 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.868+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2b1000bd30 con 0x7f2b140fb790 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.868+0000 7f2b09ffb700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 0x7f2b00070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.869+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2b1008f060 con 0x7f2b140fb790 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.869+0000 7f2b18ac5700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 0x7f2b00070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:23.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.869+0000 7f2b18ac5700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 0x7f2b00070850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2b141158d0 tx=0x7f2b04009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:23.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:23.873+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b1005bd90 con 0x7f2b140fb790 2024-03-22T02:43:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:23 smithi122 ceph-mon[83368]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:24.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:23 smithi122 ceph-mon[83368]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:24.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:23 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2084230401' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:43:24.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:23 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2543003605' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:43:24.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.151+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2b14110660 con 0x7f2b140fb790 2024-03-22T02:43:24.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.152+0000 7f2b09ffb700 1 -- 172.21.15.103:0/1907224200 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2b1005b920 con 0x7f2b140fb790 2024-03-22T02:43:24.152 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:43:24.153 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:43:24.153 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:43:24.153 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:43:24.153 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:43:24.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 msgr2=0x7f2b00070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:24.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 0x7f2b00070850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2b141158d0 tx=0x7f2b04009ce0 comp rx=0 tx=0).stop 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 msgr2=0x7f2b1411a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000b3b0 tx=0x7f2b10009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 shutdown_connections 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b0006e3a0 0x7f2b00070850 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b140fb790 0x7f2b1411a030 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 --2- 172.21.15.103:0/1907224200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b140fc0a0 0x7f2b1411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 >> 172.21.15.103:0/1907224200 conn(0x7f2b14071530 msgr2=0x7f2b14102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 shutdown_connections 2024-03-22T02:43:24.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:24.154+0000 7f2b1b52a700 1 -- 172.21.15.103:0/1907224200 wait complete. 2024-03-22T02:43:25.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:24 smithi122 ceph-mon[83368]: pgmap v356: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:24 smithi122 ceph-mon[83368]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:24 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1907224200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:43:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:24 smithi103 ceph-mon[115211]: pgmap v356: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:24 smithi103 ceph-mon[115211]: from='client.34724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:24 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1907224200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:43:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:26 smithi122 ceph-mon[83368]: pgmap v357: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:26 smithi103 ceph-mon[115211]: pgmap v357: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:29.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:28 smithi122 ceph-mon[83368]: pgmap v358: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:28 smithi103 ceph-mon[115211]: pgmap v358: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:30 smithi103 ceph-mon[115211]: pgmap v359: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:30.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:30 smithi122 ceph-mon[83368]: pgmap v359: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:32 smithi122 ceph-mon[83368]: pgmap v360: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:33.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:32 smithi103 ceph-mon[115211]: pgmap v360: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:35.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:34 smithi122 ceph-mon[83368]: pgmap v361: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:34 smithi103 ceph-mon[115211]: pgmap v361: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:36 smithi122 ceph-mon[83368]: pgmap v362: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:36 smithi103 ceph-mon[115211]: pgmap v362: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:38 smithi122 ceph-mon[83368]: pgmap v363: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:39.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:38 smithi103 ceph-mon[115211]: pgmap v363: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:40 smithi103 ceph-mon[115211]: pgmap v364: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:41.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:40 smithi122 ceph-mon[83368]: pgmap v364: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:43.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:42 smithi122 ceph-mon[83368]: pgmap v365: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:42 smithi103 ceph-mon[115211]: pgmap v365: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:44 smithi122 ceph-mon[83368]: pgmap v366: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:44 smithi103 ceph-mon[115211]: pgmap v366: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:47.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:46 smithi122 ceph-mon[83368]: pgmap v367: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:46 smithi103 ceph-mon[115211]: pgmap v367: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:43:49.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:48 smithi122 ceph-mon[83368]: pgmap v368: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:48 smithi103 ceph-mon[115211]: pgmap v368: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:50 smithi103 ceph-mon[115211]: pgmap v369: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:50 smithi122 ceph-mon[83368]: pgmap v369: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:53.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:52 smithi122 ceph-mon[83368]: pgmap v370: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:52 smithi103 ceph-mon[115211]: pgmap v370: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 -- 172.21.15.103:0/1147091474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b81026c0 msgr2=0x7f12b80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 --2- 172.21.15.103:0/1147091474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b81026c0 0x7f12b80fbd30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f12a8004940 tx=0x7f12a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 -- 172.21.15.103:0/1147091474 shutdown_connections 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 --2- 172.21.15.103:0/1147091474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b81026c0 0x7f12b80fbd30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 --2- 172.21.15.103:0/1147091474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b8101d20 0x7f12b81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 -- 172.21.15.103:0/1147091474 >> 172.21.15.103:0/1147091474 conn(0x7f12b80f7870 msgr2=0x7f12b80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 -- 172.21.15.103:0/1147091474 shutdown_connections 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.274+0000 7f12bf984700 1 -- 172.21.15.103:0/1147091474 wait complete. 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 Processor -- start 2024-03-22T02:43:54.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 -- start start 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 0x7f12b8112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81134d0 con 0x7f12b8101d20 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bf984700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b810ca40 con 0x7f12b81026c0 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58562/0 (socket says 172.21.15.103:58562) 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 -- 172.21.15.103:0/473458023 learned_addr learned my addr 172.21.15.103:0/473458023 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bcf1f700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 0x7f12b8112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 -- 172.21.15.103:0/473458023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 msgr2=0x7f12b8112df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 0x7f12b8112df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 -- 172.21.15.103:0/473458023 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8005040 con 0x7f12b8101d20 2024-03-22T02:43:54.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bcf1f700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 0x7f12b8112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.275+0000 7f12bd720700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b790 tx=0x7f12b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.276+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12b400f770 con 0x7f12b8101d20 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.276+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12b400fda0 con 0x7f12b8101d20 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.276+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12b400d310 con 0x7f12b8101d20 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.276+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b810cd20 con 0x7f12b8101d20 2024-03-22T02:43:54.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.276+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b810d2e0 con 0x7f12b8101d20 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.277+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b80fd470 con 0x7f12b8101d20 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.277+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f12b4010420 con 0x7f12b8101d20 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.278+0000 7f12adffb700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 0x7f12a4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.278+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f12b408dd10 con 0x7f12b8101d20 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.278+0000 7f12bcf1f700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 0x7f12a4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.278+0000 7f12bcf1f700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 0x7f12a4070850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f12b810dc00 tx=0x7f12a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.282+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12b405ab80 con 0x7f12b8101d20 2024-03-22T02:43:54.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.496+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f12b8108ee0 con 0x7f12a406e3a0 2024-03-22T02:43:54.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.497+0000 7f12adffb700 1 -- 172.21.15.103:0/473458023 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f12b8108ee0 con 0x7f12a406e3a0 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 msgr2=0x7f12a4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 0x7f12a4070850 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f12b810dc00 tx=0x7f12a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 msgr2=0x7f12b81128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b790 tx=0x7f12b4009250 comp rx=0 tx=0).stop 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 shutdown_connections 2024-03-22T02:43:54.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f12a406e3a0 0x7f12a4070850 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12b8101d20 0x7f12b81128b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 --2- 172.21.15.103:0/473458023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12b81026c0 0x7f12b8112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.499+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 >> 172.21.15.103:0/473458023 conn(0x7f12b80f7870 msgr2=0x7f12b80f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:54.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.500+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 shutdown_connections 2024-03-22T02:43:54.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.500+0000 7f12bf984700 1 -- 172.21.15.103:0/473458023 wait complete. 2024-03-22T02:43:54.511 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:43:54.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3770444991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b41026b0 msgr2=0x7f49b40fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3770444991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b41026b0 0x7f49b40fbd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f49a4004940 tx=0x7f49a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:54.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3770444991 shutdown_connections 2024-03-22T02:43:54.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3770444991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b41026b0 0x7f49b40fbd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3770444991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b4101d10 0x7f49b41020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3770444991 >> 172.21.15.103:0/3770444991 conn(0x7f49b40f7880 msgr2=0x7f49b40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3770444991 shutdown_connections 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.602+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3770444991 wait complete. 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 Processor -- start 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 -- start start 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b41026b0 0x7f49b41143c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b4114a10 con 0x7f49b4101d10 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49bc6e5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b410de30 con 0x7f49b41026b0 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58592/0 (socket says 172.21.15.103:58592) 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 -- 172.21.15.103:0/3671278407 learned_addr learned my addr 172.21.15.103:0/3671278407 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49b9c80700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b41026b0 0x7f49b41143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 -- 172.21.15.103:0/3671278407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b41026b0 msgr2=0x7f49b41143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b41026b0 0x7f49b41143c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.603+0000 7f49ba481700 1 -- 172.21.15.103:0/3671278407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4005040 con 0x7f49b4101d10 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49ba481700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49b0001d80 tx=0x7f49b0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b0007d10 con 0x7f49b4101d10 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49b0001e80 con 0x7f49b4101d10 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b410e110 con 0x7f49b4101d10 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b0014640 con 0x7f49b4101d10 2024-03-22T02:43:54.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.604+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b410e6a0 con 0x7f49b4101d10 2024-03-22T02:43:54.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.605+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b41041e0 con 0x7f49b4101d10 2024-03-22T02:43:54.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.605+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f49b000e960 con 0x7f49b4101d10 2024-03-22T02:43:54.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.606+0000 7f49aaffd700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 0x7f49a0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.606+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49b008db70 con 0x7f49b4101d10 2024-03-22T02:43:54.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.606+0000 7f49b9c80700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 0x7f49a0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.606+0000 7f49b9c80700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 0x7f49a0070850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49a4004890 tx=0x7f49a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.610+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49b005a950 con 0x7f49b4101d10 2024-03-22T02:43:54.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.827+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49b4109ee0 con 0x7f49a006e3a0 2024-03-22T02:43:54.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.829+0000 7f49aaffd700 1 -- 172.21.15.103:0/3671278407 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f49b4109ee0 con 0x7f49a006e3a0 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 msgr2=0x7f49a0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 0x7f49a0070850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49a4004890 tx=0x7f49a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 msgr2=0x7f49b4113e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49b0001d80 tx=0x7f49b0009ec0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 shutdown_connections 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49a006e3a0 0x7f49a0070850 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49b4101d10 0x7f49b4113e80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 --2- 172.21.15.103:0/3671278407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49b41026b0 0x7f49b41143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 >> 172.21.15.103:0/3671278407 conn(0x7f49b40f7880 msgr2=0x7f49b4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 shutdown_connections 2024-03-22T02:43:54.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.831+0000 7f49bc6e5700 1 -- 172.21.15.103:0/3671278407 wait complete. 2024-03-22T02:43:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:54 smithi103 ceph-mon[115211]: pgmap v371: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:54.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.932+0000 7fb1df57c700 1 -- 172.21.15.103:0/2178122842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d8102f50 msgr2=0x7fb1d8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.932+0000 7fb1df57c700 1 --2- 172.21.15.103:0/2178122842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d8102f50 0x7fb1d8103320 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4004940 tx=0x7fb1d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:54.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- 172.21.15.103:0/2178122842 shutdown_connections 2024-03-22T02:43:54.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 --2- 172.21.15.103:0/2178122842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d80fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 --2- 172.21.15.103:0/2178122842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d8102f50 0x7fb1d8103320 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- 172.21.15.103:0/2178122842 >> 172.21.15.103:0/2178122842 conn(0x7fb1d80714c0 msgr2=0x7fb1d80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- 172.21.15.103:0/2178122842 shutdown_connections 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- 172.21.15.103:0/2178122842 wait complete. 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 Processor -- start 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- start start 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d8102f50 0x7fb1d8113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81145b0 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1df57c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d810db90 con 0x7fb1d8102f50 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.933+0000 7fb1dd318700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58602/0 (socket says 172.21.15.103:58602) 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 -- 172.21.15.103:0/1165988034 learned_addr learned my addr 172.21.15.103:0/1165988034 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dcb17700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d8102f50 0x7fb1d8113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 -- 172.21.15.103:0/1165988034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d8102f50 msgr2=0x7fb1d8113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d8102f50 0x7fb1d8113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 -- 172.21.15.103:0/1165988034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4005040 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1dd318700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009100 tx=0x7fb1d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d400e040 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d810de10 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1d4009af0 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d401c3f0 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.934+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d810e350 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.935+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d8100200 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.936+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb1d400baf0 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.936+0000 7fb1cdffb700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 0x7fb1c4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.936+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb1d408f030 con 0x7fb1d80fb4d0 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.936+0000 7fb1dcb17700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 0x7fb1c4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:54.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.937+0000 7fb1dcb17700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 0x7fb1c4070850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1d810f2c0 tx=0x7fb1c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:54.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:54.941+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1d405bde0 con 0x7fb1d80fb4d0 2024-03-22T02:43:55.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:54 smithi122 ceph-mon[83368]: pgmap v371: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:55.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.147+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1d810a110 con 0x7fb1c406e3a0 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.157+0000 7fb1cdffb700 1 -- 172.21.15.103:0/1165988034 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb1d810a110 con 0x7fb1c406e3a0 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (12m) 6m ago 23m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (10m) 6m ago 23m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (10m) 5m ago 22m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (12m) 6m ago 23m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:43:55.159 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (6m) 6m ago 18m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (6m) 6m ago 18m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (5m) 5m ago 17m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (5m) 5m ago 18m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (14m) 6m ago 25m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (13m) 5m ago 22m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (11m) 6m ago 25m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (10m) 5m ago 22m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (13m) 6m ago 23m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (13m) 5m ago 22m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (9m) 6m ago 21m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (9m) 6m ago 21m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (8m) 6m ago 20m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (7m) 5m ago 20m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (7m) 5m ago 20m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:43:55.160 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (6m) 5m ago 19m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:43:55.161 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (12m) 6m ago 23m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:43:55.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.159+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 msgr2=0x7fb1c4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.159+0000 7fb1df57c700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 0x7fb1c4070850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1d810f2c0 tx=0x7fb1c800b040 comp rx=0 tx=0).stop 2024-03-22T02:43:55.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.159+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 msgr2=0x7fb1d8113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.159+0000 7fb1df57c700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009100 tx=0x7fb1d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 shutdown_connections 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb1c406e3a0 0x7fb1c4070850 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1d80fb4d0 0x7fb1d8113a20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 --2- 172.21.15.103:0/1165988034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb1d8102f50 0x7fb1d8113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 >> 172.21.15.103:0/1165988034 conn(0x7fb1d80714c0 msgr2=0x7fb1d80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 shutdown_connections 2024-03-22T02:43:55.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.160+0000 7fb1df57c700 1 -- 172.21.15.103:0/1165988034 wait complete. 2024-03-22T02:43:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.259+0000 7f997cf59700 1 -- 172.21.15.103:0/619555612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 msgr2=0x7f99780fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.259+0000 7f997cf59700 1 --2- 172.21.15.103:0/619555612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99780fce20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f996c004940 tx=0x7f996c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- 172.21.15.103:0/619555612 shutdown_connections 2024-03-22T02:43:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 --2- 172.21.15.103:0/619555612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99780fce20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 --2- 172.21.15.103:0/619555612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 0x7f99781031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- 172.21.15.103:0/619555612 >> 172.21.15.103:0/619555612 conn(0x7f99780f8980 msgr2=0x7f99780fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- 172.21.15.103:0/619555612 shutdown_connections 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- 172.21.15.103:0/619555612 wait complete. 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 Processor -- start 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- start start 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 0x7f9978113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978114700 con 0x7f99781037b0 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997cf59700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997810db40 con 0x7f9978102e10 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f9975d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f9975d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58620/0 (socket says 172.21.15.103:58620) 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f9975d9b700 1 -- 172.21.15.103:0/144731114 learned_addr learned my addr 172.21.15.103:0/144731114 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.260+0000 7f997659c700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 0x7f9978113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9975d9b700 1 -- 172.21.15.103:0/144731114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 msgr2=0x7f9978113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9975d9b700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 0x7f9978113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9975d9b700 1 -- 172.21.15.103:0/144731114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c005040 con 0x7f99781037b0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9975d9b700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f996c009100 tx=0x7f996c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f996c00e040 con 0x7f99781037b0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f996c004290 con 0x7f99781037b0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997810ddc0 con 0x7f99781037b0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f996c01c3f0 con 0x7f99781037b0 2024-03-22T02:43:55.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.261+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997810e380 con 0x7f99781037b0 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.262+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99781052e0 con 0x7f99781037b0 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.263+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f996c013690 con 0x7f99781037b0 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.263+0000 7f9966ffd700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 0x7f9960070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.263+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f996c08ef90 con 0x7f99781037b0 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.263+0000 7f997659c700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 0x7f9960070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.264+0000 7f997659c700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 0x7f9960070860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9970007a40 tx=0x7f9970009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:55.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.268+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f996c0c3740 con 0x7f99781037b0 2024-03-22T02:43:55.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.545+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f997810f1c0 con 0x7f99781037b0 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.546+0000 7f9966ffd700 1 -- 172.21.15.103:0/144731114 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f996c05bd00 con 0x7f99781037b0 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:43:55.548 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:43:55.549 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:43:55.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 msgr2=0x7f9960070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 0x7f9960070860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9970007a40 tx=0x7f9970009da0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 msgr2=0x7f99781140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f996c009100 tx=0x7f996c009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 shutdown_connections 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f996006e3b0 0x7f9960070860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9978102e10 0x7f9978113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 --2- 172.21.15.103:0/144731114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99781037b0 0x7f99781140b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 >> 172.21.15.103:0/144731114 conn(0x7f99780f8980 msgr2=0x7f99781021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 shutdown_connections 2024-03-22T02:43:55.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.548+0000 7f997cf59700 1 -- 172.21.15.103:0/144731114 wait complete. 2024-03-22T02:43:55.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- 172.21.15.103:0/3095469626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c102e10 msgr2=0x7fe24c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 --2- 172.21.15.103:0/3095469626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c102e10 0x7fe24c1031e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe244009ce0 tx=0x7fe244004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- 172.21.15.103:0/3095469626 shutdown_connections 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 --2- 172.21.15.103:0/3095469626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c1037b0 0x7fe24c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 --2- 172.21.15.103:0/3095469626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c102e10 0x7fe24c1031e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- 172.21.15.103:0/3095469626 >> 172.21.15.103:0/3095469626 conn(0x7fe24c0f89a0 msgr2=0x7fe24c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- 172.21.15.103:0/3095469626 shutdown_connections 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- 172.21.15.103:0/3095469626 wait complete. 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 Processor -- start 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.649+0000 7fe2540da700 1 -- start start 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c102e10 0x7fe24c113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c114720 con 0x7fe24c1037b0 2024-03-22T02:43:55.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c10db80 con 0x7fe24c102e10 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58634/0 (socket says 172.21.15.103:58634) 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 -- 172.21.15.103:0/2214925679 learned_addr learned my addr 172.21.15.103:0/2214925679 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251e76700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c102e10 0x7fe24c113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 -- 172.21.15.103:0/2214925679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c102e10 msgr2=0x7fe24c113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c102e10 0x7fe24c113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 -- 172.21.15.103:0/2214925679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244009940 con 0x7fe24c1037b0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe251675700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe23c00eb40 tx=0x7fe23c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe23c007cc0 con 0x7fe24c1037b0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c10de00 con 0x7fe24c1037b0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe23c00f040 con 0x7fe24c1037b0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe23c00c9d0 con 0x7fe24c1037b0 2024-03-22T02:43:55.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.650+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c10e3c0 con 0x7fe24c1037b0 2024-03-22T02:43:55.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.652+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c100120 con 0x7fe24c1037b0 2024-03-22T02:43:55.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.652+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe23c00cc40 con 0x7fe24c1037b0 2024-03-22T02:43:55.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.652+0000 7fe2427fc700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 0x7fe238070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:55.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.652+0000 7fe251e76700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 0x7fe238070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:55.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.652+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe23c08d830 con 0x7fe24c1037b0 2024-03-22T02:43:55.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.653+0000 7fe251e76700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 0x7fe238070850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe244004f80 tx=0x7fe244004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:55.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.657+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe23c056ae0 con 0x7fe24c1037b0 2024-03-22T02:43:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:55 smithi103 ceph-mon[115211]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:55 smithi103 ceph-mon[115211]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:55 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/144731114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:43:55.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.900+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe24c10f1e0 con 0x7fe24c1037b0 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.901+0000 7fe2427fc700 1 -- 172.21.15.103:0/2214925679 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fe23c08be60 con 0x7fe24c1037b0 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:43:55.903 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:43:55.904 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:55.905 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:43:55.906 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:43:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 msgr2=0x7fe238070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 0x7fe238070850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe244004f80 tx=0x7fe244004d10 comp rx=0 tx=0).stop 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 msgr2=0x7fe24c1140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe23c00eb40 tx=0x7fe23c00b040 comp rx=0 tx=0).stop 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 shutdown_connections 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe23806e3a0 0x7fe238070850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe24c102e10 0x7fe24c113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 --2- 172.21.15.103:0/2214925679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe24c1037b0 0x7fe24c1140d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 >> 172.21.15.103:0/2214925679 conn(0x7fe24c0f89a0 msgr2=0x7fe24c0facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 shutdown_connections 2024-03-22T02:43:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:55.903+0000 7fe2540da700 1 -- 172.21.15.103:0/2214925679 wait complete. 2024-03-22T02:43:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:43:56.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/568054418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 msgr2=0x7fcf340fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/568054418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf340fce10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcf24004940 tx=0x7fcf24009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/568054418 shutdown_connections 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/568054418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf340fce10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/568054418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf341031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/568054418 >> 172.21.15.103:0/568054418 conn(0x7fcf340f89b0 msgr2=0x7fcf340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/568054418 shutdown_connections 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.004+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/568054418 wait complete. 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 Processor -- start 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 -- start start 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf34113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34114510 con 0x7fcf341037a0 2024-03-22T02:43:56.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3ba8f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410db10 con 0x7fcf34102e00 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3902a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3902a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58646/0 (socket says 172.21.15.103:58646) 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3902a700 1 -- 172.21.15.103:0/2203299841 learned_addr learned my addr 172.21.15.103:0/2203299841 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3982b700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf34113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3902a700 1 -- 172.21.15.103:0/2203299841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 msgr2=0x7fcf34113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.005+0000 7fcf3902a700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf34113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf3902a700 1 -- 172.21.15.103:0/2203299841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24005040 con 0x7fcf341037a0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf3982b700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf34113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf3902a700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009100 tx=0x7fcf24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf2400e040 con 0x7fcf341037a0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf24013420 con 0x7fcf341037a0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3410dd90 con 0x7fcf341037a0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf2401c610 con 0x7fcf341037a0 2024-03-22T02:43:56.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.006+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3410e350 con 0x7fcf341037a0 2024-03-22T02:43:56.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.007+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf341052d0 con 0x7fcf341037a0 2024-03-22T02:43:56.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.007+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcf24013590 con 0x7fcf341037a0 2024-03-22T02:43:56.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.008+0000 7fcf2a7fc700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 0x7fcf20070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.008+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcf2408f090 con 0x7fcf341037a0 2024-03-22T02:43:56.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.008+0000 7fcf3982b700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 0x7fcf20070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:56.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.008+0000 7fcf3982b700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 0x7fcf20070850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf300041d0 tx=0x7fcf30009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:56.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.012+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf2405be40 con 0x7fcf341037a0 2024-03-22T02:43:56.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:55 smithi122 ceph-mon[83368]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:56.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:55 smithi122 ceph-mon[83368]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:56.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:55 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/144731114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:43:56.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.230+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf3410efd0 con 0x7fcf2006e3a0 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.231+0000 7fcf2a7fc700 1 -- 172.21.15.103:0/2203299841 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcf3410efd0 con 0x7fcf2006e3a0 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:43:56.233 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:43:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 msgr2=0x7fcf20070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 0x7fcf20070850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf300041d0 tx=0x7fcf30009040 comp rx=0 tx=0).stop 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 msgr2=0x7fcf34113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009100 tx=0x7fcf24009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 shutdown_connections 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcf2006e3a0 0x7fcf20070850 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf34102e00 0x7fcf34113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 --2- 172.21.15.103:0/2203299841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf341037a0 0x7fcf34113ec0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 >> 172.21.15.103:0/2203299841 conn(0x7fcf340f89b0 msgr2=0x7fcf341021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:56.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 shutdown_connections 2024-03-22T02:43:56.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.233+0000 7fcf3ba8f700 1 -- 172.21.15.103:0/2203299841 wait complete. 2024-03-22T02:43:56.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 -- 172.21.15.103:0/1695653016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fe770 msgr2=0x7f66cc106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 --2- 172.21.15.103:0/1695653016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fe770 0x7f66cc106e40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f66bc004940 tx=0x7f66bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:56.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 -- 172.21.15.103:0/1695653016 shutdown_connections 2024-03-22T02:43:56.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 --2- 172.21.15.103:0/1695653016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fe770 0x7f66cc106e40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 --2- 172.21.15.103:0/1695653016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 -- 172.21.15.103:0/1695653016 >> 172.21.15.103:0/1695653016 conn(0x7f66cc0f7890 msgr2=0x7f66cc0f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:56.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 -- 172.21.15.103:0/1695653016 shutdown_connections 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.333+0000 7f66d25a4700 1 -- 172.21.15.103:0/1695653016 wait complete. 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 Processor -- start 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 -- start start 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66cc0fe770 0x7f66cc0fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0fdc70 con 0x7f66cc0fddd0 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66d25a4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0fc710 con 0x7f66cc0fe770 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:56.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58662/0 (socket says 172.21.15.103:58662) 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.334+0000 7f66cbfff700 1 -- 172.21.15.103:0/4095649305 learned_addr learned my addr 172.21.15.103:0/4095649305 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66cbfff700 1 -- 172.21.15.103:0/4095649305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66cc0fe770 msgr2=0x7f66cc0fc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66cbfff700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66cc0fe770 0x7f66cc0fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66cbfff700 1 -- 172.21.15.103:0/4095649305 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc005040 con 0x7f66cc0fddd0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66cbfff700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f66c000b790 tx=0x7f66c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66c000f770 con 0x7f66cc0fddd0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66c000fda0 con 0x7f66cc0fddd0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc0fc9f0 con 0x7f66cc0fddd0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66c000d310 con 0x7f66cc0fddd0 2024-03-22T02:43:56.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.335+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc11a140 con 0x7f66cc0fddd0 2024-03-22T02:43:56.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.336+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc066b50 con 0x7f66cc0fddd0 2024-03-22T02:43:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.336+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f66c0010420 con 0x7f66cc0fddd0 2024-03-22T02:43:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.337+0000 7f66c8ff9700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 0x7f66b4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:43:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.337+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f66c008ddf0 con 0x7f66cc0fddd0 2024-03-22T02:43:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.337+0000 7f66cb7fe700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 0x7f66b4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:43:56.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.337+0000 7f66cb7fe700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 0x7f66b4070850 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f66bc004f90 tx=0x7f66bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:43:56.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.341+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66c0057070 con 0x7f66cc0fddd0 2024-03-22T02:43:56.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.620+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f66cc0fd170 con 0x7f66cc0fddd0 2024-03-22T02:43:56.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.620+0000 7f66c8ff9700 1 -- 172.21.15.103:0/4095649305 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f66c005a730 con 0x7f66cc0fddd0 2024-03-22T02:43:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:43:56.622 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:43:56.622 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:43:56.623 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:43:56.623 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 msgr2=0x7f66b4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 0x7f66b4070850 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f66bc004f90 tx=0x7f66bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 msgr2=0x7f66cc0fbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f66c000b790 tx=0x7f66c0009250 comp rx=0 tx=0).stop 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 shutdown_connections 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f66b406e3a0 0x7f66b4070850 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66cc0fddd0 0x7f66cc0fbc90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 --2- 172.21.15.103:0/4095649305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66cc0fe770 0x7f66cc0fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:43:56.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.622+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 >> 172.21.15.103:0/4095649305 conn(0x7f66cc0f7890 msgr2=0x7f66cc0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:43:56.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.623+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 shutdown_connections 2024-03-22T02:43:56.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:43:56.623+0000 7f66d25a4700 1 -- 172.21.15.103:0/4095649305 wait complete. 2024-03-22T02:43:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:56 smithi103 ceph-mon[115211]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:56 smithi103 ceph-mon[115211]: pgmap v372: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:56 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2214925679' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:43:56.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:56 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4095649305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:43:57.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:56 smithi122 ceph-mon[83368]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:57.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:56 smithi122 ceph-mon[83368]: pgmap v372: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:57.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:56 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2214925679' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:43:57.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:56 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4095649305' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:43:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:57 smithi122 ceph-mon[83368]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:58.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:57 smithi103 ceph-mon[115211]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:43:59.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:43:58 smithi122 ceph-mon[83368]: pgmap v373: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:43:59.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:43:58 smithi103 ceph-mon[115211]: pgmap v373: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:00 smithi103 ceph-mon[115211]: pgmap v374: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:00 smithi122 ceph-mon[83368]: pgmap v374: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:02 smithi122 ceph-mon[83368]: pgmap v375: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:02 smithi103 ceph-mon[115211]: pgmap v375: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:04 smithi122 ceph-mon[83368]: pgmap v376: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:04 smithi103 ceph-mon[115211]: pgmap v376: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:07.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:06 smithi122 ceph-mon[83368]: pgmap v377: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:06 smithi103 ceph-mon[115211]: pgmap v377: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:09.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:08 smithi122 ceph-mon[83368]: pgmap v378: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:08 smithi103 ceph-mon[115211]: pgmap v378: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:10 smithi103 ceph-mon[115211]: pgmap v379: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:10 smithi122 ceph-mon[83368]: pgmap v379: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:12 smithi122 ceph-mon[83368]: pgmap v380: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:12 smithi103 ceph-mon[115211]: pgmap v380: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:15.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:14 smithi122 ceph-mon[83368]: pgmap v381: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:14 smithi103 ceph-mon[115211]: pgmap v381: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:16 smithi122 ceph-mon[83368]: pgmap v382: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:16 smithi103 ceph-mon[115211]: pgmap v382: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:18 smithi122 ceph-mon[83368]: pgmap v383: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:18 smithi103 ceph-mon[115211]: pgmap v383: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:20 smithi103 ceph-mon[115211]: pgmap v384: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:20 smithi122 ceph-mon[83368]: pgmap v384: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:22 smithi122 ceph-mon[83368]: pgmap v385: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:22 smithi103 ceph-mon[115211]: pgmap v385: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:25.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:24 smithi122 ceph-mon[83368]: pgmap v386: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:24 smithi103 ceph-mon[115211]: pgmap v386: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 -- 172.21.15.103:0/2950013856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc101d70 msgr2=0x7f39dc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 --2- 172.21.15.103:0/2950013856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc101d70 0x7f39dc102140 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f39cc004940 tx=0x7f39cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 -- 172.21.15.103:0/2950013856 shutdown_connections 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 --2- 172.21.15.103:0/2950013856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc102710 0x7f39dc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 --2- 172.21.15.103:0/2950013856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc101d70 0x7f39dc102140 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 -- 172.21.15.103:0/2950013856 >> 172.21.15.103:0/2950013856 conn(0x7f39dc072bc0 msgr2=0x7f39dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:26.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 -- 172.21.15.103:0/2950013856 shutdown_connections 2024-03-22T02:44:26.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 -- 172.21.15.103:0/2950013856 wait complete. 2024-03-22T02:44:26.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.737+0000 7f39e27e5700 1 Processor -- start 2024-03-22T02:44:26.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39e27e5700 1 -- start start 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39e27e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc101d70 0x7f39dc10eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39e27e5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39e27e5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc10f690 con 0x7f39dc102710 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39e27e5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc10f7d0 con 0x7f39dc101d70 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43672/0 (socket says 172.21.15.103:43672) 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 -- 172.21.15.103:0/1880930235 learned_addr learned my addr 172.21.15.103:0/1880930235 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39dbfff700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc101d70 0x7f39dc10eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:26.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 -- 172.21.15.103:0/1880930235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc101d70 msgr2=0x7f39dc10eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc101d70 0x7f39dc10eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 -- 172.21.15.103:0/1880930235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc005040 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39db7fe700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f39d0002a80 tx=0x7f39d0004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.738+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d0004e80 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.739+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39d000f040 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.739+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d000daa0 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.739+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc117b70 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.739+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc118050 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.740+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc0ff080 con 0x7f39dc102710 2024-03-22T02:44:26.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.740+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f39d0007cc0 con 0x7f39dc102710 2024-03-22T02:44:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.740+0000 7f39d8ff9700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 0x7f39c4070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.740+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f39d008d350 con 0x7f39dc102710 2024-03-22T02:44:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.740+0000 7f39dbfff700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 0x7f39c4070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:26.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.741+0000 7f39dbfff700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 0x7f39c4070860 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00b3b0 tx=0x7f39cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:26.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.745+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39d00565d0 con 0x7f39dc102710 2024-03-22T02:44:26.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.964+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39dc108f00 con 0x7f39c406e3b0 2024-03-22T02:44:26.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.965+0000 7f39d8ff9700 1 -- 172.21.15.103:0/1880930235 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f39dc108f00 con 0x7f39c406e3b0 2024-03-22T02:44:26.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 msgr2=0x7f39c4070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:26.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 0x7f39c4070860 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00b3b0 tx=0x7f39cc009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:26.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 msgr2=0x7f39dc10f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:26.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f39d0002a80 tx=0x7f39d0004630 comp rx=0 tx=0).stop 2024-03-22T02:44:26.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 shutdown_connections 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f39c406e3b0 0x7f39c4070860 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39dc101d70 0x7f39dc10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 --2- 172.21.15.103:0/1880930235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39dc102710 0x7f39dc10f040 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 >> 172.21.15.103:0/1880930235 conn(0x7f39dc072bc0 msgr2=0x7f39dc0fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 shutdown_connections 2024-03-22T02:44:26.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:26.967+0000 7f39e27e5700 1 -- 172.21.15.103:0/1880930235 wait complete. 2024-03-22T02:44:26.978 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 -- 172.21.15.103:0/2632934978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 msgr2=0x7f76ac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 --2- 172.21.15.103:0/2632934978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac102140 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009ce0 tx=0x7f76a8004d10 comp rx=0 tx=0).stop 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 -- 172.21.15.103:0/2632934978 shutdown_connections 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 --2- 172.21.15.103:0/2632934978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 --2- 172.21.15.103:0/2632934978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac102140 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 -- 172.21.15.103:0/2632934978 >> 172.21.15.103:0/2632934978 conn(0x7f76ac072bc0 msgr2=0x7f76ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:27.072 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:26 smithi103 ceph-mon[115211]: pgmap v387: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:27.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 -- 172.21.15.103:0/2632934978 shutdown_connections 2024-03-22T02:44:27.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.070+0000 7f76b3634700 1 -- 172.21.15.103:0/2632934978 wait complete. 2024-03-22T02:44:27.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 Processor -- start 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 -- start start 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac114230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac114880 con 0x7f76ac101d70 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b3634700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac10dc50 con 0x7f76ac102710 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43688/0 (socket says 172.21.15.103:43688) 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 -- 172.21.15.103:0/120665433 learned_addr learned my addr 172.21.15.103:0/120665433 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b0bcf700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac114230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 -- 172.21.15.103:0/120665433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 msgr2=0x7f76ac114230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac114230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b13d0700 1 -- 172.21.15.103:0/120665433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8009940 con 0x7f76ac101d70 2024-03-22T02:44:27.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.071+0000 7f76b0bcf700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76b13d0700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f76a800be90 tx=0x7f76a8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a800e040 con 0x7f76ac101d70 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76a8008ea0 con 0x7f76ac101d70 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac10ded0 con 0x7f76ac101d70 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76a801c810 con 0x7f76ac101d70 2024-03-22T02:44:27.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.072+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac10e490 con 0x7f76ac101d70 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.073+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac0fd4c0 con 0x7f76ac101d70 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.074+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f76a8012c00 con 0x7f76ac101d70 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.074+0000 7f76a1ffb700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 0x7f7698070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.074+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f76a808f9c0 con 0x7f76ac101d70 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.074+0000 7f76b0bcf700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 0x7f7698070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.074+0000 7f76b0bcf700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 0x7f7698070850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f769c007fd0 tx=0x7f769c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.079+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f76a805c6f0 con 0x7f76ac101d70 2024-03-22T02:44:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:26 smithi122 ceph-mon[83368]: pgmap v387: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:27.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.297+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f76ac108f10 con 0x7f769806e3a0 2024-03-22T02:44:27.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.298+0000 7f76a1ffb700 1 -- 172.21.15.103:0/120665433 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f769c004920 con 0x7f769806e3a0 2024-03-22T02:44:27.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 msgr2=0x7f7698070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 0x7f7698070850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f769c007fd0 tx=0x7f769c009040 comp rx=0 tx=0).stop 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 msgr2=0x7f76ac113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f76a800be90 tx=0x7f76a8004d10 comp rx=0 tx=0).stop 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 shutdown_connections 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f769806e3a0 0x7f7698070850 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76ac101d70 0x7f76ac113cf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 --2- 172.21.15.103:0/120665433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76ac102710 0x7f76ac114230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.300+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 >> 172.21.15.103:0/120665433 conn(0x7f76ac072bc0 msgr2=0x7f76ac065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.301+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 shutdown_connections 2024-03-22T02:44:27.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.301+0000 7f76b3634700 1 -- 172.21.15.103:0/120665433 wait complete. 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 -- 172.21.15.103:0/277533868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 msgr2=0x7f59b80fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 --2- 172.21.15.103:0/277533868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b80fb910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f59a8004940 tx=0x7f59a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 -- 172.21.15.103:0/277533868 shutdown_connections 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 --2- 172.21.15.103:0/277533868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b80fb910 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 --2- 172.21.15.103:0/277533868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 0x7f59b8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 -- 172.21.15.103:0/277533868 >> 172.21.15.103:0/277533868 conn(0x7f59b80714c0 msgr2=0x7f59b80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 -- 172.21.15.103:0/277533868 shutdown_connections 2024-03-22T02:44:27.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 -- 172.21.15.103:0/277533868 wait complete. 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.402+0000 7f59bfdaa700 1 Processor -- start 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bfdaa700 1 -- start start 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bfdaa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bfdaa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 0x7f59b8114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bfdaa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b8114730 con 0x7f59b80fb4a0 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bfdaa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b810da90 con 0x7f59b8102e70 2024-03-22T02:44:27.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43716/0 (socket says 172.21.15.103:43716) 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 -- 172.21.15.103:0/1991371551 learned_addr learned my addr 172.21.15.103:0/1991371551 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bd345700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 0x7f59b8114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 -- 172.21.15.103:0/1991371551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 msgr2=0x7f59b8114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 0x7f59b8114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 -- 172.21.15.103:0/1991371551 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8005040 con 0x7f59b80fb4a0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59bdb46700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f59b000bbb0 tx=0x7f59b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.403+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b00092e0 con 0x7f59b80fb4a0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.404+0000 7f59bfdaa700 1 -- 172.21.15.103:0/1991371551 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b810dd70 con 0x7f59b80fb4a0 2024-03-22T02:44:27.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.404+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59b000cbd0 con 0x7f59b80fb4a0 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.404+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59b0007400 con 0x7f59b80fb4a0 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.404+0000 7f59bfdaa700 1 -- 172.21.15.103:0/1991371551 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b810e330 con 0x7f59b80fb4a0 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.405+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b8065380 con 0x7f59b80fb4a0 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.405+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f59b0012070 con 0x7f59b80fb4a0 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.405+0000 7f59ae7fc700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 0x7f59a4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.406+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f59b008db50 con 0x7f59b80fb4a0 2024-03-22T02:44:27.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.406+0000 7f59bd345700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 0x7f59a4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.406+0000 7f59bd345700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 0x7f59a4070850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f59a800b3b0 tx=0x7f59a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.410+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59b005a800 con 0x7f59b80fb4a0 2024-03-22T02:44:27.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.614+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59b810f200 con 0x7f59a406e3a0 2024-03-22T02:44:27.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.624+0000 7f59ae7fc700 1 -- 172.21.15.103:0/1991371551 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f59b810f200 con 0x7f59a406e3a0 2024-03-22T02:44:27.624 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:44:27.624 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (13m) 6m ago 24m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (10m) 6m ago 24m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (10m) 6m ago 23m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (12m) 6m ago 23m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (7m) 6m ago 18m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (6m) 6m ago 18m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (6m) 6m ago 18m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (6m) 6m ago 18m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (14m) 6m ago 25m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (13m) 6m ago 23m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (11m) 6m ago 25m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (11m) 6m ago 23m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (13m) 6m ago 24m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (13m) 6m ago 23m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (10m) 6m ago 22m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (9m) 6m ago 21m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:44:27.625 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (9m) 6m ago 21m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:44:27.626 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (8m) 6m ago 21m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:44:27.626 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (7m) 6m ago 20m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:44:27.626 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (7m) 6m ago 20m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:44:27.626 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (13m) 6m ago 23m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 msgr2=0x7f59a4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 0x7f59a4070850 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f59a800b3b0 tx=0x7f59a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 msgr2=0x7f59b8113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f59b000bbb0 tx=0x7f59b0009840 comp rx=0 tx=0).stop 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 shutdown_connections 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f59a406e3a0 0x7f59a4070850 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59b80fb4a0 0x7f59b8113b10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 --2- 172.21.15.103:0/1991371551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59b8102e70 0x7f59b8114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 >> 172.21.15.103:0/1991371551 conn(0x7f59b80714c0 msgr2=0x7f59b80fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:27.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 shutdown_connections 2024-03-22T02:44:27.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.626+0000 7f59a3fff700 1 -- 172.21.15.103:0/1991371551 wait complete. 2024-03-22T02:44:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.725+0000 7f79fff9b700 1 -- 172.21.15.103:0/1403825988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fe7c0 msgr2=0x7f79f8106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.725+0000 7f79fff9b700 1 --2- 172.21.15.103:0/1403825988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fe7c0 0x7f79f8106e00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f79e8004940 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 -- 172.21.15.103:0/1403825988 shutdown_connections 2024-03-22T02:44:27.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 --2- 172.21.15.103:0/1403825988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fe7c0 0x7f79f8106e00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 --2- 172.21.15.103:0/1403825988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fde20 0x7f79f80fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 -- 172.21.15.103:0/1403825988 >> 172.21.15.103:0/1403825988 conn(0x7f79f80f7870 msgr2=0x7f79f80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:27.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 -- 172.21.15.103:0/1403825988 shutdown_connections 2024-03-22T02:44:27.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.726+0000 7f79fff9b700 1 -- 172.21.15.103:0/1403825988 wait complete. 2024-03-22T02:44:27.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 Processor -- start 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 -- start start 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 0x7f79f8113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f8114530 con 0x7f79f80fde20 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fff9b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f810db30 con 0x7f79f80fe7c0 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fdd37700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fdd37700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43724/0 (socket says 172.21.15.103:43724) 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fdd37700 1 -- 172.21.15.103:0/885724536 learned_addr learned my addr 172.21.15.103:0/885724536 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.727+0000 7f79fd536700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 0x7f79f8113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fdd37700 1 -- 172.21.15.103:0/885724536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 msgr2=0x7f79f8113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fdd37700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 0x7f79f8113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fdd37700 1 -- 172.21.15.103:0/885724536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8005040 con 0x7f79f80fde20 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fd536700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 0x7f79f8113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fdd37700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f79f4002a50 tx=0x7f79f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f4007cc0 con 0x7f79f80fde20 2024-03-22T02:44:27.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f810de10 con 0x7f79f80fde20 2024-03-22T02:44:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f810e320 con 0x7f79f80fde20 2024-03-22T02:44:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79f400f040 con 0x7f79f80fde20 2024-03-22T02:44:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.728+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f400da30 con 0x7f79f80fde20 2024-03-22T02:44:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.729+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc0052b0 con 0x7f79f80fde20 2024-03-22T02:44:27.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.730+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f79f40097c0 con 0x7f79f80fde20 2024-03-22T02:44:27.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.730+0000 7f79ee7fc700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 0x7f79e4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:27.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.730+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f79f408d9f0 con 0x7f79f80fde20 2024-03-22T02:44:27.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.730+0000 7f79fd536700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 0x7f79e4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:27.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.731+0000 7f79fd536700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 0x7f79e4070850 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79f810ee10 tx=0x7f79e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:27.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:27.735+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79f405a7a0 con 0x7f79f80fde20 2024-03-22T02:44:28.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.012+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f79dc005d70 con 0x7f79f80fde20 2024-03-22T02:44:28.013 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:27 smithi103 ceph-mon[115211]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.013+0000 7f79ee7fc700 1 -- 172.21.15.103:0/885724536 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f79f405a330 con 0x7f79f80fde20 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:44:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:44:28.015 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 msgr2=0x7f79e4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 0x7f79e4070850 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79f810ee10 tx=0x7f79e8009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 msgr2=0x7f79f81139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f79f4002a50 tx=0x7f79f40046b0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 shutdown_connections 2024-03-22T02:44:28.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f79e406e3a0 0x7f79e4070850 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f79f80fde20 0x7f79f81139a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 --2- 172.21.15.103:0/885724536 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79f80fe7c0 0x7f79f8113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 >> 172.21.15.103:0/885724536 conn(0x7f79f80f7870 msgr2=0x7f79f80f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 shutdown_connections 2024-03-22T02:44:28.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.015+0000 7f79fff9b700 1 -- 172.21.15.103:0/885724536 wait complete. 2024-03-22T02:44:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:27 smithi122 ceph-mon[83368]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 -- 172.21.15.103:0/2857069283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc103780 msgr2=0x7fe6fc0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 --2- 172.21.15.103:0/2857069283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc103780 0x7fe6fc0fcdf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0004940 tx=0x7fe6f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 -- 172.21.15.103:0/2857069283 shutdown_connections 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 --2- 172.21.15.103:0/2857069283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc103780 0x7fe6fc0fcdf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 --2- 172.21.15.103:0/2857069283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc102de0 0x7fe6fc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 -- 172.21.15.103:0/2857069283 >> 172.21.15.103:0/2857069283 conn(0x7fe6fc0f8950 msgr2=0x7fe6fc0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 -- 172.21.15.103:0/2857069283 shutdown_connections 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 -- 172.21.15.103:0/2857069283 wait complete. 2024-03-22T02:44:28.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.117+0000 7fe70258f700 1 Processor -- start 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe70258f700 1 -- start start 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe70258f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe70258f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc103780 0x7fe6fc1160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe70258f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc116740 con 0x7fe6fc102de0 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe70258f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc10fd20 con 0x7fe6fc103780 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43744/0 (socket says 172.21.15.103:43744) 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 -- 172.21.15.103:0/2163736954 learned_addr learned my addr 172.21.15.103:0/2163736954 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 -- 172.21.15.103:0/2163736954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc103780 msgr2=0x7fe6fc1160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fb7fe700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc103780 0x7fe6fc1160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc103780 0x7fe6fc1160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 -- 172.21.15.103:0/2163736954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0005040 con 0x7fe6fc102de0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6fbfff700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec002a50 tx=0x7fe6ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.118+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6ec007cc0 con 0x7fe6fc102de0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.119+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc110000 con 0x7fe6fc102de0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.119+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6ec00f040 con 0x7fe6fc102de0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.119+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6ec009640 con 0x7fe6fc102de0 2024-03-22T02:44:28.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.119+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc1105c0 con 0x7fe6fc102de0 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.120+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc0fe530 con 0x7fe6fc102de0 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.120+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe6ec0097a0 con 0x7fe6fc102de0 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.121+0000 7fe6f8ff9700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 0x7fe6e4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.121+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe6ec08db00 con 0x7fe6fc102de0 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.121+0000 7fe6fb7fe700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 0x7fe6e4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.121+0000 7fe6fb7fe700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 0x7fe6e4070850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b3b0 tx=0x7fe6f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.125+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe6ec05a8b0 con 0x7fe6fc102de0 2024-03-22T02:44:28.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.367+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe6fc111200 con 0x7fe6fc102de0 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.367+0000 7fe6f8ff9700 1 -- 172.21.15.103:0/2163736954 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fe6ec05a440 con 0x7fe6fc102de0 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:44:28.368 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:44:28.369 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:44:28.370 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:44:28.371 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:44:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.369+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 msgr2=0x7fe6e4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.369+0000 7fe70258f700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 0x7fe6e4070850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b3b0 tx=0x7fe6f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.369+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 msgr2=0x7fe6fc115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.369+0000 7fe70258f700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec002a50 tx=0x7fe6ec0046b0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.369+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 shutdown_connections 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe6e406e3a0 0x7fe6e4070850 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6fc102de0 0x7fe6fc115bb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 --2- 172.21.15.103:0/2163736954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6fc103780 0x7fe6fc1160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 >> 172.21.15.103:0/2163736954 conn(0x7fe6fc0f8950 msgr2=0x7fe6fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 shutdown_connections 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.370+0000 7fe70258f700 1 -- 172.21.15.103:0/2163736954 wait complete. 2024-03-22T02:44:28.373 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:44:28.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.471+0000 7feba5450700 1 -- 172.21.15.103:0/589772382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba01026c0 msgr2=0x7feba00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.471+0000 7feba5450700 1 --2- 172.21.15.103:0/589772382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba01026c0 0x7feba00fbd30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feb94004940 tx=0x7feb94009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 -- 172.21.15.103:0/589772382 shutdown_connections 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 --2- 172.21.15.103:0/589772382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba01026c0 0x7feba00fbd30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 --2- 172.21.15.103:0/589772382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba0101d20 0x7feba01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 -- 172.21.15.103:0/589772382 >> 172.21.15.103:0/589772382 conn(0x7feba00f7870 msgr2=0x7feba00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 -- 172.21.15.103:0/589772382 shutdown_connections 2024-03-22T02:44:28.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 -- 172.21.15.103:0/589772382 wait complete. 2024-03-22T02:44:28.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 Processor -- start 2024-03-22T02:44:28.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.472+0000 7feba5450700 1 -- start start 2024-03-22T02:44:28.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feba5450700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feba5450700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 0x7feba0112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feba5450700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0113440 con 0x7feba0101d20 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feba5450700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba010ca40 con 0x7feba01026c0 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43764/0 (socket says 172.21.15.103:43764) 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 -- 172.21.15.103:0/454475339 learned_addr learned my addr 172.21.15.103:0/454475339 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9e7fc700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 0x7feba0112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 -- 172.21.15.103:0/454475339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 msgr2=0x7feba0112df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 0x7feba0112df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 -- 172.21.15.103:0/454475339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94005040 con 0x7feba0101d20 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9e7fc700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 0x7feba0112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb9effd700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb9000fce0 tx=0x7feb900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9000fe90 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb900090d0 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.473+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba010ccc0 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.474+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb900126a0 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.474+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba010d280 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.474+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba00fd470 con 0x7feba0101d20 2024-03-22T02:44:28.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.475+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7feb90009240 con 0x7feba0101d20 2024-03-22T02:44:28.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.475+0000 7feb877fe700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 0x7feb88070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.476+0000 7feb9e7fc700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 0x7feb88070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.476+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb90090630 con 0x7feba0101d20 2024-03-22T02:44:28.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.476+0000 7feb9e7fc700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 0x7feb88070850 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb9400b3b0 tx=0x7feb94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.480+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feb9005d360 con 0x7feba0101d20 2024-03-22T02:44:28.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.697+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feba0108e00 con 0x7feb8806e3a0 2024-03-22T02:44:28.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.699+0000 7feb877fe700 1 -- 172.21.15.103:0/454475339 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feba0108e00 con 0x7feb8806e3a0 2024-03-22T02:44:28.699 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:44:28.699 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:44:28.700 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:44:28.701 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:44:28.701 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:44:28.701 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:44:28.701 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 msgr2=0x7feb88070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 0x7feb88070850 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb9400b3b0 tx=0x7feb94009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 msgr2=0x7feba01128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb9000fce0 tx=0x7feb900046b0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 shutdown_connections 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feb8806e3a0 0x7feb88070850 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feba0101d20 0x7feba01128b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 --2- 172.21.15.103:0/454475339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feba01026c0 0x7feba0112df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 >> 172.21.15.103:0/454475339 conn(0x7feba00f7870 msgr2=0x7feba00f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 shutdown_connections 2024-03-22T02:44:28.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.701+0000 7feba5450700 1 -- 172.21.15.103:0/454475339 wait complete. 2024-03-22T02:44:28.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 -- 172.21.15.103:0/3316325518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c102710 msgr2=0x7f391c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 --2- 172.21.15.103:0/3316325518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c102710 0x7f391c0fbd80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3910004940 tx=0x7f3910009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 -- 172.21.15.103:0/3316325518 shutdown_connections 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 --2- 172.21.15.103:0/3316325518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c102710 0x7f391c0fbd80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 --2- 172.21.15.103:0/3316325518 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c101d70 0x7f391c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 -- 172.21.15.103:0/3316325518 >> 172.21.15.103:0/3316325518 conn(0x7f391c072bc0 msgr2=0x7f391c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 -- 172.21.15.103:0/3316325518 shutdown_connections 2024-03-22T02:44:28.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 -- 172.21.15.103:0/3316325518 wait complete. 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.805+0000 7f39217b5700 1 Processor -- start 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39217b5700 1 -- start start 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39217b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39217b5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c102710 0x7f391c10f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39217b5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c10f750 con 0x7f391c101d70 2024-03-22T02:44:28.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39217b5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c10f890 con 0x7f391c102710 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43768/0 (socket says 172.21.15.103:43768) 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 -- 172.21.15.103:0/949864907 learned_addr learned my addr 172.21.15.103:0/949864907 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391a7fc700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c102710 0x7f391c10f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 -- 172.21.15.103:0/949864907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c102710 msgr2=0x7f391c10f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c102710 0x7f391c10f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 -- 172.21.15.103:0/949864907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910005040 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f391affd700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f391400bbb0 tx=0x7f3914009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.806+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39140092e0 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.807+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f391400cb20 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.807+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c117c00 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.807+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3914015d70 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.807+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c118190 con 0x7f391c101d70 2024-03-22T02:44:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.808+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c0ff080 con 0x7f391c101d70 2024-03-22T02:44:28.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.808+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3914012070 con 0x7f391c101d70 2024-03-22T02:44:28.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.808+0000 7f39037fe700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 0x7f3904070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:28.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.809+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f391408db20 con 0x7f391c101d70 2024-03-22T02:44:28.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.809+0000 7f391a7fc700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 0x7f3904070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:28.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.809+0000 7f391a7fc700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 0x7f3904070850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3910004890 tx=0x7f3910009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:28.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:28.813+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3914056d20 con 0x7f391c101d70 2024-03-22T02:44:29.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.094+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f391c108f20 con 0x7f391c101d70 2024-03-22T02:44:29.096 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:28 smithi103 ceph-mon[115211]: pgmap v388: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:29.096 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:28 smithi103 ceph-mon[115211]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:29.096 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:28 smithi103 ceph-mon[115211]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:29.096 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:28 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/885724536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:44:29.096 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:28 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2163736954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:44:29.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.095+0000 7f39037fe700 1 -- 172.21.15.103:0/949864907 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f391405a3e0 con 0x7f391c101d70 2024-03-22T02:44:29.097 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:44:29.097 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:44:29.097 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:44:29.097 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:44:29.097 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:44:29.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 msgr2=0x7f3904070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:29.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 0x7f3904070850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3910004890 tx=0x7f3910009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:29.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 msgr2=0x7f391c10eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f391400bbb0 tx=0x7f3914009840 comp rx=0 tx=0).stop 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 shutdown_connections 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f390406e3a0 0x7f3904070850 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f391c101d70 0x7f391c10eb30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 --2- 172.21.15.103:0/949864907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f391c102710 0x7f391c10f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 >> 172.21.15.103:0/949864907 conn(0x7f391c072bc0 msgr2=0x7f391c0faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.097+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 shutdown_connections 2024-03-22T02:44:29.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:29.098+0000 7f39217b5700 1 -- 172.21.15.103:0/949864907 wait complete. 2024-03-22T02:44:29.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:28 smithi122 ceph-mon[83368]: pgmap v388: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:29.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:28 smithi122 ceph-mon[83368]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:29.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:28 smithi122 ceph-mon[83368]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:29.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:28 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/885724536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:44:29.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:28 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2163736954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:44:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:29 smithi122 ceph-mon[83368]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:30.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:29 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/949864907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:44:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:29 smithi103 ceph-mon[115211]: from='client.34780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:44:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:29 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/949864907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:44:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:30 smithi103 ceph-mon[115211]: pgmap v389: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:31.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:30 smithi122 ceph-mon[83368]: pgmap v389: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:32 smithi122 ceph-mon[83368]: pgmap v390: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:32 smithi103 ceph-mon[115211]: pgmap v390: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:35.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:34 smithi122 ceph-mon[83368]: pgmap v391: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:34 smithi103 ceph-mon[115211]: pgmap v391: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:36 smithi122 ceph-mon[83368]: pgmap v392: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:36 smithi103 ceph-mon[115211]: pgmap v392: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:39.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:38 smithi122 ceph-mon[83368]: pgmap v393: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:39.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:38 smithi103 ceph-mon[115211]: pgmap v393: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:41.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:40 smithi122 ceph-mon[83368]: pgmap v394: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:41.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:40 smithi103 ceph-mon[115211]: pgmap v394: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:43.101 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:42 smithi122 ceph-mon[83368]: pgmap v395: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:42 smithi103 ceph-mon[115211]: pgmap v395: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:44 smithi122 ceph-mon[83368]: pgmap v396: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:44 smithi103 ceph-mon[115211]: pgmap v396: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:47.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:46 smithi122 ceph-mon[83368]: pgmap v397: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:46 smithi103 ceph-mon[115211]: pgmap v397: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:44:49.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:48 smithi122 ceph-mon[83368]: pgmap v398: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:48 smithi103 ceph-mon[115211]: pgmap v398: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:50 smithi122 ceph-mon[83368]: pgmap v399: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:51.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:50 smithi103 ceph-mon[115211]: pgmap v399: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:53.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:52 smithi122 ceph-mon[83368]: pgmap v400: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:52 smithi103 ceph-mon[115211]: pgmap v400: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:55.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:54 smithi122 ceph-mon[83368]: pgmap v401: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:55.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:54 smithi103 ceph-mon[115211]: pgmap v401: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:57.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:56 smithi122 ceph-mon[83368]: pgmap v402: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:57.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:56 smithi103 ceph-mon[115211]: pgmap v402: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:59.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:44:58 smithi122 ceph-mon[83368]: pgmap v403: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:44:58 smithi103 ceph-mon[115211]: pgmap v403: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 -- 172.21.15.103:0/2774099291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 msgr2=0x7ff8500fca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 --2- 172.21.15.103:0/2774099291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff8500fca50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff844004940 tx=0x7ff844009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 -- 172.21.15.103:0/2774099291 shutdown_connections 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 --2- 172.21.15.103:0/2774099291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff8500fca50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 --2- 172.21.15.103:0/2774099291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 0x7ff850102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 -- 172.21.15.103:0/2774099291 >> 172.21.15.103:0/2774099291 conn(0x7ff8500714c0 msgr2=0x7ff8500718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 -- 172.21.15.103:0/2774099291 shutdown_connections 2024-03-22T02:44:59.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.211+0000 7ff856819700 1 -- 172.21.15.103:0/2774099291 wait complete. 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 Processor -- start 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 -- start start 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 0x7ff85010f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85010f680 con 0x7ff8500fc5e0 2024-03-22T02:44:59.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff856819700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85010f7c0 con 0x7ff850101ec0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54052/0 (socket says 172.21.15.103:54052) 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84ffff700 1 -- 172.21.15.103:0/93278771 learned_addr learned my addr 172.21.15.103:0/93278771 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84f7fe700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 0x7ff85010f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84ffff700 1 -- 172.21.15.103:0/93278771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 msgr2=0x7ff85010f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.212+0000 7ff84ffff700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 0x7ff85010f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff84ffff700 1 -- 172.21.15.103:0/93278771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844005040 con 0x7ff8500fc5e0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff84ffff700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff84000eb40 tx=0x7ff84000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff840007cc0 con 0x7ff8500fc5e0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff84000f040 con 0x7ff8500fc5e0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85010fa40 con 0x7ff8500fc5e0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff84000ca10 con 0x7ff8500fc5e0 2024-03-22T02:44:59.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.213+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff850117ef0 con 0x7ff8500fc5e0 2024-03-22T02:44:59.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.214+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8501042a0 con 0x7ff8500fc5e0 2024-03-22T02:44:59.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.214+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff84000cb70 con 0x7ff8500fc5e0 2024-03-22T02:44:59.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.215+0000 7ff84cff9700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 0x7ff838070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.215+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff84008d900 con 0x7ff8500fc5e0 2024-03-22T02:44:59.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.215+0000 7ff84f7fe700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 0x7ff838070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.215+0000 7ff84f7fe700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 0x7ff838070850 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff844009b30 tx=0x7ff844009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.220+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff840056c30 con 0x7ff8500fc5e0 2024-03-22T02:44:59.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.434+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff850109020 con 0x7ff83806e3a0 2024-03-22T02:44:59.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.435+0000 7ff84cff9700 1 -- 172.21.15.103:0/93278771 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff850109020 con 0x7ff83806e3a0 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 msgr2=0x7ff838070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 0x7ff838070850 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff844009b30 tx=0x7ff844009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 msgr2=0x7ff85010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff84000eb40 tx=0x7ff84000b040 comp rx=0 tx=0).stop 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 shutdown_connections 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff83806e3a0 0x7ff838070850 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8500fc5e0 0x7ff85010ec00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 --2- 172.21.15.103:0/93278771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff850101ec0 0x7ff85010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 >> 172.21.15.103:0/93278771 conn(0x7ff8500714c0 msgr2=0x7ff8500f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 shutdown_connections 2024-03-22T02:44:59.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.437+0000 7ff856819700 1 -- 172.21.15.103:0/93278771 wait complete. 2024-03-22T02:44:59.449 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:44:59.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 -- 172.21.15.103:0/3036098005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924102de0 msgr2=0x7fb9241031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 --2- 172.21.15.103:0/3036098005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924102de0 0x7fb9241031b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb918004940 tx=0x7fb918009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 -- 172.21.15.103:0/3036098005 shutdown_connections 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 --2- 172.21.15.103:0/3036098005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924103780 0x7fb9240fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 --2- 172.21.15.103:0/3036098005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924102de0 0x7fb9241031b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 -- 172.21.15.103:0/3036098005 >> 172.21.15.103:0/3036098005 conn(0x7fb9240f8950 msgr2=0x7fb9240fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 -- 172.21.15.103:0/3036098005 shutdown_connections 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.539+0000 7fb928ea6700 1 -- 172.21.15.103:0/3036098005 wait complete. 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 Processor -- start 2024-03-22T02:44:59.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 -- start start 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924102de0 0x7fb924113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924114510 con 0x7fb924103780 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.540+0000 7fb928ea6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92410db30 con 0x7fb924102de0 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54078/0 (socket says 172.21.15.103:54078) 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 -- 172.21.15.103:0/2087491679 learned_addr learned my addr 172.21.15.103:0/2087491679 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb92259c700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924102de0 0x7fb924113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 -- 172.21.15.103:0/2087491679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924102de0 msgr2=0x7fb924113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924102de0 0x7fb924113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 -- 172.21.15.103:0/2087491679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918005040 con 0x7fb924103780 2024-03-22T02:44:59.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb921d9b700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb914002a80 tx=0x7fb914004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb914004e80 con 0x7fb924103780 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb91400f040 con 0x7fb924103780 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb91400daa0 con 0x7fb924103780 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92410de10 con 0x7fb924103780 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.541+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92410e3d0 con 0x7fb924103780 2024-03-22T02:44:59.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.542+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9241000f0 con 0x7fb924103780 2024-03-22T02:44:59.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.543+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb914007cc0 con 0x7fb924103780 2024-03-22T02:44:59.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.544+0000 7fb912ffd700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 0x7fb90c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.544+0000 7fb92259c700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 0x7fb90c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.544+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb91408d350 con 0x7fb924103780 2024-03-22T02:44:59.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.544+0000 7fb92259c700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 0x7fb90c070850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb918004f90 tx=0x7fb918009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.548+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9140565d0 con 0x7fb924103780 2024-03-22T02:44:59.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.761+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb92410f040 con 0x7fb90c06e3a0 2024-03-22T02:44:59.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.763+0000 7fb912ffd700 1 -- 172.21.15.103:0/2087491679 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb92410f040 con 0x7fb90c06e3a0 2024-03-22T02:44:59.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 msgr2=0x7fb90c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 0x7fb90c070850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb918004f90 tx=0x7fb918009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 msgr2=0x7fb924113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb914002a80 tx=0x7fb914004630 comp rx=0 tx=0).stop 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 shutdown_connections 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb90c06e3a0 0x7fb90c070850 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb924102de0 0x7fb924113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 --2- 172.21.15.103:0/2087491679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb924103780 0x7fb924113ec0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 >> 172.21.15.103:0/2087491679 conn(0x7fb9240f8950 msgr2=0x7fb9241073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 shutdown_connections 2024-03-22T02:44:59.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.765+0000 7fb928ea6700 1 -- 172.21.15.103:0/2087491679 wait complete. 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1005979313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc81037b0 msgr2=0x7f7bc80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1005979313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc81037b0 0x7f7bc80fce20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc004940 tx=0x7f7bbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1005979313 shutdown_connections 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1005979313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc81037b0 0x7f7bc80fce20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1005979313 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc8102e10 0x7f7bc81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1005979313 >> 172.21.15.103:0/1005979313 conn(0x7f7bc80f89a0 msgr2=0x7f7bc80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1005979313 shutdown_connections 2024-03-22T02:44:59.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.867+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1005979313 wait complete. 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 Processor -- start 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 -- start start 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc81037b0 0x7f7bc81140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc8114730 con 0x7f7bc8102e10 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bcce7e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc810db20 con 0x7f7bc81037b0 2024-03-22T02:44:59.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54100/0 (socket says 172.21.15.103:54100) 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 -- 172.21.15.103:0/1332217852 learned_addr learned my addr 172.21.15.103:0/1332217852 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc5d9b700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc81037b0 0x7f7bc81140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 -- 172.21.15.103:0/1332217852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc81037b0 msgr2=0x7f7bc81140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc81037b0 0x7f7bc81140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.868+0000 7f7bc659c700 1 -- 172.21.15.103:0/1332217852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc005040 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bc659c700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0001d80 tx=0x7f7bc0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc0007d10 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bc0001e80 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc0014640 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc810de00 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.869+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc810e390 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.870+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc81052e0 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.870+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7bc000e960 con 0x7f7bc8102e10 2024-03-22T02:44:59.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.871+0000 7f7bb6ffd700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 0x7f7bb0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:44:59.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.871+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7bc008db70 con 0x7f7bc8102e10 2024-03-22T02:44:59.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.871+0000 7f7bc5d9b700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 0x7f7bb0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:44:59.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.871+0000 7f7bc5d9b700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 0x7f7bb0070850 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc004890 tx=0x7f7bbc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:44:59.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:44:59.875+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7bc005a950 con 0x7f7bc8102e10 2024-03-22T02:45:00.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.084+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bc810f1f0 con 0x7f7bb006e3a0 2024-03-22T02:45:00.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.094+0000 7f7bb6ffd700 1 -- 172.21.15.103:0/1332217852 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7bc810f1f0 con 0x7f7bb006e3a0 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (13m) 7m ago 24m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (11m) 7m ago 24m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (11m) 6m ago 23m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (13m) 7m ago 24m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (7m) 7m ago 19m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (7m) 7m ago 19m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (6m) 6m ago 19m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (7m) 6m ago 19m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (15m) 7m ago 26m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (14m) 6m ago 23m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (12m) 7m ago 26m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:45:00.096 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (11m) 6m ago 23m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (14m) 7m ago 24m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (14m) 6m ago 23m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (10m) 7m ago 22m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (10m) 7m ago 22m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (9m) 7m ago 22m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (9m) 6m ago 21m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (8m) 6m ago 21m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (7m) 6m ago 20m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:45:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (13m) 7m ago 24m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 msgr2=0x7f7bb0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 0x7f7bb0070850 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc004890 tx=0x7f7bbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 msgr2=0x7f7bc8113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0001d80 tx=0x7f7bc0009ec0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 shutdown_connections 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7bb006e3a0 0x7f7bb0070850 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc8102e10 0x7f7bc8113ba0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 --2- 172.21.15.103:0/1332217852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bc81037b0 0x7f7bc81140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 >> 172.21.15.103:0/1332217852 conn(0x7f7bc80f89a0 msgr2=0x7f7bc80face0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 shutdown_connections 2024-03-22T02:45:00.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.096+0000 7f7bcce7e700 1 -- 172.21.15.103:0/1332217852 wait complete. 2024-03-22T02:45:00.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 -- 172.21.15.103:0/625371979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 msgr2=0x7f4d901033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 --2- 172.21.15.103:0/625371979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d901033a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4d80004940 tx=0x7f4d80009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 -- 172.21.15.103:0/625371979 shutdown_connections 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 --2- 172.21.15.103:0/625371979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 --2- 172.21.15.103:0/625371979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d901033a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 -- 172.21.15.103:0/625371979 >> 172.21.15.103:0/625371979 conn(0x7f4d90071530 msgr2=0x7f4d90071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 -- 172.21.15.103:0/625371979 shutdown_connections 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.196+0000 7f4d966e0700 1 -- 172.21.15.103:0/625371979 wait complete. 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 Processor -- start 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 -- start start 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d900fe670 con 0x7f4d90102fd0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d966e0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d900fd1a0 con 0x7f4d900fb540 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d8f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d8f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54118/0 (socket says 172.21.15.103:54118) 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.197+0000 7f4d8f7fe700 1 -- 172.21.15.103:0/799684289 learned_addr learned my addr 172.21.15.103:0/799684289 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8f7fe700 1 -- 172.21.15.103:0/799684289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 msgr2=0x7f4d900fc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8ffff700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8f7fe700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8f7fe700 1 -- 172.21.15.103:0/799684289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80005040 con 0x7f4d90102fd0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8ffff700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8f7fe700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c8c0 tx=0x7f4d840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d8400a980 con 0x7f4d90102fd0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d900fd480 con 0x7f4d90102fd0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d8400ce60 con 0x7f4d90102fd0 2024-03-22T02:45:00.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d840115e0 con 0x7f4d90102fd0 2024-03-22T02:45:00.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.198+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9011a0d0 con 0x7f4d90102fd0 2024-03-22T02:45:00.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.199+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d90100270 con 0x7f4d90102fd0 2024-03-22T02:45:00.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.199+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4d84012400 con 0x7f4d90102fd0 2024-03-22T02:45:00.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.200+0000 7f4d8cff9700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 0x7f4d78070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.200+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4d8408ded0 con 0x7f4d90102fd0 2024-03-22T02:45:00.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.200+0000 7f4d8ffff700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 0x7f4d78070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.200+0000 7f4d8ffff700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 0x7f4d78070850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d80004560 tx=0x7f4d80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.205+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d84057150 con 0x7f4d90102fd0 2024-03-22T02:45:00.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.494+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d900fe860 con 0x7f4d90102fd0 2024-03-22T02:45:00.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.494+0000 7f4d8cff9700 1 -- 172.21.15.103:0/799684289 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4d840126b0 con 0x7f4d90102fd0 2024-03-22T02:45:00.496 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:45:00.496 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:45:00.496 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:45:00.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:45:00.497 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 msgr2=0x7f4d78070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 0x7f4d78070850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4d80004560 tx=0x7f4d80009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 msgr2=0x7f4d900fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c8c0 tx=0x7f4d840044c0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 shutdown_connections 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4d7806e3a0 0x7f4d78070850 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d900fb540 0x7f4d900fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 --2- 172.21.15.103:0/799684289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d90102fd0 0x7f4d900fcc60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 >> 172.21.15.103:0/799684289 conn(0x7f4d90071530 msgr2=0x7f4d900fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 shutdown_connections 2024-03-22T02:45:00.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.497+0000 7f4d966e0700 1 -- 172.21.15.103:0/799684289 wait complete. 2024-03-22T02:45:00.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- 172.21.15.103:0/605803785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fde10 msgr2=0x7ff3740fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/605803785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fde10 0x7ff3740fe1e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009ce0 tx=0x7ff36c004d10 comp rx=0 tx=0).stop 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- 172.21.15.103:0/605803785 shutdown_connections 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/605803785 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fe7b0 0x7ff374106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/605803785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fde10 0x7ff3740fe1e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- 172.21.15.103:0/605803785 >> 172.21.15.103:0/605803785 conn(0x7ff3740f7870 msgr2=0x7ff3740f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- 172.21.15.103:0/605803785 shutdown_connections 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- 172.21.15.103:0/605803785 wait complete. 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 Processor -- start 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 -- start start 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.599+0000 7ff37bcb8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fde10 0x7ff374113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff37bcb8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff37bcb8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff374114520 con 0x7ff3740fe7b0 2024-03-22T02:45:00.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff37bcb8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37410db40 con 0x7ff3740fde10 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54138/0 (socket says 172.21.15.103:54138) 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 -- 172.21.15.103:0/1327711200 learned_addr learned my addr 172.21.15.103:0/1327711200 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379a54700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fde10 0x7ff374113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 -- 172.21.15.103:0/1327711200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fde10 msgr2=0x7ff374113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fde10 0x7ff374113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 -- 172.21.15.103:0/1327711200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c009940 con 0x7ff3740fe7b0 2024-03-22T02:45:00.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff379253700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff36400c830 tx=0x7ff364004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3640106f0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff364010d20 con 0x7ff3740fe7b0 2024-03-22T02:45:00.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3640113a0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37410de20 con 0x7ff3740fe7b0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.600+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37410e310 con 0x7ff3740fe7b0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.601+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff374066ad0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.602+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff364012400 con 0x7ff3740fe7b0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.602+0000 7ff36a7fc700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 0x7ff360070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.602+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff36408dd70 con 0x7ff3740fe7b0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.602+0000 7ff379a54700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 0x7ff360070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.603+0000 7ff379a54700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 0x7ff360070850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b740 tx=0x7ff36c004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.607+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff364056ff0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.849+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff374109ea0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.849+0000 7ff36a7fc700 1 -- 172.21.15.103:0/1327711200 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7ff36405a6b0 con 0x7ff3740fe7b0 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:45:00.851 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:45:00.852 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:00.853 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:00.854 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:00.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 msgr2=0x7ff360070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 0x7ff360070850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00b740 tx=0x7ff36c004d10 comp rx=0 tx=0).stop 2024-03-22T02:45:00.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 msgr2=0x7ff374113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff36400c830 tx=0x7ff364004470 comp rx=0 tx=0).stop 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 shutdown_connections 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff36006e3a0 0x7ff360070850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3740fde10 0x7ff374113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 --2- 172.21.15.103:0/1327711200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3740fe7b0 0x7ff374113ed0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 >> 172.21.15.103:0/1327711200 conn(0x7ff3740f7870 msgr2=0x7ff3740f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 shutdown_connections 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.852+0000 7ff37bcb8700 1 -- 172.21.15.103:0/1327711200 wait complete. 2024-03-22T02:45:00.855 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:45:00.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/460131233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 msgr2=0x7ff3ac106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/460131233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac106f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff394004940 tx=0x7ff394009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/460131233 shutdown_connections 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/460131233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac106f00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/460131233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 0x7ff3ac0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/460131233 >> 172.21.15.103:0/460131233 conn(0x7ff3ac072bc0 msgr2=0x7ff3ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/460131233 shutdown_connections 2024-03-22T02:45:00.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/460131233 wait complete. 2024-03-22T02:45:00.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 Processor -- start 2024-03-22T02:45:00.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.954+0000 7ff3b0e6c700 1 -- start start 2024-03-22T02:45:00.957 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3b0e6c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 0x7ff3ac113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3b0e6c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3b0e6c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac114730 con 0x7ff3ac0fe830 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3b0e6c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac10db20 con 0x7ff3ac0fde90 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54160/0 (socket says 172.21.15.103:54160) 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 -- 172.21.15.103:0/3736822804 learned_addr learned my addr 172.21.15.103:0/3736822804 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3aa59c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 0x7ff3ac113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 -- 172.21.15.103:0/3736822804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 msgr2=0x7ff3ac113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 0x7ff3ac113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 -- 172.21.15.103:0/3736822804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394005040 con 0x7ff3ac0fe830 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a1d9b700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3940093b0 tx=0x7ff394009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39400e040 con 0x7ff3ac0fe830 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac10dda0 con 0x7ff3ac0fe830 2024-03-22T02:45:00.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff394004290 con 0x7ff3ac0fe830 2024-03-22T02:45:00.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.955+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39401c3f0 con 0x7ff3ac0fe830 2024-03-22T02:45:00.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.956+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac10e2e0 con 0x7ff3ac0fe830 2024-03-22T02:45:00.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.957+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac065e30 con 0x7ff3ac0fe830 2024-03-22T02:45:00.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.957+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff3940137b0 con 0x7ff3ac0fe830 2024-03-22T02:45:00.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.957+0000 7ff3a37fe700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 0x7ff398070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:00.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.958+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff39408ef70 con 0x7ff3ac0fe830 2024-03-22T02:45:00.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.958+0000 7ff3aa59c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 0x7ff398070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:00.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.958+0000 7ff3aa59c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 0x7ff398070860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0041d0 tx=0x7ff39c0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:00.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:00.962+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3940c3740 con 0x7ff3ac0fe830 2024-03-22T02:45:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:00 smithi122 ceph-mon[83368]: pgmap v404: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:00 smithi122 ceph-mon[83368]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:00 smithi122 ceph-mon[83368]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:01.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:00 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/799684289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:45:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:00 smithi103 ceph-mon[115211]: pgmap v404: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:00 smithi103 ceph-mon[115211]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:00 smithi103 ceph-mon[115211]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:00 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/799684289' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:45:01.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.177+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3ac109f10 con 0x7ff39806e3b0 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.178+0000 7ff3a37fe700 1 -- 172.21.15.103:0/3736822804 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff3ac109f10 con 0x7ff39806e3b0 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:45:01.180 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:45:01.181 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:45:01.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 msgr2=0x7ff398070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 0x7ff398070860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0041d0 tx=0x7ff39c0043c0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 msgr2=0x7ff3ac1140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3940093b0 tx=0x7ff394009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 shutdown_connections 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.180+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff39806e3b0 0x7ff398070860 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.181+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3ac0fde90 0x7ff3ac113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.181+0000 7ff3b0e6c700 1 --2- 172.21.15.103:0/3736822804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3ac0fe830 0x7ff3ac1140e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.181+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 >> 172.21.15.103:0/3736822804 conn(0x7ff3ac072bc0 msgr2=0x7ff3ac1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.181+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 shutdown_connections 2024-03-22T02:45:01.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.181+0000 7ff3b0e6c700 1 -- 172.21.15.103:0/3736822804 wait complete. 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 -- 172.21.15.103:0/385321779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef4102e20 msgr2=0x7f1ef41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 --2- 172.21.15.103:0/385321779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef4102e20 0x7f1ef41031f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4004940 tx=0x7f1ee4009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 -- 172.21.15.103:0/385321779 shutdown_connections 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 --2- 172.21.15.103:0/385321779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef41037c0 0x7f1ef40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 --2- 172.21.15.103:0/385321779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef4102e20 0x7f1ef41031f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 -- 172.21.15.103:0/385321779 >> 172.21.15.103:0/385321779 conn(0x7f1ef40f89b0 msgr2=0x7f1ef40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:01.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 -- 172.21.15.103:0/385321779 shutdown_connections 2024-03-22T02:45:01.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.281+0000 7f1ef9388700 1 -- 172.21.15.103:0/385321779 wait complete. 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 Processor -- start 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 -- start start 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 0x7f1ef41139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef41145c0 con 0x7f1ef41037c0 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef9388700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef410db50 con 0x7f1ef4102e20 2024-03-22T02:45:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.282+0000 7f1ef27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54166/0 (socket says 172.21.15.103:54166) 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 -- 172.21.15.103:0/3084702333 learned_addr learned my addr 172.21.15.103:0/3084702333 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef2ffd700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 0x7f1ef41139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 -- 172.21.15.103:0/3084702333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 msgr2=0x7f1ef41139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 0x7f1ef41139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 -- 172.21.15.103:0/3084702333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee4005040 con 0x7f1ef41037c0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef2ffd700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 0x7f1ef41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef27fc700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800b790 tx=0x7f1ee8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee800f6f0 con 0x7f1ef41037c0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ee800fd20 con 0x7f1ef41037c0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ee80103a0 con 0x7f1ef41037c0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef410de30 con 0x7f1ef41037c0 2024-03-22T02:45:01.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.283+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef410e3f0 con 0x7f1ef41037c0 2024-03-22T02:45:01.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.284+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef40fe570 con 0x7f1ef41037c0 2024-03-22T02:45:01.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.285+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1ee8011400 con 0x7f1ef41037c0 2024-03-22T02:45:01.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.285+0000 7f1edb7fe700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 0x7f1edc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:01.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.285+0000 7f1ef2ffd700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 0x7f1edc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:01.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.285+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ee808dca0 con 0x7f1ef41037c0 2024-03-22T02:45:01.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.286+0000 7f1ef2ffd700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 0x7f1edc070850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4004f90 tx=0x7f1ee4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:01.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.290+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ee8056ea0 con 0x7f1ef41037c0 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.571+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ef4109f90 con 0x7f1ef41037c0 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.571+0000 7f1edb7fe700 1 -- 172.21.15.103:0/3084702333 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1ee805a560 con 0x7f1ef41037c0 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:45:01.573 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:45:01.574 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 msgr2=0x7f1edc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 0x7f1edc070850 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4004f90 tx=0x7f1ee4009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 msgr2=0x7f1ef4113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800b790 tx=0x7f1ee8009220 comp rx=0 tx=0).stop 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 shutdown_connections 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1edc06e3a0 0x7f1edc070850 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ef4102e20 0x7f1ef41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 --2- 172.21.15.103:0/3084702333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef41037c0 0x7f1ef4113ee0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 >> 172.21.15.103:0/3084702333 conn(0x7f1ef40f89b0 msgr2=0x7f1ef40f9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 shutdown_connections 2024-03-22T02:45:01.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:01.574+0000 7f1ef9388700 1 -- 172.21.15.103:0/3084702333 wait complete. 2024-03-22T02:45:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:01 smithi103 ceph-mon[115211]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:01 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1327711200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:45:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:01 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3084702333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:45:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:01 smithi122 ceph-mon[83368]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:01 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1327711200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:45:02.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:01 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3084702333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:45:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:02 smithi122 ceph-mon[83368]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:02 smithi122 ceph-mon[83368]: pgmap v405: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:02 smithi103 ceph-mon[115211]: from='client.34808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:02 smithi103 ceph-mon[115211]: pgmap v405: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:04 smithi122 ceph-mon[83368]: pgmap v406: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:04 smithi103 ceph-mon[115211]: pgmap v406: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:07.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:06 smithi122 ceph-mon[83368]: pgmap v407: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:06 smithi103 ceph-mon[115211]: pgmap v407: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:09.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:08 smithi122 ceph-mon[83368]: pgmap v408: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:08 smithi103 ceph-mon[115211]: pgmap v408: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:10 smithi122 ceph-mon[83368]: pgmap v409: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:10 smithi103 ceph-mon[115211]: pgmap v409: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:12 smithi122 ceph-mon[83368]: pgmap v410: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:12 smithi103 ceph-mon[115211]: pgmap v410: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:15.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:14 smithi122 ceph-mon[83368]: pgmap v411: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:14 smithi103 ceph-mon[115211]: pgmap v411: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:16 smithi122 ceph-mon[83368]: pgmap v412: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:16 smithi103 ceph-mon[115211]: pgmap v412: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:18 smithi122 ceph-mon[83368]: pgmap v413: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:18 smithi103 ceph-mon[115211]: pgmap v413: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:20 smithi122 ceph-mon[83368]: pgmap v414: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:21.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:20 smithi103 ceph-mon[115211]: pgmap v414: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:22 smithi122 ceph-mon[83368]: pgmap v415: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:22 smithi103 ceph-mon[115211]: pgmap v415: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:25.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:24 smithi122 ceph-mon[83368]: pgmap v416: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:24 smithi103 ceph-mon[115211]: pgmap v416: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:26 smithi122 ceph-mon[83368]: pgmap v417: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:26 smithi103 ceph-mon[115211]: pgmap v417: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:28 smithi103 ceph-mon[115211]: pgmap v418: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:28 smithi122 ceph-mon[83368]: pgmap v418: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:30 smithi103 ceph-mon[115211]: pgmap v419: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:31.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:30 smithi122 ceph-mon[83368]: pgmap v419: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 -- 172.21.15.103:0/2552640819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 msgr2=0x7f31780fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 --2- 172.21.15.103:0/2552640819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f31780fbd80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3160004940 tx=0x7f3160009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 -- 172.21.15.103:0/2552640819 shutdown_connections 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 --2- 172.21.15.103:0/2552640819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f31780fbd80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 --2- 172.21.15.103:0/2552640819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f3178102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 -- 172.21.15.103:0/2552640819 >> 172.21.15.103:0/2552640819 conn(0x7f3178072bc0 msgr2=0x7f3178072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:31.691 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 -- 172.21.15.103:0/2552640819 shutdown_connections 2024-03-22T02:45:31.691 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.689+0000 7f317d616700 1 -- 172.21.15.103:0/2552640819 wait complete. 2024-03-22T02:45:31.691 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 Processor -- start 2024-03-22T02:45:31.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 -- start start 2024-03-22T02:45:31.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f317810c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317810d390 con 0x7f3178102710 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f317d616700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317810d500 con 0x7f3178101d70 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f316e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f316e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45714/0 (socket says 172.21.15.103:45714) 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f316e7fc700 1 -- 172.21.15.103:0/2864687849 learned_addr learned my addr 172.21.15.103:0/2864687849 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316e7fc700 1 -- 172.21.15.103:0/2864687849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 msgr2=0x7f317810c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.690+0000 7f3176ffd700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f317810c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316e7fc700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f317810c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316e7fc700 1 -- 172.21.15.103:0/2864687849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160005040 con 0x7f3178102710 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f3176ffd700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f317810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316e7fc700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f316000b3b0 tx=0x7f3160009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316000e040 con 0x7f3178102710 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317810d780 con 0x7f3178102710 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3160009b10 con 0x7f3178102710 2024-03-22T02:45:31.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3160013420 con 0x7f3178102710 2024-03-22T02:45:31.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.691+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317811a300 con 0x7f3178102710 2024-03-22T02:45:31.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.692+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3178104240 con 0x7f3178102710 2024-03-22T02:45:31.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.692+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f316000bbc0 con 0x7f3178102710 2024-03-22T02:45:31.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.693+0000 7f316ffff700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 0x7f3164070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:31.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.693+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f316008e610 con 0x7f3178102710 2024-03-22T02:45:31.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.693+0000 7f3176ffd700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 0x7f3164070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:31.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.693+0000 7f3176ffd700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 0x7f3164070850 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3168007fd0 tx=0x7f3168009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:31.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.698+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f316005b530 con 0x7f3178102710 2024-03-22T02:45:31.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.915+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f317810d780 con 0x7f316406e3a0 2024-03-22T02:45:31.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.916+0000 7f316ffff700 1 -- 172.21.15.103:0/2864687849 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f317810d780 con 0x7f316406e3a0 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 msgr2=0x7f3164070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 0x7f3164070850 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3168007fd0 tx=0x7f3168009040 comp rx=0 tx=0).stop 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 msgr2=0x7f317810ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f316000b3b0 tx=0x7f3160009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 shutdown_connections 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f316406e3a0 0x7f3164070850 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3178101d70 0x7f317810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 --2- 172.21.15.103:0/2864687849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3178102710 0x7f317810ce50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:31.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 >> 172.21.15.103:0/2864687849 conn(0x7f3178072bc0 msgr2=0x7f31780fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:31.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 shutdown_connections 2024-03-22T02:45:31.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:31.918+0000 7f317d616700 1 -- 172.21.15.103:0/2864687849 wait complete. 2024-03-22T02:45:31.929 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- 172.21.15.103:0/2414921818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 msgr2=0x7f819c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 --2- 172.21.15.103:0/2414921818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c0fbd30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f818c004940 tx=0x7f818c009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- 172.21.15.103:0/2414921818 shutdown_connections 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 --2- 172.21.15.103:0/2414921818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c0fbd30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 --2- 172.21.15.103:0/2414921818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 0x7f819c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- 172.21.15.103:0/2414921818 >> 172.21.15.103:0/2414921818 conn(0x7f819c0f7870 msgr2=0x7f819c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- 172.21.15.103:0/2414921818 shutdown_connections 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- 172.21.15.103:0/2414921818 wait complete. 2024-03-22T02:45:32.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 Processor -- start 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.020+0000 7f81a4484700 1 -- start start 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a4484700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 0x7f819c115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a4484700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a4484700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c116790 con 0x7f819c1026c0 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a4484700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c10fd70 con 0x7f819c101d20 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45726/0 (socket says 172.21.15.103:45726) 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 -- 172.21.15.103:0/3893092893 learned_addr learned my addr 172.21.15.103:0/3893092893 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a2220700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 0x7f819c115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 -- 172.21.15.103:0/3893092893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 msgr2=0x7f819c115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 0x7f819c115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 -- 172.21.15.103:0/3893092893 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c005040 con 0x7f819c1026c0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f81a1a1f700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f818c009100 tx=0x7f818c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.021+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f818c00e040 con 0x7f819c1026c0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.022+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c10fff0 con 0x7f819c1026c0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.022+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f818c009af0 con 0x7f819c1026c0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.022+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f818c01c3f0 con 0x7f819c1026c0 2024-03-22T02:45:32.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.022+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c1105b0 con 0x7f819c1026c0 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.023+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c0fd470 con 0x7f819c1026c0 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.023+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f818c00baf0 con 0x7f819c1026c0 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.024+0000 7f8192ffd700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 0x7f8188070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.024+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f818c08eff0 con 0x7f819c1026c0 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.024+0000 7f81a2220700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 0x7f8188070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.024+0000 7f81a2220700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 0x7f8188070850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8194001370 tx=0x7f819400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.028+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f818c0c3740 con 0x7f819c1026c0 2024-03-22T02:45:32.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.247+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f819c108e90 con 0x7f818806e3a0 2024-03-22T02:45:32.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.248+0000 7f8192ffd700 1 -- 172.21.15.103:0/3893092893 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f819c108e90 con 0x7f818806e3a0 2024-03-22T02:45:32.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 msgr2=0x7f8188070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 0x7f8188070850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8194001370 tx=0x7f819400b040 comp rx=0 tx=0).stop 2024-03-22T02:45:32.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 msgr2=0x7f819c116140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f818c009100 tx=0x7f818c009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 shutdown_connections 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f818806e3a0 0x7f8188070850 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f819c101d20 0x7f819c115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 --2- 172.21.15.103:0/3893092893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f819c1026c0 0x7f819c116140 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 >> 172.21.15.103:0/3893092893 conn(0x7f819c0f7870 msgr2=0x7f819c0f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 shutdown_connections 2024-03-22T02:45:32.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.250+0000 7f81a4484700 1 -- 172.21.15.103:0/3893092893 wait complete. 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 -- 172.21.15.103:0/1211632621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fde90 msgr2=0x7f96c40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 --2- 172.21.15.103:0/1211632621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fde90 0x7f96c40fe260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009ce0 tx=0x7f96bc004d10 comp rx=0 tx=0).stop 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 -- 172.21.15.103:0/1211632621 shutdown_connections 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 --2- 172.21.15.103:0/1211632621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fe830 0x7f96c4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 --2- 172.21.15.103:0/1211632621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fde90 0x7f96c40fe260 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 -- 172.21.15.103:0/1211632621 >> 172.21.15.103:0/1211632621 conn(0x7f96c4072bc0 msgr2=0x7f96c4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 -- 172.21.15.103:0/1211632621 shutdown_connections 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.352+0000 7f96c9725700 1 -- 172.21.15.103:0/1211632621 wait complete. 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 Processor -- start 2024-03-22T02:45:32.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 -- start start 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 0x7f96c4113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41147a0 con 0x7f96c40fe830 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c9725700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c410db50 con 0x7f96c40fde90 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45750/0 (socket says 172.21.15.103:45750) 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 -- 172.21.15.103:0/2023418540 learned_addr learned my addr 172.21.15.103:0/2023418540 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c2ffd700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 0x7f96c4113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 -- 172.21.15.103:0/2023418540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 msgr2=0x7f96c4113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 0x7f96c4113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c27fc700 1 -- 172.21.15.103:0/2023418540 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc009940 con 0x7f96c40fe830 2024-03-22T02:45:32.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.353+0000 7f96c2ffd700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 0x7f96c4113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96c27fc700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b790 tx=0x7f96b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b800f770 con 0x7f96c40fe830 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c410de30 con 0x7f96c40fe830 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96b800fda0 con 0x7f96c40fe830 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c410e370 con 0x7f96c40fe830 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.354+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b800d310 con 0x7f96c40fe830 2024-03-22T02:45:32.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.355+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b00052b0 con 0x7f96c40fe830 2024-03-22T02:45:32.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.356+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f96b800f8d0 con 0x7f96c40fe830 2024-03-22T02:45:32.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.356+0000 7f96ab7fe700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 0x7f96ac070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.356+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f96b808dd80 con 0x7f96c40fe830 2024-03-22T02:45:32.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.356+0000 7f96c2ffd700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 0x7f96ac070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.356+0000 7f96c2ffd700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 0x7f96ac070850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00b740 tx=0x7f96bc004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.361+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96b805aab0 con 0x7f96c40fe830 2024-03-22T02:45:32.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.564+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f96b0000bc0 con 0x7f96ac06e3a0 2024-03-22T02:45:32.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.574+0000 7f96ab7fe700 1 -- 172.21.15.103:0/2023418540 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f96b0000bc0 con 0x7f96ac06e3a0 2024-03-22T02:45:32.575 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (14m) 7m ago 25m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (11m) 7m ago 25m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (11m) 7m ago 24m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (13m) 7m ago 24m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (8m) 7m ago 19m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (7m) 7m ago 19m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (7m) 7m ago 19m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (7m) 7m ago 19m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (15m) 7m ago 26m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (15m) 7m ago 24m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (12m) 7m ago 26m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (12m) 7m ago 24m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (14m) 7m ago 25m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:45:32.576 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (14m) 7m ago 24m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (11m) 7m ago 23m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (10m) 7m ago 23m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (10m) 7m ago 22m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (9m) 7m ago 22m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (8m) 7m ago 21m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (8m) 7m ago 21m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:45:32.577 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (14m) 7m ago 24m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 msgr2=0x7f96ac070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 0x7f96ac070850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00b740 tx=0x7f96bc004d10 comp rx=0 tx=0).stop 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 msgr2=0x7f96c41140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b790 tx=0x7f96b8009250 comp rx=0 tx=0).stop 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 shutdown_connections 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f96ac06e3a0 0x7f96ac070850 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96c40fde90 0x7f96c4113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 --2- 172.21.15.103:0/2023418540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96c40fe830 0x7f96c41140c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 >> 172.21.15.103:0/2023418540 conn(0x7f96c4072bc0 msgr2=0x7f96c40ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 shutdown_connections 2024-03-22T02:45:32.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.576+0000 7f96c9725700 1 -- 172.21.15.103:0/2023418540 wait complete. 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 -- 172.21.15.103:0/3638022896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2748101d20 msgr2=0x7f27481020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 --2- 172.21.15.103:0/3638022896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2748101d20 0x7f27481020f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2738004940 tx=0x7f2738009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 -- 172.21.15.103:0/3638022896 shutdown_connections 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 --2- 172.21.15.103:0/3638022896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27481026c0 0x7f27480fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 --2- 172.21.15.103:0/3638022896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2748101d20 0x7f27481020f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.676+0000 7f274e86a700 1 -- 172.21.15.103:0/3638022896 >> 172.21.15.103:0/3638022896 conn(0x7f27480f7870 msgr2=0x7f27480f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 -- 172.21.15.103:0/3638022896 shutdown_connections 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 -- 172.21.15.103:0/3638022896 wait complete. 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 Processor -- start 2024-03-22T02:45:32.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 -- start start 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2748101d20 0x7f2748102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2748104ad0 con 0x7f27481026c0 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.677+0000 7f274e86a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2748103600 con 0x7f2748101d20 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45778/0 (socket says 172.21.15.103:45778) 2024-03-22T02:45:32.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 -- 172.21.15.103:0/1183118676 learned_addr learned my addr 172.21.15.103:0/1183118676 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f2747fff700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2748101d20 0x7f2748102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 -- 172.21.15.103:0/1183118676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2748101d20 msgr2=0x7f2748102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2748101d20 0x7f2748102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 -- 172.21.15.103:0/1183118676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738005040 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f27477fe700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c800 tx=0x7f273c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f273c00a760 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27481038e0 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f273c00ad90 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f273c0113a0 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.678+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274811a180 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.679+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27480fd470 con 0x7f27481026c0 2024-03-22T02:45:32.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.680+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f273c012400 con 0x7f27481026c0 2024-03-22T02:45:32.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.680+0000 7f2744ff9700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 0x7f2730070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:32.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.680+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f273c08de00 con 0x7f27481026c0 2024-03-22T02:45:32.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.680+0000 7f2747fff700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 0x7f2730070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:32.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.681+0000 7f2747fff700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 0x7f2730070850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2738004f90 tx=0x7f2738009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:32.685 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.685+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f273c057080 con 0x7f27481026c0 2024-03-22T02:45:32.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.967+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f274811a3d0 con 0x7f27481026c0 2024-03-22T02:45:32.968 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:32 smithi103 ceph-mon[115211]: pgmap v420: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:32.968 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:32.968 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:32 smithi103 ceph-mon[115211]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.967+0000 7f2744ff9700 1 -- 172.21.15.103:0/1183118676 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f273c05a740 con 0x7f27481026c0 2024-03-22T02:45:32.968 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:45:32.968 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:45:32.969 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:45:32.970 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:45:32.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 msgr2=0x7f2730070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 0x7f2730070850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2738004f90 tx=0x7f2738009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 msgr2=0x7f27481030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c800 tx=0x7f273c004470 comp rx=0 tx=0).stop 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 shutdown_connections 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f273006e3a0 0x7f2730070850 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2748101d20 0x7f2748102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 --2- 172.21.15.103:0/1183118676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27481026c0 0x7f27481030c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.969+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 >> 172.21.15.103:0/1183118676 conn(0x7f27480f7870 msgr2=0x7f27480f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.970+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 shutdown_connections 2024-03-22T02:45:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:32.970+0000 7f274e86a700 1 -- 172.21.15.103:0/1183118676 wait complete. 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/599903367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4101ce0 msgr2=0x7fd2f41020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/599903367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4101ce0 0x7fd2f41020b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0004940 tx=0x7fd2f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/599903367 shutdown_connections 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/599903367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4102680 0x7fd2f40fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/599903367 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4101ce0 0x7fd2f41020b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/599903367 >> 172.21.15.103:0/599903367 conn(0x7fd2f40f7890 msgr2=0x7fd2f40f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/599903367 shutdown_connections 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/599903367 wait complete. 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 Processor -- start 2024-03-22T02:45:33.071 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- start start 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4101ce0 0x7fd2f4102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f4104ae0 con 0x7fd2f4102680 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.070+0000 7fd2fb2e2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f41035c0 con 0x7fd2f4101ce0 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45790/0 (socket says 172.21.15.103:45790) 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 -- 172.21.15.103:0/1376761004 learned_addr learned my addr 172.21.15.103:0/1376761004 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:33.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f907e700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4101ce0 0x7fd2f4102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 -- 172.21.15.103:0/1376761004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4101ce0 msgr2=0x7fd2f4102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4101ce0 0x7fd2f4102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 -- 172.21.15.103:0/1376761004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0005040 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2f887d700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4002a50 tx=0x7fd2e4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e4007cc0 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2e400f040 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f4103870 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e400da40 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.071+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f411a130 con 0x7fd2f4102680 2024-03-22T02:45:33.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.072+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f40fd430 con 0x7fd2f4102680 2024-03-22T02:45:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.073+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd2e40097c0 con 0x7fd2f4102680 2024-03-22T02:45:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.073+0000 7fd2e9ffb700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 0x7fd2e0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.073+0000 7fd2f907e700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 0x7fd2e0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.073+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd2e408dc20 con 0x7fd2f4102680 2024-03-22T02:45:33.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.074+0000 7fd2f907e700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 0x7fd2e0070850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0004f90 tx=0x7fd2f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.078 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.078+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2e405a950 con 0x7fd2f4102680 2024-03-22T02:45:33.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.318+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd2f4108e70 con 0x7fd2f4102680 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.318+0000 7fd2e9ffb700 1 -- 172.21.15.103:0/1376761004 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fd2e405fe60 con 0x7fd2f4102680 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:45:33.319 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:45:33.320 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:33.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:33.322 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 msgr2=0x7fd2e0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 0x7fd2e0070850 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0004f90 tx=0x7fd2f0009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 msgr2=0x7fd2f4103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4002a50 tx=0x7fd2e4004600 comp rx=0 tx=0).stop 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 shutdown_connections 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd2e006e3a0 0x7fd2e0070850 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2f4101ce0 0x7fd2f4102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 --2- 172.21.15.103:0/1376761004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2f4102680 0x7fd2f4103080 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 >> 172.21.15.103:0/1376761004 conn(0x7fd2f40f7890 msgr2=0x7fd2f40662f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:33.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 shutdown_connections 2024-03-22T02:45:33.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.320+0000 7fd2fb2e2700 1 -- 172.21.15.103:0/1376761004 wait complete. 2024-03-22T02:45:33.324 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:45:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:32 smithi122 ceph-mon[83368]: pgmap v420: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:32 smithi122 ceph-mon[83368]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:33.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 -- 172.21.15.103:0/928799373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 msgr2=0x7fe3240fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 --2- 172.21.15.103:0/928799373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe3240fcc40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe30c004940 tx=0x7fe30c009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:33.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 -- 172.21.15.103:0/928799373 shutdown_connections 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 --2- 172.21.15.103:0/928799373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe3240fcc40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 --2- 172.21.15.103:0/928799373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe324103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 -- 172.21.15.103:0/928799373 >> 172.21.15.103:0/928799373 conn(0x7fe3240f8760 msgr2=0x7fe3240fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 -- 172.21.15.103:0/928799373 shutdown_connections 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.422+0000 7fe328b9d700 1 -- 172.21.15.103:0/928799373 wait complete. 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 Processor -- start 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 -- start start 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe3241139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324114570 con 0x7fe3241035d0 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe328b9d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32410d960 con 0x7fe324102c30 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45806/0 (socket says 172.21.15.103:45806) 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 -- 172.21.15.103:0/2301794006 learned_addr learned my addr 172.21.15.103:0/2301794006 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:33.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe3237fe700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe3241139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 -- 172.21.15.103:0/2301794006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 msgr2=0x7fe3241139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe3241139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 -- 172.21.15.103:0/2301794006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c005040 con 0x7fe3241035d0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe3237fe700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe3241139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.423+0000 7fe31bfff700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009100 tx=0x7fe30c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c00e040 con 0x7fe3241035d0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe30c004290 con 0x7fe3241035d0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32410dbe0 con 0x7fe3241035d0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30c01c3f0 con 0x7fe3241035d0 2024-03-22T02:45:33.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32410e1a0 con 0x7fe3241035d0 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.424+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3240fe380 con 0x7fe3241035d0 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.425+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe30c013790 con 0x7fe3241035d0 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.425+0000 7fe320ff9700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 0x7fe310070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.426+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe30c08f100 con 0x7fe3241035d0 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.426+0000 7fe3237fe700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 0x7fe310070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.426+0000 7fe3237fe700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 0x7fe310070850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe314009fd0 tx=0x7fe314009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.430+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe30c05be30 con 0x7fe3241035d0 2024-03-22T02:45:33.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.649+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe324109d00 con 0x7fe31006e3a0 2024-03-22T02:45:33.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.650+0000 7fe320ff9700 1 -- 172.21.15.103:0/2301794006 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe324109d00 con 0x7fe31006e3a0 2024-03-22T02:45:33.651 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:45:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:45:33.651 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:45:33.652 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 msgr2=0x7fe310070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 0x7fe310070850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe314009fd0 tx=0x7fe314009ee0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 msgr2=0x7fe324113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009100 tx=0x7fe30c009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 shutdown_connections 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe31006e3a0 0x7fe310070850 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe324102c30 0x7fe3241139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 --2- 172.21.15.103:0/2301794006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3241035d0 0x7fe324113f20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.652+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 >> 172.21.15.103:0/2301794006 conn(0x7fe3240f8760 msgr2=0x7fe3240f9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:33.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.653+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 shutdown_connections 2024-03-22T02:45:33.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.653+0000 7fe328b9d700 1 -- 172.21.15.103:0/2301794006 wait complete. 2024-03-22T02:45:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- 172.21.15.103:0/3240559690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c102e10 msgr2=0x7f739c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 --2- 172.21.15.103:0/3240559690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c102e10 0x7f739c1031e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7390004940 tx=0x7f7390009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- 172.21.15.103:0/3240559690 shutdown_connections 2024-03-22T02:45:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 --2- 172.21.15.103:0/3240559690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c1037b0 0x7f739c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 --2- 172.21.15.103:0/3240559690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c102e10 0x7f739c1031e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- 172.21.15.103:0/3240559690 >> 172.21.15.103:0/3240559690 conn(0x7f739c0f89a0 msgr2=0x7f739c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- 172.21.15.103:0/3240559690 shutdown_connections 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- 172.21.15.103:0/3240559690 wait complete. 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 Processor -- start 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.753+0000 7f73a309b700 1 -- start start 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a309b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 0x7f739c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a309b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a309b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c070e20 con 0x7f739c1037b0 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a309b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c070f60 con 0x7f739c102e10 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33972/0 (socket says 172.21.15.103:33972) 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 -- 172.21.15.103:0/2311148724 learned_addr learned my addr 172.21.15.103:0/2311148724 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a0e37700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 0x7f739c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 -- 172.21.15.103:0/2311148724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 msgr2=0x7f739c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 0x7f739c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 -- 172.21.15.103:0/2311148724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390005040 con 0x7f739c1037b0 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f73a0e37700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 0x7f739c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:45:33.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.754+0000 7f739bfff700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f738800b790 tx=0x7f7388009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.755+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f738800f6f0 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.755+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f738800fd20 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.755+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c119e90 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.755+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73880103a0 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.755+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c11a370 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.756+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c0fe560 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.756+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7388010500 con 0x7f739c1037b0 2024-03-22T02:45:33.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.756+0000 7f73997fa700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 0x7f738c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:45:33.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.757+0000 7f73a0e37700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 0x7f738c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:45:33.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.757+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f738808dc80 con 0x7f739c1037b0 2024-03-22T02:45:33.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.757+0000 7f73a0e37700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 0x7f738c070850 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7390004560 tx=0x7f7390009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:45:33.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:33.761+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7388056e80 con 0x7f739c1037b0 2024-03-22T02:45:34.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.044+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f739c05f5f0 con 0x7f739c1037b0 2024-03-22T02:45:34.045 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:33 smithi103 ceph-mon[115211]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:34.045 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:33 smithi103 ceph-mon[115211]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:34.045 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:33 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1183118676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:45:34.045 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:33 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1376761004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:45:34.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.044+0000 7f73997fa700 1 -- 172.21.15.103:0/2311148724 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f738805a540 con 0x7f739c1037b0 2024-03-22T02:45:34.045 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:45:34.046 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:45:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:45:34.046 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:45:34.046 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 msgr2=0x7f738c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 0x7f738c070850 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7390004560 tx=0x7f7390009e30 comp rx=0 tx=0).stop 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 msgr2=0x7f739c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f738800b790 tx=0x7f7388009220 comp rx=0 tx=0).stop 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 shutdown_connections 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f738c06e3a0 0x7f738c070850 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f739c102e10 0x7f739c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 --2- 172.21.15.103:0/2311148724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f739c1037b0 0x7f739c070800 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 >> 172.21.15.103:0/2311148724 conn(0x7f739c0f89a0 msgr2=0x7f739c0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 shutdown_connections 2024-03-22T02:45:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:45:34.046+0000 7f73a309b700 1 -- 172.21.15.103:0/2311148724 wait complete. 2024-03-22T02:45:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:33 smithi122 ceph-mon[83368]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:33 smithi122 ceph-mon[83368]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:33 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1183118676' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:45:34.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:33 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1376761004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:45:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:34 smithi103 ceph-mon[115211]: pgmap v421: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:34 smithi103 ceph-mon[115211]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:34 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2311148724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:45:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:34 smithi122 ceph-mon[83368]: pgmap v421: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:34 smithi122 ceph-mon[83368]: from='client.34836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:45:35.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:34 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2311148724' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:45:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:36 smithi103 ceph-mon[115211]: pgmap v422: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:36 smithi122 ceph-mon[83368]: pgmap v422: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:39.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:38 smithi103 ceph-mon[115211]: pgmap v423: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:38 smithi122 ceph-mon[83368]: pgmap v423: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:41.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:40 smithi103 ceph-mon[115211]: pgmap v424: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:40 smithi122 ceph-mon[83368]: pgmap v424: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:42 smithi103 ceph-mon[115211]: pgmap v425: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:42 smithi122 ceph-mon[83368]: pgmap v425: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:44 smithi122 ceph-mon[83368]: pgmap v426: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:44 smithi103 ceph-mon[115211]: pgmap v426: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:46 smithi122 ceph-mon[83368]: pgmap v427: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:46 smithi103 ceph-mon[115211]: pgmap v427: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:48.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:45:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:48 smithi122 ceph-mon[83368]: pgmap v428: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:49.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:48 smithi103 ceph-mon[115211]: pgmap v428: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:50 smithi122 ceph-mon[83368]: pgmap v429: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:50 smithi103 ceph-mon[115211]: pgmap v429: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:53.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:52 smithi122 ceph-mon[83368]: pgmap v430: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:53.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:52 smithi103 ceph-mon[115211]: pgmap v430: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:54 smithi122 ceph-mon[83368]: pgmap v431: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:54 smithi103 ceph-mon[115211]: pgmap v431: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:57.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:56 smithi122 ceph-mon[83368]: pgmap v432: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:56 smithi103 ceph-mon[115211]: pgmap v432: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:59.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:45:58 smithi122 ceph-mon[83368]: pgmap v433: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:45:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:45:58 smithi103 ceph-mon[115211]: pgmap v433: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:00 smithi122 ceph-mon[83368]: pgmap v434: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:00 smithi103 ceph-mon[115211]: pgmap v434: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:02 smithi122 ceph-mon[83368]: pgmap v435: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:02 smithi103 ceph-mon[115211]: pgmap v435: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- 172.21.15.103:0/273316075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4103780 msgr2=0x7f61d40fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 --2- 172.21.15.103:0/273316075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4103780 0x7f61d40fcdf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f61c0004940 tx=0x7f61c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- 172.21.15.103:0/273316075 shutdown_connections 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 --2- 172.21.15.103:0/273316075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4103780 0x7f61d40fcdf0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 --2- 172.21.15.103:0/273316075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4102de0 0x7f61d41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- 172.21.15.103:0/273316075 >> 172.21.15.103:0/273316075 conn(0x7f61d40f8950 msgr2=0x7f61d40fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- 172.21.15.103:0/273316075 shutdown_connections 2024-03-22T02:46:04.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- 172.21.15.103:0/273316075 wait complete. 2024-03-22T02:46:04.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 Processor -- start 2024-03-22T02:46:04.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.162+0000 7f61dab27700 1 -- start start 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61dab27700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61dab27700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 0x7f61d4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61dab27700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d4072210 con 0x7f61d4102de0 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61dab27700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d4070d40 con 0x7f61d4103780 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55020/0 (socket says 172.21.15.103:55020) 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 -- 172.21.15.103:0/2337724251 learned_addr learned my addr 172.21.15.103:0/2337724251 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d3fff700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 0x7f61d4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 -- 172.21.15.103:0/2337724251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 msgr2=0x7f61d4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 0x7f61d4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 -- 172.21.15.103:0/2337724251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c0005040 con 0x7f61d4102de0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d3fff700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 0x7f61d4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.163+0000 7f61d88c3700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c8c0 tx=0x7f61c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.164+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61c800a980 con 0x7f61d4102de0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.164+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61c800ce60 con 0x7f61d4102de0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.164+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61c80115e0 con 0x7f61d4102de0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.164+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d4071020 con 0x7f61d4102de0 2024-03-22T02:46:04.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.164+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d411a120 con 0x7f61d4102de0 2024-03-22T02:46:04.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.165+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d40fe530 con 0x7f61d4102de0 2024-03-22T02:46:04.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.165+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f61c8012400 con 0x7f61d4102de0 2024-03-22T02:46:04.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.166+0000 7f61d17fa700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 0x7f61c4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.166+0000 7f61d3fff700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 0x7f61c4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.166+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f61c808ddb0 con 0x7f61d4102de0 2024-03-22T02:46:04.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.166+0000 7f61d3fff700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 0x7f61c4070850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.171+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f61c80570f0 con 0x7f61d4102de0 2024-03-22T02:46:04.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.389+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d40f9fd0 con 0x7f61c406e3a0 2024-03-22T02:46:04.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.391+0000 7f61d17fa700 1 -- 172.21.15.103:0/2337724251 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f61d40f9fd0 con 0x7f61c406e3a0 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 msgr2=0x7f61c4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 0x7f61c4070850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c0009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 msgr2=0x7f61d40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c8c0 tx=0x7f61c80044c0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 shutdown_connections 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61c406e3a0 0x7f61c4070850 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f61d4102de0 0x7f61d40702c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 --2- 172.21.15.103:0/2337724251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61d4103780 0x7f61d4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 >> 172.21.15.103:0/2337724251 conn(0x7f61d40f8950 msgr2=0x7f61d40f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 shutdown_connections 2024-03-22T02:46:04.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.393+0000 7f61dab27700 1 -- 172.21.15.103:0/2337724251 wait complete. 2024-03-22T02:46:04.404 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:46:04.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- 172.21.15.103:0/3129108522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 msgr2=0x7f42bc0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3129108522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc0fbd30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f42b0004940 tx=0x7f42b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- 172.21.15.103:0/3129108522 shutdown_connections 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3129108522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc0fbd30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3129108522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 0x7f42bc1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- 172.21.15.103:0/3129108522 >> 172.21.15.103:0/3129108522 conn(0x7f42bc0f7870 msgr2=0x7f42bc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- 172.21.15.103:0/3129108522 shutdown_connections 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- 172.21.15.103:0/3129108522 wait complete. 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 Processor -- start 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.496+0000 7f42c0d00700 1 -- start start 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 0x7f42bc10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc10f5e0 con 0x7f42bc1026c0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc10f720 con 0x7f42bc101d20 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55044/0 (socket says 172.21.15.103:55044) 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 -- 172.21.15.103:0/3612536195 learned_addr learned my addr 172.21.15.103:0/3612536195 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42ba59c700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 0x7f42bc10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 -- 172.21.15.103:0/3612536195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 msgr2=0x7f42bc10eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 0x7f42bc10eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 -- 172.21.15.103:0/3612536195 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0005040 con 0x7f42bc1026c0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42b9d9b700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009100 tx=0x7f42b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42b000e040 con 0x7f42bc1026c0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc10f9a0 con 0x7f42bc1026c0 2024-03-22T02:46:04.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.497+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc117ed0 con 0x7f42bc1026c0 2024-03-22T02:46:04.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.498+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42b0009af0 con 0x7f42bc1026c0 2024-03-22T02:46:04.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.498+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42b001c3f0 con 0x7f42bc1026c0 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.499+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c0052b0 con 0x7f42bc1026c0 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.499+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f42b001c660 con 0x7f42bc1026c0 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.499+0000 7f42aaffd700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 0x7f42a4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.499+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f42b008ee80 con 0x7f42bc1026c0 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.500+0000 7f42ba59c700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 0x7f42a4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.500+0000 7f42ba59c700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 0x7f42a4070850 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f42ac004eb0 tx=0x7f42ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.504+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42b0058270 con 0x7f42bc1026c0 2024-03-22T02:46:04.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.721+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f429c000bc0 con 0x7f42a406e3a0 2024-03-22T02:46:04.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.722+0000 7f42aaffd700 1 -- 172.21.15.103:0/3612536195 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f429c000bc0 con 0x7f42a406e3a0 2024-03-22T02:46:04.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 msgr2=0x7f42a4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 0x7f42a4070850 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f42ac004eb0 tx=0x7f42ac00b040 comp rx=0 tx=0).stop 2024-03-22T02:46:04.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 msgr2=0x7f42bc10f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009100 tx=0x7f42b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:04.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 shutdown_connections 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42a406e3a0 0x7f42a4070850 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42bc101d20 0x7f42bc10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 --2- 172.21.15.103:0/3612536195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42bc1026c0 0x7f42bc10f0a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 >> 172.21.15.103:0/3612536195 conn(0x7f42bc0f7870 msgr2=0x7f42bc0f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 shutdown_connections 2024-03-22T02:46:04.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.724+0000 7f42c0d00700 1 -- 172.21.15.103:0/3612536195 wait complete. 2024-03-22T02:46:04.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 -- 172.21.15.103:0/1551373557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 msgr2=0x7efe800fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 --2- 172.21.15.103:0/1551373557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe800fce10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efe74004940 tx=0x7efe74009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:04.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 -- 172.21.15.103:0/1551373557 shutdown_connections 2024-03-22T02:46:04.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 --2- 172.21.15.103:0/1551373557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe800fce10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 --2- 172.21.15.103:0/1551373557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 0x7efe801031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 -- 172.21.15.103:0/1551373557 >> 172.21.15.103:0/1551373557 conn(0x7efe800f89f0 msgr2=0x7efe800fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:04.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 -- 172.21.15.103:0/1551373557 shutdown_connections 2024-03-22T02:46:04.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.825+0000 7efe85174700 1 -- 172.21.15.103:0/1551373557 wait complete. 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 Processor -- start 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 -- start start 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 0x7efe80113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe801144b0 con 0x7efe801037a0 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe85174700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8010dab0 con 0x7efe80102e00 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe7e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.826+0000 7efe7e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55050/0 (socket says 172.21.15.103:55050) 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7e59c700 1 -- 172.21.15.103:0/1248529495 learned_addr learned my addr 172.21.15.103:0/1248529495 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7ed9d700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 0x7efe80113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7e59c700 1 -- 172.21.15.103:0/1248529495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 msgr2=0x7efe80113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7e59c700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 0x7efe80113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7e59c700 1 -- 172.21.15.103:0/1248529495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74005040 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe7e59c700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efe74000c00 tx=0x7efe74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe74021070 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe74013c00 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe7400e070 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8010dd30 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.827+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8010e210 con 0x7efe801037a0 2024-03-22T02:46:04.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.828+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe801052d0 con 0x7efe801037a0 2024-03-22T02:46:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.829+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efe7401ca60 con 0x7efe801037a0 2024-03-22T02:46:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.829+0000 7efe677fe700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 0x7efe68070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.829+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efe7408f1d0 con 0x7efe801037a0 2024-03-22T02:46:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.829+0000 7efe7ed9d700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 0x7efe68070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:04.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.830+0000 7efe7ed9d700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 0x7efe68070850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe70009b80 tx=0x7efe7000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:04.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:04.834+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe7405bf80 con 0x7efe801037a0 2024-03-22T02:46:05.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.041+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efe8010efe0 con 0x7efe6806e3a0 2024-03-22T02:46:05.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.051+0000 7efe677fe700 1 -- 172.21.15.103:0/1248529495 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efe8010efe0 con 0x7efe6806e3a0 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (14m) 8m ago 25m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (12m) 8m ago 26m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (12m) 7m ago 24m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (14m) 8m ago 25m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (8m) 8m ago 20m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (8m) 8m ago 20m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (7m) 7m ago 20m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (8m) 7m ago 20m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (16m) 8m ago 27m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:46:05.052 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (15m) 7m ago 24m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (13m) 8m ago 27m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (13m) 7m ago 24m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (15m) 8m ago 25m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (15m) 7m ago 24m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (11m) 8m ago 24m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (11m) 8m ago 23m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (10m) 8m ago 23m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (10m) 7m ago 22m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (9m) 7m ago 22m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (8m) 7m ago 21m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:46:05.053 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (14m) 8m ago 25m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 msgr2=0x7efe68070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 0x7efe68070850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe70009b80 tx=0x7efe7000b040 comp rx=0 tx=0).stop 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 msgr2=0x7efe80113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efe74000c00 tx=0x7efe74009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 shutdown_connections 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efe6806e3a0 0x7efe68070850 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe80102e00 0x7efe80113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 --2- 172.21.15.103:0/1248529495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe801037a0 0x7efe80113e60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 >> 172.21.15.103:0/1248529495 conn(0x7efe800f89f0 msgr2=0x7efe800fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 shutdown_connections 2024-03-22T02:46:05.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.053+0000 7efe85174700 1 -- 172.21.15.103:0/1248529495 wait complete. 2024-03-22T02:46:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:04 smithi103 ceph-mon[115211]: pgmap v436: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 -- 172.21.15.103:0/2734466442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 msgr2=0x7ff960103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 --2- 172.21.15.103:0/2734466442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960103320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff950004940 tx=0x7ff950009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 -- 172.21.15.103:0/2734466442 shutdown_connections 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 --2- 172.21.15.103:0/2734466442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 0x7ff9600fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 --2- 172.21.15.103:0/2734466442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960103320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 -- 172.21.15.103:0/2734466442 >> 172.21.15.103:0/2734466442 conn(0x7ff9600714c0 msgr2=0x7ff9600718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 -- 172.21.15.103:0/2734466442 shutdown_connections 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.153+0000 7ff965154700 1 -- 172.21.15.103:0/2734466442 wait complete. 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 Processor -- start 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 -- start start 2024-03-22T02:46:05.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 0x7ff960113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960114590 con 0x7ff960102f50 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff965154700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010dbe0 con 0x7ff9600fb4d0 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff95e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff95e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55074/0 (socket says 172.21.15.103:55074) 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff95e59c700 1 -- 172.21.15.103:0/774042515 learned_addr learned my addr 172.21.15.103:0/774042515 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.154+0000 7ff95ed9d700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 0x7ff960113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff95e59c700 1 -- 172.21.15.103:0/774042515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 msgr2=0x7ff960113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff95e59c700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 0x7ff960113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff95e59c700 1 -- 172.21.15.103:0/774042515 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff950005040 con 0x7ff960102f50 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff95e59c700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c810 tx=0x7ff954004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff954012810 con 0x7ff960102f50 2024-03-22T02:46:05.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff954012e40 con 0x7ff960102f50 2024-03-22T02:46:05.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96010dec0 con 0x7ff960102f50 2024-03-22T02:46:05.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9540102e0 con 0x7ff960102f50 2024-03-22T02:46:05.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.155+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96010e400 con 0x7ff960102f50 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.156+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960100200 con 0x7ff960102f50 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.157+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff954011380 con 0x7ff960102f50 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.157+0000 7ff9477fe700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 0x7ff948070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.157+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff95408e1d0 con 0x7ff960102f50 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.157+0000 7ff95ed9d700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 0x7ff948070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.157+0000 7ff95ed9d700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 0x7ff948070850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff950004890 tx=0x7ff950009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.162+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff95405aec0 con 0x7ff960102f50 2024-03-22T02:46:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:04 smithi122 ceph-mon[83368]: pgmap v436: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:05.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.442+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff96010efe0 con 0x7ff960102f50 2024-03-22T02:46:05.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.442+0000 7ff9477fe700 1 -- 172.21.15.103:0/774042515 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff954011630 con 0x7ff960102f50 2024-03-22T02:46:05.443 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:46:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:46:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:46:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:46:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:46:05.445 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:46:05.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 msgr2=0x7ff948070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 0x7ff948070850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff950004890 tx=0x7ff950009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 msgr2=0x7ff960113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c810 tx=0x7ff954004470 comp rx=0 tx=0).stop 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 shutdown_connections 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff94806e3a0 0x7ff948070850 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9600fb4d0 0x7ff960113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 --2- 172.21.15.103:0/774042515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff960102f50 0x7ff960113f40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.444+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 >> 172.21.15.103:0/774042515 conn(0x7ff9600714c0 msgr2=0x7ff9600fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.445+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 shutdown_connections 2024-03-22T02:46:05.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.445+0000 7ff965154700 1 -- 172.21.15.103:0/774042515 wait complete. 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 -- 172.21.15.103:0/2808288516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0102680 msgr2=0x7fc4d00fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 --2- 172.21.15.103:0/2808288516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0102680 0x7fc4d00fbcf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4004940 tx=0x7fc4c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 -- 172.21.15.103:0/2808288516 shutdown_connections 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 --2- 172.21.15.103:0/2808288516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0102680 0x7fc4d00fbcf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 --2- 172.21.15.103:0/2808288516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0101ce0 0x7fc4d01020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 -- 172.21.15.103:0/2808288516 >> 172.21.15.103:0/2808288516 conn(0x7fc4d00f7870 msgr2=0x7fc4d00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 -- 172.21.15.103:0/2808288516 shutdown_connections 2024-03-22T02:46:05.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.545+0000 7fc4d647a700 1 -- 172.21.15.103:0/2808288516 wait complete. 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 Processor -- start 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 -- start start 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 0x7fc4d010f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d010f570 con 0x7fc4d0101ce0 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4d647a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d010f6b0 con 0x7fc4d0102680 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4cffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55092/0 (socket says 172.21.15.103:55092) 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.546+0000 7fc4cffff700 1 -- 172.21.15.103:0/1368233099 learned_addr learned my addr 172.21.15.103:0/1368233099 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cf7fe700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 0x7fc4d010f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cffff700 1 -- 172.21.15.103:0/1368233099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 msgr2=0x7fc4d010f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cffff700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 0x7fc4d010f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cffff700 1 -- 172.21.15.103:0/1368233099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4005040 con 0x7fc4d0101ce0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cf7fe700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 0x7fc4d010f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4cffff700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000c830 tx=0x7fc4c0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c000a760 con 0x7fc4d0101ce0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d0117b60 con 0x7fc4d0101ce0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4c000ad90 con 0x7fc4d0101ce0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d0117f30 con 0x7fc4d0101ce0 2024-03-22T02:46:05.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.547+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c00113a0 con 0x7fc4d0101ce0 2024-03-22T02:46:05.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.548+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc0052b0 con 0x7fc4d0101ce0 2024-03-22T02:46:05.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.549+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc4c0012400 con 0x7fc4d0101ce0 2024-03-22T02:46:05.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.549+0000 7fc4ccff9700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 0x7fc4b8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.549+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc4c008de20 con 0x7fc4d0101ce0 2024-03-22T02:46:05.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.549+0000 7fc4cf7fe700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 0x7fc4b8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.549+0000 7fc4cf7fe700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 0x7fc4b8070850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400b3b0 tx=0x7fc4c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.554+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc4c00570a0 con 0x7fc4d0101ce0 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.794+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc4bc005d70 con 0x7fc4d0101ce0 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.794+0000 7fc4ccff9700 1 -- 172.21.15.103:0/1368233099 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fc4c005a760 con 0x7fc4d0101ce0 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:46:05.795 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:46:05.796 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:46:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:05.798 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 msgr2=0x7fc4b8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 0x7fc4b8070850 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400b3b0 tx=0x7fc4c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 msgr2=0x7fc4d010eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000c830 tx=0x7fc4c0004470 comp rx=0 tx=0).stop 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 shutdown_connections 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc4b806e3a0 0x7fc4b8070850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4d0101ce0 0x7fc4d010eaf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 --2- 172.21.15.103:0/1368233099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d0102680 0x7fc4d010f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 >> 172.21.15.103:0/1368233099 conn(0x7fc4d00f7870 msgr2=0x7fc4d00f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 shutdown_connections 2024-03-22T02:46:05.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.797+0000 7fc4d647a700 1 -- 172.21.15.103:0/1368233099 wait complete. 2024-03-22T02:46:05.800 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:46:05.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.897+0000 7f873f9b6700 1 -- 172.21.15.103:0/3630932862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102e00 msgr2=0x7f87381031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3630932862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102e00 0x7f87381031d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8734004940 tx=0x7f8734009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- 172.21.15.103:0/3630932862 shutdown_connections 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3630932862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87381037a0 0x7f87380fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3630932862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102e00 0x7f87381031d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- 172.21.15.103:0/3630932862 >> 172.21.15.103:0/3630932862 conn(0x7f87380f89b0 msgr2=0x7f87380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- 172.21.15.103:0/3630932862 shutdown_connections 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- 172.21.15.103:0/3630932862 wait complete. 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 Processor -- start 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- start start 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 0x7f8738113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738114510 con 0x7f87381037a0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.898+0000 7f873f9b6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873810db30 con 0x7f8738102e00 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55104/0 (socket says 172.21.15.103:55104) 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 -- 172.21.15.103:0/3229795869 learned_addr learned my addr 172.21.15.103:0/3229795869 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873d752700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 0x7f8738113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 -- 172.21.15.103:0/3229795869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 msgr2=0x7f8738113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 0x7f8738113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 -- 172.21.15.103:0/3229795869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734005040 con 0x7f87381037a0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873d752700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 0x7f8738113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873cf51700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f872800c7e0 tx=0x7f8728007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8728012810 con 0x7f87381037a0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8728012e40 con 0x7f87381037a0 2024-03-22T02:46:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872800a550 con 0x7f87381037a0 2024-03-22T02:46:05.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873810de10 con 0x7f87381037a0 2024-03-22T02:46:05.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.899+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873810e380 con 0x7f87381037a0 2024-03-22T02:46:05.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.900+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8738100110 con 0x7f87381037a0 2024-03-22T02:46:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.901+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8728011380 con 0x7f87381037a0 2024-03-22T02:46:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.901+0000 7f872dffb700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 0x7f8724070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.901+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f872808e210 con 0x7f87381037a0 2024-03-22T02:46:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.901+0000 7f873d752700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 0x7f8724070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:05.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.901+0000 7f873d752700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 0x7f8724070850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f873400b3b0 tx=0x7f8734009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:05.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:05.906+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8728057410 con 0x7f87381037a0 2024-03-22T02:46:06.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.120+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f873810efd0 con 0x7f872406e3a0 2024-03-22T02:46:06.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:05 smithi103 ceph-mon[115211]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:05 smithi103 ceph-mon[115211]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:05 smithi103 ceph-mon[115211]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:05 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/774042515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:46:06.121 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:05 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1368233099' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.121+0000 7f872dffb700 1 -- 172.21.15.103:0/3229795869 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f873810efd0 con 0x7f872406e3a0 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:46:06.122 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:46:06.123 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 msgr2=0x7f8724070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 0x7f8724070850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f873400b3b0 tx=0x7f8734009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 msgr2=0x7f8738113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f872800c7e0 tx=0x7f8728007f90 comp rx=0 tx=0).stop 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 shutdown_connections 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f872406e3a0 0x7f8724070850 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8738102e00 0x7f8738113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 --2- 172.21.15.103:0/3229795869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87381037a0 0x7f8738113ec0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.123+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 >> 172.21.15.103:0/3229795869 conn(0x7f87380f89b0 msgr2=0x7f87381073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.124+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 shutdown_connections 2024-03-22T02:46:06.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.124+0000 7f873f9b6700 1 -- 172.21.15.103:0/3229795869 wait complete. 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- 172.21.15.103:0/2880553164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3081037c0 msgr2=0x7fb3080fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 --2- 172.21.15.103:0/2880553164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3081037c0 0x7fb3080fce30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8004940 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- 172.21.15.103:0/2880553164 shutdown_connections 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 --2- 172.21.15.103:0/2880553164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3081037c0 0x7fb3080fce30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 --2- 172.21.15.103:0/2880553164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb308102e20 0x7fb3081031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- 172.21.15.103:0/2880553164 >> 172.21.15.103:0/2880553164 conn(0x7fb3080f89d0 msgr2=0x7fb3080fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- 172.21.15.103:0/2880553164 shutdown_connections 2024-03-22T02:46:06.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- 172.21.15.103:0/2880553164 wait complete. 2024-03-22T02:46:06.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 Processor -- start 2024-03-22T02:46:06.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.224+0000 7fb30fade700 1 -- start start 2024-03-22T02:46:06.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 0x7fb308070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb308070d90 con 0x7fb308102e20 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb308070ed0 con 0x7fb3081037c0 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:55118/0 (socket says 172.21.15.103:55118) 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 -- 172.21.15.103:0/3257471834 learned_addr learned my addr 172.21.15.103:0/3257471834 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d079700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 0x7fb308070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 -- 172.21.15.103:0/3257471834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 msgr2=0x7fb308070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 0x7fb308070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 -- 172.21.15.103:0/3257471834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8005040 con 0x7fb308102e20 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d079700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 0x7fb308070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:46:06.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30d87a700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb304002a50 tx=0x7fb3040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb304007cc0 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb30400f040 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb308119ef0 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb304009640 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.225+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30811a370 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.226+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3081052f0 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.227+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb3040097a0 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.227+0000 7fb2fe7fc700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 0x7fb2f4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.227+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb30408dae0 con 0x7fb308102e20 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.227+0000 7fb30d079700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 0x7fb2f4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:06.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.228+0000 7fb30d079700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 0x7fb2f4070850 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009fd0 tx=0x7fb2f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:06.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.232+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb304056d60 con 0x7fb308102e20 2024-03-22T02:46:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:05 smithi122 ceph-mon[83368]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:05 smithi122 ceph-mon[83368]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:05 smithi122 ceph-mon[83368]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:05 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/774042515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:46:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:05 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1368233099' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.512+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb3080fe570 con 0x7fb308102e20 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.512+0000 7fb2fe7fc700 1 -- 172.21.15.103:0/3257471834 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb30405a420 con 0x7fb308102e20 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:46:06.513 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:46:06.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 msgr2=0x7fb2f4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 0x7fb2f4070850 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009fd0 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:06.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 msgr2=0x7fb3080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:06.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb304002a50 tx=0x7fb3040046b0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 shutdown_connections 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb2f406e3a0 0x7fb2f4070850 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb308102e20 0x7fb3080702c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 --2- 172.21.15.103:0/3257471834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb3081037c0 0x7fb308070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 >> 172.21.15.103:0/3257471834 conn(0x7fb3080f89d0 msgr2=0x7fb308102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 shutdown_connections 2024-03-22T02:46:06.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:06.514+0000 7fb30fade700 1 -- 172.21.15.103:0/3257471834 wait complete. 2024-03-22T02:46:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:06 smithi122 ceph-mon[83368]: pgmap v437: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:07.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:06 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3257471834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:46:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:06 smithi103 ceph-mon[115211]: pgmap v437: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:06 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3257471834' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:46:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:07 smithi122 ceph-mon[83368]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:07 smithi103 ceph-mon[115211]: from='client.34864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:08 smithi122 ceph-mon[83368]: pgmap v438: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:09.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:08 smithi103 ceph-mon[115211]: pgmap v438: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:10 smithi122 ceph-mon[83368]: pgmap v439: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:10 smithi103 ceph-mon[115211]: pgmap v439: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:12 smithi122 ceph-mon[83368]: pgmap v440: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:12 smithi103 ceph-mon[115211]: pgmap v440: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:14 smithi122 ceph-mon[83368]: pgmap v441: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:14 smithi103 ceph-mon[115211]: pgmap v441: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:17.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:16 smithi122 ceph-mon[83368]: pgmap v442: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:17.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:16 smithi103 ceph-mon[115211]: pgmap v442: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:18 smithi122 ceph-mon[83368]: pgmap v443: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:19.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:18 smithi103 ceph-mon[115211]: pgmap v443: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:20 smithi122 ceph-mon[83368]: pgmap v444: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:20 smithi103 ceph-mon[115211]: pgmap v444: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:22 smithi122 ceph-mon[83368]: pgmap v445: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:22 smithi103 ceph-mon[115211]: pgmap v445: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:24 smithi122 ceph-mon[83368]: pgmap v446: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:25.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:24 smithi103 ceph-mon[115211]: pgmap v446: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:26 smithi122 ceph-mon[83368]: pgmap v447: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:26 smithi103 ceph-mon[115211]: pgmap v447: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:28 smithi122 ceph-mon[83368]: pgmap v448: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:28 smithi103 ceph-mon[115211]: pgmap v448: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:31.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:30 smithi122 ceph-mon[83368]: pgmap v449: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:30 smithi103 ceph-mon[115211]: pgmap v449: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:32 smithi122 ceph-mon[83368]: pgmap v450: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:32 smithi103 ceph-mon[115211]: pgmap v450: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:34 smithi122 ceph-mon[83368]: pgmap v451: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:34 smithi103 ceph-mon[115211]: pgmap v451: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:36.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 -- 172.21.15.103:0/567140744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 msgr2=0x7f99a81020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 --2- 172.21.15.103:0/567140744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a81020d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9998009ce0 tx=0x7f9998004d10 comp rx=0 tx=0).stop 2024-03-22T02:46:36.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 -- 172.21.15.103:0/567140744 shutdown_connections 2024-03-22T02:46:36.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 --2- 172.21.15.103:0/567140744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 0x7f99a80fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 --2- 172.21.15.103:0/567140744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a81020d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 -- 172.21.15.103:0/567140744 >> 172.21.15.103:0/567140744 conn(0x7f99a80f7850 msgr2=0x7f99a80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:36.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 -- 172.21.15.103:0/567140744 shutdown_connections 2024-03-22T02:46:36.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 -- 172.21.15.103:0/567140744 wait complete. 2024-03-22T02:46:36.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.628+0000 7f99adc2f700 1 Processor -- start 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99adc2f700 1 -- start start 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99adc2f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99adc2f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 0x7f99a8113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99adc2f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8114620 con 0x7f99a8101d00 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99adc2f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a810dc00 con 0x7f99a81026a0 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99a77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99a77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39532/0 (socket says 172.21.15.103:39532) 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99a77fe700 1 -- 172.21.15.103:0/2863100129 learned_addr learned my addr 172.21.15.103:0/2863100129 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.629+0000 7f99a6ffd700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 0x7f99a8113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99a77fe700 1 -- 172.21.15.103:0/2863100129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 msgr2=0x7f99a8113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99a77fe700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 0x7f99a8113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99a77fe700 1 -- 172.21.15.103:0/2863100129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9998009940 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99a77fe700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9998003400 tx=0x7f9998004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f999800e040 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9998004610 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a810de80 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f999801c640 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.630+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a810e440 con 0x7f99a8101d00 2024-03-22T02:46:36.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.631+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a80fd450 con 0x7f99a8101d00 2024-03-22T02:46:36.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.632+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9998012a70 con 0x7f99a8101d00 2024-03-22T02:46:36.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.632+0000 7f998ffff700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 0x7f9990070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.632+0000 7f99a6ffd700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 0x7f9990070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.632+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f999808f730 con 0x7f99a8101d00 2024-03-22T02:46:36.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.632+0000 7f99a6ffd700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 0x7f9990070850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f99a810f330 tx=0x7f999c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:36.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.637+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f999805c4e0 con 0x7f99a8101d00 2024-03-22T02:46:36.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.854+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99a810f0e0 con 0x7f999006e3a0 2024-03-22T02:46:36.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.855+0000 7f998ffff700 1 -- 172.21.15.103:0/2863100129 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f99a810f0e0 con 0x7f999006e3a0 2024-03-22T02:46:36.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 msgr2=0x7f9990070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 0x7f9990070850 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f99a810f330 tx=0x7f999c00b040 comp rx=0 tx=0).stop 2024-03-22T02:46:36.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 msgr2=0x7f99a8113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9998003400 tx=0x7f9998004d10 comp rx=0 tx=0).stop 2024-03-22T02:46:36.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 shutdown_connections 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f999006e3a0 0x7f9990070850 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99a8101d00 0x7f99a8113a90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 --2- 172.21.15.103:0/2863100129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f99a81026a0 0x7f99a8113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.857+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 >> 172.21.15.103:0/2863100129 conn(0x7f99a80f7850 msgr2=0x7f99a80f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.858+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 shutdown_connections 2024-03-22T02:46:36.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.858+0000 7f99adc2f700 1 -- 172.21.15.103:0/2863100129 wait complete. 2024-03-22T02:46:36.869 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 -- 172.21.15.103:0/3607418922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fe830 msgr2=0x7fde4c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 --2- 172.21.15.103:0/3607418922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fe830 0x7fde4c106f00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fde3c004940 tx=0x7fde3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 -- 172.21.15.103:0/3607418922 shutdown_connections 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 --2- 172.21.15.103:0/3607418922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fe830 0x7fde4c106f00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 --2- 172.21.15.103:0/3607418922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fde90 0x7fde4c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 -- 172.21.15.103:0/3607418922 >> 172.21.15.103:0/3607418922 conn(0x7fde4c072bc0 msgr2=0x7fde4c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:36.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 -- 172.21.15.103:0/3607418922 shutdown_connections 2024-03-22T02:46:36.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.960+0000 7fde525b4700 1 -- 172.21.15.103:0/3607418922 wait complete. 2024-03-22T02:46:36.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 Processor -- start 2024-03-22T02:46:36.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 -- start start 2024-03-22T02:46:36.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fe830 0x7fde4c114180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c1147d0 con 0x7fde4c0fde90 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde525b4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c10dba0 con 0x7fde4c0fe830 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde4bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde4bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39548/0 (socket says 172.21.15.103:39548) 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde4bfff700 1 -- 172.21.15.103:0/2731158821 learned_addr learned my addr 172.21.15.103:0/2731158821 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.961+0000 7fde4b7fe700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fe830 0x7fde4c114180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde4bfff700 1 -- 172.21.15.103:0/2731158821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fe830 msgr2=0x7fde4c114180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:36.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde4bfff700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fe830 0x7fde4c114180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde4bfff700 1 -- 172.21.15.103:0/2731158821 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c005040 con 0x7fde4c0fde90 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde4bfff700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fde40002a50 tx=0x7fde400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde40007cc0 con 0x7fde4c0fde90 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde4000f040 con 0x7fde4c0fde90 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c10de80 con 0x7fde4c0fde90 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde4000da70 con 0x7fde4c0fde90 2024-03-22T02:46:36.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.962+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c10e370 con 0x7fde4c0fde90 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.963+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c065e30 con 0x7fde4c0fde90 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.963+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fde4000dc10 con 0x7fde4c0fde90 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.964+0000 7fde48ff9700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 0x7fde34070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.964+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde4008d420 con 0x7fde4c0fde90 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.964+0000 7fde4b7fe700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 0x7fde34070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:36.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.964+0000 7fde4b7fe700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 0x7fde34070850 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde4c10f4e0 tx=0x7fde3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:36.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:36.970+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde4005a150 con 0x7fde4c0fde90 2024-03-22T02:46:37.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.186+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde4c04bed0 con 0x7fde3406e3a0 2024-03-22T02:46:37.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.187+0000 7fde48ff9700 1 -- 172.21.15.103:0/2731158821 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fde4c04bed0 con 0x7fde3406e3a0 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 msgr2=0x7fde34070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 0x7fde34070850 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde4c10f4e0 tx=0x7fde3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 msgr2=0x7fde4c113c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fde40002a50 tx=0x7fde400046b0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 shutdown_connections 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fde3406e3a0 0x7fde34070850 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde4c0fde90 0x7fde4c113c40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 --2- 172.21.15.103:0/2731158821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde4c0fe830 0x7fde4c114180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 >> 172.21.15.103:0/2731158821 conn(0x7fde4c072bc0 msgr2=0x7fde4c103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:37.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 shutdown_connections 2024-03-22T02:46:37.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.189+0000 7fde525b4700 1 -- 172.21.15.103:0/2731158821 wait complete. 2024-03-22T02:46:37.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 -- 172.21.15.103:0/2721272561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 msgr2=0x7faef40fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2721272561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef40fb820 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faee8004940 tx=0x7faee8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:37.293 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:36 smithi103 ceph-mon[115211]: pgmap v452: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:37.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 -- 172.21.15.103:0/2721272561 shutdown_connections 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2721272561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef40fb820 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2721272561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 0x7faef4103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 -- 172.21.15.103:0/2721272561 >> 172.21.15.103:0/2721272561 conn(0x7faef4072310 msgr2=0x7faef4072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.290+0000 7faef9f3d700 1 -- 172.21.15.103:0/2721272561 shutdown_connections 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 -- 172.21.15.103:0/2721272561 wait complete. 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 Processor -- start 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 -- start start 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 0x7faef4113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef4114590 con 0x7faef40fb3b0 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.291+0000 7faef9f3d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef410db70 con 0x7faef4102f70 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39576/0 (socket says 172.21.15.103:39576) 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 -- 172.21.15.103:0/2526089094 learned_addr learned my addr 172.21.15.103:0/2526089094 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef2ffd700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 0x7faef4113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 -- 172.21.15.103:0/2526089094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 msgr2=0x7faef4113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 0x7faef4113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 -- 172.21.15.103:0/2526089094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8005040 con 0x7faef40fb3b0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef37fe700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faee400b790 tx=0x7faee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee400f770 con 0x7faef40fb3b0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faee400fda0 con 0x7faef40fb3b0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef410de50 con 0x7faef40fb3b0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.292+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee400d310 con 0x7faef40fb3b0 2024-03-22T02:46:37.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.293+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef410e340 con 0x7faef40fb3b0 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.294+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef41053b0 con 0x7faef40fb3b0 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.294+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7faee4010420 con 0x7faef40fb3b0 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.294+0000 7faedbfff700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 0x7faedc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.295+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faee408de10 con 0x7faef40fb3b0 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.295+0000 7faef2ffd700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 0x7faedc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.295+0000 7faef2ffd700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 0x7faedc070850 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faee8004f90 tx=0x7faee8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:37.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.299+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faee4057090 con 0x7faef40fb3b0 2024-03-22T02:46:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:36 smithi122 ceph-mon[83368]: pgmap v452: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:37.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.502+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faef410a0f0 con 0x7faedc06e3a0 2024-03-22T02:46:37.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.512+0000 7faedbfff700 1 -- 172.21.15.103:0/2526089094 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7faef410a0f0 con 0x7faedc06e3a0 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (15m) 8m ago 26m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (12m) 8m ago 26m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (12m) 8m ago 25m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (14m) 8m ago 26m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (9m) 8m ago 20m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (8m) 8m ago 20m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (8m) 8m ago 20m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (8m) 8m ago 20m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (16m) 8m ago 27m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (16m) 8m ago 25m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (13m) 8m ago 27m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:46:37.514 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (13m) 8m ago 25m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (15m) 8m ago 26m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (15m) 8m ago 25m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (12m) 8m ago 24m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (11m) 8m ago 24m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (11m) 8m ago 23m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (10m) 8m ago 23m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (10m) 8m ago 22m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (9m) 8m ago 22m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:46:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (15m) 8m ago 25m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 msgr2=0x7faedc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 0x7faedc070850 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faee8004f90 tx=0x7faee8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 msgr2=0x7faef4113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faee400b790 tx=0x7faee4009250 comp rx=0 tx=0).stop 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 shutdown_connections 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7faedc06e3a0 0x7faedc070850 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef40fb3b0 0x7faef4113970 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 --2- 172.21.15.103:0/2526089094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faef4102f70 0x7faef4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 >> 172.21.15.103:0/2526089094 conn(0x7faef4072310 msgr2=0x7faef40f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:37.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 shutdown_connections 2024-03-22T02:46:37.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.514+0000 7faef9f3d700 1 -- 172.21.15.103:0/2526089094 wait complete. 2024-03-22T02:46:37.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 -- 172.21.15.103:0/3782774977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c101d10 msgr2=0x7f609c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 --2- 172.21.15.103:0/3782774977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c101d10 0x7f609c1020e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6094004940 tx=0x7f6094009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 -- 172.21.15.103:0/3782774977 shutdown_connections 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 --2- 172.21.15.103:0/3782774977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c1026b0 0x7f609c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 --2- 172.21.15.103:0/3782774977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c101d10 0x7f609c1020e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 -- 172.21.15.103:0/3782774977 >> 172.21.15.103:0/3782774977 conn(0x7f609c0f7880 msgr2=0x7f609c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 -- 172.21.15.103:0/3782774977 shutdown_connections 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.614+0000 7f60a33bb700 1 -- 172.21.15.103:0/3782774977 wait complete. 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 Processor -- start 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 -- start start 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c101d10 0x7f609c10cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c10d730 con 0x7f609c1026b0 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a33bb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c10d8a0 con 0x7f609c101d10 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39592/0 (socket says 172.21.15.103:39592) 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 -- 172.21.15.103:0/472726747 learned_addr learned my addr 172.21.15.103:0/472726747 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a1157700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c101d10 0x7f609c10cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 -- 172.21.15.103:0/472726747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c101d10 msgr2=0x7f609c10cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c101d10 0x7f609c10cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 -- 172.21.15.103:0/472726747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094005040 con 0x7f609c1026b0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.615+0000 7f60a0956700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f608c00b790 tx=0x7f608c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.616+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608c00f6f0 con 0x7f609c1026b0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.616+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f608c00fd20 con 0x7f609c1026b0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.616+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c11a290 con 0x7f609c1026b0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.616+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f608c0103a0 con 0x7f609c1026b0 2024-03-22T02:46:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.616+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c11a710 con 0x7f609c1026b0 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.617+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c0fd460 con 0x7f609c1026b0 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.617+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f608c011400 con 0x7f609c1026b0 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.618+0000 7f6091ffb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 0x7f6088070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.618+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f608c08dc00 con 0x7f609c1026b0 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.618+0000 7f60a1157700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 0x7f6088070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.618+0000 7f60a1157700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 0x7f6088070860 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f609400b020 tx=0x7f6094009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:37.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.624+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f608c05a930 con 0x7f609c1026b0 2024-03-22T02:46:37.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.903+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f609c05f5f0 con 0x7f609c1026b0 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.904+0000 7f6091ffb700 1 -- 172.21.15.103:0/472726747 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f608c05a4c0 con 0x7f609c1026b0 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:46:37.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:46:37.907 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 msgr2=0x7f6088070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 0x7f6088070860 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f609400b020 tx=0x7f6094009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 msgr2=0x7f609c10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f608c00b790 tx=0x7f608c009220 comp rx=0 tx=0).stop 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 shutdown_connections 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f608806e3b0 0x7f6088070860 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f609c101d10 0x7f609c10cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 --2- 172.21.15.103:0/472726747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f609c1026b0 0x7f609c10d160 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:37.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 >> 172.21.15.103:0/472726747 conn(0x7f609c0f7880 msgr2=0x7f609c0f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:37.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 shutdown_connections 2024-03-22T02:46:37.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:37.906+0000 7f60a33bb700 1 -- 172.21.15.103:0/472726747 wait complete. 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 -- 172.21.15.103:0/2995221240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 msgr2=0x7f78600fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 --2- 172.21.15.103:0/2995221240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fe260 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7858009ce0 tx=0x7f7858004d10 comp rx=0 tx=0).stop 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 -- 172.21.15.103:0/2995221240 shutdown_connections 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 --2- 172.21.15.103:0/2995221240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 0x7f7860106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 --2- 172.21.15.103:0/2995221240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fe260 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 -- 172.21.15.103:0/2995221240 >> 172.21.15.103:0/2995221240 conn(0x7f7860072bc0 msgr2=0x7f7860072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 -- 172.21.15.103:0/2995221240 shutdown_connections 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 -- 172.21.15.103:0/2995221240 wait complete. 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.007+0000 7f7864f76700 1 Processor -- start 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 -- start start 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 0x7f78600fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78600fc820 con 0x7f78600fde90 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78600fc960 con 0x7f78600fe830 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39616/0 (socket says 172.21.15.103:39616) 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 -- 172.21.15.103:0/1635367025 learned_addr learned my addr 172.21.15.103:0/1635367025 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785dd9b700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 0x7f78600fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 -- 172.21.15.103:0/1635367025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 msgr2=0x7f78600fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 0x7f78600fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 -- 172.21.15.103:0/1635367025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858009940 con 0x7f78600fde90 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f785e59c700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f785800b740 tx=0x7f7858004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785800e040 con 0x7f78600fde90 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78580049b0 con 0x7f78600fde90 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.008+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78600fcbb0 con 0x7f78600fde90 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.009+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785801c6d0 con 0x7f78600fde90 2024-03-22T02:46:38.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.009+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786011a290 con 0x7f78600fde90 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.009+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7860065e30 con 0x7f78600fde90 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.010+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7858012b00 con 0x7f78600fde90 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.010+0000 7f7856ffd700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.010+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f785808fa10 con 0x7f78600fde90 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.010+0000 7f785dd9b700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.011+0000 7f785dd9b700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7848009fd0 tx=0x7f7848009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.015+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7858058c10 con 0x7f78600fde90 2024-03-22T02:46:38.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.257+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f786004bf00 con 0x7f78600fde90 2024-03-22T02:46:38.259 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:37 smithi103 ceph-mon[115211]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:38.260 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:37 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/472726747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.258+0000 7f7856ffd700 1 -- 172.21.15.103:0/1635367025 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f785805c2d0 con 0x7f78600fde90 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:46:38.260 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:46:38.261 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:46:38.262 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:38.263 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:46:38.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 msgr2=0x7f784c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7848009fd0 tx=0x7f7848009f20 comp rx=0 tx=0).stop 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 msgr2=0x7f78600fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f785800b740 tx=0x7f7858004d10 comp rx=0 tx=0).stop 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 shutdown_connections 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f784c06e3a0 0x7f784c070850 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78600fde90 0x7f78600fbd50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 --2- 172.21.15.103:0/1635367025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78600fe830 0x7f78600fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 >> 172.21.15.103:0/1635367025 conn(0x7f7860072bc0 msgr2=0x7f7860103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 shutdown_connections 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.260+0000 7f7864f76700 1 -- 172.21.15.103:0/1635367025 wait complete. 2024-03-22T02:46:38.265 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:46:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:37 smithi122 ceph-mon[83368]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:37 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/472726747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 -- 172.21.15.103:0/3202427064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fe830 msgr2=0x7f3ad0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 --2- 172.21.15.103:0/3202427064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fe830 0x7f3ad0106f00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8004940 tx=0x7f3ab8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 -- 172.21.15.103:0/3202427064 shutdown_connections 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 --2- 172.21.15.103:0/3202427064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fe830 0x7f3ad0106f00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 --2- 172.21.15.103:0/3202427064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fde90 0x7f3ad00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 -- 172.21.15.103:0/3202427064 >> 172.21.15.103:0/3202427064 conn(0x7f3ad0072bc0 msgr2=0x7f3ad0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 -- 172.21.15.103:0/3202427064 shutdown_connections 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 -- 172.21.15.103:0/3202427064 wait complete. 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.365+0000 7f3ad49de700 1 Processor -- start 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ad49de700 1 -- start start 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ad49de700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ad49de700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fe830 0x7f3ad01140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ad49de700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad0114730 con 0x7f3ad00fde90 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ad49de700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad010db20 con 0x7f3ad00fe830 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39638/0 (socket says 172.21.15.103:39638) 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 -- 172.21.15.103:0/404061403 learned_addr learned my addr 172.21.15.103:0/404061403 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3acdd9b700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fe830 0x7f3ad01140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 -- 172.21.15.103:0/404061403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fe830 msgr2=0x7f3ad01140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fe830 0x7f3ad01140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 -- 172.21.15.103:0/404061403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8005040 con 0x7f3ad00fde90 2024-03-22T02:46:38.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ace59c700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c830 tx=0x7f3ac0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.366+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac00106f0 con 0x7f3ad00fde90 2024-03-22T02:46:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.367+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad010de00 con 0x7f3ad00fde90 2024-03-22T02:46:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.367+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ac0010d20 con 0x7f3ad00fde90 2024-03-22T02:46:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.367+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ac00113a0 con 0x7f3ad00fde90 2024-03-22T02:46:38.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.367+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad010e340 con 0x7f3ad00fde90 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.367+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad0065e30 con 0x7f3ad00fde90 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.368+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3ac0012400 con 0x7f3ad00fde90 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.368+0000 7f3ac6ffd700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 0x7f3abc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.369+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ac008dc60 con 0x7f3ad00fde90 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.369+0000 7f3acdd9b700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 0x7f3abc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.369+0000 7f3acdd9b700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 0x7f3abc070850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ad010ece0 tx=0x7f3ab8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.373+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ac005aad0 con 0x7f3ad00fde90 2024-03-22T02:46:38.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.588+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ad004bed0 con 0x7f3abc06e3a0 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.589+0000 7f3ac6ffd700 1 -- 172.21.15.103:0/404061403 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ad004bed0 con 0x7f3abc06e3a0 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:46:38.591 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:46:38.592 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:46:38.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 msgr2=0x7f3abc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 0x7f3abc070850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3ad010ece0 tx=0x7f3ab8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 msgr2=0x7f3ad0113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000c830 tx=0x7f3ac0004470 comp rx=0 tx=0).stop 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 shutdown_connections 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3abc06e3a0 0x7f3abc070850 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ad00fde90 0x7f3ad0113ba0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 --2- 172.21.15.103:0/404061403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ad00fe830 0x7f3ad01140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 >> 172.21.15.103:0/404061403 conn(0x7f3ad0072bc0 msgr2=0x7f3ad01037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 shutdown_connections 2024-03-22T02:46:38.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.592+0000 7f3ad49de700 1 -- 172.21.15.103:0/404061403 wait complete. 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 -- 172.21.15.103:0/4063038980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10102710 msgr2=0x7f8c100fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 --2- 172.21.15.103:0/4063038980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10102710 0x7f8c100fbd80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8004940 tx=0x7f8bf8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 -- 172.21.15.103:0/4063038980 shutdown_connections 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 --2- 172.21.15.103:0/4063038980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10102710 0x7f8c100fbd80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 --2- 172.21.15.103:0/4063038980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10101d70 0x7f8c10102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 -- 172.21.15.103:0/4063038980 >> 172.21.15.103:0/4063038980 conn(0x7f8c10072bc0 msgr2=0x7f8c10072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 -- 172.21.15.103:0/4063038980 shutdown_connections 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 -- 172.21.15.103:0/4063038980 wait complete. 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.696+0000 7f8c15193700 1 Processor -- start 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 -- start start 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10102710 0x7f8c101140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c10114710 con 0x7f8c10101d70 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1010db20 con 0x7f8c10102710 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39662/0 (socket says 172.21.15.103:39662) 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 -- 172.21.15.103:0/41472830 learned_addr learned my addr 172.21.15.103:0/41472830 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0e59c700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10102710 0x7f8c101140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 -- 172.21.15.103:0/41472830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10102710 msgr2=0x7f8c101140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10102710 0x7f8c101140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 -- 172.21.15.103:0/41472830 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf8005040 con 0x7f8c10101d70 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c0ed9d700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8c08001d80 tx=0x7f8c08009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c08007d10 con 0x7f8c10101d70 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c08007e70 con 0x7f8c10101d70 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1010de00 con 0x7f8c10101d70 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c08014640 con 0x7f8c10101d70 2024-03-22T02:46:38.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.697+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1010e3c0 con 0x7f8c10101d70 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.698+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c100fd4c0 con 0x7f8c10101d70 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.699+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8c0800e960 con 0x7f8c10101d70 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.699+0000 7f8c077fe700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 0x7f8bfc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.700+0000 7f8c0e59c700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 0x7f8bfc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.700+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c0808dbc0 con 0x7f8c10101d70 2024-03-22T02:46:38.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.700+0000 7f8c0e59c700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 0x7f8bfc070850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800b3b0 tx=0x7f8bf8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:46:38.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.704+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c08056e70 con 0x7f8c10101d70 2024-03-22T02:46:38.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.985+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8c1005f5f0 con 0x7f8c10101d70 2024-03-22T02:46:38.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.986+0000 7f8c077fe700 1 -- 172.21.15.103:0/41472830 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8c0805a530 con 0x7f8c10101d70 2024-03-22T02:46:38.988 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:46:38.988 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:46:38.988 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:46:38.988 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:46:38.988 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:46:38.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 msgr2=0x7f8bfc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 0x7f8bfc070850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800b3b0 tx=0x7f8bf8009e30 comp rx=0 tx=0).stop 2024-03-22T02:46:38.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 msgr2=0x7f8c10113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8c08001d80 tx=0x7f8c08009ec0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 shutdown_connections 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8bfc06e3a0 0x7f8bfc070850 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c10101d70 0x7f8c10113b80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 --2- 172.21.15.103:0/41472830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c10102710 0x7f8c101140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 >> 172.21.15.103:0/41472830 conn(0x7f8c10072bc0 msgr2=0x7f8c100659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 shutdown_connections 2024-03-22T02:46:38.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:46:38.988+0000 7f8c15193700 1 -- 172.21.15.103:0/41472830 wait complete. 2024-03-22T02:46:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:38 smithi122 ceph-mon[83368]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:38 smithi122 ceph-mon[83368]: pgmap v453: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:38 smithi122 ceph-mon[83368]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:39.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:38 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1635367025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:46:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:38 smithi103 ceph-mon[115211]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:38 smithi103 ceph-mon[115211]: pgmap v453: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:38 smithi103 ceph-mon[115211]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:38 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1635367025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:46:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:39 smithi122 ceph-mon[83368]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:39 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/41472830' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:46:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:39 smithi103 ceph-mon[115211]: from='client.34892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:46:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:39 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/41472830' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:46:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:40 smithi122 ceph-mon[83368]: pgmap v454: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:41.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:40 smithi103 ceph-mon[115211]: pgmap v454: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:42 smithi122 ceph-mon[83368]: pgmap v455: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:43.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:42 smithi103 ceph-mon[115211]: pgmap v455: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:45.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:44 smithi122 ceph-mon[83368]: pgmap v456: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:44 smithi103 ceph-mon[115211]: pgmap v456: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:46 smithi122 ceph-mon[83368]: pgmap v457: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:47.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:46 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:46 smithi103 ceph-mon[115211]: pgmap v457: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:46 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:46:49.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:49 smithi122 ceph-mon[83368]: pgmap v458: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:49.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:49 smithi103 ceph-mon[115211]: pgmap v458: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:51 smithi122 ceph-mon[83368]: pgmap v459: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:51.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:51 smithi103 ceph-mon[115211]: pgmap v459: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:53.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:53 smithi122 ceph-mon[83368]: pgmap v460: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:53.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:53 smithi103 ceph-mon[115211]: pgmap v460: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:55 smithi122 ceph-mon[83368]: pgmap v461: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:55 smithi103 ceph-mon[115211]: pgmap v461: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:57.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:57 smithi122 ceph-mon[83368]: pgmap v462: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:57 smithi103 ceph-mon[115211]: pgmap v462: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:59.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:46:59 smithi122 ceph-mon[83368]: pgmap v463: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:46:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:46:59 smithi103 ceph-mon[115211]: pgmap v463: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:01 smithi122 ceph-mon[83368]: pgmap v464: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:01 smithi103 ceph-mon[115211]: pgmap v464: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:03 smithi122 ceph-mon[83368]: pgmap v465: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:03 smithi103 ceph-mon[115211]: pgmap v465: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:05 smithi122 ceph-mon[83368]: pgmap v466: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:05 smithi103 ceph-mon[115211]: pgmap v466: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:07.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:07 smithi122 ceph-mon[83368]: pgmap v467: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:07.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:07 smithi103 ceph-mon[115211]: pgmap v467: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/791283781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb381037a0 msgr2=0x7fdb380fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/791283781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb381037a0 0x7fdb380fce10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdb34004940 tx=0x7fdb34009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/791283781 shutdown_connections 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/791283781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb381037a0 0x7fdb380fce10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/791283781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb38102e00 0x7fdb381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/791283781 >> 172.21.15.103:0/791283781 conn(0x7fdb380f89b0 msgr2=0x7fdb380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/791283781 shutdown_connections 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.105+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/791283781 wait complete. 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 Processor -- start 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 -- start start 2024-03-22T02:47:09.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb381037a0 0x7fdb38113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381144a0 con 0x7fdb38102e00 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3fd5a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3810daf0 con 0x7fdb381037a0 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3daf6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3daf6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52222/0 (socket says 172.21.15.103:52222) 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.106+0000 7fdb3daf6700 1 -- 172.21.15.103:0/1099903550 learned_addr learned my addr 172.21.15.103:0/1099903550 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3d2f5700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb381037a0 0x7fdb38113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3daf6700 1 -- 172.21.15.103:0/1099903550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb381037a0 msgr2=0x7fdb38113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3daf6700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb381037a0 0x7fdb38113e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3daf6700 1 -- 172.21.15.103:0/1099903550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34005040 con 0x7fdb38102e00 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3daf6700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800eb40 tx=0x7fdb2800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb28007cc0 con 0x7fdb38102e00 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb2800f040 con 0x7fdb38102e00 2024-03-22T02:47:09.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3810dd70 con 0x7fdb38102e00 2024-03-22T02:47:09.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb28009620 con 0x7fdb38102e00 2024-03-22T02:47:09.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.107+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3810e280 con 0x7fdb38102e00 2024-03-22T02:47:09.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.108+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb381052d0 con 0x7fdb38102e00 2024-03-22T02:47:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.108+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fdb280097a0 con 0x7fdb38102e00 2024-03-22T02:47:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.109+0000 7fdb2e7fc700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 0x7fdb24070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.109+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb2808d980 con 0x7fdb38102e00 2024-03-22T02:47:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.109+0000 7fdb3d2f5700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 0x7fdb24070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.109+0000 7fdb3d2f5700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 0x7fdb24070850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009b30 tx=0x7fdb34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.114+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb2805a7e0 con 0x7fdb38102e00 2024-03-22T02:47:09.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.332+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb3810efd0 con 0x7fdb2406e3a0 2024-03-22T02:47:09.333 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:09 smithi103 ceph-mon[115211]: pgmap v468: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:09.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.333+0000 7fdb2e7fc700 1 -- 172.21.15.103:0/1099903550 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdb3810efd0 con 0x7fdb2406e3a0 2024-03-22T02:47:09.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 msgr2=0x7fdb24070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 0x7fdb24070850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009b30 tx=0x7fdb34009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 msgr2=0x7fdb38113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800eb40 tx=0x7fdb2800b040 comp rx=0 tx=0).stop 2024-03-22T02:47:09.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 shutdown_connections 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdb2406e3a0 0x7fdb24070850 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb38102e00 0x7fdb38113910 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 --2- 172.21.15.103:0/1099903550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdb381037a0 0x7fdb38113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 >> 172.21.15.103:0/1099903550 conn(0x7fdb380f89b0 msgr2=0x7fdb381021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 shutdown_connections 2024-03-22T02:47:09.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.335+0000 7fdb3fd5a700 1 -- 172.21.15.103:0/1099903550 wait complete. 2024-03-22T02:47:09.346 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:47:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:09 smithi122 ceph-mon[83368]: pgmap v468: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:09.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3641903694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a8101d20 msgr2=0x7fc1a81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3641903694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a8101d20 0x7fc1a81020f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4004940 tx=0x7fc1a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3641903694 shutdown_connections 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3641903694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a81026c0 0x7fc1a80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3641903694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a8101d20 0x7fc1a81020f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.437+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3641903694 >> 172.21.15.103:0/3641903694 conn(0x7fc1a80f7870 msgr2=0x7fc1a80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3641903694 shutdown_connections 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3641903694 wait complete. 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 Processor -- start 2024-03-22T02:47:09.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 -- start start 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a8101d20 0x7fc1a8113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8114520 con 0x7fc1a81026c0 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1afd2a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a810db40 con 0x7fc1a8101d20 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1ad2c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1ad2c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52232/0 (socket says 172.21.15.103:52232) 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.438+0000 7fc1ad2c5700 1 -- 172.21.15.103:0/3906089033 learned_addr learned my addr 172.21.15.103:0/3906089033 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1adac6700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a8101d20 0x7fc1a8113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1ad2c5700 1 -- 172.21.15.103:0/3906089033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a8101d20 msgr2=0x7fc1a8113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1ad2c5700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a8101d20 0x7fc1a8113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1ad2c5700 1 -- 172.21.15.103:0/3906089033 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4005040 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1ad2c5700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c800 tx=0x7fc198004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19800a760 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc19800ad90 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a810de20 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1980113a0 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.439+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a810e3e0 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.440+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a80fd470 con 0x7fc1a81026c0 2024-03-22T02:47:09.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.440+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc198012400 con 0x7fc1a81026c0 2024-03-22T02:47:09.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.441+0000 7fc19e7fc700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 0x7fc194070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.441+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc19808dd70 con 0x7fc1a81026c0 2024-03-22T02:47:09.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.441+0000 7fc1adac6700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 0x7fc194070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.441+0000 7fc1adac6700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 0x7fc194070850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4004f90 tx=0x7fc1a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.445+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc198057030 con 0x7fc1a81026c0 2024-03-22T02:47:09.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.661+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1a8108e20 con 0x7fc19406e3a0 2024-03-22T02:47:09.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.662+0000 7fc19e7fc700 1 -- 172.21.15.103:0/3906089033 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc1a8108e20 con 0x7fc19406e3a0 2024-03-22T02:47:09.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.664+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 msgr2=0x7fc194070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.664+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 0x7fc194070850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4004f90 tx=0x7fc1a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.664+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 msgr2=0x7fc1a8113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.664+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c800 tx=0x7fc198004470 comp rx=0 tx=0).stop 2024-03-22T02:47:09.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 shutdown_connections 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc19406e3a0 0x7fc194070850 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1a8101d20 0x7fc1a8113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 --2- 172.21.15.103:0/3906089033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1a81026c0 0x7fc1a8113ed0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 >> 172.21.15.103:0/3906089033 conn(0x7fc1a80f7870 msgr2=0x7fc1a80f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 shutdown_connections 2024-03-22T02:47:09.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.665+0000 7fc1afd2a700 1 -- 172.21.15.103:0/3906089033 wait complete. 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- 172.21.15.103:0/1444626043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 msgr2=0x7fe2c80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 --2- 172.21.15.103:0/1444626043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c80fbd30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4004940 tx=0x7fe2c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- 172.21.15.103:0/1444626043 shutdown_connections 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 --2- 172.21.15.103:0/1444626043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c80fbd30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 --2- 172.21.15.103:0/1444626043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 0x7fe2c81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- 172.21.15.103:0/1444626043 >> 172.21.15.103:0/1444626043 conn(0x7fe2c80f7870 msgr2=0x7fe2c80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- 172.21.15.103:0/1444626043 shutdown_connections 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- 172.21.15.103:0/1444626043 wait complete. 2024-03-22T02:47:09.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 Processor -- start 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.765+0000 7fe2d0547700 1 -- start start 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2d0547700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 0x7fe2c810eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2d0547700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2d0547700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810f5c0 con 0x7fe2c81026c0 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2d0547700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810f700 con 0x7fe2c8101d20 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52252/0 (socket says 172.21.15.103:52252) 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 -- 172.21.15.103:0/2445380425 learned_addr learned my addr 172.21.15.103:0/2445380425 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2ce2e3700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 0x7fe2c810eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 -- 172.21.15.103:0/2445380425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 msgr2=0x7fe2c810eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 0x7fe2c810eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 -- 172.21.15.103:0/2445380425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4005040 con 0x7fe2c81026c0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2cdae2700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009100 tx=0x7fe2c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2c400e040 con 0x7fe2c81026c0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2c4013420 con 0x7fe2c81026c0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2c401c610 con 0x7fe2c81026c0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.766+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c810f980 con 0x7fe2c81026c0 2024-03-22T02:47:09.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.767+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c8117eb0 con 0x7fe2c81026c0 2024-03-22T02:47:09.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.767+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c80fd470 con 0x7fe2c81026c0 2024-03-22T02:47:09.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.768+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe2c4013590 con 0x7fe2c81026c0 2024-03-22T02:47:09.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.768+0000 7fe2beffd700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 0x7fe2b4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:09.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.768+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe2c408f090 con 0x7fe2c81026c0 2024-03-22T02:47:09.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.768+0000 7fe2ce2e3700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 0x7fe2b4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:09.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.769+0000 7fe2ce2e3700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 0x7fe2b4070850 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009fd0 tx=0x7fe2b8009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:09.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.773+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe2c4058310 con 0x7fe2c81026c0 2024-03-22T02:47:09.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.979+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c8108e70 con 0x7fe2b406e3a0 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.989+0000 7fe2beffd700 1 -- 172.21.15.103:0/2445380425 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe2c8108e70 con 0x7fe2b406e3a0 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (15m) 9m ago 26m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (13m) 9m ago 27m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (13m) 8m ago 26m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:47:09.990 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (15m) 9m ago 26m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (9m) 9m ago 21m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (9m) 9m ago 21m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (8m) 8m ago 21m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (9m) 8m ago 21m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (17m) 9m ago 28m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (16m) 8m ago 25m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (14m) 9m ago 28m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (14m) 8m ago 25m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (16m) 9m ago 27m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (16m) 8m ago 25m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (12m) 9m ago 25m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:47:09.991 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (12m) 9m ago 24m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (11m) 9m ago 24m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (11m) 8m ago 23m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (10m) 8m ago 23m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (10m) 8m ago 22m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (16m) 9m ago 26m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:47:09.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 msgr2=0x7fe2b4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 0x7fe2b4070850 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009fd0 tx=0x7fe2b8009e50 comp rx=0 tx=0).stop 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 msgr2=0x7fe2c810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009100 tx=0x7fe2c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 shutdown_connections 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe2b406e3a0 0x7fe2b4070850 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2c8101d20 0x7fe2c810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 --2- 172.21.15.103:0/2445380425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2c81026c0 0x7fe2c810f080 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 >> 172.21.15.103:0/2445380425 conn(0x7fe2c80f7870 msgr2=0x7fe2c80f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 shutdown_connections 2024-03-22T02:47:09.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:09.991+0000 7fe2d0547700 1 -- 172.21.15.103:0/2445380425 wait complete. 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1927649244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 msgr2=0x7f9d681092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1927649244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d681092f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c004940 tx=0x7f9d5c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1927649244 shutdown_connections 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1927649244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d681092f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1927649244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d680fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1927649244 >> 172.21.15.103:0/1927649244 conn(0x7f9d68072bc0 msgr2=0x7f9d68072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1927649244 shutdown_connections 2024-03-22T02:47:10.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.092+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1927649244 wait complete. 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 Processor -- start 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 -- start start 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d68115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d681168c0 con 0x7f9d680fc590 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6c88a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6810fc90 con 0x7f9d680fbc80 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52278/0 (socket says 172.21.15.103:52278) 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 -- 172.21.15.103:0/1626831376 learned_addr learned my addr 172.21.15.103:0/1626831376 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6659c700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d68115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 -- 172.21.15.103:0/1626831376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 msgr2=0x7f9d68115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d68115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d65d9b700 1 -- 172.21.15.103:0/1626831376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c005040 con 0x7f9d680fc590 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.093+0000 7f9d6659c700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d68115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d65d9b700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009150 tx=0x7f9d5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c00e040 con 0x7f9d680fc590 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6810ff10 con 0x7f9d680fc590 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d5c013420 con 0x7f9d680fc590 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c01c630 con 0x7f9d680fc590 2024-03-22T02:47:10.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.094+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d681104d0 con 0x7f9d680fc590 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.095+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d68065e30 con 0x7f9d680fc590 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.095+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9d5c013590 con 0x7f9d680fc590 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.096+0000 7f9d56ffd700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 0x7f9d50070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.096+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9d5c08f070 con 0x7f9d680fc590 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.096+0000 7f9d6659c700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 0x7f9d50070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.096+0000 7f9d6659c700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 0x7f9d50070860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009fd0 tx=0x7f9d5800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.100+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d5c05be20 con 0x7f9d680fc590 2024-03-22T02:47:10.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.383+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9d6810c160 con 0x7f9d680fc590 2024-03-22T02:47:10.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.384+0000 7f9d56ffd700 1 -- 172.21.15.103:0/1626831376 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9d5c05b9b0 con 0x7f9d680fc590 2024-03-22T02:47:10.385 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:47:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:47:10.385 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:47:10.386 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:47:10.387 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:47:10.388 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:47:10.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 msgr2=0x7f9d50070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 0x7f9d50070860 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009fd0 tx=0x7f9d5800b040 comp rx=0 tx=0).stop 2024-03-22T02:47:10.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 msgr2=0x7f9d68116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009150 tx=0x7f9d5c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 shutdown_connections 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9d5006e3b0 0x7f9d50070860 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d680fbc80 0x7f9d68115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 --2- 172.21.15.103:0/1626831376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d680fc590 0x7f9d68116270 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 >> 172.21.15.103:0/1626831376 conn(0x7f9d68072bc0 msgr2=0x7f9d68105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:10.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 shutdown_connections 2024-03-22T02:47:10.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.386+0000 7f9d6c88a700 1 -- 172.21.15.103:0/1626831376 wait complete. 2024-03-22T02:47:10.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- 172.21.15.103:0/3772033834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 msgr2=0x7f87a40fbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 --2- 172.21.15.103:0/3772033834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a40fbb60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8794004940 tx=0x7f8794009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:10.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- 172.21.15.103:0/3772033834 shutdown_connections 2024-03-22T02:47:10.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 --2- 172.21.15.103:0/3772033834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 0x7f87a410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 --2- 172.21.15.103:0/3772033834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a40fbb60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- 172.21.15.103:0/3772033834 >> 172.21.15.103:0/3772033834 conn(0x7f87a4071530 msgr2=0x7f87a4071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:10.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- 172.21.15.103:0/3772033834 shutdown_connections 2024-03-22T02:47:10.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- 172.21.15.103:0/3772033834 wait complete. 2024-03-22T02:47:10.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 Processor -- start 2024-03-22T02:47:10.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.488+0000 7f87aa353700 1 -- start start 2024-03-22T02:47:10.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 0x7f87a411a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a411abc0 con 0x7f87a40fb790 2024-03-22T02:47:10.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41141f0 con 0x7f87a40fc0a0 2024-03-22T02:47:10.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52286/0 (socket says 172.21.15.103:52286) 2024-03-22T02:47:10.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 -- 172.21.15.103:0/936393589 learned_addr learned my addr 172.21.15.103:0/936393589 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:10.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a37fe700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 0x7f87a411a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 -- 172.21.15.103:0/936393589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 msgr2=0x7f87a411a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 0x7f87a411a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 -- 172.21.15.103:0/936393589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8794005040 con 0x7f87a40fb790 2024-03-22T02:47:10.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a3fff700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f879400b3b0 tx=0x7f8794009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879400e040 con 0x7f87a40fb790 2024-03-22T02:47:10.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f879400bbc0 con 0x7f87a40fb790 2024-03-22T02:47:10.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879401c3f0 con 0x7f87a40fb790 2024-03-22T02:47:10.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a4114470 con 0x7f87a40fb790 2024-03-22T02:47:10.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.489+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a41149b0 con 0x7f87a40fb790 2024-03-22T02:47:10.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.490+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a40fe110 con 0x7f87a40fb790 2024-03-22T02:47:10.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.491+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f879400bd30 con 0x7f87a40fb790 2024-03-22T02:47:10.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.491+0000 7f87a0ff9700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 0x7f878c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.491+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f879408ef90 con 0x7f87a40fb790 2024-03-22T02:47:10.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.491+0000 7f87a37fe700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 0x7f878c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.492+0000 7f87a37fe700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 0x7f878c070850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87a41158d0 tx=0x7f8798009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.496+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f87940c3740 con 0x7f87a40fb790 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.737+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f87a4110660 con 0x7f87a40fb790 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.738+0000 7f87a0ff9700 1 -- 172.21.15.103:0/936393589 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f879405bd40 con 0x7f87a40fb790 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:47:10.740 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:47:10.741 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:47:10.742 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:10.743 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 msgr2=0x7f878c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 0x7f878c070850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87a41158d0 tx=0x7f8798009ee0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 msgr2=0x7f87a411a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f879400b3b0 tx=0x7f8794009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 shutdown_connections 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f878c06e3a0 0x7f878c070850 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87a40fb790 0x7f87a411a030 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 --2- 172.21.15.103:0/936393589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87a40fc0a0 0x7f87a411a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 >> 172.21.15.103:0/936393589 conn(0x7f87a4071530 msgr2=0x7f87a4102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:10.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 shutdown_connections 2024-03-22T02:47:10.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.740+0000 7f87aa353700 1 -- 172.21.15.103:0/936393589 wait complete. 2024-03-22T02:47:10.745 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- 172.21.15.103:0/3989656948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc1035d0 msgr2=0x7f26cc0fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 --2- 172.21.15.103:0/3989656948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc1035d0 0x7f26cc0fcc40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f26bc004940 tx=0x7f26bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- 172.21.15.103:0/3989656948 shutdown_connections 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 --2- 172.21.15.103:0/3989656948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc1035d0 0x7f26cc0fcc40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 --2- 172.21.15.103:0/3989656948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc102c30 0x7f26cc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- 172.21.15.103:0/3989656948 >> 172.21.15.103:0/3989656948 conn(0x7f26cc0f8760 msgr2=0x7f26cc0fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- 172.21.15.103:0/3989656948 shutdown_connections 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- 172.21.15.103:0/3989656948 wait complete. 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 Processor -- start 2024-03-22T02:47:10.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.844+0000 7f26d19a0700 1 -- start start 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc1035d0 0x7f26cc113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc114570 con 0x7f26cc102c30 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc10d960 con 0x7f26cc1035d0 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52310/0 (socket says 172.21.15.103:52310) 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 -- 172.21.15.103:0/2903302702 learned_addr learned my addr 172.21.15.103:0/2903302702 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26cbfff700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc1035d0 0x7f26cc113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 -- 172.21.15.103:0/2903302702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc1035d0 msgr2=0x7f26cc113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc1035d0 0x7f26cc113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:10.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 -- 172.21.15.103:0/2903302702 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26bc005040 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d099e700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26b4002a50 tx=0x7f26b40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b4007cc0 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26cc10dc40 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26b400f040 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b400da30 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.845+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc10e1d0 con 0x7f26cc102c30 2024-03-22T02:47:10.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.846+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc0fe380 con 0x7f26cc102c30 2024-03-22T02:47:10.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.847+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f26b400db90 con 0x7f26cc102c30 2024-03-22T02:47:10.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.847+0000 7f26c97fa700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 0x7f26b8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:10.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.847+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f26b408db80 con 0x7f26cc102c30 2024-03-22T02:47:10.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.847+0000 7f26cbfff700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 0x7f26b8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:10.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.848+0000 7f26cbfff700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 0x7f26b8070850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f26bc004890 tx=0x7f26bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:10.852+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26b405a8b0 con 0x7f26cc102c30 2024-03-22T02:47:11.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.068+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f26cc109d00 con 0x7f26b806e3a0 2024-03-22T02:47:11.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.069+0000 7f26c97fa700 1 -- 172.21.15.103:0/2903302702 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f26cc109d00 con 0x7f26b806e3a0 2024-03-22T02:47:11.070 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:47:11.070 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:47:11.071 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:47:11.072 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:47:11.072 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:47:11.072 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:47:11.072 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:47:11.072 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:47:11.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 msgr2=0x7f26b8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 0x7f26b8070850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f26bc004890 tx=0x7f26bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:11.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 msgr2=0x7f26cc1139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26b4002a50 tx=0x7f26b40046b0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 shutdown_connections 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f26b806e3a0 0x7f26b8070850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26cc102c30 0x7f26cc1139e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 --2- 172.21.15.103:0/2903302702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26cc1035d0 0x7f26cc113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 >> 172.21.15.103:0/2903302702 conn(0x7f26cc0f8760 msgr2=0x7f26cc0f9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 shutdown_connections 2024-03-22T02:47:11.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.071+0000 7f26d19a0700 1 -- 172.21.15.103:0/2903302702 wait complete. 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.172+0000 7f7377d40700 1 -- 172.21.15.103:0/205499779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73701026c0 msgr2=0x7f73700fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.172+0000 7f7377d40700 1 --2- 172.21.15.103:0/205499779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73701026c0 0x7f73700fbd30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7360004940 tx=0x7f7360009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- 172.21.15.103:0/205499779 shutdown_connections 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 --2- 172.21.15.103:0/205499779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73701026c0 0x7f73700fbd30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 --2- 172.21.15.103:0/205499779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7370101d20 0x7f73701020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- 172.21.15.103:0/205499779 >> 172.21.15.103:0/205499779 conn(0x7f73700f7870 msgr2=0x7f73700f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- 172.21.15.103:0/205499779 shutdown_connections 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- 172.21.15.103:0/205499779 wait complete. 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 Processor -- start 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- start start 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 0x7f737010f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737010f5e0 con 0x7f7370101d20 2024-03-22T02:47:11.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7377d40700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737010f720 con 0x7f73701026c0 2024-03-22T02:47:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.173+0000 7f7375adc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52320/0 (socket says 172.21.15.103:52320) 2024-03-22T02:47:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 -- 172.21.15.103:0/2600801296 learned_addr learned my addr 172.21.15.103:0/2600801296 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f73752db700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 0x7f737010f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 -- 172.21.15.103:0/2600801296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 msgr2=0x7f737010f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 0x7f737010f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 -- 172.21.15.103:0/2600801296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7360005040 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f73752db700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 0x7f737010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7375adc700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f736c00c8c0 tx=0x7f736c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f736c00a980 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f736c00ce60 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7370117bd0 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f736c0115e0 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.174+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7370117fa0 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.175+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73700fd470 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.175+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f736c012400 con 0x7f7370101d20 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.176+0000 7f73667fc700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 0x7f735c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:11.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.176+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f736c08ded0 con 0x7f7370101d20 2024-03-22T02:47:11.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.176+0000 7f73752db700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 0x7f735c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:11.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.176+0000 7f73752db700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 0x7f735c070850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7360000c00 tx=0x7f7360009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:11.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.181+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f736c057150 con 0x7f7370101d20 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: pgmap v469: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1626831376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:47:11.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:11 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/936393589' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:47:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: pgmap v469: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1626831376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:47:11.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:11 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/936393589' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:47:11.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.458+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7370066880 con 0x7f7370101d20 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.459+0000 7f73667fc700 1 -- 172.21.15.103:0/2600801296 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f736c0126b0 con 0x7f7370101d20 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:47:11.460 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 msgr2=0x7f735c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 0x7f735c070850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7360000c00 tx=0x7f7360009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 msgr2=0x7f737010eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f736c00c8c0 tx=0x7f736c0044c0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 shutdown_connections 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f735c06e3a0 0x7f735c070850 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7370101d20 0x7f737010eb60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 --2- 172.21.15.103:0/2600801296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73701026c0 0x7f737010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 >> 172.21.15.103:0/2600801296 conn(0x7f73700f7870 msgr2=0x7f73700f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 shutdown_connections 2024-03-22T02:47:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:11.461+0000 7f7377d40700 1 -- 172.21.15.103:0/2600801296 wait complete. 2024-03-22T02:47:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:12 smithi122 ceph-mon[83368]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:12 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2600801296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:47:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:12 smithi103 ceph-mon[115211]: from='client.34920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:12 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2600801296' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:47:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:13 smithi122 ceph-mon[83368]: pgmap v470: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:13 smithi103 ceph-mon[115211]: pgmap v470: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:15.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:15 smithi122 ceph-mon[83368]: pgmap v471: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:15 smithi103 ceph-mon[115211]: pgmap v471: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:17 smithi122 ceph-mon[83368]: pgmap v472: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:17 smithi103 ceph-mon[115211]: pgmap v472: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:19 smithi122 ceph-mon[83368]: pgmap v473: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:19.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:19 smithi103 ceph-mon[115211]: pgmap v473: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:21 smithi122 ceph-mon[83368]: pgmap v474: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:21.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:21 smithi103 ceph-mon[115211]: pgmap v474: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:23.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:23 smithi122 ceph-mon[83368]: pgmap v475: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:23 smithi103 ceph-mon[115211]: pgmap v475: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:25 smithi122 ceph-mon[83368]: pgmap v476: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:25.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:25 smithi103 ceph-mon[115211]: pgmap v476: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:27 smithi122 ceph-mon[83368]: pgmap v477: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:27.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:27 smithi103 ceph-mon[115211]: pgmap v477: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:29.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:29 smithi122 ceph-mon[83368]: pgmap v478: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:29.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:29 smithi103 ceph-mon[115211]: pgmap v478: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:31.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:31 smithi122 ceph-mon[83368]: pgmap v479: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:31.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:31 smithi103 ceph-mon[115211]: pgmap v479: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:33 smithi122 ceph-mon[83368]: pgmap v480: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:33 smithi103 ceph-mon[115211]: pgmap v480: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:35.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:35 smithi122 ceph-mon[83368]: pgmap v481: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:35.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:35 smithi103 ceph-mon[115211]: pgmap v481: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:37.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:37 smithi122 ceph-mon[83368]: pgmap v482: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:37.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:37 smithi103 ceph-mon[115211]: pgmap v482: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:39.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:39 smithi122 ceph-mon[83368]: pgmap v483: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:39.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:39 smithi103 ceph-mon[115211]: pgmap v483: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:41.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:41 smithi122 ceph-mon[83368]: pgmap v484: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:41.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:41 smithi103 ceph-mon[115211]: pgmap v484: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 -- 172.21.15.103:0/2047321325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0134102e00 msgr2=0x7f01341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 --2- 172.21.15.103:0/2047321325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0134102e00 0x7f01341031d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0124004940 tx=0x7f0124009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 -- 172.21.15.103:0/2047321325 shutdown_connections 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 --2- 172.21.15.103:0/2047321325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01341037a0 0x7f01340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 --2- 172.21.15.103:0/2047321325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0134102e00 0x7f01341031d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 -- 172.21.15.103:0/2047321325 >> 172.21.15.103:0/2047321325 conn(0x7f01340f89b0 msgr2=0x7f01340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:41.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 -- 172.21.15.103:0/2047321325 shutdown_connections 2024-03-22T02:47:41.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.575+0000 7f0139ad2700 1 -- 172.21.15.103:0/2047321325 wait complete. 2024-03-22T02:47:41.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 Processor -- start 2024-03-22T02:47:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 -- start start 2024-03-22T02:47:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 0x7f0134113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0134114520 con 0x7f01341037a0 2024-03-22T02:47:41.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0139ad2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013410db70 con 0x7f0134102e00 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0132ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0132ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51676/0 (socket says 172.21.15.103:51676) 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.576+0000 7f0132ffd700 1 -- 172.21.15.103:0/1937409703 learned_addr learned my addr 172.21.15.103:0/1937409703 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f01337fe700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 0x7f0134113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0132ffd700 1 -- 172.21.15.103:0/1937409703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 msgr2=0x7f0134113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0132ffd700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 0x7f0134113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0132ffd700 1 -- 172.21.15.103:0/1937409703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124005040 con 0x7f01341037a0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f01337fe700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 0x7f0134113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0132ffd700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0128002a50 tx=0x7f0128004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0128007cc0 con 0x7f01341037a0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f012800f040 con 0x7f01341037a0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013410de50 con 0x7f01341037a0 2024-03-22T02:47:41.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f012800da40 con 0x7f01341037a0 2024-03-22T02:47:41.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.577+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013410e410 con 0x7f01341037a0 2024-03-22T02:47:41.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.578+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0134100110 con 0x7f01341037a0 2024-03-22T02:47:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.579+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f01280097c0 con 0x7f01341037a0 2024-03-22T02:47:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.579+0000 7f011bfff700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 0x7f011c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.579+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f012808db50 con 0x7f01341037a0 2024-03-22T02:47:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.579+0000 7f01337fe700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 0x7f011c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:41.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.579+0000 7f01337fe700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 0x7f011c070850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0124004560 tx=0x7f0124009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:41.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.584+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f012805a900 con 0x7f01341037a0 2024-03-22T02:47:41.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.799+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0134113660 con 0x7f011c06e3a0 2024-03-22T02:47:41.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.800+0000 7f011bfff700 1 -- 172.21.15.103:0/1937409703 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0134113660 con 0x7f011c06e3a0 2024-03-22T02:47:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 msgr2=0x7f011c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 0x7f011c070850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0124004560 tx=0x7f0124009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:41.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 msgr2=0x7f0134113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0128002a50 tx=0x7f0128004600 comp rx=0 tx=0).stop 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 shutdown_connections 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f011c06e3a0 0x7f011c070850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0134102e00 0x7f0134113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 --2- 172.21.15.103:0/1937409703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01341037a0 0x7f0134113ed0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.802+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 >> 172.21.15.103:0/1937409703 conn(0x7f01340f89b0 msgr2=0x7f01341073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.803+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 shutdown_connections 2024-03-22T02:47:41.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.803+0000 7f0139ad2700 1 -- 172.21.15.103:0/1937409703 wait complete. 2024-03-22T02:47:41.814 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:47:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 -- 172.21.15.103:0/9802013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 msgr2=0x7fed080fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/9802013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed080fbd30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fecfc004940 tx=0x7fecfc009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 -- 172.21.15.103:0/9802013 shutdown_connections 2024-03-22T02:47:41.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/9802013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed080fbd30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/9802013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed08101d20 0x7fed081020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 -- 172.21.15.103:0/9802013 >> 172.21.15.103:0/9802013 conn(0x7fed080f7870 msgr2=0x7fed080f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.905+0000 7fed0f8d6700 1 -- 172.21.15.103:0/9802013 shutdown_connections 2024-03-22T02:47:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 -- 172.21.15.103:0/9802013 wait complete. 2024-03-22T02:47:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 Processor -- start 2024-03-22T02:47:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 -- start start 2024-03-22T02:47:41.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed08101d20 0x7fed081128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08113440 con 0x7fed081026c0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0f8d6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0810ca40 con 0x7fed08101d20 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0ce71700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0ce71700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51706/0 (socket says 172.21.15.103:51706) 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.906+0000 7fed0ce71700 1 -- 172.21.15.103:0/1483297170 learned_addr learned my addr 172.21.15.103:0/1483297170 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0ce71700 1 -- 172.21.15.103:0/1483297170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed08101d20 msgr2=0x7fed081128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0ce71700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed08101d20 0x7fed081128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0ce71700 1 -- 172.21.15.103:0/1483297170 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc005040 con 0x7fed081026c0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0ce71700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009180 tx=0x7fecfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecfc00e040 con 0x7fed081026c0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecfc00bbc0 con 0x7fed081026c0 2024-03-22T02:47:41.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0810ccc0 con 0x7fed081026c0 2024-03-22T02:47:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecfc01c3f0 con 0x7fed081026c0 2024-03-22T02:47:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.907+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0810d280 con 0x7fed081026c0 2024-03-22T02:47:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.908+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed080fd470 con 0x7fed081026c0 2024-03-22T02:47:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.908+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fecfc00bd30 con 0x7fed081026c0 2024-03-22T02:47:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.909+0000 7fed05ffb700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 0x7fecf0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.909+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fecfc08f190 con 0x7fed081026c0 2024-03-22T02:47:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.909+0000 7fed0d672700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 0x7fecf0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:41.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.909+0000 7fed0d672700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 0x7fecf0070850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fed00004d60 tx=0x7fed00007b30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:41.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:41.914+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fecfc05bec0 con 0x7fed081026c0 2024-03-22T02:47:42.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.128+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed0810df70 con 0x7fecf006e3a0 2024-03-22T02:47:42.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.129+0000 7fed05ffb700 1 -- 172.21.15.103:0/1483297170 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fed0810df70 con 0x7fecf006e3a0 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 msgr2=0x7fecf0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 0x7fecf0070850 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fed00004d60 tx=0x7fed00007b30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 msgr2=0x7fed08112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009180 tx=0x7fecfc009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 shutdown_connections 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fecf006e3a0 0x7fecf0070850 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed08101d20 0x7fed081128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 --2- 172.21.15.103:0/1483297170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed081026c0 0x7fed08112df0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 >> 172.21.15.103:0/1483297170 conn(0x7fed080f7870 msgr2=0x7fed080f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 shutdown_connections 2024-03-22T02:47:42.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.131+0000 7fed0f8d6700 1 -- 172.21.15.103:0/1483297170 wait complete. 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 -- 172.21.15.103:0/4024517428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56081037b0 msgr2=0x7f56080fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 --2- 172.21.15.103:0/4024517428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56081037b0 0x7f56080fce20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f55f8004940 tx=0x7f55f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 -- 172.21.15.103:0/4024517428 shutdown_connections 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 --2- 172.21.15.103:0/4024517428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56081037b0 0x7f56080fce20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 --2- 172.21.15.103:0/4024517428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5608102e10 0x7f56081031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 -- 172.21.15.103:0/4024517428 >> 172.21.15.103:0/4024517428 conn(0x7f56080f89a0 msgr2=0x7f56080fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 -- 172.21.15.103:0/4024517428 shutdown_connections 2024-03-22T02:47:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.233+0000 7f560e7a0700 1 -- 172.21.15.103:0/4024517428 wait complete. 2024-03-22T02:47:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 Processor -- start 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 -- start start 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56081037b0 0x7f56081140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5608114730 con 0x7f5608102e10 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f560e7a0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560810db20 con 0x7f56081037b0 2024-03-22T02:47:42.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f5607fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f5607fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51728/0 (socket says 172.21.15.103:51728) 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f5607fff700 1 -- 172.21.15.103:0/1136849083 learned_addr learned my addr 172.21.15.103:0/1136849083 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.234+0000 7f56077fe700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56081037b0 0x7f56081140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5607fff700 1 -- 172.21.15.103:0/1136849083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56081037b0 msgr2=0x7f56081140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5607fff700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56081037b0 0x7f56081140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5607fff700 1 -- 172.21.15.103:0/1136849083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f8005040 con 0x7f5608102e10 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5607fff700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c8c0 tx=0x7f55fc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55fc00a760 con 0x7f5608102e10 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55fc00ad90 con 0x7f5608102e10 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560810de00 con 0x7f5608102e10 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55fc0113a0 con 0x7f5608102e10 2024-03-22T02:47:42.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.235+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560810e390 con 0x7f5608102e10 2024-03-22T02:47:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.236+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56081052e0 con 0x7f5608102e10 2024-03-22T02:47:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.237+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f55fc012400 con 0x7f5608102e10 2024-03-22T02:47:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.237+0000 7f5604ff9700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 0x7f55f0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.237+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f55fc08de20 con 0x7f5608102e10 2024-03-22T02:47:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.237+0000 7f56077fe700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 0x7f55f0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.238+0000 7f56077fe700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 0x7f55f0070850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f560810ee00 tx=0x7f55f8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.242+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55fc05ab90 con 0x7f5608102e10 2024-03-22T02:47:42.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.451+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f560810f260 con 0x7f55f006e3a0 2024-03-22T02:47:42.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.461+0000 7f5604ff9700 1 -- 172.21.15.103:0/1136849083 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f560810f260 con 0x7f55f006e3a0 2024-03-22T02:47:42.461 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:47:42.461 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (16m) 9m ago 27m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:47:42.461 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (13m) 9m ago 27m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (13m) 9m ago 26m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (15m) 9m ago 27m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (10m) 9m ago 21m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (10m) 9m ago 21m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (9m) 9m ago 21m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (9m) 9m ago 21m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (17m) 9m ago 28m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (17m) 9m ago 26m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (15m) 9m ago 29m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (14m) 9m ago 26m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (16m) 9m ago 27m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (16m) 9m ago 26m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (13m) 9m ago 25m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:47:42.462 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (12m) 9m ago 25m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (12m) 9m ago 24m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (11m) 9m ago 24m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (11m) 9m ago 23m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (10m) 9m ago 23m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (16m) 9m ago 26m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:47:42.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 msgr2=0x7f55f0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 0x7f55f0070850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f560810ee00 tx=0x7f55f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 msgr2=0x7f5608113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c8c0 tx=0x7f55fc0044c0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 shutdown_connections 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f55f006e3a0 0x7f55f0070850 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5608102e10 0x7f5608113ba0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 --2- 172.21.15.103:0/1136849083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56081037b0 0x7f56081140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 >> 172.21.15.103:0/1136849083 conn(0x7f56080f89a0 msgr2=0x7f5608102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 shutdown_connections 2024-03-22T02:47:42.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.463+0000 7f560e7a0700 1 -- 172.21.15.103:0/1136849083 wait complete. 2024-03-22T02:47:42.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.562+0000 7f2d81ad7700 1 -- 172.21.15.103:0/1500273586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 msgr2=0x7f2d7c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.562+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/1500273586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c1020f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c004940 tx=0x7f2d6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 -- 172.21.15.103:0/1500273586 shutdown_connections 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/1500273586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 0x7f2d7c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/1500273586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c1020f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 -- 172.21.15.103:0/1500273586 >> 172.21.15.103:0/1500273586 conn(0x7f2d7c0f7870 msgr2=0x7f2d7c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 -- 172.21.15.103:0/1500273586 shutdown_connections 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 -- 172.21.15.103:0/1500273586 wait complete. 2024-03-22T02:47:42.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 Processor -- start 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.563+0000 7f2d81ad7700 1 -- start start 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d81ad7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d81ad7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 0x7f2d7c113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d81ad7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1144f0 con 0x7f2d7c101d20 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d81ad7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c10db40 con 0x7f2d7c1026c0 2024-03-22T02:47:42.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51750/0 (socket says 172.21.15.103:51750) 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 -- 172.21.15.103:0/4112922684 learned_addr learned my addr 172.21.15.103:0/4112922684 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7affd700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 0x7f2d7c113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 -- 172.21.15.103:0/4112922684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 msgr2=0x7f2d7c113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 0x7f2d7c113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 -- 172.21.15.103:0/4112922684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c005040 con 0x7f2d7c101d20 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d7b7fe700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00b8b0 tx=0x7f2d6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d6c00e040 con 0x7f2d7c101d20 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c10ddc0 con 0x7f2d7c101d20 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.564+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d6c004290 con 0x7f2d7c101d20 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.565+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d6c01c3f0 con 0x7f2d7c101d20 2024-03-22T02:47:42.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.565+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c10e380 con 0x7f2d7c101d20 2024-03-22T02:47:42.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.565+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c0fd470 con 0x7f2d7c101d20 2024-03-22T02:47:42.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.566+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2d6c013420 con 0x7f2d7c101d20 2024-03-22T02:47:42.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.566+0000 7f2d63fff700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 0x7f2d64070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.566+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d6c08f050 con 0x7f2d7c101d20 2024-03-22T02:47:42.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.567+0000 7f2d7affd700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 0x7f2d64070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.567+0000 7f2d7affd700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 0x7f2d64070850 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0f9130 tx=0x7f2d70009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.571+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d6c0582d0 con 0x7f2d7c101d20 2024-03-22T02:47:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.854+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2d7c10efe0 con 0x7f2d7c101d20 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.854+0000 7f2d63fff700 1 -- 172.21.15.103:0/4112922684 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2d6c05b990 con 0x7f2d7c101d20 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:42.855 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:47:42.856 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:47:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 msgr2=0x7f2d64070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 0x7f2d64070850 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0f9130 tx=0x7f2d70009040 comp rx=0 tx=0).stop 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 msgr2=0x7f2d7c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00b8b0 tx=0x7f2d6c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 shutdown_connections 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6406e3a0 0x7f2d64070850 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d7c101d20 0x7f2d7c113a00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 --2- 172.21.15.103:0/4112922684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d7c1026c0 0x7f2d7c113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.856+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 >> 172.21.15.103:0/4112922684 conn(0x7f2d7c0f7870 msgr2=0x7f2d7c0662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.857+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 shutdown_connections 2024-03-22T02:47:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.857+0000 7f2d81ad7700 1 -- 172.21.15.103:0/4112922684 wait complete. 2024-03-22T02:47:42.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 -- 172.21.15.103:0/388858377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530103780 msgr2=0x7fd5300fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 --2- 172.21.15.103:0/388858377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530103780 0x7fd5300fcdf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd524004940 tx=0x7fd524009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 -- 172.21.15.103:0/388858377 shutdown_connections 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 --2- 172.21.15.103:0/388858377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530103780 0x7fd5300fcdf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 --2- 172.21.15.103:0/388858377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530102de0 0x7fd5301031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 -- 172.21.15.103:0/388858377 >> 172.21.15.103:0/388858377 conn(0x7fd5300f8970 msgr2=0x7fd5300fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 -- 172.21.15.103:0/388858377 shutdown_connections 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.957+0000 7fd5354e8700 1 -- 172.21.15.103:0/388858377 wait complete. 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 Processor -- start 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 -- start start 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530103780 0x7fd530070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd530070d90 con 0x7fd530102de0 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd5354e8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd530070ed0 con 0x7fd530103780 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51766/0 (socket says 172.21.15.103:51766) 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 -- 172.21.15.103:0/1996438578 learned_addr learned my addr 172.21.15.103:0/1996438578 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52e7fc700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530103780 0x7fd530070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 -- 172.21.15.103:0/1996438578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530103780 msgr2=0x7fd530070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530103780 0x7fd530070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.958+0000 7fd52effd700 1 -- 172.21.15.103:0/1996438578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524005040 con 0x7fd530102de0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd52effd700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd52800ec90 tx=0x7fd52800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd528007b30 con 0x7fd530102de0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd530119df0 con 0x7fd530102de0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd52800f040 con 0x7fd530102de0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd52800cc80 con 0x7fd530102de0 2024-03-22T02:47:42.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.959+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53011a1f0 con 0x7fd530102de0 2024-03-22T02:47:42.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.960+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5300fe530 con 0x7fd530102de0 2024-03-22T02:47:42.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.960+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd528009070 con 0x7fd530102de0 2024-03-22T02:47:42.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.961+0000 7fd5177fe700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 0x7fd518070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:42.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.961+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd52808e250 con 0x7fd530102de0 2024-03-22T02:47:42.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.961+0000 7fd52e7fc700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 0x7fd518070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:42.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.961+0000 7fd52e7fc700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 0x7fd518070850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd524004890 tx=0x7fd524009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:42.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:42.965+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd528057450 con 0x7fd530102de0 2024-03-22T02:47:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:43 smithi103 ceph-mon[115211]: pgmap v485: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:43 smithi103 ceph-mon[115211]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:43 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4112922684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:47:43.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.206+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd5300719b0 con 0x7fd530102de0 2024-03-22T02:47:43.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.207+0000 7fd5177fe700 1 -- 172.21.15.103:0/1996438578 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fd52805ab10 con 0x7fd530102de0 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:47:43.208 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:47:43.209 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:43.210 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:47:43.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.209+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 msgr2=0x7fd518070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 0x7fd518070850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd524004890 tx=0x7fd524009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:43.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 msgr2=0x7fd5300702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd52800ec90 tx=0x7fd52800b040 comp rx=0 tx=0).stop 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 shutdown_connections 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd51806e3a0 0x7fd518070850 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd530102de0 0x7fd5300702c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 --2- 172.21.15.103:0/1996438578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd530103780 0x7fd530070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 >> 172.21.15.103:0/1996438578 conn(0x7fd5300f8970 msgr2=0x7fd5300f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 shutdown_connections 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.210+0000 7fd5354e8700 1 -- 172.21.15.103:0/1996438578 wait complete. 2024-03-22T02:47:43.212 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:47:43.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- 172.21.15.103:0/3288158334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 msgr2=0x7f82180fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 --2- 172.21.15.103:0/3288158334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f82180fce30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8204004940 tx=0x7f8204009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- 172.21.15.103:0/3288158334 shutdown_connections 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 --2- 172.21.15.103:0/3288158334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f82180fce30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 --2- 172.21.15.103:0/3288158334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 0x7f82181031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- 172.21.15.103:0/3288158334 >> 172.21.15.103:0/3288158334 conn(0x7f82180f89b0 msgr2=0x7f82180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- 172.21.15.103:0/3288158334 shutdown_connections 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- 172.21.15.103:0/3288158334 wait complete. 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 Processor -- start 2024-03-22T02:47:43.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- start start 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 0x7f82180702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218072210 con 0x7f82181037c0 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.313+0000 7f821ecfd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218070d40 con 0x7f8218102e20 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51780/0 (socket says 172.21.15.103:51780) 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 -- 172.21.15.103:0/120847179 learned_addr learned my addr 172.21.15.103:0/120847179 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f821ca99700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 0x7f82180702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 -- 172.21.15.103:0/120847179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 msgr2=0x7f82180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 0x7f82180702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 -- 172.21.15.103:0/120847179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204005040 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f8217fff700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8204009100 tx=0x7f8204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820400e040 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8204013420 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8218070fc0 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820401c630 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.314+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821811a0f0 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.315+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82181052f0 con 0x7f82181037c0 2024-03-22T02:47:43.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.316+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8204013a00 con 0x7f82181037c0 2024-03-22T02:47:43.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.316+0000 7f82157fa700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 0x7f8208070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.316+0000 7f821ca99700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 0x7f8208070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.316+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f820408edb0 con 0x7f82181037c0 2024-03-22T02:47:43.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.317+0000 7f821ca99700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 0x7f8208070850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f820c004f40 tx=0x7f820c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:43.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.321+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82040580f0 con 0x7f82181037c0 2024-03-22T02:47:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:43 smithi122 ceph-mon[83368]: pgmap v485: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:43 smithi122 ceph-mon[83368]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:43.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:43 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4112922684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:47:43.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.540+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8218109f00 con 0x7f820806e3a0 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.541+0000 7f82157fa700 1 -- 172.21.15.103:0/120847179 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8218109f00 con 0x7f820806e3a0 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:47:43.542 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:47:43.543 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:47:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 msgr2=0x7f8208070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 0x7f8208070850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f820c004f40 tx=0x7f820c00b040 comp rx=0 tx=0).stop 2024-03-22T02:47:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 msgr2=0x7f8218070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8204009100 tx=0x7f8204009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 shutdown_connections 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f820806e3a0 0x7f8208070850 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8218102e20 0x7f82180702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.543+0000 7f821ecfd700 1 --2- 172.21.15.103:0/120847179 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82181037c0 0x7f8218070800 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.544+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 >> 172.21.15.103:0/120847179 conn(0x7f82180f89b0 msgr2=0x7f82181021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.544+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 shutdown_connections 2024-03-22T02:47:43.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.544+0000 7f821ecfd700 1 -- 172.21.15.103:0/120847179 wait complete. 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/3639074911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98102710 msgr2=0x7f4f980fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/3639074911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98102710 0x7f4f980fbd80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c004940 tx=0x7f4f8c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/3639074911 shutdown_connections 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/3639074911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98102710 0x7f4f980fbd80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/3639074911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98101d70 0x7f4f98102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/3639074911 >> 172.21.15.103:0/3639074911 conn(0x7f4f98072bc0 msgr2=0x7f4f98072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.643+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/3639074911 shutdown_connections 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/3639074911 wait complete. 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 Processor -- start 2024-03-22T02:47:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 -- start start 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98102710 0x7f4f980fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980fe470 con 0x7f4f98101d70 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f9e4f6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980fe5e0 con 0x7f4f98102710 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.644+0000 7f4f97fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51812/0 (socket says 172.21.15.103:51812) 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 -- 172.21.15.103:0/2757233796 learned_addr learned my addr 172.21.15.103:0/2757233796 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f977fe700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98102710 0x7f4f980fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 -- 172.21.15.103:0/2757233796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98102710 msgr2=0x7f4f980fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98102710 0x7f4f980fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 -- 172.21.15.103:0/2757233796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c005040 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f97fff700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4f88002a50 tx=0x7f4f880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f88007cc0 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f98119e60 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f8800f040 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8800da30 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.645+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9811a260 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.646+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f980fd4c0 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.647+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4f8800db90 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.647+0000 7f4f94ff9700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 0x7f4f80070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.647+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f8808db80 con 0x7f4f98101d70 2024-03-22T02:47:43.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.647+0000 7f4f977fe700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 0x7f4f80070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:47:43.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.648+0000 7f4f977fe700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 0x7f4f80070850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c004890 tx=0x7f4f8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:47:43.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.652+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f88056d80 con 0x7f4f98101d70 2024-03-22T02:47:43.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.931+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4f9805f5f0 con 0x7f4f98101d70 2024-03-22T02:47:43.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.931+0000 7f4f94ff9700 1 -- 172.21.15.103:0/2757233796 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4f9805f5f0 con 0x7f4f98101d70 2024-03-22T02:47:43.934 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:47:43.934 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:47:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:47:43.934 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:47:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 msgr2=0x7f4f80070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 0x7f4f80070850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c004890 tx=0x7f4f8c009e30 comp rx=0 tx=0).stop 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 msgr2=0x7f4f980fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4f88002a50 tx=0x7f4f880046b0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 shutdown_connections 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f8006e3a0 0x7f4f80070850 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f98101d70 0x7f4f980fd9f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 --2- 172.21.15.103:0/2757233796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f98102710 0x7f4f980fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 >> 172.21.15.103:0/2757233796 conn(0x7f4f98072bc0 msgr2=0x7f4f980659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 shutdown_connections 2024-03-22T02:47:43.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:47:43.933+0000 7f4f9e4f6700 1 -- 172.21.15.103:0/2757233796 wait complete. 2024-03-22T02:47:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:44 smithi122 ceph-mon[83368]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:44 smithi122 ceph-mon[83368]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:44 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1996438578' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:47:44.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:44 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2757233796' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:47:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:44 smithi103 ceph-mon[115211]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:44 smithi103 ceph-mon[115211]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:44 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1996438578' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:47:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:44 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2757233796' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:47:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:45 smithi122 ceph-mon[83368]: pgmap v486: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:45.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:45 smithi122 ceph-mon[83368]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:45 smithi103 ceph-mon[115211]: pgmap v486: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:45.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:45 smithi103 ceph-mon[115211]: from='client.34948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:47:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:46 smithi103 ceph-mon[115211]: pgmap v487: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:46.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:46 smithi122 ceph-mon[83368]: pgmap v487: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:47:48.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:48 smithi103 ceph-mon[115211]: pgmap v488: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:48.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:48 smithi122 ceph-mon[83368]: pgmap v488: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:50.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:50 smithi122 ceph-mon[83368]: pgmap v489: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:50.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:50 smithi103 ceph-mon[115211]: pgmap v489: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:52.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:52 smithi122 ceph-mon[83368]: pgmap v490: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:52.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:52 smithi103 ceph-mon[115211]: pgmap v490: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:54.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:54 smithi122 ceph-mon[83368]: pgmap v491: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:54.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:54 smithi103 ceph-mon[115211]: pgmap v491: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:56.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:56 smithi122 ceph-mon[83368]: pgmap v492: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:56.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:56 smithi103 ceph-mon[115211]: pgmap v492: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:58.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:47:58 smithi122 ceph-mon[83368]: pgmap v493: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:47:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:47:58 smithi103 ceph-mon[115211]: pgmap v493: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:00.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:00 smithi122 ceph-mon[83368]: pgmap v494: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:00.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:00 smithi103 ceph-mon[115211]: pgmap v494: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:02.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:02 smithi122 ceph-mon[83368]: pgmap v495: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:02.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:02 smithi103 ceph-mon[115211]: pgmap v495: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:04.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:04 smithi122 ceph-mon[83368]: pgmap v496: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:04.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:04 smithi103 ceph-mon[115211]: pgmap v496: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:06.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:06 smithi122 ceph-mon[83368]: pgmap v497: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:06.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:06 smithi103 ceph-mon[115211]: pgmap v497: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:08.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:08 smithi122 ceph-mon[83368]: pgmap v498: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:08.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:08 smithi103 ceph-mon[115211]: pgmap v498: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:10.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:10 smithi122 ceph-mon[83368]: pgmap v499: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:10.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:10 smithi103 ceph-mon[115211]: pgmap v499: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:12.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:12 smithi122 ceph-mon[83368]: pgmap v500: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:12.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:12 smithi103 ceph-mon[115211]: pgmap v500: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 -- 172.21.15.103:0/627258528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fe7e0 msgr2=0x7fd180106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 --2- 172.21.15.103:0/627258528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fe7e0 0x7fd180106eb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd16c004940 tx=0x7fd16c009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 -- 172.21.15.103:0/627258528 shutdown_connections 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 --2- 172.21.15.103:0/627258528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fe7e0 0x7fd180106eb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 --2- 172.21.15.103:0/627258528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fde40 0x7fd1800fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 -- 172.21.15.103:0/627258528 >> 172.21.15.103:0/627258528 conn(0x7fd1800f7880 msgr2=0x7fd1800f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 -- 172.21.15.103:0/627258528 shutdown_connections 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.048+0000 7fd186b4f700 1 -- 172.21.15.103:0/627258528 wait complete. 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 Processor -- start 2024-03-22T02:48:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 -- start start 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fe7e0 0x7fd180114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd180114b20 con 0x7fd1800fde40 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd186b4f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18010de60 con 0x7fd1800fe7e0 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd1848eb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd1848eb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49218/0 (socket says 172.21.15.103:49218) 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd1848eb700 1 -- 172.21.15.103:0/4099933632 learned_addr learned my addr 172.21.15.103:0/4099933632 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.049+0000 7fd17ffff700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fe7e0 0x7fd180114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd1848eb700 1 -- 172.21.15.103:0/4099933632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fe7e0 msgr2=0x7fd180114440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd1848eb700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fe7e0 0x7fd180114440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd1848eb700 1 -- 172.21.15.103:0/4099933632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c005040 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd1848eb700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd17400b790 tx=0x7fd174009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17400f6f0 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18010e0e0 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd17400fd20 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1740103a0 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.050+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18010e620 con 0x7fd1800fde40 2024-03-22T02:48:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.051+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1800666b0 con 0x7fd1800fde40 2024-03-22T02:48:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.052+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd174011400 con 0x7fd1800fde40 2024-03-22T02:48:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.052+0000 7fd17d7fa700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 0x7fd170070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.052+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd17408dc00 con 0x7fd1800fde40 2024-03-22T02:48:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.052+0000 7fd17ffff700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 0x7fd170070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.053+0000 7fd17ffff700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 0x7fd170070860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd18010f7a0 tx=0x7fd16c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.057+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd17405a9b0 con 0x7fd1800fde40 2024-03-22T02:48:14.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.272+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd18004bf40 con 0x7fd17006e3b0 2024-03-22T02:48:14.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.273+0000 7fd17d7fa700 1 -- 172.21.15.103:0/4099933632 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd18004bf40 con 0x7fd17006e3b0 2024-03-22T02:48:14.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 msgr2=0x7fd170070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 0x7fd170070860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd18010f7a0 tx=0x7fd16c009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 msgr2=0x7fd180113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd17400b790 tx=0x7fd174009250 comp rx=0 tx=0).stop 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 shutdown_connections 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd17006e3b0 0x7fd170070860 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1800fde40 0x7fd180113f00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 --2- 172.21.15.103:0/4099933632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1800fe7e0 0x7fd180114440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.275+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 >> 172.21.15.103:0/4099933632 conn(0x7fd1800f7880 msgr2=0x7fd180103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.276+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 shutdown_connections 2024-03-22T02:48:14.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.276+0000 7fd186b4f700 1 -- 172.21.15.103:0/4099933632 wait complete. 2024-03-22T02:48:14.287 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 -- 172.21.15.103:0/2645952911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 msgr2=0x7fefac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 --2- 172.21.15.103:0/2645952911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac0fce10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fef98004940 tx=0x7fef98009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 -- 172.21.15.103:0/2645952911 shutdown_connections 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 --2- 172.21.15.103:0/2645952911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac0fce10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 --2- 172.21.15.103:0/2645952911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 0x7fefac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 -- 172.21.15.103:0/2645952911 >> 172.21.15.103:0/2645952911 conn(0x7fefac0f89b0 msgr2=0x7fefac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 -- 172.21.15.103:0/2645952911 shutdown_connections 2024-03-22T02:48:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 -- 172.21.15.103:0/2645952911 wait complete. 2024-03-22T02:48:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.379+0000 7fefb2c63700 1 Processor -- start 2024-03-22T02:48:14.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb2c63700 1 -- start start 2024-03-22T02:48:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb2c63700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 0x7fefac113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb2c63700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb2c63700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac114510 con 0x7fefac1037a0 2024-03-22T02:48:14.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb2c63700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac10db10 con 0x7fefac102e00 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49240/0 (socket says 172.21.15.103:49240) 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 -- 172.21.15.103:0/586236969 learned_addr learned my addr 172.21.15.103:0/586236969 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefb09ff700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 0x7fefac113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 -- 172.21.15.103:0/586236969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 msgr2=0x7fefac113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 0x7fefac113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 -- 172.21.15.103:0/586236969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef98005040 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.380+0000 7fefabfff700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fef98009100 tx=0x7fef98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.381+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9800e040 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.381+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef98013420 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.381+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac10dd90 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.381+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9801c610 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.381+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac10e350 con 0x7fefac1037a0 2024-03-22T02:48:14.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.382+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac1052d0 con 0x7fefac1037a0 2024-03-22T02:48:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.382+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fef98013590 con 0x7fefac1037a0 2024-03-22T02:48:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.382+0000 7fefa97fa700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 0x7fef9c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.383+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fef9808f090 con 0x7fefac1037a0 2024-03-22T02:48:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.383+0000 7fefb09ff700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 0x7fef9c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.383+0000 7fefb09ff700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 0x7fef9c070850 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004cc0 tx=0x7fefa000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.387+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef9805be40 con 0x7fefac1037a0 2024-03-22T02:48:14.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:14 smithi122 ceph-mon[83368]: pgmap v501: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:14.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.604+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefac10efd0 con 0x7fef9c06e3a0 2024-03-22T02:48:14.605 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:14 smithi103 ceph-mon[115211]: pgmap v501: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:14.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.605+0000 7fefa97fa700 1 -- 172.21.15.103:0/586236969 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fefac10efd0 con 0x7fef9c06e3a0 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 msgr2=0x7fef9c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 0x7fef9c070850 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fefa0004cc0 tx=0x7fefa000b040 comp rx=0 tx=0).stop 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 msgr2=0x7fefac113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fef98009100 tx=0x7fef98009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 shutdown_connections 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fef9c06e3a0 0x7fef9c070850 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefac102e00 0x7fefac113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 --2- 172.21.15.103:0/586236969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefac1037a0 0x7fefac113ec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 >> 172.21.15.103:0/586236969 conn(0x7fefac0f89b0 msgr2=0x7fefac1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 shutdown_connections 2024-03-22T02:48:14.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.607+0000 7fefb2c63700 1 -- 172.21.15.103:0/586236969 wait complete. 2024-03-22T02:48:14.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 -- 172.21.15.103:0/235500556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0364102e00 msgr2=0x7f03641031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 --2- 172.21.15.103:0/235500556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0364102e00 0x7f03641031d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0354004940 tx=0x7f0354009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 -- 172.21.15.103:0/235500556 shutdown_connections 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 --2- 172.21.15.103:0/235500556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03641037a0 0x7f03640fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 --2- 172.21.15.103:0/235500556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0364102e00 0x7f03641031d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.708+0000 7f036a5c4700 1 -- 172.21.15.103:0/235500556 >> 172.21.15.103:0/235500556 conn(0x7f03640f89b0 msgr2=0x7f03640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.709+0000 7f036a5c4700 1 -- 172.21.15.103:0/235500556 shutdown_connections 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.709+0000 7f036a5c4700 1 -- 172.21.15.103:0/235500556 wait complete. 2024-03-22T02:48:14.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.709+0000 7f036a5c4700 1 Processor -- start 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.709+0000 7f036a5c4700 1 -- start start 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0364102e00 0x7f03640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0364072210 con 0x7f03641037a0 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0364070d40 con 0x7f0364102e00 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49246/0 (socket says 172.21.15.103:49246) 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 -- 172.21.15.103:0/1584594141 learned_addr learned my addr 172.21.15.103:0/1584594141 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:14.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f0363fff700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0364102e00 0x7f03640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 -- 172.21.15.103:0/1584594141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0364102e00 msgr2=0x7f03640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0364102e00 0x7f03640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 -- 172.21.15.103:0/1584594141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0354005040 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f03637fe700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0358002a80 tx=0x7f0358004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0358007cc0 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f035800f040 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0364071020 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f035800daf0 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.710+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036411a170 con 0x7f03641037a0 2024-03-22T02:48:14.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.711+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0364100110 con 0x7f03641037a0 2024-03-22T02:48:14.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.712+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f035800dc90 con 0x7f03641037a0 2024-03-22T02:48:14.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.712+0000 7f0360ff9700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 0x7f034c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:14.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.712+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f035808d410 con 0x7f03641037a0 2024-03-22T02:48:14.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.712+0000 7f0363fff700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 0x7f034c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:14.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.713+0000 7f0363fff700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 0x7f034c070850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f035400b3b0 tx=0x7f0354009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:14.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.717+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0358056700 con 0x7f03641037a0 2024-03-22T02:48:14.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.926+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0364109f20 con 0x7f034c06e3a0 2024-03-22T02:48:14.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.936+0000 7f0360ff9700 1 -- 172.21.15.103:0/1584594141 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0364109f20 con 0x7f034c06e3a0 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (16m) 10m ago 27m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (14m) 10m ago 28m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (14m) 9m ago 27m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (16m) 10m ago 27m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (10m) 10m ago 22m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (10m) 10m ago 22m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (10m) 9m ago 22m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (10m) 9m ago 22m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (18m) 10m ago 29m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (17m) 9m ago 26m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:48:14.938 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (15m) 10m ago 29m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (15m) 9m ago 26m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (17m) 10m ago 28m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (17m) 9m ago 27m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (14m) 10m ago 26m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (13m) 10m ago 25m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (12m) 10m ago 25m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (12m) 9m ago 24m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (11m) 9m ago 24m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (11m) 9m ago 23m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:48:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (17m) 10m ago 27m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 msgr2=0x7f034c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 0x7f034c070850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f035400b3b0 tx=0x7f0354009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 msgr2=0x7f0364070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0358002a80 tx=0x7f0358004630 comp rx=0 tx=0).stop 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 shutdown_connections 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f034c06e3a0 0x7f034c070850 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0364102e00 0x7f03640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 --2- 172.21.15.103:0/1584594141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03641037a0 0x7f0364070800 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:14.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 >> 172.21.15.103:0/1584594141 conn(0x7f03640f89b0 msgr2=0x7f03641073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:14.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 shutdown_connections 2024-03-22T02:48:14.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:14.938+0000 7f036a5c4700 1 -- 172.21.15.103:0/1584594141 wait complete. 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.037+0000 7f03d3fd2700 1 -- 172.21.15.103:0/3010735239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fde20 msgr2=0x7f03cc0fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.037+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/3010735239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fde20 0x7f03cc0fe1f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f03c8004940 tx=0x7f03c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- 172.21.15.103:0/3010735239 shutdown_connections 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/3010735239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/3010735239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fde20 0x7f03cc0fe1f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- 172.21.15.103:0/3010735239 >> 172.21.15.103:0/3010735239 conn(0x7f03cc0f7870 msgr2=0x7f03cc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- 172.21.15.103:0/3010735239 shutdown_connections 2024-03-22T02:48:15.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- 172.21.15.103:0/3010735239 wait complete. 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 Processor -- start 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- start start 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fde20 0x7f03cc113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc114490 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.038+0000 7f03d3fd2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc10da90 con 0x7f03cc0fde20 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49264/0 (socket says 172.21.15.103:49264) 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 -- 172.21.15.103:0/2237075385 learned_addr learned my addr 172.21.15.103:0/2237075385 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d1d6e700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fde20 0x7f03cc113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 -- 172.21.15.103:0/2237075385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fde20 msgr2=0x7f03cc113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fde20 0x7f03cc113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 -- 172.21.15.103:0/2237075385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8005040 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d156d700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00eb40 tx=0x7f03bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc007cc0 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03bc00f040 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc10dd10 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc009620 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.039+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc10e250 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.040+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc066b50 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.041+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f03bc009780 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.041+0000 7f03c27fc700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 0x7f03b8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.041+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f03bc08daf0 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.041+0000 7f03d1d6e700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 0x7f03b8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.042+0000 7f03d1d6e700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 0x7f03b8070850 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03c800b0d0 tx=0x7f03c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.046+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f03bc05a850 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.332+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03cc10ef80 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.333+0000 7f03c27fc700 1 -- 172.21.15.103:0/2237075385 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f03bc05a3e0 con 0x7f03cc0fe7c0 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:15.335 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:48:15.336 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 msgr2=0x7f03b8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 0x7f03b8070850 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03c800b0d0 tx=0x7f03c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 msgr2=0x7f03cc113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00eb40 tx=0x7f03bc00b040 comp rx=0 tx=0).stop 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 shutdown_connections 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f03b806e3a0 0x7f03b8070850 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03cc0fde20 0x7f03cc113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 --2- 172.21.15.103:0/2237075385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03cc0fe7c0 0x7f03cc113e40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 >> 172.21.15.103:0/2237075385 conn(0x7f03cc0f7870 msgr2=0x7f03cc0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:15.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 shutdown_connections 2024-03-22T02:48:15.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.335+0000 7f03d3fd2700 1 -- 172.21.15.103:0/2237075385 wait complete. 2024-03-22T02:48:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 -- 172.21.15.103:0/2011165308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c102e00 msgr2=0x7fc52c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 --2- 172.21.15.103:0/2011165308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c102e00 0x7fc52c1031d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc528004940 tx=0x7fc528009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 -- 172.21.15.103:0/2011165308 shutdown_connections 2024-03-22T02:48:15.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 --2- 172.21.15.103:0/2011165308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c1037a0 0x7fc52c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 --2- 172.21.15.103:0/2011165308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c102e00 0x7fc52c1031d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.435+0000 7fc533518700 1 -- 172.21.15.103:0/2011165308 >> 172.21.15.103:0/2011165308 conn(0x7fc52c0f89b0 msgr2=0x7fc52c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:15.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 -- 172.21.15.103:0/2011165308 shutdown_connections 2024-03-22T02:48:15.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 -- 172.21.15.103:0/2011165308 wait complete. 2024-03-22T02:48:15.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 Processor -- start 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 -- start start 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c102e00 0x7fc52c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c114520 con 0x7fc52c1037a0 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc533518700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c10db70 con 0x7fc52c102e00 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc530ab3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc530ab3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49282/0 (socket says 172.21.15.103:49282) 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc530ab3700 1 -- 172.21.15.103:0/2731593340 learned_addr learned my addr 172.21.15.103:0/2731593340 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.436+0000 7fc5312b4700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c102e00 0x7fc52c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc530ab3700 1 -- 172.21.15.103:0/2731593340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c102e00 msgr2=0x7fc52c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc530ab3700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c102e00 0x7fc52c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc530ab3700 1 -- 172.21.15.103:0/2731593340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528005040 con 0x7fc52c1037a0 2024-03-22T02:48:15.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc530ab3700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc51c002a50 tx=0x7fc51c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc51c007cc0 con 0x7fc52c1037a0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c10de50 con 0x7fc52c1037a0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc51c00f040 con 0x7fc52c1037a0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc51c00d9c0 con 0x7fc52c1037a0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.437+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c10e410 con 0x7fc52c1037a0 2024-03-22T02:48:15.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.438+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c100110 con 0x7fc52c1037a0 2024-03-22T02:48:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.438+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc51c009730 con 0x7fc52c1037a0 2024-03-22T02:48:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.439+0000 7fc521ffb700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 0x7fc518070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.439+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc51c08dbc0 con 0x7fc52c1037a0 2024-03-22T02:48:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.439+0000 7fc5312b4700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 0x7fc518070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.439+0000 7fc5312b4700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 0x7fc518070850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc528004f90 tx=0x7fc528009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.443+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc51c05a870 con 0x7fc52c1037a0 2024-03-22T02:48:15.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:15 smithi122 ceph-mon[83368]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:15 smithi122 ceph-mon[83368]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:15 smithi122 ceph-mon[83368]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:15 smithi103 ceph-mon[115211]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:15 smithi103 ceph-mon[115211]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:15 smithi103 ceph-mon[115211]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:15.687 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.686+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc52c0fe550 con 0x7fc52c1037a0 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.686+0000 7fc521ffb700 1 -- 172.21.15.103:0/2731593340 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fc51c08d770 con 0x7fc52c1037a0 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:48:15.688 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:48:15.689 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:15.690 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:48:15.691 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:15.691 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:15.691 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:15.691 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:15.691 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 msgr2=0x7fc518070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 0x7fc518070850 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc528004f90 tx=0x7fc528009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 msgr2=0x7fc52c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc51c002a50 tx=0x7fc51c004600 comp rx=0 tx=0).stop 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 shutdown_connections 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc51806e3a0 0x7fc518070850 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc52c102e00 0x7fc52c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 --2- 172.21.15.103:0/2731593340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc52c1037a0 0x7fc52c113ed0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 >> 172.21.15.103:0/2731593340 conn(0x7fc52c0f89b0 msgr2=0x7fc52c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 shutdown_connections 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.689+0000 7fc533518700 1 -- 172.21.15.103:0/2731593340 wait complete. 2024-03-22T02:48:15.692 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:48:15.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- 172.21.15.103:0/3406681825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 msgr2=0x7f8ad41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/3406681825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad41020f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4004940 tx=0x7f8ac4009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:15.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- 172.21.15.103:0/3406681825 shutdown_connections 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/3406681825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/3406681825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad41020f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- 172.21.15.103:0/3406681825 >> 172.21.15.103:0/3406681825 conn(0x7f8ad40f7870 msgr2=0x7f8ad40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- 172.21.15.103:0/3406681825 shutdown_connections 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- 172.21.15.103:0/3406681825 wait complete. 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 Processor -- start 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.791+0000 7f8ad8f06700 1 -- start start 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad4113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad4114590 con 0x7f8ad4101d20 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad410db20 con 0x7f8ad41026c0 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49302/0 (socket says 172.21.15.103:49302) 2024-03-22T02:48:15.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 -- 172.21.15.103:0/1563954007 learned_addr learned my addr 172.21.15.103:0/1563954007 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ac9d9b700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad4113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 -- 172.21.15.103:0/1563954007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 msgr2=0x7f8ad4113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad4113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 -- 172.21.15.103:0/1563954007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4005040 con 0x7f8ad4101d20 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ac9d9b700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad259c700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400b3b0 tx=0x7f8ac4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ac400e040 con 0x7f8ad4101d20 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ac400bbc0 con 0x7f8ad4101d20 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad410dda0 con 0x7f8ad4101d20 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ac401c3f0 con 0x7f8ad4101d20 2024-03-22T02:48:15.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.792+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad410e360 con 0x7f8ad4101d20 2024-03-22T02:48:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.793+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad40fd470 con 0x7f8ad4101d20 2024-03-22T02:48:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.794+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8ac400bd30 con 0x7f8ad4101d20 2024-03-22T02:48:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.794+0000 7f8acb7fe700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 0x7f8ac0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.794+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8ac408ef90 con 0x7f8ad4101d20 2024-03-22T02:48:15.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.794+0000 7f8ac9d9b700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 0x7f8ac0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:15.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.795+0000 7f8ac9d9b700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 0x7f8ac0070850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8ad410f210 tx=0x7f8abc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:15.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:15.799+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8ac40c3740 con 0x7f8ad4101d20 2024-03-22T02:48:16.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.011+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ad410e5d0 con 0x7f8ac006e3a0 2024-03-22T02:48:16.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.013+0000 7f8acb7fe700 1 -- 172.21.15.103:0/1563954007 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8ad410e5d0 con 0x7f8ac006e3a0 2024-03-22T02:48:16.014 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:48:16.015 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:48:16.016 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 msgr2=0x7f8ac0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 0x7f8ac0070850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8ad410f210 tx=0x7f8abc008040 comp rx=0 tx=0).stop 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 msgr2=0x7f8ad4113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400b3b0 tx=0x7f8ac4009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 shutdown_connections 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8ac006e3a0 0x7f8ac0070850 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ad4101d20 0x7f8ad4113970 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 --2- 172.21.15.103:0/1563954007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8ad41026c0 0x7f8ad4113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 >> 172.21.15.103:0/1563954007 conn(0x7f8ad40f7870 msgr2=0x7f8ad40f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:16.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 shutdown_connections 2024-03-22T02:48:16.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.016+0000 7f8ad8f06700 1 -- 172.21.15.103:0/1563954007 wait complete. 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- 172.21.15.103:0/2484266703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 msgr2=0x7f81e8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2484266703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8103240 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009ce0 tx=0x7f81e4004d10 comp rx=0 tx=0).stop 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- 172.21.15.103:0/2484266703 shutdown_connections 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2484266703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2484266703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8103240 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- 172.21.15.103:0/2484266703 >> 172.21.15.103:0/2484266703 conn(0x7f81e80714c0 msgr2=0x7f81e80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- 172.21.15.103:0/2484266703 shutdown_connections 2024-03-22T02:48:16.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- 172.21.15.103:0/2484266703 wait complete. 2024-03-22T02:48:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 Processor -- start 2024-03-22T02:48:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.119+0000 7f81efcd5700 1 -- start start 2024-03-22T02:48:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e8113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8114860 con 0x7f81e8102e70 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e810dc30 con 0x7f81e80fb4a0 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49320/0 (socket says 172.21.15.103:49320) 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 -- 172.21.15.103:0/2200473911 learned_addr learned my addr 172.21.15.103:0/2200473911 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81eda71700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e8113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 -- 172.21.15.103:0/2200473911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 msgr2=0x7f81e8113cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e8113cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 -- 172.21.15.103:0/2200473911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4009940 con 0x7f81e8102e70 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81eda71700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e8113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81ed270700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c810 tx=0x7f81d8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81d8012810 con 0x7f81e8102e70 2024-03-22T02:48:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81d8012e40 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81d800a370 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e810df10 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.120+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e810e4d0 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.121+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e8100110 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.122+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f81d8011810 con 0x7f81e8102e70 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.122+0000 7f81de7fc700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 0x7f81d4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.122+0000 7f81eda71700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 0x7f81d4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.122+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f81d808e160 con 0x7f81e8102e70 2024-03-22T02:48:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.123+0000 7f81eda71700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 0x7f81d4070850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f81e40048c0 tx=0x7f81e4004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.127+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f81d805af10 con 0x7f81e8102e70 2024-03-22T02:48:16.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.415+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f81e810f370 con 0x7f81e8102e70 2024-03-22T02:48:16.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.415+0000 7f81de7fc700 1 -- 172.21.15.103:0/2200473911 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f81d805aaa0 con 0x7f81e8102e70 2024-03-22T02:48:16.417 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:48:16.417 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:48:16.417 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:48:16.418 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:48:16.418 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:48:16.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 msgr2=0x7f81d4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 0x7f81d4070850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f81e40048c0 tx=0x7f81e4004d10 comp rx=0 tx=0).stop 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 msgr2=0x7f81e8114210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c810 tx=0x7f81d8004470 comp rx=0 tx=0).stop 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 shutdown_connections 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f81d406e3a0 0x7f81d4070850 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e80fb4a0 0x7f81e8113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 --2- 172.21.15.103:0/2200473911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e8102e70 0x7f81e8114210 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.417+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 >> 172.21.15.103:0/2200473911 conn(0x7f81e80714c0 msgr2=0x7f81e80fa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.418+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 shutdown_connections 2024-03-22T02:48:16.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:16.418+0000 7f81efcd5700 1 -- 172.21.15.103:0/2200473911 wait complete. 2024-03-22T02:48:16.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:16 smithi122 ceph-mon[83368]: pgmap v502: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:16.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2237075385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:48:16.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2731593340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:48:16.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:16 smithi122 ceph-mon[83368]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:16.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:16 smithi103 ceph-mon[115211]: pgmap v502: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:16.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2237075385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:48:16.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2731593340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:48:16.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:16 smithi103 ceph-mon[115211]: from='client.34976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:17.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:17 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2200473911' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:48:17.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:17 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2200473911' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:48:17.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:18.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:18 smithi122 ceph-mon[83368]: pgmap v503: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:18.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:18 smithi103 ceph-mon[115211]: pgmap v503: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:20.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:20 smithi122 ceph-mon[83368]: pgmap v504: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:20 smithi103 ceph-mon[115211]: pgmap v504: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:22.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:22 smithi122 ceph-mon[83368]: pgmap v505: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:22.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:22 smithi103 ceph-mon[115211]: pgmap v505: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:24.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:24 smithi122 ceph-mon[83368]: pgmap v506: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:24.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:24 smithi103 ceph-mon[115211]: pgmap v506: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:26.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:26 smithi103 ceph-mon[115211]: pgmap v507: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:26 smithi122 ceph-mon[83368]: pgmap v507: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:28.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:28 smithi103 ceph-mon[115211]: pgmap v508: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:28.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:28 smithi122 ceph-mon[83368]: pgmap v508: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:30.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:30 smithi103 ceph-mon[115211]: pgmap v509: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:30.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:30 smithi122 ceph-mon[83368]: pgmap v509: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:32 smithi103 ceph-mon[115211]: pgmap v510: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:32.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:32.650 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:32 smithi122 ceph-mon[83368]: pgmap v510: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:32.651 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:34.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:34 smithi103 ceph-mon[115211]: pgmap v511: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:34.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:34 smithi122 ceph-mon[83368]: pgmap v511: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:36.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:36 smithi103 ceph-mon[115211]: pgmap v512: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:36.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:36 smithi122 ceph-mon[83368]: pgmap v512: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:38 smithi103 ceph-mon[115211]: pgmap v513: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:38.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:38 smithi122 ceph-mon[83368]: pgmap v513: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:40.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:40 smithi103 ceph-mon[115211]: pgmap v514: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:40.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:40 smithi122 ceph-mon[83368]: pgmap v514: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:42.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:42 smithi103 ceph-mon[115211]: pgmap v515: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:42 smithi122 ceph-mon[83368]: pgmap v515: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:44 smithi122 ceph-mon[83368]: pgmap v516: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:44.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:44 smithi103 ceph-mon[115211]: pgmap v516: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 -- 172.21.15.103:0/3022066034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 msgr2=0x7fc498102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 --2- 172.21.15.103:0/3022066034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc498102140 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc48c004940 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 -- 172.21.15.103:0/3022066034 shutdown_connections 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 --2- 172.21.15.103:0/3022066034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc4980fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 --2- 172.21.15.103:0/3022066034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc498102140 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 -- 172.21.15.103:0/3022066034 >> 172.21.15.103:0/3022066034 conn(0x7fc498072bc0 msgr2=0x7fc498072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:46.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 -- 172.21.15.103:0/3022066034 shutdown_connections 2024-03-22T02:48:46.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 -- 172.21.15.103:0/3022066034 wait complete. 2024-03-22T02:48:46.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.530+0000 7fc49eb35700 1 Processor -- start 2024-03-22T02:48:46.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49eb35700 1 -- start start 2024-03-22T02:48:46.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49eb35700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49eb35700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc49810f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49eb35700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49810f6d0 con 0x7fc498101d70 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49eb35700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49810f810 con 0x7fc498102710 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40746/0 (socket says 172.21.15.103:40746) 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 -- 172.21.15.103:0/628815319 learned_addr learned my addr 172.21.15.103:0/628815319 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc497fff700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc49810f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 -- 172.21.15.103:0/628815319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 msgr2=0x7fc49810f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc49810f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 -- 172.21.15.103:0/628815319 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c005040 con 0x7fc498101d70 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc497fff700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc49810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.531+0000 7fc49c8d1700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009150 tx=0x7fc48c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:46.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.532+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc48c00e040 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.532+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc48c013420 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.532+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc498117b60 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.532+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc48c01c630 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.532+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc498118120 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.533+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4980ff080 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.533+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc48c013590 con 0x7fc498101d70 2024-03-22T02:48:46.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.533+0000 7fc4957fa700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 0x7fc488070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.534+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc48c08ef50 con 0x7fc498101d70 2024-03-22T02:48:46.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.534+0000 7fc497fff700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 0x7fc488070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.534+0000 7fc497fff700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 0x7fc488070860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc484006fd0 tx=0x7fc484008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:46.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.538+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc48c0c3740 con 0x7fc498101d70 2024-03-22T02:48:46.676 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:46 smithi103 ceph-mon[115211]: pgmap v517: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:46.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.754+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc49804bed0 con 0x7fc48806e3b0 2024-03-22T02:48:46.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.755+0000 7fc4957fa700 1 -- 172.21.15.103:0/628815319 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc49804bed0 con 0x7fc48806e3b0 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 msgr2=0x7fc488070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 0x7fc488070860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc484006fd0 tx=0x7fc484008040 comp rx=0 tx=0).stop 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 msgr2=0x7fc49810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009150 tx=0x7fc48c009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 shutdown_connections 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc48806e3b0 0x7fc488070860 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc498101d70 0x7fc49810ec50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 --2- 172.21.15.103:0/628815319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc498102710 0x7fc49810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 >> 172.21.15.103:0/628815319 conn(0x7fc498072bc0 msgr2=0x7fc4980fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:46.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 shutdown_connections 2024-03-22T02:48:46.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.758+0000 7fc49eb35700 1 -- 172.21.15.103:0/628815319 wait complete. 2024-03-22T02:48:46.769 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:48:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:46 smithi122 ceph-mon[83368]: pgmap v517: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:46.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- 172.21.15.103:0/927876451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 msgr2=0x7f52780fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 --2- 172.21.15.103:0/927876451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f52780fb910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5268004940 tx=0x7f5268009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- 172.21.15.103:0/927876451 shutdown_connections 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 --2- 172.21.15.103:0/927876451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f52780fb910 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 --2- 172.21.15.103:0/927876451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- 172.21.15.103:0/927876451 >> 172.21.15.103:0/927876451 conn(0x7f52780714c0 msgr2=0x7f52780718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- 172.21.15.103:0/927876451 shutdown_connections 2024-03-22T02:48:46.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- 172.21.15.103:0/927876451 wait complete. 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 Processor -- start 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.860+0000 7f527f9e8700 1 -- start start 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527f9e8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527f9e8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527f9e8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52781146e0 con 0x7f52780fb4a0 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527f9e8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527810dad0 con 0x7f5278102e70 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40768/0 (socket says 172.21.15.103:40768) 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 -- 172.21.15.103:0/818103506 learned_addr learned my addr 172.21.15.103:0/818103506 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:46.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527cf83700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 -- 172.21.15.103:0/818103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 msgr2=0x7f5278114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 -- 172.21.15.103:0/818103506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268005040 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527cf83700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527d784700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5274001db0 tx=0x7f5274009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f527400ed50 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5274007df0 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527810ddb0 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.861+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5274014640 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.862+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527810e2f0 con 0x7f52780fb4a0 2024-03-22T02:48:46.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.862+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5278100170 con 0x7f52780fb4a0 2024-03-22T02:48:46.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.863+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5274009360 con 0x7f52780fb4a0 2024-03-22T02:48:46.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.863+0000 7f526dffb700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 0x7f5264070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:46.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.863+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f527408dc90 con 0x7f52780fb4a0 2024-03-22T02:48:46.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.863+0000 7f527cf83700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 0x7f5264070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:46.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.864+0000 7f527cf83700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 0x7f5264070850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5268009fd0 tx=0x7f5268009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:46.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:46.868+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f527405aa70 con 0x7f52780fb4a0 2024-03-22T02:48:47.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.083+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f527810f210 con 0x7f526406e3a0 2024-03-22T02:48:47.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.085+0000 7f526dffb700 1 -- 172.21.15.103:0/818103506 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f527810f210 con 0x7f526406e3a0 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 msgr2=0x7f5264070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 0x7f5264070850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5268009fd0 tx=0x7f5268009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 msgr2=0x7f5278113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5274001db0 tx=0x7f5274009ee0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 shutdown_connections 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f526406e3a0 0x7f5264070850 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52780fb4a0 0x7f5278113b50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 --2- 172.21.15.103:0/818103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5278102e70 0x7f5278114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 >> 172.21.15.103:0/818103506 conn(0x7f52780714c0 msgr2=0x7f52780fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 shutdown_connections 2024-03-22T02:48:47.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.087+0000 7f527f9e8700 1 -- 172.21.15.103:0/818103506 wait complete. 2024-03-22T02:48:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/2704646019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 msgr2=0x7f6ff80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/2704646019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff80fbd80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6fec004940 tx=0x7f6fec009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/2704646019 shutdown_connections 2024-03-22T02:48:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/2704646019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff80fbd80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/2704646019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/2704646019 >> 172.21.15.103:0/2704646019 conn(0x7f6ff8072bc0 msgr2=0x7f6ff8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/2704646019 shutdown_connections 2024-03-22T02:48:47.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.188+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/2704646019 wait complete. 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 Processor -- start 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 -- start start 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff810ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810f760 con 0x7f6ff8102710 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ffd6d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810f8a0 con 0x7f6ff8101d70 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ff67fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ff67fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40776/0 (socket says 172.21.15.103:40776) 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.189+0000 7f6ff67fc700 1 -- 172.21.15.103:0/67719718 learned_addr learned my addr 172.21.15.103:0/67719718 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff6ffd700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff810ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff67fc700 1 -- 172.21.15.103:0/67719718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 msgr2=0x7f6ff810ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff67fc700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff810ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff67fc700 1 -- 172.21.15.103:0/67719718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec005040 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff6ffd700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ff67fc700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009180 tx=0x7f6fec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fec00e040 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8117b90 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fec00bbc0 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fec01c3f0 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.190+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8118120 con 0x7f6ff8102710 2024-03-22T02:48:47.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.191+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8104240 con 0x7f6ff8102710 2024-03-22T02:48:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.192+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6fec00bd30 con 0x7f6ff8102710 2024-03-22T02:48:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.192+0000 7f6fdf7fe700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 0x7f6fe0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.192+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6fec08f110 con 0x7f6ff8102710 2024-03-22T02:48:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.192+0000 7f6ff6ffd700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 0x7f6fe0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.193+0000 7f6ff6ffd700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 0x7f6fe0070850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8007fd0 tx=0x7f6fe8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.197+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6fec058390 con 0x7f6ff8102710 2024-03-22T02:48:47.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.401+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ff8065e70 con 0x7f6fe006e3a0 2024-03-22T02:48:47.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.411+0000 7f6fdf7fe700 1 -- 172.21.15.103:0/67719718 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6ff8065e70 con 0x7f6fe006e3a0 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (17m) 11m ago 28m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (15m) 11m ago 28m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (14m) 10m ago 27m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (17m) 11m ago 28m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (11m) 11m ago 23m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (11m) 11m ago 22m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (10m) 10m ago 22m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (10m) 10m ago 22m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (18m) 11m ago 29m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (18m) 10m ago 27m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:48:47.413 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (16m) 11m ago 30m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (15m) 10m ago 27m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (18m) 11m ago 28m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (17m) 10m ago 27m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (14m) 11m ago 26m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (14m) 11m ago 26m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (13m) 11m ago 25m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (12m) 10m ago 25m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (12m) 10m ago 24m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (11m) 10m ago 24m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:48:47.414 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (17m) 11m ago 28m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.413+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 msgr2=0x7f6fe0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 0x7f6fe0070850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8007fd0 tx=0x7f6fe8009040 comp rx=0 tx=0).stop 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 msgr2=0x7f6ff810f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009180 tx=0x7f6fec009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 shutdown_connections 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6fe006e3a0 0x7f6fe0070850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ff8101d70 0x7f6ff810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 --2- 172.21.15.103:0/67719718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ff8102710 0x7f6ff810f220 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 >> 172.21.15.103:0/67719718 conn(0x7f6ff8072bc0 msgr2=0x7f6ff80fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 shutdown_connections 2024-03-22T02:48:47.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.414+0000 7f6ffd6d2700 1 -- 172.21.15.103:0/67719718 wait complete. 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- 172.21.15.103:0/175197129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 msgr2=0x7f3f240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 --2- 172.21.15.103:0/175197129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f240fbd80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3f18004940 tx=0x7f3f18009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- 172.21.15.103:0/175197129 shutdown_connections 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 --2- 172.21.15.103:0/175197129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f240fbd80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 --2- 172.21.15.103:0/175197129 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 0x7f3f24102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- 172.21.15.103:0/175197129 >> 172.21.15.103:0/175197129 conn(0x7f3f24072bc0 msgr2=0x7f3f24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- 172.21.15.103:0/175197129 shutdown_connections 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- 172.21.15.103:0/175197129 wait complete. 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 Processor -- start 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.514+0000 7f3f298fc700 1 -- start start 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 0x7f3f24112a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241136b0 con 0x7f3f24102710 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2410ca10 con 0x7f3f24101d70 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40788/0 (socket says 172.21.15.103:40788) 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 -- 172.21.15.103:0/2851134994 learned_addr learned my addr 172.21.15.103:0/2851134994 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f22ffd700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 0x7f3f24112a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 -- 172.21.15.103:0/2851134994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 msgr2=0x7f3f24112a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 0x7f3f24112a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 -- 172.21.15.103:0/2851134994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18005040 con 0x7f3f24102710 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f227fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009100 tx=0x7f3f18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1800e040 con 0x7f3f24102710 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f18004290 con 0x7f3f24102710 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2410cc90 con 0x7f3f24102710 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f18013660 con 0x7f3f24102710 2024-03-22T02:48:47.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.515+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2410d1f0 con 0x7f3f24102710 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.516+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f240fd4c0 con 0x7f3f24102710 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.517+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3f1800bd00 con 0x7f3f24102710 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.517+0000 7f3f0b7fe700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 0x7f3f0c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.517+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f18091c20 con 0x7f3f24102710 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.517+0000 7f3f22ffd700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 0x7f3f0c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.518+0000 7f3f22ffd700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 0x7f3f0c070850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400a900 tx=0x7f3f14008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.522+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f1805e950 con 0x7f3f24102710 2024-03-22T02:48:47.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.805+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f24108ec0 con 0x7f3f24102710 2024-03-22T02:48:47.806 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:47 smithi103 ceph-mon[115211]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:47.806 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:47.806 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:47 smithi103 ceph-mon[115211]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:47.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.805+0000 7f3f0b7fe700 1 -- 172.21.15.103:0/2851134994 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3f18063e60 con 0x7f3f24102710 2024-03-22T02:48:47.806 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:48:47.806 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:48:47.807 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:48:47.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 msgr2=0x7f3f0c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 0x7f3f0c070850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400a900 tx=0x7f3f14008040 comp rx=0 tx=0).stop 2024-03-22T02:48:47.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 msgr2=0x7f3f24112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009100 tx=0x7f3f18009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 shutdown_connections 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f0c06e3a0 0x7f3f0c070850 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f24101d70 0x7f3f24112a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 --2- 172.21.15.103:0/2851134994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f24102710 0x7f3f24112fd0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.807+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 >> 172.21.15.103:0/2851134994 conn(0x7f3f24072bc0 msgr2=0x7f3f240faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.808+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 shutdown_connections 2024-03-22T02:48:47.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.808+0000 7f3f298fc700 1 -- 172.21.15.103:0/2851134994 wait complete. 2024-03-22T02:48:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:47 smithi122 ceph-mon[83368]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:48:47.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:47 smithi122 ceph-mon[83368]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- 172.21.15.103:0/2808756504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 msgr2=0x7fa180100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 --2- 172.21.15.103:0/2808756504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa180100cd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa174004940 tx=0x7fa174009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- 172.21.15.103:0/2808756504 shutdown_connections 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 --2- 172.21.15.103:0/2808756504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa180100cd0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 --2- 172.21.15.103:0/2808756504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 0x7fa1800fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- 172.21.15.103:0/2808756504 >> 172.21.15.103:0/2808756504 conn(0x7fa180072bc0 msgr2=0x7fa180072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- 172.21.15.103:0/2808756504 shutdown_connections 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- 172.21.15.103:0/2808756504 wait complete. 2024-03-22T02:48:47.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 Processor -- start 2024-03-22T02:48:47.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.909+0000 7fa186746700 1 -- start start 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa186746700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 0x7fa18011a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa186746700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa186746700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18011ad60 con 0x7fa1800fc620 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa186746700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180114130 con 0x7fa1800fbc80 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40804/0 (socket says 172.21.15.103:40804) 2024-03-22T02:48:47.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 -- 172.21.15.103:0/2594232987 learned_addr learned my addr 172.21.15.103:0/2594232987 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17ffff700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 0x7fa18011a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 -- 172.21.15.103:0/2594232987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 msgr2=0x7fa18011a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 0x7fa18011a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 -- 172.21.15.103:0/2594232987 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174005040 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17f7fe700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa17400b3b0 tx=0x7fa174009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17400e040 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa174013420 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17401c630 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.910+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1801143b0 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.911+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1801148f0 con 0x7fa1800fc620 2024-03-22T02:48:47.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.912+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180065e30 con 0x7fa1800fc620 2024-03-22T02:48:47.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.912+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa174013590 con 0x7fa1800fc620 2024-03-22T02:48:47.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.912+0000 7fa17cff9700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 0x7fa168070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:47.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.912+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa17408efe0 con 0x7fa1800fc620 2024-03-22T02:48:47.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.912+0000 7fa17ffff700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 0x7fa168070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:47.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.913+0000 7fa17ffff700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 0x7fa168070850 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa170007fd0 tx=0x7fa170009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:47.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:47.917+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa17405bd90 con 0x7fa1800fc620 2024-03-22T02:48:48.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.160+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa180110560 con 0x7fa1800fc620 2024-03-22T02:48:48.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.161+0000 7fa17cff9700 1 -- 172.21.15.103:0/2594232987 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fa17405b920 con 0x7fa1800fc620 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:48:48.162 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:48:48.163 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:48.164 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:48.165 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:48:48.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 msgr2=0x7fa168070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 0x7fa168070850 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa170007fd0 tx=0x7fa170009040 comp rx=0 tx=0).stop 2024-03-22T02:48:48.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 msgr2=0x7fa18011a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa17400b3b0 tx=0x7fa174009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 shutdown_connections 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa16806e3a0 0x7fa168070850 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1800fbc80 0x7fa18011a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 --2- 172.21.15.103:0/2594232987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa1800fc620 0x7fa18011a710 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 >> 172.21.15.103:0/2594232987 conn(0x7fa180072bc0 msgr2=0x7fa18010aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 shutdown_connections 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.163+0000 7fa186746700 1 -- 172.21.15.103:0/2594232987 wait complete. 2024-03-22T02:48:48.166 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.268+0000 7f3b0462b700 1 -- 172.21.15.103:0/2446454358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc102f50 msgr2=0x7f3afc103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.268+0000 7f3b0462b700 1 --2- 172.21.15.103:0/2446454358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc102f50 0x7f3afc103320 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3af8004940 tx=0x7f3af8009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- 172.21.15.103:0/2446454358 shutdown_connections 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 --2- 172.21.15.103:0/2446454358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 --2- 172.21.15.103:0/2446454358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc102f50 0x7f3afc103320 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- 172.21.15.103:0/2446454358 >> 172.21.15.103:0/2446454358 conn(0x7f3afc0714c0 msgr2=0x7f3afc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- 172.21.15.103:0/2446454358 shutdown_connections 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- 172.21.15.103:0/2446454358 wait complete. 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 Processor -- start 2024-03-22T02:48:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- start start 2024-03-22T02:48:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc102f50 0x7f3afc113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc114530 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b0462b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc10db30 con 0x7f3afc102f50 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b023c7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.269+0000 7f3b023c7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40820/0 (socket says 172.21.15.103:40820) 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b023c7700 1 -- 172.21.15.103:0/3371589288 learned_addr learned my addr 172.21.15.103:0/3371589288 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b01bc6700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc102f50 0x7f3afc113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b023c7700 1 -- 172.21.15.103:0/3371589288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc102f50 msgr2=0x7f3afc113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b023c7700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc102f50 0x7f3afc113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b023c7700 1 -- 172.21.15.103:0/3371589288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8005040 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b023c7700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3af800b3b0 tx=0x7f3af8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3af800e040 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3af8009b10 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc10ddb0 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3af8013820 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.270+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc10e2f0 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.271+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc100200 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.272+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3af801abd0 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.272+0000 7f3af2ffd700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 0x7f3ae8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.272+0000 7f3b01bc6700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 0x7f3ae8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.272+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3af8092dd0 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.273+0000 7f3b01bc6700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 0x7f3ae8070850 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3afc10ecf0 tx=0x7f3aec009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:48.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.277+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3af805fb80 con 0x7f3afc0fb4d0 2024-03-22T02:48:48.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.491+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3afc10a0a0 con 0x7f3ae806e3a0 2024-03-22T02:48:48.492 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:48 smithi103 ceph-mon[115211]: pgmap v518: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:48.492 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:48 smithi103 ceph-mon[115211]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:48.492 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:48 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2851134994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:48:48.492 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:48 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2594232987' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.492+0000 7f3af2ffd700 1 -- 172.21.15.103:0/3371589288 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3afc10a0a0 con 0x7f3ae806e3a0 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:48:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:48:48.494 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:48:48.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 msgr2=0x7f3ae8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 0x7f3ae8070850 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3afc10ecf0 tx=0x7f3aec009ee0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 msgr2=0x7f3afc1139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3af800b3b0 tx=0x7f3af8009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 shutdown_connections 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3ae806e3a0 0x7f3ae8070850 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3afc0fb4d0 0x7f3afc1139a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 --2- 172.21.15.103:0/3371589288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3afc102f50 0x7f3afc113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 >> 172.21.15.103:0/3371589288 conn(0x7f3afc0714c0 msgr2=0x7f3afc0fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 shutdown_connections 2024-03-22T02:48:48.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.495+0000 7f3b0462b700 1 -- 172.21.15.103:0/3371589288 wait complete. 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- 172.21.15.103:0/516804874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 msgr2=0x7f06180fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 --2- 172.21.15.103:0/516804874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06180fb710 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0600004940 tx=0x7f0600009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- 172.21.15.103:0/516804874 shutdown_connections 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 --2- 172.21.15.103:0/516804874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06180fb710 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 --2- 172.21.15.103:0/516804874 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 0x7f06181030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- 172.21.15.103:0/516804874 >> 172.21.15.103:0/516804874 conn(0x7f0618071300 msgr2=0x7f0618071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- 172.21.15.103:0/516804874 shutdown_connections 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- 172.21.15.103:0/516804874 wait complete. 2024-03-22T02:48:48.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 Processor -- start 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.597+0000 7f0617fff700 1 -- start start 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0617fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0617fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 0x7f0618113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0617fff700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618114530 con 0x7f06180fb2a0 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0617fff700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061810d920 con 0x7f0618102d20 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40848/0 (socket says 172.21.15.103:40848) 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 -- 172.21.15.103:0/3762947648 learned_addr learned my addr 172.21.15.103:0/3762947648 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f060e7fc700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 0x7f0618113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 -- 172.21.15.103:0/3762947648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 msgr2=0x7f0618113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 0x7f0618113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 -- 172.21.15.103:0/3762947648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0600005040 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0616ffd700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0608002a80 tx=0x7f0608004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0608007cc0 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f060800f040 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061810dc00 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.598+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f060800dad0 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.599+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061810e1c0 con 0x7f06180fb2a0 2024-03-22T02:48:48.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.599+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0618105100 con 0x7f06180fb2a0 2024-03-22T02:48:48.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.600+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f060800dc30 con 0x7f06180fb2a0 2024-03-22T02:48:48.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.600+0000 7f060ffff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 0x7f0604070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:48:48.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.600+0000 7f060e7fc700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 0x7f0604070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:48:48.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.600+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f060808db90 con 0x7f06180fb2a0 2024-03-22T02:48:48.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.601+0000 7f060e7fc700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 0x7f0604070850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f060000b3b0 tx=0x7f0600009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:48:48.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.605+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f060805a8c0 con 0x7f06180fb2a0 2024-03-22T02:48:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:48 smithi122 ceph-mon[83368]: pgmap v518: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:48 smithi122 ceph-mon[83368]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:48.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:48 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2851134994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:48:48.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:48 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2594232987' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:48:48.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.888+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f061810eff0 con 0x7f06180fb2a0 2024-03-22T02:48:48.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.888+0000 7f060ffff700 1 -- 172.21.15.103:0/3762947648 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f060805a450 con 0x7f06180fb2a0 2024-03-22T02:48:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:48:48.889 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:48:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:48:48.890 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:48:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:48:48.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.890+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 msgr2=0x7f0604070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.890+0000 7f0617fff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 0x7f0604070850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f060000b3b0 tx=0x7f0600009e30 comp rx=0 tx=0).stop 2024-03-22T02:48:48.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.890+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 msgr2=0x7f06181139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.890+0000 7f0617fff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0608002a80 tx=0x7f0608004600 comp rx=0 tx=0).stop 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 shutdown_connections 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f060406e3a0 0x7f0604070850 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06180fb2a0 0x7f06181139a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 --2- 172.21.15.103:0/3762947648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0618102d20 0x7f0618113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 >> 172.21.15.103:0/3762947648 conn(0x7f0618071300 msgr2=0x7f06180fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 shutdown_connections 2024-03-22T02:48:48.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:48:48.891+0000 7f0617fff700 1 -- 172.21.15.103:0/3762947648 wait complete. 2024-03-22T02:48:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:49 smithi122 ceph-mon[83368]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:49.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:49 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3762947648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:48:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:49 smithi103 ceph-mon[115211]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:48:49.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:49 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3762947648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:48:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:50 smithi122 ceph-mon[83368]: pgmap v519: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:50 smithi103 ceph-mon[115211]: pgmap v519: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:52 smithi122 ceph-mon[83368]: pgmap v520: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:52.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:52 smithi103 ceph-mon[115211]: pgmap v520: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:54 smithi122 ceph-mon[83368]: pgmap v521: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:54 smithi103 ceph-mon[115211]: pgmap v521: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:56.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:56 smithi122 ceph-mon[83368]: pgmap v522: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:56 smithi103 ceph-mon[115211]: pgmap v522: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:58.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:48:58 smithi122 ceph-mon[83368]: pgmap v523: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:48:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:48:58 smithi103 ceph-mon[115211]: pgmap v523: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:00 smithi122 ceph-mon[83368]: pgmap v524: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:00 smithi103 ceph-mon[115211]: pgmap v524: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:02 smithi122 ceph-mon[83368]: pgmap v525: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:02 smithi103 ceph-mon[115211]: pgmap v525: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:04.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:04 smithi122 ceph-mon[83368]: pgmap v526: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:04.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:04 smithi103 ceph-mon[115211]: pgmap v526: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:06 smithi122 ceph-mon[83368]: pgmap v527: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:06 smithi103 ceph-mon[115211]: pgmap v527: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:08.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:08 smithi122 ceph-mon[83368]: pgmap v528: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:08.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:08 smithi103 ceph-mon[115211]: pgmap v528: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:10.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:10 smithi122 ceph-mon[83368]: pgmap v529: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:10 smithi103 ceph-mon[115211]: pgmap v529: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:12.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:12 smithi122 ceph-mon[83368]: pgmap v530: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:12.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:12 smithi103 ceph-mon[115211]: pgmap v530: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:14.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:14 smithi122 ceph-mon[83368]: pgmap v531: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:14.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:14 smithi103 ceph-mon[115211]: pgmap v531: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:16 smithi122 ceph-mon[83368]: pgmap v532: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:16 smithi103 ceph-mon[115211]: pgmap v532: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:17.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:18.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:18 smithi122 ceph-mon[83368]: pgmap v533: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:18 smithi103 ceph-mon[115211]: pgmap v533: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:19.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3572757692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 msgr2=0x7ff4f00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3572757692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fb910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0004940 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3572757692 shutdown_connections 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3572757692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fb910 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3572757692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 0x7ff4f0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3572757692 >> 172.21.15.103:0/3572757692 conn(0x7ff4f00714c0 msgr2=0x7ff4f00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3572757692 shutdown_connections 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.007+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3572757692 wait complete. 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 Processor -- start 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 -- start start 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 0x7ff4f00fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f00fd150 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f7aaa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f00fd290 con 0x7ff4f0102e70 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35050/0 (socket says 172.21.15.103:35050) 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 -- 172.21.15.103:0/3552718466 learned_addr learned my addr 172.21.15.103:0/3552718466 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5045700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 0x7ff4f00fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 -- 172.21.15.103:0/3552718466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 msgr2=0x7ff4f00fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 0x7ff4f00fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.008+0000 7ff4f5846700 1 -- 172.21.15.103:0/3552718466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0005040 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4f5846700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00ec90 tx=0x7ff4ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ec007b30 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f0119e50 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4ec00f040 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4ec00cc80 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.009+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f011a250 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.010+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f0100170 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.010+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff4ec007c90 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.011+0000 7ff4e67fc700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 0x7ff4dc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.011+0000 7ff4f5045700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 0x7ff4dc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.011+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4ec08e210 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.011+0000 7ff4f5045700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 0x7ff4dc070850 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009b30 tx=0x7ff4e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.016+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4ec057410 con 0x7ff4f00fb4a0 2024-03-22T02:49:19.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.238+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4f004bed0 con 0x7ff4dc06e3a0 2024-03-22T02:49:19.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.239+0000 7ff4e67fc700 1 -- 172.21.15.103:0/3552718466 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4f004bed0 con 0x7ff4dc06e3a0 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 msgr2=0x7ff4dc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 0x7ff4dc070850 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009b30 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 msgr2=0x7ff4f00fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00ec90 tx=0x7ff4ec00b040 comp rx=0 tx=0).stop 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 shutdown_connections 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff4dc06e3a0 0x7ff4dc070850 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4f00fb4a0 0x7ff4f00fc680 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 --2- 172.21.15.103:0/3552718466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4f0102e70 0x7ff4f00fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 >> 172.21.15.103:0/3552718466 conn(0x7ff4f00714c0 msgr2=0x7ff4f00fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 shutdown_connections 2024-03-22T02:49:19.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.241+0000 7ff4f7aaa700 1 -- 172.21.15.103:0/3552718466 wait complete. 2024-03-22T02:49:19.253 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:49:19.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 -- 172.21.15.103:0/1463003714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d74101d10 msgr2=0x7f2d741020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/1463003714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d74101d10 0x7f2d741020e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2d64004940 tx=0x7f2d64009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 -- 172.21.15.103:0/1463003714 shutdown_connections 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/1463003714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d741026b0 0x7f2d740fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/1463003714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d74101d10 0x7f2d741020e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 -- 172.21.15.103:0/1463003714 >> 172.21.15.103:0/1463003714 conn(0x7f2d740f7880 msgr2=0x7f2d740f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 -- 172.21.15.103:0/1463003714 shutdown_connections 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.344+0000 7f2d7a82d700 1 -- 172.21.15.103:0/1463003714 wait complete. 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 Processor -- start 2024-03-22T02:49:19.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 -- start start 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d74101d10 0x7f2d7410ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7410fa50 con 0x7f2d741026b0 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d7a82d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7410fb90 con 0x7f2d74101d10 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d737fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d737fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35058/0 (socket says 172.21.15.103:35058) 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.345+0000 7f2d737fe700 1 -- 172.21.15.103:0/3007041876 learned_addr learned my addr 172.21.15.103:0/3007041876 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d73fff700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d74101d10 0x7f2d7410ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d737fe700 1 -- 172.21.15.103:0/3007041876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d74101d10 msgr2=0x7f2d7410ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d737fe700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d74101d10 0x7f2d7410ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d737fe700 1 -- 172.21.15.103:0/3007041876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64005040 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d737fe700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00b790 tx=0x7f2d5c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d5c00f6f0 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d74117f00 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d5c00fd20 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d5c0103a0 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.346+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d74118490 con 0x7f2d741026b0 2024-03-22T02:49:19.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.347+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d740fd460 con 0x7f2d741026b0 2024-03-22T02:49:19.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.348+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2d5c00f850 con 0x7f2d741026b0 2024-03-22T02:49:19.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.348+0000 7f2d70ff9700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 0x7f2d60070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.348+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2d5c08dc50 con 0x7f2d741026b0 2024-03-22T02:49:19.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.348+0000 7f2d73fff700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 0x7f2d60070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.349+0000 7f2d73fff700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 0x7f2d60070850 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d64004f90 tx=0x7f2d64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.353+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d5c056e50 con 0x7f2d741026b0 2024-03-22T02:49:19.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.568+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d74108ef0 con 0x7f2d6006e3a0 2024-03-22T02:49:19.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.570+0000 7f2d70ff9700 1 -- 172.21.15.103:0/3007041876 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2d74108ef0 con 0x7f2d6006e3a0 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 msgr2=0x7f2d60070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 0x7f2d60070850 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d64004f90 tx=0x7f2d64009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 msgr2=0x7f2d7410f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00b790 tx=0x7f2d5c009220 comp rx=0 tx=0).stop 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 shutdown_connections 2024-03-22T02:49:19.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2d6006e3a0 0x7f2d60070850 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d74101d10 0x7f2d7410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 --2- 172.21.15.103:0/3007041876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d741026b0 0x7f2d7410f370 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 >> 172.21.15.103:0/3007041876 conn(0x7f2d740f7880 msgr2=0x7f2d740f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 shutdown_connections 2024-03-22T02:49:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.572+0000 7f2d7a82d700 1 -- 172.21.15.103:0/3007041876 wait complete. 2024-03-22T02:49:19.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 -- 172.21.15.103:0/3261765238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308102800 msgr2=0x7f83080fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 --2- 172.21.15.103:0/3261765238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308102800 0x7f83080fc650 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8304004940 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 -- 172.21.15.103:0/3261765238 shutdown_connections 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 --2- 172.21.15.103:0/3261765238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308102800 0x7f83080fc650 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 --2- 172.21.15.103:0/3261765238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308101e60 0x7f8308102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 -- 172.21.15.103:0/3261765238 >> 172.21.15.103:0/3261765238 conn(0x7f8308071530 msgr2=0x7f8308071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 -- 172.21.15.103:0/3261765238 shutdown_connections 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.673+0000 7f830fb97700 1 -- 172.21.15.103:0/3261765238 wait complete. 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 Processor -- start 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 -- start start 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308102800 0x7f8308112d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83081133d0 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830fb97700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830810c9d0 con 0x7f8308102800 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830d933700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830d933700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35076/0 (socket says 172.21.15.103:35076) 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830d933700 1 -- 172.21.15.103:0/1205013094 learned_addr learned my addr 172.21.15.103:0/1205013094 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.674+0000 7f830d132700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308102800 0x7f8308112d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830d933700 1 -- 172.21.15.103:0/1205013094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308102800 msgr2=0x7f8308112d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830d933700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308102800 0x7f8308112d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830d933700 1 -- 172.21.15.103:0/1205013094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304005040 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830d933700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82f8002a50 tx=0x7f82f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f8007cc0 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82f800f040 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f800da30 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830810ccb0 con 0x7f8308101e60 2024-03-22T02:49:19.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.675+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830810d240 con 0x7f8308101e60 2024-03-22T02:49:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.676+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83080ff1c0 con 0x7f8308101e60 2024-03-22T02:49:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.676+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f82f800db90 con 0x7f8308101e60 2024-03-22T02:49:19.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.677+0000 7f82fe7fc700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 0x7f82f4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:19.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.677+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f82f808db80 con 0x7f8308101e60 2024-03-22T02:49:19.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.677+0000 7f830d132700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 0x7f82f4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:19.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.677+0000 7f830d132700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 0x7f82f4070850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8304004890 tx=0x7f8304009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:19.683 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.681+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82f805a8b0 con 0x7f8308101e60 2024-03-22T02:49:19.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.886+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f830810df00 con 0x7f82f406e3a0 2024-03-22T02:49:19.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.896+0000 7f82fe7fc700 1 -- 172.21.15.103:0/1205013094 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f830810df00 con 0x7f82f406e3a0 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (17m) 11m ago 29m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (15m) 11m ago 29m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (15m) 11m ago 28m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (17m) 11m ago 28m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (11m) 11m ago 23m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (11m) 11m ago 23m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (11m) 11m ago 23m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (11m) 11m ago 23m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (19m) 11m ago 30m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:49:19.898 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (18m) 11m ago 28m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (16m) 11m ago 30m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (16m) 11m ago 27m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (18m) 11m ago 29m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (18m) 11m ago 28m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (15m) 11m ago 27m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (14m) 11m ago 26m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (13m) 11m ago 26m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (13m) 11m ago 25m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (12m) 11m ago 25m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (12m) 11m ago 25m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:49:19.899 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (18m) 11m ago 28m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:49:19.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 msgr2=0x7f82f4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 0x7f82f4070850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8304004890 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:19.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 msgr2=0x7f8308112840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:19.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82f8002a50 tx=0x7f82f80046b0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 shutdown_connections 2024-03-22T02:49:19.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f82f406e3a0 0x7f82f4070850 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8308101e60 0x7f8308112840 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 --2- 172.21.15.103:0/1205013094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8308102800 0x7f8308112d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:19.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.898+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 >> 172.21.15.103:0/1205013094 conn(0x7f8308071530 msgr2=0x7f83080fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:19.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.899+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 shutdown_connections 2024-03-22T02:49:19.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.899+0000 7f830fb97700 1 -- 172.21.15.103:0/1205013094 wait complete. 2024-03-22T02:49:20.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 -- 172.21.15.103:0/2799499739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 msgr2=0x7f62600fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 --2- 172.21.15.103:0/2799499739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f62600fcdf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6254004940 tx=0x7f6254009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 -- 172.21.15.103:0/2799499739 shutdown_connections 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 --2- 172.21.15.103:0/2799499739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f62600fcdf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 --2- 172.21.15.103:0/2799499739 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62601031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 -- 172.21.15.103:0/2799499739 >> 172.21.15.103:0/2799499739 conn(0x7f62600f8970 msgr2=0x7f62600fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.998+0000 7f6265903700 1 -- 172.21.15.103:0/2799499739 shutdown_connections 2024-03-22T02:49:20.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.999+0000 7f6265903700 1 -- 172.21.15.103:0/2799499739 wait complete. 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.999+0000 7f6265903700 1 Processor -- start 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:19.999+0000 7f6265903700 1 -- start start 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f6265903700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f6265903700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f6265903700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6260070dd0 con 0x7f6260103780 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f6265903700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6260070f10 con 0x7f6260102de0 2024-03-22T02:49:20.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35088/0 (socket says 172.21.15.103:35088) 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 -- 172.21.15.103:0/196731435 learned_addr learned my addr 172.21.15.103:0/196731435 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625effd700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 -- 172.21.15.103:0/196731435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 msgr2=0x7f62600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 -- 172.21.15.103:0/196731435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6254005040 con 0x7f6260103780 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625effd700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f625e7fc700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6254009100 tx=0x7f6254009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f625400e040 con 0x7f6260103780 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6260119ee0 con 0x7f6260103780 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6254004290 con 0x7f6260103780 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.000+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f625401c3f0 con 0x7f6260103780 2024-03-22T02:49:20.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.001+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626011a360 con 0x7f6260103780 2024-03-22T02:49:20.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.001+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62601000f0 con 0x7f6260103780 2024-03-22T02:49:20.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.002+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6254013790 con 0x7f6260103780 2024-03-22T02:49:20.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.002+0000 7f62477fe700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 0x7f6248070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.003+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f625408f080 con 0x7f6260103780 2024-03-22T02:49:20.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.003+0000 7f625effd700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 0x7f6248070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.003+0000 7f625effd700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 0x7f6248070860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6250006fd0 tx=0x7f6250008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.007+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6254058300 con 0x7f6260103780 2024-03-22T02:49:20.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.289+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6260071960 con 0x7f6260103780 2024-03-22T02:49:20.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.290+0000 7f62477fe700 1 -- 172.21.15.103:0/196731435 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6254013420 con 0x7f6260103780 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:49:20.292 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:49:20.293 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 msgr2=0x7f6248070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 0x7f6248070860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6250006fd0 tx=0x7f6250008040 comp rx=0 tx=0).stop 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 msgr2=0x7f6260070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6254009100 tx=0x7f6254009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 shutdown_connections 2024-03-22T02:49:20.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f624806e3b0 0x7f6248070860 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6260102de0 0x7f62600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 --2- 172.21.15.103:0/196731435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6260103780 0x7f6260070800 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 >> 172.21.15.103:0/196731435 conn(0x7f62600f8970 msgr2=0x7f6260107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 shutdown_connections 2024-03-22T02:49:20.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.292+0000 7f6265903700 1 -- 172.21.15.103:0/196731435 wait complete. 2024-03-22T02:49:20.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 -- 172.21.15.103:0/952379318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 msgr2=0x7fdfcc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/952379318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc1031e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc004940 tx=0x7fdfbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 -- 172.21.15.103:0/952379318 shutdown_connections 2024-03-22T02:49:20.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/952379318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 0x7fdfcc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/952379318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc1031e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 -- 172.21.15.103:0/952379318 >> 172.21.15.103:0/952379318 conn(0x7fdfcc0f89a0 msgr2=0x7fdfcc0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 -- 172.21.15.103:0/952379318 shutdown_connections 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.393+0000 7fdfd1f33700 1 -- 172.21.15.103:0/952379318 wait complete. 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 Processor -- start 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 -- start start 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 0x7fdfcc1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc114720 con 0x7fdfcc102e10 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfd1f33700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc10db80 con 0x7fdfcc1037b0 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfcb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfcb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35104/0 (socket says 172.21.15.103:35104) 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfcb7fe700 1 -- 172.21.15.103:0/1827082312 learned_addr learned my addr 172.21.15.103:0/1827082312 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfc3fff700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 0x7fdfcc1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfcb7fe700 1 -- 172.21.15.103:0/1827082312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 msgr2=0x7fdfcc1140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.394+0000 7fdfcb7fe700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 0x7fdfcc1140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfcb7fe700 1 -- 172.21.15.103:0/1827082312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc005040 con 0x7fdfcc102e10 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfcb7fe700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00b3b0 tx=0x7fdfbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfbc00e040 con 0x7fdfcc102e10 2024-03-22T02:49:20.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfbc013420 con 0x7fdfcc102e10 2024-03-22T02:49:20.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc10de00 con 0x7fdfcc102e10 2024-03-22T02:49:20.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfbc01c630 con 0x7fdfcc102e10 2024-03-22T02:49:20.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.395+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc10e3c0 con 0x7fdfcc102e10 2024-03-22T02:49:20.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.396+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc100120 con 0x7fdfcc102e10 2024-03-22T02:49:20.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.396+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fdfbc013590 con 0x7fdfcc102e10 2024-03-22T02:49:20.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.397+0000 7fdfc8ff9700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 0x7fdfb8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.397+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdfbc08efe0 con 0x7fdfcc102e10 2024-03-22T02:49:20.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.397+0000 7fdfc3fff700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 0x7fdfb8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.397+0000 7fdfc3fff700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 0x7fdfb8070850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc10f430 tx=0x7fdfb400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.402+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdfbc058260 con 0x7fdfcc102e10 2024-03-22T02:49:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:20 smithi103 ceph-mon[115211]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:20 smithi103 ceph-mon[115211]: pgmap v534: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:20 smithi103 ceph-mon[115211]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:20 smithi103 ceph-mon[115211]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:20 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/196731435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:49:20.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.647+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdfcc04bed0 con 0x7fdfcc102e10 2024-03-22T02:49:20.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.648+0000 7fdfc8ff9700 1 -- 172.21.15.103:0/1827082312 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fdfbc05b920 con 0x7fdfcc102e10 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:49:20.650 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:49:20.651 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:20.652 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:20.653 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:20.653 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:20.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 msgr2=0x7fdfb8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.653 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 0x7fdfb8070850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc10f430 tx=0x7fdfb400b040 comp rx=0 tx=0).stop 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 msgr2=0x7fdfcc113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00b3b0 tx=0x7fdfbc009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 shutdown_connections 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fdfb806e3a0 0x7fdfb8070850 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfcc102e10 0x7fdfcc113b90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 --2- 172.21.15.103:0/1827082312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc1037b0 0x7fdfcc1140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 >> 172.21.15.103:0/1827082312 conn(0x7fdfcc0f89a0 msgr2=0x7fdfcc0facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 shutdown_connections 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.650+0000 7fdfd1f33700 1 -- 172.21.15.103:0/1827082312 wait complete. 2024-03-22T02:49:20.654 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:49:20.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- 172.21.15.103:0/1759891591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 msgr2=0x7f660c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 --2- 172.21.15.103:0/1759891591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c106df0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f65fc004940 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- 172.21.15.103:0/1759891591 shutdown_connections 2024-03-22T02:49:20.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 --2- 172.21.15.103:0/1759891591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c106df0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 --2- 172.21.15.103:0/1759891591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 0x7f660c0fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- 172.21.15.103:0/1759891591 >> 172.21.15.103:0/1759891591 conn(0x7f660c0f7880 msgr2=0x7f660c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- 172.21.15.103:0/1759891591 shutdown_connections 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- 172.21.15.103:0/1759891591 wait complete. 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 Processor -- start 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.752+0000 7f6613bc2700 1 -- start start 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f6613bc2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 0x7f660c113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f6613bc2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f6613bc2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c114aa0 con 0x7f660c0fe7b0 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f6613bc2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c10dde0 con 0x7f660c0fde10 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35132/0 (socket says 172.21.15.103:35132) 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 -- 172.21.15.103:0/144762153 learned_addr learned my addr 172.21.15.103:0/144762153 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:20.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661195e700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 0x7f660c113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 -- 172.21.15.103:0/144762153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 msgr2=0x7f660c113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 0x7f660c113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 -- 172.21.15.103:0/144762153 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc005040 con 0x7f660c0fe7b0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f661115d700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f65fc004890 tx=0x7f65fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65fc00e040 con 0x7f660c0fe7b0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65fc009af0 con 0x7f660c0fe7b0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660c10e000 con 0x7f660c0fe7b0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.753+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65fc01c630 con 0x7f660c0fe7b0 2024-03-22T02:49:20.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.754+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660c10e540 con 0x7f660c0fe7b0 2024-03-22T02:49:20.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.755+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660c101760 con 0x7f660c0fe7b0 2024-03-22T02:49:20.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.755+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f65fc00bbc0 con 0x7f660c0fe7b0 2024-03-22T02:49:20.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.755+0000 7f66027fc700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 0x7f65f8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:20.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.756+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f65fc08f0d0 con 0x7f660c0fe7b0 2024-03-22T02:49:20.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.756+0000 7f661195e700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 0x7f65f8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:20.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.756+0000 7f661195e700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 0x7f65f8070850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6604001250 tx=0x7f6604009620 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:20.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.760+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65fc05be80 con 0x7f660c0fe7b0 2024-03-22T02:49:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:20 smithi122 ceph-mon[83368]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:20 smithi122 ceph-mon[83368]: pgmap v534: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:20.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:20 smithi122 ceph-mon[83368]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:20 smithi122 ceph-mon[83368]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:20.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:20 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/196731435' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:49:20.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.970+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f660c10f540 con 0x7f65f806e3a0 2024-03-22T02:49:20.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.972+0000 7f66027fc700 1 -- 172.21.15.103:0/144762153 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f660c10f540 con 0x7f65f806e3a0 2024-03-22T02:49:20.973 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:49:20.973 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:49:20.974 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 msgr2=0x7f65f8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 0x7f65f8070850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6604001250 tx=0x7f6604009620 comp rx=0 tx=0).stop 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 msgr2=0x7f660c1143c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f65fc004890 tx=0x7f65fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 shutdown_connections 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65f806e3a0 0x7f65f8070850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f660c0fde10 0x7f660c113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 --2- 172.21.15.103:0/144762153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f660c0fe7b0 0x7f660c1143c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:20.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 >> 172.21.15.103:0/144762153 conn(0x7f660c0f7880 msgr2=0x7f660c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:20.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 shutdown_connections 2024-03-22T02:49:20.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:20.974+0000 7f6613bc2700 1 -- 172.21.15.103:0/144762153 wait complete. 2024-03-22T02:49:21.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.075+0000 7fcf165cb700 1 -- 172.21.15.103:0/1171258030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf10102f70 msgr2=0x7fcf10103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:21.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.075+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1171258030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf10102f70 0x7fcf10103340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcf00004940 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:21.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.075+0000 7fcf165cb700 1 -- 172.21.15.103:0/1171258030 shutdown_connections 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1171258030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fb3b0 0x7fcf100fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1171258030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf10102f70 0x7fcf10103340 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- 172.21.15.103:0/1171258030 >> 172.21.15.103:0/1171258030 conn(0x7fcf10072310 msgr2=0x7fcf10072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- 172.21.15.103:0/1171258030 shutdown_connections 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- 172.21.15.103:0/1171258030 wait complete. 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 Processor -- start 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- start start 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf10102f70 0x7fcf10113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf101144d0 con 0x7fcf100fb3b0 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.076+0000 7fcf165cb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1010dad0 con 0x7fcf10102f70 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35158/0 (socket says 172.21.15.103:35158) 2024-03-22T02:49:21.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 -- 172.21.15.103:0/1712811625 learned_addr learned my addr 172.21.15.103:0/1712811625 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0f7fe700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf10102f70 0x7fcf10113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 -- 172.21.15.103:0/1712811625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf10102f70 msgr2=0x7fcf10113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf10102f70 0x7fcf10113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 -- 172.21.15.103:0/1712811625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00005040 con 0x7fcf100fb3b0 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0ffff700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000b8b0 tx=0x7fcf00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf0000e040 con 0x7fcf100fb3b0 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf00004290 con 0x7fcf100fb3b0 2024-03-22T02:49:21.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1010dd50 con 0x7fcf100fb3b0 2024-03-22T02:49:21.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf0001c3f0 con 0x7fcf100fb3b0 2024-03-22T02:49:21.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.077+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1010e310 con 0x7fcf100fb3b0 2024-03-22T02:49:21.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.078+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf101053b0 con 0x7fcf100fb3b0 2024-03-22T02:49:21.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.079+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcf00013420 con 0x7fcf100fb3b0 2024-03-22T02:49:21.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.079+0000 7fcf0cff9700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 0x7fcef8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:21.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.079+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcf0008f050 con 0x7fcf100fb3b0 2024-03-22T02:49:21.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.079+0000 7fcf0f7fe700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 0x7fcef8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:21.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.080+0000 7fcf0f7fe700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 0x7fcef8070850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcf1010ec90 tx=0x7fcf04009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:21.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.084+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf0005be00 con 0x7fcf100fb3b0 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.363+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcf1010ef90 con 0x7fcf100fb3b0 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.363+0000 7fcf0cff9700 1 -- 172.21.15.103:0/1712811625 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fcf0005b990 con 0x7fcf100fb3b0 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:49:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 msgr2=0x7fcef8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 0x7fcef8070850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcf1010ec90 tx=0x7fcf04009040 comp rx=0 tx=0).stop 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 msgr2=0x7fcf10113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000b8b0 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 shutdown_connections 2024-03-22T02:49:21.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcef806e3a0 0x7fcef8070850 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf100fb3b0 0x7fcf10113940 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 --2- 172.21.15.103:0/1712811625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf10102f70 0x7fcf10113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:21.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.365+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 >> 172.21.15.103:0/1712811625 conn(0x7fcf10072310 msgr2=0x7fcf100f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:21.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.366+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 shutdown_connections 2024-03-22T02:49:21.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:21.366+0000 7fcf165cb700 1 -- 172.21.15.103:0/1712811625 wait complete. 2024-03-22T02:49:21.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1827082312' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:49:21.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:21 smithi103 ceph-mon[115211]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:21.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1712811625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:49:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1827082312' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:49:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:21 smithi122 ceph-mon[83368]: from='client.35032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:21.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1712811625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:49:22.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:22 smithi122 ceph-mon[83368]: pgmap v535: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:22 smithi103 ceph-mon[115211]: pgmap v535: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:24.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:24 smithi122 ceph-mon[83368]: pgmap v536: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:24.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:24 smithi103 ceph-mon[115211]: pgmap v536: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:26.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:26 smithi122 ceph-mon[83368]: pgmap v537: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:26 smithi103 ceph-mon[115211]: pgmap v537: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:28.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:28 smithi122 ceph-mon[83368]: pgmap v538: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:28 smithi103 ceph-mon[115211]: pgmap v538: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:30.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:30 smithi122 ceph-mon[83368]: pgmap v539: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:30 smithi103 ceph-mon[115211]: pgmap v539: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:32 smithi122 ceph-mon[83368]: pgmap v540: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:32 smithi103 ceph-mon[115211]: pgmap v540: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:34.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:34 smithi122 ceph-mon[83368]: pgmap v541: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:34.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:34 smithi103 ceph-mon[115211]: pgmap v541: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:36.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:36 smithi122 ceph-mon[83368]: pgmap v542: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:36.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:36 smithi103 ceph-mon[115211]: pgmap v542: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:38.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:38 smithi122 ceph-mon[83368]: pgmap v543: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:38 smithi103 ceph-mon[115211]: pgmap v543: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:40.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:40 smithi122 ceph-mon[83368]: pgmap v544: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:40 smithi103 ceph-mon[115211]: pgmap v544: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:42 smithi122 ceph-mon[83368]: pgmap v545: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:42 smithi103 ceph-mon[115211]: pgmap v545: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:44.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:44 smithi122 ceph-mon[83368]: pgmap v546: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:44.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:44 smithi103 ceph-mon[115211]: pgmap v546: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:46.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:46 smithi122 ceph-mon[83368]: pgmap v547: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:46.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:46 smithi103 ceph-mon[115211]: pgmap v547: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:49:48.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:48 smithi122 ceph-mon[83368]: pgmap v548: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:48.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:48 smithi103 ceph-mon[115211]: pgmap v548: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:50.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:50 smithi122 ceph-mon[83368]: pgmap v549: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:50 smithi103 ceph-mon[115211]: pgmap v549: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:51.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 -- 172.21.15.103:0/862952490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4101d70 msgr2=0x7feef4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 --2- 172.21.15.103:0/862952490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4101d70 0x7feef4102140 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7feee4004940 tx=0x7feee4009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:51.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 -- 172.21.15.103:0/862952490 shutdown_connections 2024-03-22T02:49:51.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 --2- 172.21.15.103:0/862952490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4102710 0x7feef40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 --2- 172.21.15.103:0/862952490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4101d70 0x7feef4102140 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 -- 172.21.15.103:0/862952490 >> 172.21.15.103:0/862952490 conn(0x7feef4072bc0 msgr2=0x7feef4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:51.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 -- 172.21.15.103:0/862952490 shutdown_connections 2024-03-22T02:49:51.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.479+0000 7feefa6da700 1 -- 172.21.15.103:0/862952490 wait complete. 2024-03-22T02:49:51.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 Processor -- start 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 -- start start 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4101d70 0x7feef410ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef410f6d0 con 0x7feef4102710 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feefa6da700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef410f810 con 0x7feef4101d70 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feef37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feef37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35522/0 (socket says 172.21.15.103:35522) 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feef37fe700 1 -- 172.21.15.103:0/1576780306 learned_addr learned my addr 172.21.15.103:0/1576780306 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.480+0000 7feef3fff700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4101d70 0x7feef410ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef37fe700 1 -- 172.21.15.103:0/1576780306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4101d70 msgr2=0x7feef410ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef37fe700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4101d70 0x7feef410ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef37fe700 1 -- 172.21.15.103:0/1576780306 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4005040 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef37fe700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feee800b790 tx=0x7feee8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee800f6f0 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feee800fd20 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feee80103a0 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef4117b60 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.481+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef4118120 con 0x7feef4102710 2024-03-22T02:49:51.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.482+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef40ff080 con 0x7feef4102710 2024-03-22T02:49:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.483+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7feee8011400 con 0x7feef4102710 2024-03-22T02:49:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.483+0000 7feef0ff9700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 0x7feedc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.483+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feee808dc70 con 0x7feef4102710 2024-03-22T02:49:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.483+0000 7feef3fff700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 0x7feedc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.483+0000 7feef3fff700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 0x7feedc070850 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feee4004f90 tx=0x7feee4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:51.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.488+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feee8056e70 con 0x7feef4102710 2024-03-22T02:49:51.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.703+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feef4065dd0 con 0x7feedc06e3a0 2024-03-22T02:49:51.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.705+0000 7feef0ff9700 1 -- 172.21.15.103:0/1576780306 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feef4065dd0 con 0x7feedc06e3a0 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 msgr2=0x7feedc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 0x7feedc070850 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feee4004f90 tx=0x7feee4009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 msgr2=0x7feef410f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feee800b790 tx=0x7feee8009220 comp rx=0 tx=0).stop 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 shutdown_connections 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7feedc06e3a0 0x7feedc070850 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef4101d70 0x7feef410ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 --2- 172.21.15.103:0/1576780306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef4102710 0x7feef410f190 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 >> 172.21.15.103:0/1576780306 conn(0x7feef4072bc0 msgr2=0x7feef40fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:51.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 shutdown_connections 2024-03-22T02:49:51.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.707+0000 7feefa6da700 1 -- 172.21.15.103:0/1576780306 wait complete. 2024-03-22T02:49:51.719 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:49:51.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- 172.21.15.103:0/3947849688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 msgr2=0x7f4260102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 --2- 172.21.15.103:0/3947849688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260102140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4250009ce0 tx=0x7f4250004d10 comp rx=0 tx=0).stop 2024-03-22T02:49:51.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- 172.21.15.103:0/3947849688 shutdown_connections 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 --2- 172.21.15.103:0/3947849688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42600fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 --2- 172.21.15.103:0/3947849688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260102140 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- 172.21.15.103:0/3947849688 >> 172.21.15.103:0/3947849688 conn(0x7f4260072bc0 msgr2=0x7f4260072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- 172.21.15.103:0/3947849688 shutdown_connections 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- 172.21.15.103:0/3947849688 wait complete. 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 Processor -- start 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.811+0000 7f4264812700 1 -- start start 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4264812700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4264812700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42601141c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4264812700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601148a0 con 0x7f4260101d70 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4264812700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426010dc90 con 0x7f4260102710 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35538/0 (socket says 172.21.15.103:35538) 2024-03-22T02:49:51.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 -- 172.21.15.103:0/1290739330 learned_addr learned my addr 172.21.15.103:0/1290739330 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4255d9b700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42601141c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 -- 172.21.15.103:0/1290739330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 msgr2=0x7f42601141c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42601141c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 -- 172.21.15.103:0/1290739330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009940 con 0x7f4260101d70 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4255d9b700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42601141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f425e59c700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4250005010 tx=0x7f4250004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425000e040 con 0x7f4260101d70 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4250008e20 con 0x7f4260101d70 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426010df10 con 0x7f4260101d70 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.812+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425001c660 con 0x7f4260101d70 2024-03-22T02:49:51.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.813+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426010e4d0 con 0x7f4260101d70 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.813+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42600fd4c0 con 0x7f4260101d70 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.814+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4250012a70 con 0x7f4260101d70 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.814+0000 7f42577fe700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 0x7f424c070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.814+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f425008f9c0 con 0x7f4260101d70 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.814+0000 7f4255d9b700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 0x7f424c070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:51.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.815+0000 7f4255d9b700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 0x7f424c070860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4248006fd0 tx=0x7f4248008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:51.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:51.819+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4250058bc0 con 0x7f4260101d70 2024-03-22T02:49:52.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.033+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4260108f50 con 0x7f424c06e3b0 2024-03-22T02:49:52.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.034+0000 7f42577fe700 1 -- 172.21.15.103:0/1290739330 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4260108f50 con 0x7f424c06e3b0 2024-03-22T02:49:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 msgr2=0x7f424c070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 0x7f424c070860 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4248006fd0 tx=0x7f4248008040 comp rx=0 tx=0).stop 2024-03-22T02:49:52.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 msgr2=0x7f4260113c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4250005010 tx=0x7f4250004d10 comp rx=0 tx=0).stop 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 shutdown_connections 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f424c06e3b0 0x7f424c070860 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4260101d70 0x7f4260113c80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 --2- 172.21.15.103:0/1290739330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4260102710 0x7f42601141c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.036+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 >> 172.21.15.103:0/1290739330 conn(0x7f4260072bc0 msgr2=0x7f4260065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.037+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 shutdown_connections 2024-03-22T02:49:52.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.037+0000 7f4264812700 1 -- 172.21.15.103:0/1290739330 wait complete. 2024-03-22T02:49:52.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/1337481060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4101d70 msgr2=0x7fbbb4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/1337481060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4101d70 0x7fbbb4102140 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004940 tx=0x7fbbb0009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:52.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/1337481060 shutdown_connections 2024-03-22T02:49:52.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/1337481060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4102710 0x7fbbb40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/1337481060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4101d70 0x7fbbb4102140 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.137+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/1337481060 >> 172.21.15.103:0/1337481060 conn(0x7fbbb4072bc0 msgr2=0x7fbbb4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/1337481060 shutdown_connections 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/1337481060 wait complete. 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 Processor -- start 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 -- start start 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4101d70 0x7fbbb410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb410f670 con 0x7fbbb4102710 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbbbb4e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb410f7b0 con 0x7fbbb4101d70 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.138+0000 7fbbb90e9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35544/0 (socket says 172.21.15.103:35544) 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 -- 172.21.15.103:0/4255920273 learned_addr learned my addr 172.21.15.103:0/4255920273 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb98ea700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4101d70 0x7fbbb410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 -- 172.21.15.103:0/4255920273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4101d70 msgr2=0x7fbbb410eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4101d70 0x7fbbb410eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 -- 172.21.15.103:0/4255920273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0005040 con 0x7fbbb4102710 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbb90e9700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbba4002a50 tx=0x7fbba4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba4007cc0 con 0x7fbbb4102710 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbba400f040 con 0x7fbbb4102710 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb4117b50 con 0x7fbbb4102710 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba400d9c0 con 0x7fbbb4102710 2024-03-22T02:49:52.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.139+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb4118030 con 0x7fbbb4102710 2024-03-22T02:49:52.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.140+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb40ff080 con 0x7fbbb4102710 2024-03-22T02:49:52.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.141+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fbba4009730 con 0x7fbbb4102710 2024-03-22T02:49:52.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.141+0000 7fbbaa7fc700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 0x7fbba0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.141+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbba408da80 con 0x7fbbb4102710 2024-03-22T02:49:52.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.141+0000 7fbbb98ea700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 0x7fbba0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.142+0000 7fbbb98ea700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 0x7fbba0070850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004f90 tx=0x7fbbb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.146+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbba4056d00 con 0x7fbbb4102710 2024-03-22T02:49:52.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.357+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbb4108ee0 con 0x7fbba006e3a0 2024-03-22T02:49:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.367+0000 7fbbaa7fc700 1 -- 172.21.15.103:0/4255920273 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbbb4108ee0 con 0x7fbba006e3a0 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (18m) 12m ago 29m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (16m) 12m ago 29m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (16m) 11m ago 28m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (18m) 12m ago 29m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (12m) 12m ago 24m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (12m) 12m ago 23m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (11m) 11m ago 23m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:49:52.370 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (11m) 11m ago 24m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (19m) 12m ago 31m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (19m) 11m ago 28m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (17m) 12m ago 31m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (16m) 11m ago 28m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (19m) 12m ago 29m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (19m) 11m ago 28m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (15m) 12m ago 27m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (15m) 12m ago 27m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (14m) 12m ago 26m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (13m) 11m ago 26m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (13m) 11m ago 26m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (12m) 11m ago 25m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:49:52.371 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (18m) 12m ago 29m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 msgr2=0x7fbba0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 0x7fbba0070850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004f90 tx=0x7fbbb0009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 msgr2=0x7fbbb410f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbba4002a50 tx=0x7fbba4004600 comp rx=0 tx=0).stop 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 shutdown_connections 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fbba006e3a0 0x7fbba0070850 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbb4101d70 0x7fbbb410eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 --2- 172.21.15.103:0/4255920273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbbb4102710 0x7fbbb410f020 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 >> 172.21.15.103:0/4255920273 conn(0x7fbbb4072bc0 msgr2=0x7fbbb40fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 shutdown_connections 2024-03-22T02:49:52.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.369+0000 7fbbbbb4e700 1 -- 172.21.15.103:0/4255920273 wait complete. 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 -- 172.21.15.103:0/2462684034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 msgr2=0x7f665c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 --2- 172.21.15.103:0/2462684034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c106f00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6644004940 tx=0x7f6644009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 -- 172.21.15.103:0/2462684034 shutdown_connections 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 --2- 172.21.15.103:0/2462684034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c106f00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 --2- 172.21.15.103:0/2462684034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 -- 172.21.15.103:0/2462684034 >> 172.21.15.103:0/2462684034 conn(0x7f665c072bc0 msgr2=0x7f665c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 -- 172.21.15.103:0/2462684034 shutdown_connections 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 -- 172.21.15.103:0/2462684034 wait complete. 2024-03-22T02:49:52.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.469+0000 7f66619ea700 1 Processor -- start 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66619ea700 1 -- start start 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66619ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66619ea700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66619ea700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0fc820 con 0x7f665c0fe830 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66619ea700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0fc960 con 0x7f665c0fde90 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35560/0 (socket says 172.21.15.103:35560) 2024-03-22T02:49:52.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 -- 172.21.15.103:0/996268184 learned_addr learned my addr 172.21.15.103:0/996268184 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f665affd700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 -- 172.21.15.103:0/996268184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 msgr2=0x7f665c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 -- 172.21.15.103:0/996268184 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6644005040 con 0x7f665c0fe830 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f665affd700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.470+0000 7f66527fc700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f664400b3b0 tx=0x7f6644009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.471+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f664400e040 con 0x7f665c0fe830 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.471+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c0fcbb0 con 0x7f665c0fe830 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.471+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6644013420 con 0x7f665c0fe830 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.471+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c11a250 con 0x7f665c0fe830 2024-03-22T02:49:52.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.471+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66440255e0 con 0x7f665c0fe830 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.472+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c0052b0 con 0x7f665c0fe830 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.472+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f664400ba90 con 0x7f665c0fe830 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.473+0000 7f6653fff700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 0x7f6648070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.473+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f664408efa0 con 0x7f665c0fe830 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.473+0000 7f665affd700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 0x7f6648070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.473+0000 7f665affd700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 0x7f6648070850 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f664c0041d0 tx=0x7f664c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.480+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f66440c3740 con 0x7f665c0fe830 2024-03-22T02:49:52.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.759+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f663c005d70 con 0x7f665c0fe830 2024-03-22T02:49:52.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:52 smithi103 ceph-mon[115211]: pgmap v550: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:52.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:52 smithi103 ceph-mon[115211]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:52.762 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:52 smithi103 ceph-mon[115211]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:52.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.760+0000 7f6653fff700 1 -- 172.21.15.103:0/996268184 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6644058220 con 0x7f665c0fe830 2024-03-22T02:49:52.762 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:49:52.762 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:49:52.762 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:49:52.763 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:49:52.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 msgr2=0x7f6648070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 0x7f6648070850 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f664c0041d0 tx=0x7f664c009040 comp rx=0 tx=0).stop 2024-03-22T02:49:52.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 msgr2=0x7f665c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f664400b3b0 tx=0x7f6644009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 shutdown_connections 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f664806e3a0 0x7f6648070850 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f665c0fde90 0x7f665c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 --2- 172.21.15.103:0/996268184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f665c0fe830 0x7f665c0fc290 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 >> 172.21.15.103:0/996268184 conn(0x7f665c072bc0 msgr2=0x7f665c1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.762+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 shutdown_connections 2024-03-22T02:49:52.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.763+0000 7f66619ea700 1 -- 172.21.15.103:0/996268184 wait complete. 2024-03-22T02:49:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:52 smithi122 ceph-mon[83368]: pgmap v550: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:52.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:52 smithi122 ceph-mon[83368]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:52.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:52 smithi122 ceph-mon[83368]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 -- 172.21.15.103:0/572405236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d41026c0 msgr2=0x7f62d40fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 --2- 172.21.15.103:0/572405236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d41026c0 0x7f62d40fbd30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009ce0 tx=0x7f62c8004d10 comp rx=0 tx=0).stop 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 -- 172.21.15.103:0/572405236 shutdown_connections 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 --2- 172.21.15.103:0/572405236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d41026c0 0x7f62d40fbd30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 --2- 172.21.15.103:0/572405236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d4101d20 0x7f62d41020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 -- 172.21.15.103:0/572405236 >> 172.21.15.103:0/572405236 conn(0x7f62d40f7870 msgr2=0x7f62d40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 -- 172.21.15.103:0/572405236 shutdown_connections 2024-03-22T02:49:52.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.864+0000 7f62d99cb700 1 -- 172.21.15.103:0/572405236 wait complete. 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 Processor -- start 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 -- start start 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d41026c0 0x7f62d4114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d4114750 con 0x7f62d4101d20 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d99cb700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d410dca0 con 0x7f62d41026c0 2024-03-22T02:49:52.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35582/0 (socket says 172.21.15.103:35582) 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 -- 172.21.15.103:0/2090481141 learned_addr learned my addr 172.21.15.103:0/2090481141 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d27fc700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d41026c0 0x7f62d4114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 -- 172.21.15.103:0/2090481141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d41026c0 msgr2=0x7f62d4114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d41026c0 0x7f62d4114070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.865+0000 7f62d2ffd700 1 -- 172.21.15.103:0/2090481141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8009940 con 0x7f62d4101d20 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62d2ffd700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b790 tx=0x7f62cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc00f6f0 con 0x7f62d4101d20 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62cc00fd20 con 0x7f62d4101d20 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d410df20 con 0x7f62d4101d20 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc0103a0 con 0x7f62d4101d20 2024-03-22T02:49:52.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.866+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d410e4b0 con 0x7f62d4101d20 2024-03-22T02:49:52.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.867+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d40fd470 con 0x7f62d4101d20 2024-03-22T02:49:52.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.867+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f62cc011400 con 0x7f62d4101d20 2024-03-22T02:49:52.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.868+0000 7f62bb7fe700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 0x7f62bc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:52.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.868+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f62cc08da90 con 0x7f62d4101d20 2024-03-22T02:49:52.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.868+0000 7f62d27fc700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 0x7f62bc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:52.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.868+0000 7f62d27fc700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 0x7f62bc070850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f62c800b740 tx=0x7f62c8004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:52.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:52.872+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f62cc05a900 con 0x7f62d4101d20 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.114+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f62d410f180 con 0x7f62d4101d20 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.114+0000 7f62bb7fe700 1 -- 172.21.15.103:0/2090481141 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f62cc05a490 con 0x7f62d4101d20 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:49:53.116 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:49:53.117 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:49:53.118 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:53.119 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 msgr2=0x7f62bc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 0x7f62bc070850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f62c800b740 tx=0x7f62c8004d10 comp rx=0 tx=0).stop 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 msgr2=0x7f62d4113b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b790 tx=0x7f62cc009250 comp rx=0 tx=0).stop 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 shutdown_connections 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f62bc06e3a0 0x7f62bc070850 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4101d20 0x7f62d4113b30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 --2- 172.21.15.103:0/2090481141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62d41026c0 0x7f62d4114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 >> 172.21.15.103:0/2090481141 conn(0x7f62d40f7870 msgr2=0x7f62d40f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:53.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 shutdown_connections 2024-03-22T02:49:53.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.117+0000 7f62d99cb700 1 -- 172.21.15.103:0/2090481141 wait complete. 2024-03-22T02:49:53.121 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:49:53.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 -- 172.21.15.103:0/2500330604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 msgr2=0x7f84fc0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 --2- 172.21.15.103:0/2500330604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc0fce30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f84e4004940 tx=0x7f84e4009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 -- 172.21.15.103:0/2500330604 shutdown_connections 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 --2- 172.21.15.103:0/2500330604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc0fce30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 --2- 172.21.15.103:0/2500330604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 0x7f84fc1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 -- 172.21.15.103:0/2500330604 >> 172.21.15.103:0/2500330604 conn(0x7f84fc0f89b0 msgr2=0x7f84fc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 -- 172.21.15.103:0/2500330604 shutdown_connections 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.219+0000 7f8500cbf700 1 -- 172.21.15.103:0/2500330604 wait complete. 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 Processor -- start 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 -- start start 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 0x7f84fc113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1145b0 con 0x7f84fc1037c0 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f8500cbf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc10db90 con 0x7f84fc102e20 2024-03-22T02:49:53.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35598/0 (socket says 172.21.15.103:35598) 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 -- 172.21.15.103:0/157773080 learned_addr learned my addr 172.21.15.103:0/157773080 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84fa59c700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 0x7f84fc113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 -- 172.21.15.103:0/157773080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 msgr2=0x7f84fc113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 0x7f84fc113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 -- 172.21.15.103:0/157773080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4005040 con 0x7f84fc1037c0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.220+0000 7f84f1d9b700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009180 tx=0x7f84e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e400e040 con 0x7f84fc1037c0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84e400bbc0 con 0x7f84fc1037c0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc10de10 con 0x7f84fc1037c0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84e401c3f0 con 0x7f84fc1037c0 2024-03-22T02:49:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc10e3d0 con 0x7f84fc1037c0 2024-03-22T02:49:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.221+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc1052f0 con 0x7f84fc1037c0 2024-03-22T02:49:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.222+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f84e400bd30 con 0x7f84fc1037c0 2024-03-22T02:49:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.222+0000 7f84f37fe700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 0x7f84e8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.223+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f84e408f110 con 0x7f84fc1037c0 2024-03-22T02:49:53.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.223+0000 7f84fa59c700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 0x7f84e8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.223+0000 7f84fa59c700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 0x7f84e8070850 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f84ec004990 tx=0x7f84ec009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.227+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84e405bec0 con 0x7f84fc1037c0 2024-03-22T02:49:53.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.443+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84fc10f0e0 con 0x7f84e806e3a0 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.444+0000 7f84f37fe700 1 -- 172.21.15.103:0/157773080 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f84fc10f0e0 con 0x7f84e806e3a0 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:49:53.446 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:49:53.447 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 msgr2=0x7f84e8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 0x7f84e8070850 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f84ec004990 tx=0x7f84ec009ee0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 msgr2=0x7f84fc113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009180 tx=0x7f84e4009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 shutdown_connections 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f84e806e3a0 0x7f84e8070850 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84fc102e20 0x7f84fc113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 --2- 172.21.15.103:0/157773080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84fc1037c0 0x7f84fc113f60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 >> 172.21.15.103:0/157773080 conn(0x7f84fc0f89b0 msgr2=0x7f84fc102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:53.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 shutdown_connections 2024-03-22T02:49:53.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.446+0000 7f8500cbf700 1 -- 172.21.15.103:0/157773080 wait complete. 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 -- 172.21.15.103:0/3774039977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 msgr2=0x7ff9ac106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 --2- 172.21.15.103:0/3774039977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac106f00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff994004940 tx=0x7ff994009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 -- 172.21.15.103:0/3774039977 shutdown_connections 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 --2- 172.21.15.103:0/3774039977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac106f00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 --2- 172.21.15.103:0/3774039977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 0x7ff9ac0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 -- 172.21.15.103:0/3774039977 >> 172.21.15.103:0/3774039977 conn(0x7ff9ac072bc0 msgr2=0x7ff9ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 -- 172.21.15.103:0/3774039977 shutdown_connections 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.548+0000 7ff9b2190700 1 -- 172.21.15.103:0/3774039977 wait complete. 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 Processor -- start 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 -- start start 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 0x7ff9ac113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac1146b0 con 0x7ff9ac0fe830 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9b2190700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac10daa0 con 0x7ff9ac0fde90 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9aaffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9aaffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35610/0 (socket says 172.21.15.103:35610) 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9aaffd700 1 -- 172.21.15.103:0/1665587253 learned_addr learned my addr 172.21.15.103:0/1665587253 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9ab7fe700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 0x7ff9ac113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9aaffd700 1 -- 172.21.15.103:0/1665587253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 msgr2=0x7ff9ac113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.549+0000 7ff9aaffd700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 0x7ff9ac113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9aaffd700 1 -- 172.21.15.103:0/1665587253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff994005040 con 0x7ff9ac0fe830 2024-03-22T02:49:53.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9aaffd700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff99400e040 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac10dd20 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff994013420 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff99401c630 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.550+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac10e260 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.551+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac065e30 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.551+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff994013a90 con 0x7ff9ac0fe830 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.552+0000 7ff9a3fff700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 0x7ff998070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:49:53.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.552+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff99408ecc0 con 0x7ff9ac0fe830 2024-03-22T02:49:53.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.552+0000 7ff9ab7fe700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 0x7ff998070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:49:53.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.552+0000 7ff9ab7fe700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 0x7ff998070860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4007b20 tx=0x7ff9a4009e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:49:53.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.557+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff99405ba70 con 0x7ff9ac0fe830 2024-03-22T02:49:53.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.837+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff9ac109f10 con 0x7ff9ac0fe830 2024-03-22T02:49:53.839 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:53 smithi103 ceph-mon[115211]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:53.839 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:53 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/996268184' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:49:53.839 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:53 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2090481141' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:49:53.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.837+0000 7ff9a3fff700 1 -- 172.21.15.103:0/1665587253 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff99405b600 con 0x7ff9ac0fe830 2024-03-22T02:49:53.839 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:49:53.840 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:49:53.840 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:49:53.840 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:49:53.840 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:49:53.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.839+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 msgr2=0x7ff998070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.839+0000 7ff9b2190700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 0x7ff998070860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4007b20 tx=0x7ff9a4009e20 comp rx=0 tx=0).stop 2024-03-22T02:49:53.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.839+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 msgr2=0x7ff9ac114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:49:53.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.839+0000 7ff9b2190700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff994009e30 comp rx=0 tx=0).stop 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 shutdown_connections 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff99806e3b0 0x7ff998070860 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ac0fde90 0x7ff9ac113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 --2- 172.21.15.103:0/1665587253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9ac0fe830 0x7ff9ac114060 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 >> 172.21.15.103:0/1665587253 conn(0x7ff9ac072bc0 msgr2=0x7ff9ac103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 shutdown_connections 2024-03-22T02:49:53.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:49:53.840+0000 7ff9b2190700 1 -- 172.21.15.103:0/1665587253 wait complete. 2024-03-22T02:49:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:53 smithi122 ceph-mon[83368]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:53 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/996268184' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:49:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:53 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2090481141' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:49:54.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:54 smithi122 ceph-mon[83368]: pgmap v551: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:54 smithi122 ceph-mon[83368]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:54.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:54 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1665587253' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:49:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:54 smithi103 ceph-mon[115211]: pgmap v551: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:54 smithi103 ceph-mon[115211]: from='client.35060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:49:54.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:54 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1665587253' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:49:56.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:56 smithi122 ceph-mon[83368]: pgmap v552: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:56.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:56 smithi103 ceph-mon[115211]: pgmap v552: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:58.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:49:58 smithi122 ceph-mon[83368]: pgmap v553: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:49:58.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:49:58 smithi103 ceph-mon[115211]: pgmap v553: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:00 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T02:49:59.999+0000 7f4cf3d54700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:50:00.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:00 smithi122 ceph-mon[83368]: pgmap v554: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:00.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:00 smithi122 ceph-mon[83368]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:50:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:00 smithi103 ceph-mon[115211]: pgmap v554: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:00 smithi103 ceph-mon[115211]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:50:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:02 smithi122 ceph-mon[83368]: pgmap v555: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:02.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:02 smithi103 ceph-mon[115211]: pgmap v555: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:04.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:04 smithi122 ceph-mon[83368]: pgmap v556: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:04.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:04 smithi103 ceph-mon[115211]: pgmap v556: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:06.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:06 smithi122 ceph-mon[83368]: pgmap v557: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:06.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:06 smithi103 ceph-mon[115211]: pgmap v557: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:08.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:08 smithi122 ceph-mon[83368]: pgmap v558: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:08.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:08 smithi103 ceph-mon[115211]: pgmap v558: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:10.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:10 smithi122 ceph-mon[83368]: pgmap v559: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:10.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:10 smithi103 ceph-mon[115211]: pgmap v559: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:12.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:12 smithi122 ceph-mon[83368]: pgmap v560: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:12.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:12 smithi103 ceph-mon[115211]: pgmap v560: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:14.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:14 smithi122 ceph-mon[83368]: pgmap v561: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:14.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:14 smithi103 ceph-mon[115211]: pgmap v561: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:16.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:16 smithi122 ceph-mon[83368]: pgmap v562: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:16.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:16 smithi103 ceph-mon[115211]: pgmap v562: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:18.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:18 smithi122 ceph-mon[83368]: pgmap v563: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:18.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:18 smithi103 ceph-mon[115211]: pgmap v563: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:20.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:20 smithi122 ceph-mon[83368]: pgmap v564: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:20 smithi103 ceph-mon[115211]: pgmap v564: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:22.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:22 smithi122 ceph-mon[83368]: pgmap v565: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:22.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:22 smithi103 ceph-mon[115211]: pgmap v565: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 -- 172.21.15.103:0/1985351031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 msgr2=0x7f65f00fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 --2- 172.21.15.103:0/1985351031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f00fe1f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f65e0004940 tx=0x7f65e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 -- 172.21.15.103:0/1985351031 shutdown_connections 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 --2- 172.21.15.103:0/1985351031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 --2- 172.21.15.103:0/1985351031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f00fe1f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 -- 172.21.15.103:0/1985351031 >> 172.21.15.103:0/1985351031 conn(0x7f65f00f7870 msgr2=0x7f65f00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 -- 172.21.15.103:0/1985351031 shutdown_connections 2024-03-22T02:50:23.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.955+0000 7f65f5afc700 1 -- 172.21.15.103:0/1985351031 wait complete. 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 Processor -- start 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 -- start start 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01144a0 con 0x7f65f00fde20 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65f5afc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f010daa0 con 0x7f65f00fe7c0 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38124/0 (socket says 172.21.15.103:38124) 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 -- 172.21.15.103:0/2013879578 learned_addr learned my addr 172.21.15.103:0/2013879578 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 -- 172.21.15.103:0/2013879578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 msgr2=0x7f65f0113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65eeffd700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0113e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65ef7fe700 1 -- 172.21.15.103:0/2013879578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0005040 con 0x7f65f00fde20 2024-03-22T02:50:23.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.956+0000 7f65eeffd700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65ef7fe700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009180 tx=0x7f65e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e000e040 con 0x7f65f00fde20 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f010dd20 con 0x7f65f00fde20 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65e0013420 con 0x7f65f00fde20 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65e001c630 con 0x7f65f00fde20 2024-03-22T02:50:23.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.957+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f010e260 con 0x7f65f00fde20 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.958+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f0066b70 con 0x7f65f00fde20 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.958+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f65e0013590 con 0x7f65f00fde20 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.959+0000 7f65d7fff700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 0x7f65d8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.959+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f65e008f190 con 0x7f65f00fde20 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.959+0000 7f65eeffd700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 0x7f65d8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:23.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.959+0000 7f65eeffd700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 0x7f65d8070850 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009b80 tx=0x7f65e400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:23.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:23.964+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65e005bf40 con 0x7f65f00fde20 2024-03-22T02:50:24.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.182+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f65f0109eb0 con 0x7f65d806e3a0 2024-03-22T02:50:24.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.183+0000 7f65d7fff700 1 -- 172.21.15.103:0/2013879578 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f65f0109eb0 con 0x7f65d806e3a0 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 msgr2=0x7f65d8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 0x7f65d8070850 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009b80 tx=0x7f65e400b040 comp rx=0 tx=0).stop 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 msgr2=0x7f65f0113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009180 tx=0x7f65e0009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 shutdown_connections 2024-03-22T02:50:24.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65d806e3a0 0x7f65d8070850 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fde20 0x7f65f0113910 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 --2- 172.21.15.103:0/2013879578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fe7c0 0x7f65f0113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.185+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 >> 172.21.15.103:0/2013879578 conn(0x7f65f00f7870 msgr2=0x7f65f00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.186+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 shutdown_connections 2024-03-22T02:50:24.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.186+0000 7f65f5afc700 1 -- 172.21.15.103:0/2013879578 wait complete. 2024-03-22T02:50:24.196 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- 172.21.15.103:0/200063096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e4101d20 msgr2=0x7f15e41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 --2- 172.21.15.103:0/200063096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e4101d20 0x7f15e41020f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f15d4004940 tx=0x7f15d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- 172.21.15.103:0/200063096 shutdown_connections 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 --2- 172.21.15.103:0/200063096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e41026c0 0x7f15e40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 --2- 172.21.15.103:0/200063096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e4101d20 0x7f15e41020f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- 172.21.15.103:0/200063096 >> 172.21.15.103:0/200063096 conn(0x7f15e40f7870 msgr2=0x7f15e40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- 172.21.15.103:0/200063096 shutdown_connections 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- 172.21.15.103:0/200063096 wait complete. 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 Processor -- start 2024-03-22T02:50:24.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.288+0000 7f15eae08700 1 -- start start 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15eae08700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e4101d20 0x7f15e4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15eae08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15eae08700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4104ad0 con 0x7f15e41026c0 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15eae08700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4103600 con 0x7f15e4101d20 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38142/0 (socket says 172.21.15.103:38142) 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 -- 172.21.15.103:0/4104651440 learned_addr learned my addr 172.21.15.103:0/4104651440 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e8ba4700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e4101d20 0x7f15e4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 -- 172.21.15.103:0/4104651440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e4101d20 msgr2=0x7f15e4102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e4101d20 0x7f15e4102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 -- 172.21.15.103:0/4104651440 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d4005040 con 0x7f15e41026c0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.289+0000 7f15e3fff700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c800 tx=0x7f15d8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.290+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d800a760 con 0x7f15e41026c0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.290+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e41038e0 con 0x7f15e41026c0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.290+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15d800ad90 con 0x7f15e41026c0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.290+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d80113a0 con 0x7f15e41026c0 2024-03-22T02:50:24.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.290+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e411a180 con 0x7f15e41026c0 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.291+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e40fd470 con 0x7f15e41026c0 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.291+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f15d8012400 con 0x7f15e41026c0 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.292+0000 7f15e17fa700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 0x7f15cc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.292+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f15d808dcd0 con 0x7f15e41026c0 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.292+0000 7f15e8ba4700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 0x7f15cc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.292+0000 7f15e8ba4700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 0x7f15cc070850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f15d4004f90 tx=0x7f15d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.296+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15d8057010 con 0x7f15e41026c0 2024-03-22T02:50:24.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.510+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e40f9640 con 0x7f15cc06e3a0 2024-03-22T02:50:24.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.511+0000 7f15e17fa700 1 -- 172.21.15.103:0/4104651440 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f15e40f9640 con 0x7f15cc06e3a0 2024-03-22T02:50:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.513+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 msgr2=0x7f15cc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.513+0000 7f15eae08700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 0x7f15cc070850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f15d4004f90 tx=0x7f15d4009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.513+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 msgr2=0x7f15e41030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.513+0000 7f15eae08700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c800 tx=0x7f15d8004470 comp rx=0 tx=0).stop 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 shutdown_connections 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f15cc06e3a0 0x7f15cc070850 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e4101d20 0x7f15e4102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 --2- 172.21.15.103:0/4104651440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e41026c0 0x7f15e41030c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 >> 172.21.15.103:0/4104651440 conn(0x7f15e40f7870 msgr2=0x7f15e40f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 shutdown_connections 2024-03-22T02:50:24.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.514+0000 7f15eae08700 1 -- 172.21.15.103:0/4104651440 wait complete. 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 -- 172.21.15.103:0/1587166794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58102710 msgr2=0x7f8c580fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 --2- 172.21.15.103:0/1587166794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58102710 0x7f8c580fbd80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8c48004940 tx=0x7f8c48009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 -- 172.21.15.103:0/1587166794 shutdown_connections 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 --2- 172.21.15.103:0/1587166794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58102710 0x7f8c580fbd80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 --2- 172.21.15.103:0/1587166794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58101d70 0x7f8c58102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 -- 172.21.15.103:0/1587166794 >> 172.21.15.103:0/1587166794 conn(0x7f8c58072bc0 msgr2=0x7f8c58072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 -- 172.21.15.103:0/1587166794 shutdown_connections 2024-03-22T02:50:24.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.615+0000 7f8c5e756700 1 -- 172.21.15.103:0/1587166794 wait complete. 2024-03-22T02:50:24.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 Processor -- start 2024-03-22T02:50:24.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 -- start start 2024-03-22T02:50:24.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58102710 0x7f8c58112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58113610 con 0x7f8c58101d70 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c5e756700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5810ca00 con 0x7f8c58102710 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38170/0 (socket says 172.21.15.103:38170) 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 -- 172.21.15.103:0/4198558286 learned_addr learned my addr 172.21.15.103:0/4198558286 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c577fe700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58102710 0x7f8c58112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 -- 172.21.15.103:0/4198558286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58102710 msgr2=0x7f8c58112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58102710 0x7f8c58112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.616+0000 7f8c57fff700 1 -- 172.21.15.103:0/4198558286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48005040 con 0x7f8c58101d70 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c57fff700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c810 tx=0x7f8c4c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4c012810 con 0x7f8c58101d70 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c4c012e40 con 0x7f8c58101d70 2024-03-22T02:50:24.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5810cce0 con 0x7f8c58101d70 2024-03-22T02:50:24.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4c00a370 con 0x7f8c58101d70 2024-03-22T02:50:24.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.617+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5810d2a0 con 0x7f8c58101d70 2024-03-22T02:50:24.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.618+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58104240 con 0x7f8c58101d70 2024-03-22T02:50:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.618+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8c4c011810 con 0x7f8c58101d70 2024-03-22T02:50:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.619+0000 7f8c54ff9700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 0x7f8c40070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.619+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c4c08e260 con 0x7f8c58101d70 2024-03-22T02:50:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.619+0000 7f8c577fe700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 0x7f8c40070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.619+0000 7f8c577fe700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 0x7f8c40070850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c5810dbc0 tx=0x7f8c48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.623+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c4c05af90 con 0x7f8c58101d70 2024-03-22T02:50:24.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.828+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c5810e0d0 con 0x7f8c4006e3a0 2024-03-22T02:50:24.829 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:24 smithi103 ceph-mon[115211]: pgmap v566: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:24.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.839+0000 7f8c54ff9700 1 -- 172.21.15.103:0/4198558286 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8c5810e0d0 con 0x7f8c4006e3a0 2024-03-22T02:50:24.839 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (18m) 12m ago 30m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (16m) 12m ago 30m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (16m) 12m ago 29m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (18m) 12m ago 29m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (13m) 12m ago 24m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (12m) 12m ago 24m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (12m) 12m ago 24m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (12m) 12m ago 24m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (20m) 12m ago 31m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (19m) 12m ago 29m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (17m) 12m ago 31m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (17m) 12m ago 29m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (19m) 12m ago 30m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (19m) 12m ago 29m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:50:24.840 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (16m) 12m ago 28m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (15m) 12m ago 27m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (15m) 12m ago 27m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (14m) 12m ago 27m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (13m) 12m ago 26m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (13m) 12m ago 26m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:50:24.841 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (19m) 12m ago 29m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 msgr2=0x7f8c40070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 0x7f8c40070850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c5810dbc0 tx=0x7f8c48009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 msgr2=0x7f8c58112a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c810 tx=0x7f8c4c004470 comp rx=0 tx=0).stop 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 shutdown_connections 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c4006e3a0 0x7f8c40070850 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c58101d70 0x7f8c58112a80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 --2- 172.21.15.103:0/4198558286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c58102710 0x7f8c58112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 >> 172.21.15.103:0/4198558286 conn(0x7f8c58072bc0 msgr2=0x7f8c580fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 shutdown_connections 2024-03-22T02:50:24.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.841+0000 7f8c5e756700 1 -- 172.21.15.103:0/4198558286 wait complete. 2024-03-22T02:50:24.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:24 smithi122 ceph-mon[83368]: pgmap v566: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 -- 172.21.15.103:0/4026541764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 msgr2=0x7f7b60070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/4026541764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60070d10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7b50004970 tx=0x7f7b50009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 -- 172.21.15.103:0/4026541764 shutdown_connections 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/4026541764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60070d10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/4026541764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b60070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.940+0000 7f7b67cf8700 1 -- 172.21.15.103:0/4026541764 >> 172.21.15.103:0/4026541764 conn(0x7f7b600f89b0 msgr2=0x7f7b600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 -- 172.21.15.103:0/4026541764 shutdown_connections 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 -- 172.21.15.103:0/4026541764 wait complete. 2024-03-22T02:50:24.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 Processor -- start 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 -- start start 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b6011a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6011ac00 con 0x7f7b600708a0 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b67cf8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60114150 con 0x7f7b60071f90 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b65a94700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b65a94700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38184/0 (socket says 172.21.15.103:38184) 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.941+0000 7f7b65a94700 1 -- 172.21.15.103:0/3451691060 learned_addr learned my addr 172.21.15.103:0/3451691060 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65293700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b6011a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65a94700 1 -- 172.21.15.103:0/3451691060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 msgr2=0x7f7b6011a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65a94700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b6011a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65a94700 1 -- 172.21.15.103:0/3451691060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50005040 con 0x7f7b600708a0 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65293700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b6011a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:50:24.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b65a94700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00b790 tx=0x7f7b5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c00f770 con 0x7f7b600708a0 2024-03-22T02:50:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b601143d0 con 0x7f7b600708a0 2024-03-22T02:50:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b5c00fda0 con 0x7f7b600708a0 2024-03-22T02:50:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c00d310 con 0x7f7b600708a0 2024-03-22T02:50:24.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.942+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b60114910 con 0x7f7b600708a0 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.943+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b60071aa0 con 0x7f7b600708a0 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.944+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7b5c00d4f0 con 0x7f7b600708a0 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.944+0000 7f7b567fc700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 0x7f7b4c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.944+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b5c08de40 con 0x7f7b600708a0 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.944+0000 7f7b65293700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 0x7f7b4c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:24.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.944+0000 7f7b65293700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 0x7f7b4c070850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000b370 tx=0x7f7b50009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:24.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:24.949+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b5c05abf0 con 0x7f7b600708a0 2024-03-22T02:50:25.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.228+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b601156c0 con 0x7f7b600708a0 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.229+0000 7f7b567fc700 1 -- 172.21.15.103:0/3451691060 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7b5c05a780 con 0x7f7b600708a0 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:50:25.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:50:25.231 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 msgr2=0x7f7b4c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 0x7f7b4c070850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000b370 tx=0x7f7b50009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 msgr2=0x7f7b60119fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00b790 tx=0x7f7b5c009250 comp rx=0 tx=0).stop 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 shutdown_connections 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b4c06e3a0 0x7f7b4c070850 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b600708a0 0x7f7b60119fe0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 --2- 172.21.15.103:0/3451691060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b60071f90 0x7f7b6011a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 >> 172.21.15.103:0/3451691060 conn(0x7f7b600f89b0 msgr2=0x7f7b600f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 shutdown_connections 2024-03-22T02:50:25.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.231+0000 7f7b67cf8700 1 -- 172.21.15.103:0/3451691060 wait complete. 2024-03-22T02:50:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 -- 172.21.15.103:0/689847943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd10101d20 msgr2=0x7efd101020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 --2- 172.21.15.103:0/689847943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd10101d20 0x7efd101020f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efd00004940 tx=0x7efd00009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 -- 172.21.15.103:0/689847943 shutdown_connections 2024-03-22T02:50:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 --2- 172.21.15.103:0/689847943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd101026c0 0x7efd100fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 --2- 172.21.15.103:0/689847943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd10101d20 0x7efd101020f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 -- 172.21.15.103:0/689847943 >> 172.21.15.103:0/689847943 conn(0x7efd100f7870 msgr2=0x7efd100f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 -- 172.21.15.103:0/689847943 shutdown_connections 2024-03-22T02:50:25.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.332+0000 7efd169ad700 1 -- 172.21.15.103:0/689847943 wait complete. 2024-03-22T02:50:25.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 Processor -- start 2024-03-22T02:50:25.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 -- start start 2024-03-22T02:50:25.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd10101d20 0x7efd1010ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1010f550 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.333+0000 7efd169ad700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1010f690 con 0x7efd10101d20 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38192/0 (socket says 172.21.15.103:38192) 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 -- 172.21.15.103:0/2289567769 learned_addr learned my addr 172.21.15.103:0/2289567769 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0ffff700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd10101d20 0x7efd1010ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 -- 172.21.15.103:0/2289567769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd10101d20 msgr2=0x7efd1010ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd10101d20 0x7efd1010ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 -- 172.21.15.103:0/2289567769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00005040 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0f7fe700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c800 tx=0x7efd04004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd040106f0 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd04010d20 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10117b40 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd040113a0 con 0x7efd101026c0 2024-03-22T02:50:25.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.334+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd10117f60 con 0x7efd101026c0 2024-03-22T02:50:25.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.335+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd100fd470 con 0x7efd101026c0 2024-03-22T02:50:25.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.336+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efd04012400 con 0x7efd101026c0 2024-03-22T02:50:25.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.336+0000 7efd0cff9700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 0x7efcf8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.336+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd0408dd00 con 0x7efd101026c0 2024-03-22T02:50:25.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.336+0000 7efd0ffff700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 0x7efcf8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.336+0000 7efd0ffff700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 0x7efcf8070850 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd00004f90 tx=0x7efd00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:25.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.341+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd04056f80 con 0x7efd101026c0 2024-03-22T02:50:25.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.589+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efd10108e20 con 0x7efd101026c0 2024-03-22T02:50:25.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.589+0000 7efd0cff9700 1 -- 172.21.15.103:0/2289567769 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7efd0405a640 con 0x7efd101026c0 2024-03-22T02:50:25.590 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:50:25.590 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:50:25.591 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:50:25.592 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:25.593 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.592+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 msgr2=0x7efcf8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.592+0000 7efd169ad700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 0x7efcf8070850 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd00004f90 tx=0x7efd00009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.592+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 msgr2=0x7efd1010f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.592+0000 7efd169ad700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c800 tx=0x7efd04004470 comp rx=0 tx=0).stop 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 shutdown_connections 2024-03-22T02:50:25.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efcf806e3a0 0x7efcf8070850 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd10101d20 0x7efd1010ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 --2- 172.21.15.103:0/2289567769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd101026c0 0x7efd1010f010 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 >> 172.21.15.103:0/2289567769 conn(0x7efd100f7870 msgr2=0x7efd100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 shutdown_connections 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.593+0000 7efd169ad700 1 -- 172.21.15.103:0/2289567769 wait complete. 2024-03-22T02:50:25.595 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 -- 172.21.15.103:0/1384182379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4102710 msgr2=0x7fc3b40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 --2- 172.21.15.103:0/1384182379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4102710 0x7fc3b40fbd80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8004940 tx=0x7fc3a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 -- 172.21.15.103:0/1384182379 shutdown_connections 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 --2- 172.21.15.103:0/1384182379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4102710 0x7fc3b40fbd80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 --2- 172.21.15.103:0/1384182379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4101d70 0x7fc3b4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 -- 172.21.15.103:0/1384182379 >> 172.21.15.103:0/1384182379 conn(0x7fc3b4072bc0 msgr2=0x7fc3b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 -- 172.21.15.103:0/1384182379 shutdown_connections 2024-03-22T02:50:25.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 -- 172.21.15.103:0/1384182379 wait complete. 2024-03-22T02:50:25.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.694+0000 7fc3ba840700 1 Processor -- start 2024-03-22T02:50:25.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3ba840700 1 -- start start 2024-03-22T02:50:25.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3ba840700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3ba840700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 0x7fc3b4112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3ba840700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b4113630 con 0x7fc3b4101d70 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3ba840700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b410ca20 con 0x7fc3b4102710 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38202/0 (socket says 172.21.15.103:38202) 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 -- 172.21.15.103:0/272521610 learned_addr learned my addr 172.21.15.103:0/272521610 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b37fe700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 0x7fc3b4112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 -- 172.21.15.103:0/272521610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 msgr2=0x7fc3b4112fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 0x7fc3b4112fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 -- 172.21.15.103:0/272521610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8005040 con 0x7fc3b4101d70 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b37fe700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 0x7fc3b4112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.695+0000 7fc3b3fff700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4002a50 tx=0x7fc3a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.696+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a4007cc0 con 0x7fc3b4101d70 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.696+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b410cd00 con 0x7fc3b4101d70 2024-03-22T02:50:25.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.696+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3a400f040 con 0x7fc3b4101d70 2024-03-22T02:50:25.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.696+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3a4009640 con 0x7fc3b4101d70 2024-03-22T02:50:25.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.696+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b410d2c0 con 0x7fc3b4101d70 2024-03-22T02:50:25.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.697+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b4104240 con 0x7fc3b4101d70 2024-03-22T02:50:25.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.697+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc3a40097a0 con 0x7fc3b4101d70 2024-03-22T02:50:25.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.698+0000 7fc3b0ff9700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 0x7fc39c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:25.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.698+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc3a408dbd0 con 0x7fc3b4101d70 2024-03-22T02:50:25.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.698+0000 7fc3b37fe700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 0x7fc39c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:25.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.698+0000 7fc3b37fe700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 0x7fc39c070850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800b3b0 tx=0x7fc3a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:25.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.702+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc3a405a900 con 0x7fc3b4101d70 2024-03-22T02:50:25.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:25 smithi122 ceph-mon[83368]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:25 smithi122 ceph-mon[83368]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:25 smithi122 ceph-mon[83368]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:25 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3451691060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:50:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:25 smithi103 ceph-mon[115211]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:25 smithi103 ceph-mon[115211]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:25 smithi103 ceph-mon[115211]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:25 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3451691060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:50:25.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.915+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3b410e160 con 0x7fc39c06e3a0 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.916+0000 7fc3b0ff9700 1 -- 172.21.15.103:0/272521610 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc3b410e160 con 0x7fc39c06e3a0 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:50:25.917 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:50:25.918 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:50:25.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.918+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 msgr2=0x7fc39c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.918+0000 7fc3ba840700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 0x7fc39c070850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800b3b0 tx=0x7fc3a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:25.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 msgr2=0x7fc3b4112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4002a50 tx=0x7fc3a40046b0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 shutdown_connections 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc39c06e3a0 0x7fc39c070850 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3b4101d70 0x7fc3b4112aa0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 --2- 172.21.15.103:0/272521610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3b4102710 0x7fc3b4112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 >> 172.21.15.103:0/272521610 conn(0x7fc3b4072bc0 msgr2=0x7fc3b40fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 shutdown_connections 2024-03-22T02:50:25.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:25.919+0000 7fc3ba840700 1 -- 172.21.15.103:0/272521610 wait complete. 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- 172.21.15.103:0/3503817343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d900fb570 msgr2=0x7f5d900fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 --2- 172.21.15.103:0/3503817343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d900fb570 0x7f5d900fb9e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5d80004940 tx=0x7f5d80009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- 172.21.15.103:0/3503817343 shutdown_connections 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 --2- 172.21.15.103:0/3503817343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d900fb570 0x7f5d900fb9e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 --2- 172.21.15.103:0/3503817343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d90101530 0x7f5d90101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- 172.21.15.103:0/3503817343 >> 172.21.15.103:0/3503817343 conn(0x7f5d900714c0 msgr2=0x7f5d900718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- 172.21.15.103:0/3503817343 shutdown_connections 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- 172.21.15.103:0/3503817343 wait complete. 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 Processor -- start 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- start start 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d900fb570 0x7f5d90113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90114510 con 0x7f5d90101530 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.020+0000 7f5d98384700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9010db10 con 0x7f5d900fb570 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38220/0 (socket says 172.21.15.103:38220) 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 -- 172.21.15.103:0/3656479941 learned_addr learned my addr 172.21.15.103:0/3656479941 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d96120700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d900fb570 0x7f5d90113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 -- 172.21.15.103:0/3656479941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d900fb570 msgr2=0x7f5d90113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:26.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d900fb570 0x7f5d90113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 -- 172.21.15.103:0/3656479941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80005040 con 0x7f5d90101530 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d9591f700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009150 tx=0x7f5d80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8000e040 con 0x7f5d90101530 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d80013420 con 0x7f5d90101530 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9010dd90 con 0x7f5d90101530 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d8001c630 con 0x7f5d90101530 2024-03-22T02:50:26.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.021+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9010e350 con 0x7f5d90101530 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.022+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d901059f0 con 0x7f5d90101530 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.023+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5d80013590 con 0x7f5d90101530 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.023+0000 7f5d86ffd700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 0x7f5d7c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.023+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5d8008ef30 con 0x7f5d90101530 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.023+0000 7f5d96120700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 0x7f5d7c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:26.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.024+0000 7f5d96120700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 0x7f5d7c070850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c007fd0 tx=0x7f5d8c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:26.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.028+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d800c3740 con 0x7f5d90101530 2024-03-22T02:50:26.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.306+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5d9004c740 con 0x7f5d90101530 2024-03-22T02:50:26.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.306+0000 7f5d86ffd700 1 -- 172.21.15.103:0/3656479941 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5d8005bda0 con 0x7f5d90101530 2024-03-22T02:50:26.307 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:50:26.307 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:50:26.307 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:50:26.308 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:50:26.308 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:50:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.308+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 msgr2=0x7f5d7c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.308+0000 7f5d98384700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 0x7f5d7c070850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c007fd0 tx=0x7f5d8c009040 comp rx=0 tx=0).stop 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.308+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 msgr2=0x7f5d90113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.308+0000 7f5d98384700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009150 tx=0x7f5d80009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 shutdown_connections 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f5d7c06e3a0 0x7f5d7c070850 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d900fb570 0x7f5d90113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 --2- 172.21.15.103:0/3656479941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d90101530 0x7f5d90113ec0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 >> 172.21.15.103:0/3656479941 conn(0x7f5d900714c0 msgr2=0x7f5d900fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 shutdown_connections 2024-03-22T02:50:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:26.309+0000 7f5d98384700 1 -- 172.21.15.103:0/3656479941 wait complete. 2024-03-22T02:50:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:26 smithi122 ceph-mon[83368]: pgmap v567: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:26 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2289567769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:50:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:26 smithi122 ceph-mon[83368]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:26 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3656479941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:50:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:26 smithi103 ceph-mon[115211]: pgmap v567: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:26 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2289567769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:50:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:26 smithi103 ceph-mon[115211]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:26 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3656479941' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:50:28.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:28 smithi122 ceph-mon[83368]: pgmap v568: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:28.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:28 smithi103 ceph-mon[115211]: pgmap v568: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:30.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:30 smithi122 ceph-mon[83368]: pgmap v569: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:30.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:30 smithi103 ceph-mon[115211]: pgmap v569: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:32 smithi122 ceph-mon[83368]: pgmap v570: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:32 smithi103 ceph-mon[115211]: pgmap v570: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:34.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:34 smithi122 ceph-mon[83368]: pgmap v571: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:34.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:34 smithi103 ceph-mon[115211]: pgmap v571: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:36.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:36 smithi122 ceph-mon[83368]: pgmap v572: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:36.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:36 smithi103 ceph-mon[115211]: pgmap v572: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:38.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:38 smithi122 ceph-mon[83368]: pgmap v573: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:38.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:38 smithi103 ceph-mon[115211]: pgmap v573: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:40.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:40 smithi122 ceph-mon[83368]: pgmap v574: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:40 smithi103 ceph-mon[115211]: pgmap v574: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:42.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:42 smithi122 ceph-mon[83368]: pgmap v575: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:42.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:42 smithi103 ceph-mon[115211]: pgmap v575: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:44 smithi122 ceph-mon[83368]: pgmap v576: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:44 smithi103 ceph-mon[115211]: pgmap v576: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:46.926 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:46 smithi103 ceph-mon[115211]: pgmap v577: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:47.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:46 smithi122 ceph-mon[83368]: pgmap v577: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:48.126 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:50:49.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:48 smithi122 ceph-mon[83368]: pgmap v578: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:48 smithi103 ceph-mon[115211]: pgmap v578: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:50 smithi103 ceph-mon[115211]: pgmap v579: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:50 smithi122 ceph-mon[83368]: pgmap v579: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:53.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:52 smithi122 ceph-mon[83368]: pgmap v580: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:52 smithi103 ceph-mon[115211]: pgmap v580: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:55.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:54 smithi122 ceph-mon[83368]: pgmap v581: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:55.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:54 smithi103 ceph-mon[115211]: pgmap v581: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:56.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1759950050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b81037b0 msgr2=0x7f21b80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1759950050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b81037b0 0x7f21b80fce20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f21a0004940 tx=0x7f21a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:56.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1759950050 shutdown_connections 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1759950050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b81037b0 0x7f21b80fce20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1759950050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b8102e10 0x7f21b81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1759950050 >> 172.21.15.103:0/1759950050 conn(0x7f21b80f8980 msgr2=0x7f21b80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1759950050 shutdown_connections 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.422+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1759950050 wait complete. 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 Processor -- start 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 -- start start 2024-03-22T02:50:56.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b81037b0 0x7f21b81140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8114710 con 0x7f21b8102e10 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21bcbd5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b810db00 con 0x7f21b81037b0 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60776/0 (socket says 172.21.15.103:60776) 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 -- 172.21.15.103:0/1421400206 learned_addr learned my addr 172.21.15.103:0/1421400206 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21add9b700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b81037b0 0x7f21b81140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 -- 172.21.15.103:0/1421400206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b81037b0 msgr2=0x7f21b81140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b81037b0 0x7f21b81140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.423+0000 7f21b659c700 1 -- 172.21.15.103:0/1421400206 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0005040 con 0x7f21b8102e10 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21b659c700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f21a800c8c0 tx=0x7f21a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a800a760 con 0x7f21b8102e10 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21a800ad90 con 0x7f21b8102e10 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b810dde0 con 0x7f21b8102e10 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a80113a0 con 0x7f21b8102e10 2024-03-22T02:50:56.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.424+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b810e370 con 0x7f21b8102e10 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.425+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b81052e0 con 0x7f21b8102e10 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.425+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f21a8012400 con 0x7f21b8102e10 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.426+0000 7f21af7fe700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 0x7f21a4070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.426+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f21a808dde0 con 0x7f21b8102e10 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.426+0000 7f21add9b700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 0x7f21a4070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.426+0000 7f21add9b700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 0x7f21a4070850 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f21b810ede0 tx=0x7f21a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:56.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.431+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21a805ab90 con 0x7f21b8102e10 2024-03-22T02:50:56.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.644+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21b810f1d0 con 0x7f21a406e3a0 2024-03-22T02:50:56.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.645+0000 7f21af7fe700 1 -- 172.21.15.103:0/1421400206 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f21b810f1d0 con 0x7f21a406e3a0 2024-03-22T02:50:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 msgr2=0x7f21a4070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 0x7f21a4070850 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f21b810ede0 tx=0x7f21a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 msgr2=0x7f21b8113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f21a800c8c0 tx=0x7f21a80044c0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 shutdown_connections 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f21a406e3a0 0x7f21a4070850 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21b8102e10 0x7f21b8113b80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 --2- 172.21.15.103:0/1421400206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21b81037b0 0x7f21b81140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 >> 172.21.15.103:0/1421400206 conn(0x7f21b80f8980 msgr2=0x7f21b81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 shutdown_connections 2024-03-22T02:50:56.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.647+0000 7f21bcbd5700 1 -- 172.21.15.103:0/1421400206 wait complete. 2024-03-22T02:50:56.661 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:50:56.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3559245987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 msgr2=0x7f3f400706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3559245987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f400706d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3f30004970 tx=0x7f3f30009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:56.753 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:56 smithi103 ceph-mon[115211]: pgmap v582: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3559245987 shutdown_connections 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3559245987 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 0x7f3f40109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3559245987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f400706d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3559245987 >> 172.21.15.103:0/3559245987 conn(0x7f3f400f89b0 msgr2=0x7f3f400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3559245987 shutdown_connections 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3559245987 wait complete. 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.751+0000 7f3f45ebc700 1 Processor -- start 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f45ebc700 1 -- start start 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f45ebc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f45ebc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 0x7f3f40116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f45ebc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401166d0 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f45ebc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4010fcb0 con 0x7f3f40070c10 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60790/0 (socket says 172.21.15.103:60790) 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 -- 172.21.15.103:0/3150048204 learned_addr learned my addr 172.21.15.103:0/3150048204 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f36ffd700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 0x7f3f40116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 -- 172.21.15.103:0/3150048204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 msgr2=0x7f3f40116080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 0x7f3f40116080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.752+0000 7f3f3f7fe700 1 -- 172.21.15.103:0/3150048204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30005040 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f3f7fe700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3000e040 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4010ff30 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f3000bc10 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3001c3f0 con 0x7f3f40070300 2024-03-22T02:50:56.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.753+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f40110470 con 0x7f3f40070300 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.754+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f40071a40 con 0x7f3f40070300 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.754+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f3f30013420 con 0x7f3f40070300 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.755+0000 7f3f3cff9700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 0x7f3f2c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.755+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f3008efd0 con 0x7f3f40070300 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.755+0000 7f3f36ffd700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 0x7f3f2c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:56.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.755+0000 7f3f36ffd700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 0x7f3f2c070850 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f401113e0 tx=0x7f3f2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:56.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.759+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f3005bd80 con 0x7f3f40070300 2024-03-22T02:50:56.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.976+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f40111200 con 0x7f3f2c06e3a0 2024-03-22T02:50:56.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.977+0000 7f3f3cff9700 1 -- 172.21.15.103:0/3150048204 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3f40111200 con 0x7f3f2c06e3a0 2024-03-22T02:50:56.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 msgr2=0x7f3f2c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 0x7f3f2c070850 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f401113e0 tx=0x7f3f2800b040 comp rx=0 tx=0).stop 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 msgr2=0x7f3f40115b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f30009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 shutdown_connections 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f3f2c06e3a0 0x7f3f2c070850 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40070300 0x7f3f40115b40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 --2- 172.21.15.103:0/3150048204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f40070c10 0x7f3f40116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 >> 172.21.15.103:0/3150048204 conn(0x7f3f400f89b0 msgr2=0x7f3f400f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 shutdown_connections 2024-03-22T02:50:56.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:56.980+0000 7f3f45ebc700 1 -- 172.21.15.103:0/3150048204 wait complete. 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 -- 172.21.15.103:0/3818364038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 msgr2=0x7f89100fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 --2- 172.21.15.103:0/3818364038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f89100fbd80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8904004940 tx=0x7f8904009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 -- 172.21.15.103:0/3818364038 shutdown_connections 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 --2- 172.21.15.103:0/3818364038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f89100fbd80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 --2- 172.21.15.103:0/3818364038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 -- 172.21.15.103:0/3818364038 >> 172.21.15.103:0/3818364038 conn(0x7f8910072bc0 msgr2=0x7f8910072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 -- 172.21.15.103:0/3818364038 shutdown_connections 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.080+0000 7f8915246700 1 -- 172.21.15.103:0/3818364038 wait complete. 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 Processor -- start 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 -- start start 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910112a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910113610 con 0x7f8910102710 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f8915246700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010ca00 con 0x7f8910101d70 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60802/0 (socket says 172.21.15.103:60802) 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 -- 172.21.15.103:0/2364879740 learned_addr learned my addr 172.21.15.103:0/2364879740 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890ed9d700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910112a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 -- 172.21.15.103:0/2364879740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 msgr2=0x7f8910112a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910112a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890e59c700 1 -- 172.21.15.103:0/2364879740 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8904005040 con 0x7f8910102710 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.081+0000 7f890ed9d700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910112a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f890e59c700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8904009100 tx=0x7f8904009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f890400e040 con 0x7f8910102710 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8904009af0 con 0x7f8910102710 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891010cc80 con 0x7f8910102710 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f890401c3f0 con 0x7f8910102710 2024-03-22T02:50:57.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.082+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891010d240 con 0x7f8910102710 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.083+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89100fd4c0 con 0x7f8910102710 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.083+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f890400baf0 con 0x7f8910102710 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.084+0000 7f88f77fe700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 0x7f88f8072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.084+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f890408eff0 con 0x7f8910102710 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.084+0000 7f890ed9d700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 0x7f88f8072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.084+0000 7f890ed9d700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 0x7f88f8072a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8900004fd0 tx=0x7f890000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.088+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f890405bda0 con 0x7f8910102710 2024-03-22T02:50:57.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:56 smithi122 ceph-mon[83368]: pgmap v582: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:57.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.297+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f891010e170 con 0x7f88f80705b0 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.307+0000 7f88f77fe700 1 -- 172.21.15.103:0/2364879740 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f891010e170 con 0x7f88f80705b0 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (19m) 13m ago 30m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (17m) 13m ago 30m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (17m) 12m ago 29m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (19m) 13m ago 30m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (13m) 13m ago 25m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (13m) 13m ago 25m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (12m) 12m ago 25m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (13m) 12m ago 25m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:50:57.309 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (21m) 13m ago 32m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (20m) 12m ago 29m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (18m) 13m ago 32m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (17m) 12m ago 29m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (20m) 13m ago 30m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (20m) 12m ago 29m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (16m) 13m ago 28m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (16m) 13m ago 28m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (15m) 13m ago 27m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (14m) 12m ago 27m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (14m) 12m ago 27m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (13m) 12m ago 26m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:50:57.310 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (19m) 13m ago 30m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:50:57.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 msgr2=0x7f88f8072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 0x7f88f8072a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8900004fd0 tx=0x7f890000b040 comp rx=0 tx=0).stop 2024-03-22T02:50:57.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 msgr2=0x7f8910112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8904009100 tx=0x7f8904009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 shutdown_connections 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88f80705b0 0x7f88f8072a60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8910101d70 0x7f8910112a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 --2- 172.21.15.103:0/2364879740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8910102710 0x7f8910112fc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 >> 172.21.15.103:0/2364879740 conn(0x7f8910072bc0 msgr2=0x7f89100fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 shutdown_connections 2024-03-22T02:50:57.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.309+0000 7f8915246700 1 -- 172.21.15.103:0/2364879740 wait complete. 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 -- 172.21.15.103:0/1400235088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 msgr2=0x7fd7580fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/1400235088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7580fe250 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd748004940 tx=0x7fd748009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 -- 172.21.15.103:0/1400235088 shutdown_connections 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/1400235088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/1400235088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7580fe250 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.409+0000 7fd75e8bf700 1 -- 172.21.15.103:0/1400235088 >> 172.21.15.103:0/1400235088 conn(0x7fd7580f7850 msgr2=0x7fd7580f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 -- 172.21.15.103:0/1400235088 shutdown_connections 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 -- 172.21.15.103:0/1400235088 wait complete. 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 Processor -- start 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 -- start start 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd758114480 con 0x7fd7580fde80 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.410+0000 7fd75e8bf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75810da80 con 0x7fd7580fe820 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60806/0 (socket says 172.21.15.103:60806) 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 -- 172.21.15.103:0/2707928532 learned_addr learned my addr 172.21.15.103:0/2707928532 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd7577fe700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 -- 172.21.15.103:0/2707928532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 msgr2=0x7fd758113e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758113e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 -- 172.21.15.103:0/2707928532 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748005040 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd7577fe700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd757fff700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd748009380 tx=0x7fd748009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd74800e040 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75810dd60 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75810e2a0 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.411+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd748009af0 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.412+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd74801c630 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.412+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd758066af0 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.413+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd748013420 con 0x7fd7580fde80 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.413+0000 7fd754ff9700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 0x7fd740070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.413+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd74808f110 con 0x7fd7580fde80 2024-03-22T02:50:57.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.413+0000 7fd7577fe700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 0x7fd740070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.413+0000 7fd7577fe700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 0x7fd740070640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009fd0 tx=0x7fd74c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.418+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd74805bec0 con 0x7fd7580fde80 2024-03-22T02:50:57.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.699+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd75810ef40 con 0x7fd7580fde80 2024-03-22T02:50:57.701 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:57 smithi103 ceph-mon[115211]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:57.701 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:57 smithi103 ceph-mon[115211]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.699+0000 7fd754ff9700 1 -- 172.21.15.103:0/2707928532 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd748004490 con 0x7fd7580fde80 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:50:57.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:50:57.703 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 msgr2=0x7fd740070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 0x7fd740070640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009fd0 tx=0x7fd74c00b040 comp rx=0 tx=0).stop 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 msgr2=0x7fd7581138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd748009380 tx=0x7fd748009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 shutdown_connections 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd74006e190 0x7fd740070640 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd7580fde80 0x7fd7581138f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 --2- 172.21.15.103:0/2707928532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7580fe820 0x7fd758113e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 >> 172.21.15.103:0/2707928532 conn(0x7fd7580f7850 msgr2=0x7fd7580f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:57.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 shutdown_connections 2024-03-22T02:50:57.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.702+0000 7fd75e8bf700 1 -- 172.21.15.103:0/2707928532 wait complete. 2024-03-22T02:50:57.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- 172.21.15.103:0/4024842041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 msgr2=0x7f83d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 --2- 172.21.15.103:0/4024842041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d41031d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f83c4004940 tx=0x7f83c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:57.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- 172.21.15.103:0/4024842041 shutdown_connections 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 --2- 172.21.15.103:0/4024842041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 0x7f83d40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 --2- 172.21.15.103:0/4024842041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d41031d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- 172.21.15.103:0/4024842041 >> 172.21.15.103:0/4024842041 conn(0x7f83d40f89b0 msgr2=0x7f83d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- 172.21.15.103:0/4024842041 shutdown_connections 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- 172.21.15.103:0/4024842041 wait complete. 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 Processor -- start 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.804+0000 7f83da677700 1 -- start start 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83da677700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83da677700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 0x7f83d4113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83da677700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d4114520 con 0x7f83d4102e00 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83da677700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d410db70 con 0x7f83d41037a0 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60822/0 (socket says 172.21.15.103:60822) 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 -- 172.21.15.103:0/180893245 learned_addr learned my addr 172.21.15.103:0/180893245 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:57.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d37fe700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 0x7f83d4113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 -- 172.21.15.103:0/180893245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 msgr2=0x7f83d4113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 0x7f83d4113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 -- 172.21.15.103:0/180893245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c4005040 con 0x7f83d4102e00 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d3fff700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b8b0 tx=0x7f83c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83c400e040 con 0x7f83d4102e00 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d410ddf0 con 0x7f83d4102e00 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c4004290 con 0x7f83d4102e00 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.805+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83c401c3f0 con 0x7f83d4102e00 2024-03-22T02:50:57.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.806+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d410e3b0 con 0x7f83d4102e00 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.807+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d4100110 con 0x7f83d4102e00 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.807+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f83c4013420 con 0x7f83d4102e00 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.808+0000 7f83d0ff9700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 0x7f83bc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.808+0000 7f83d37fe700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 0x7f83bc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.808+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f83c408f050 con 0x7f83d4102e00 2024-03-22T02:50:57.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.808+0000 7f83d37fe700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 0x7f83bc070850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f83d410ece0 tx=0x7f83c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:57.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:57.812+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f83c405be00 con 0x7f83d4102e00 2024-03-22T02:50:58.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.053+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83d410efe0 con 0x7f83d4102e00 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.054+0000 7f83d0ff9700 1 -- 172.21.15.103:0/180893245 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f83c405b990 con 0x7f83d4102e00 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:50:58.056 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:50:58.057 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:50:58.058 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:58.059 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:50:58.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:50:58.060 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:58.060 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:58.060 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:58.060 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:50:58.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 msgr2=0x7f83bc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.061 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 0x7f83bc070850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f83d410ece0 tx=0x7f83c800b040 comp rx=0 tx=0).stop 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 msgr2=0x7f83d4113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b8b0 tx=0x7f83c4009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 shutdown_connections 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f83bc06e3a0 0x7f83bc070850 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83d4102e00 0x7f83d4113990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 --2- 172.21.15.103:0/180893245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83d41037a0 0x7f83d4113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 >> 172.21.15.103:0/180893245 conn(0x7f83d40f89b0 msgr2=0x7f83d41073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:58.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 shutdown_connections 2024-03-22T02:50:58.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.056+0000 7f83da677700 1 -- 172.21.15.103:0/180893245 wait complete. 2024-03-22T02:50:58.063 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:50:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:57 smithi122 ceph-mon[83368]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:57 smithi122 ceph-mon[83368]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:58.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 -- 172.21.15.103:0/2014261122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f641037a0 msgr2=0x7f2f640fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 --2- 172.21.15.103:0/2014261122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f641037a0 0x7f2f640fce10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2f54004940 tx=0x7f2f54009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:58.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 -- 172.21.15.103:0/2014261122 shutdown_connections 2024-03-22T02:50:58.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 --2- 172.21.15.103:0/2014261122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f641037a0 0x7f2f640fce10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 --2- 172.21.15.103:0/2014261122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f64102e00 0x7f2f641031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 -- 172.21.15.103:0/2014261122 >> 172.21.15.103:0/2014261122 conn(0x7f2f640f89b0 msgr2=0x7f2f640fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:58.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 -- 172.21.15.103:0/2014261122 shutdown_connections 2024-03-22T02:50:58.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.157+0000 7f2f692d6700 1 -- 172.21.15.103:0/2014261122 wait complete. 2024-03-22T02:50:58.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 Processor -- start 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 -- start start 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f641037a0 0x7f2f64113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64114530 con 0x7f2f64102e00 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f692d6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6410db50 con 0x7f2f641037a0 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f62ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f62ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60848/0 (socket says 172.21.15.103:60848) 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.158+0000 7f2f62ffd700 1 -- 172.21.15.103:0/1684704950 learned_addr learned my addr 172.21.15.103:0/1684704950 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f627fc700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f641037a0 0x7f2f64113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f62ffd700 1 -- 172.21.15.103:0/1684704950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f641037a0 msgr2=0x7f2f64113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f62ffd700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f641037a0 0x7f2f64113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f62ffd700 1 -- 172.21.15.103:0/1684704950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f54005040 con 0x7f2f64102e00 2024-03-22T02:50:58.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f62ffd700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2f58002a50 tx=0x7f2f580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f58007cc0 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f5800f040 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6410de30 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5800da70 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.159+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6410e3f0 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.160+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f641052d0 con 0x7f2f64102e00 2024-03-22T02:50:58.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.161+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2f5800dcb0 con 0x7f2f64102e00 2024-03-22T02:50:58.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.161+0000 7f2f4b7fe700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 0x7f2f4c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.161+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f5808d430 con 0x7f2f64102e00 2024-03-22T02:50:58.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.161+0000 7f2f627fc700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 0x7f2f4c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.161+0000 7f2f627fc700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 0x7f2f4c070850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2f6410f240 tx=0x7f2f54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:58.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.166+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f58056630 con 0x7f2f64102e00 2024-03-22T02:50:58.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.381+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f64109f70 con 0x7f2f4c06e3a0 2024-03-22T02:50:58.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.382+0000 7f2f4b7fe700 1 -- 172.21.15.103:0/1684704950 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2f64109f70 con 0x7f2f4c06e3a0 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:50:58.384 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:50:58.385 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 msgr2=0x7f2f4c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 0x7f2f4c070850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2f6410f240 tx=0x7f2f54009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 msgr2=0x7f2f641139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2f58002a50 tx=0x7f2f580046b0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 shutdown_connections 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f4c06e3a0 0x7f2f4c070850 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f64102e00 0x7f2f641139a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 --2- 172.21.15.103:0/1684704950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f641037a0 0x7f2f64113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 >> 172.21.15.103:0/1684704950 conn(0x7f2f640f89b0 msgr2=0x7f2f641021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:58.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 shutdown_connections 2024-03-22T02:50:58.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.384+0000 7f2f692d6700 1 -- 172.21.15.103:0/1684704950 wait complete. 2024-03-22T02:50:58.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 -- 172.21.15.103:0/5036373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 msgr2=0x7f60b40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 --2- 172.21.15.103:0/5036373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b40fbd80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f60a8004940 tx=0x7f60a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 -- 172.21.15.103:0/5036373 shutdown_connections 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 --2- 172.21.15.103:0/5036373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b40fbd80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 --2- 172.21.15.103:0/5036373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 0x7f60b4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 -- 172.21.15.103:0/5036373 >> 172.21.15.103:0/5036373 conn(0x7f60b4072bc0 msgr2=0x7f60b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 -- 172.21.15.103:0/5036373 shutdown_connections 2024-03-22T02:50:58.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 -- 172.21.15.103:0/5036373 wait complete. 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.484+0000 7f60ba581700 1 Processor -- start 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60ba581700 1 -- start start 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60ba581700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 0x7f60b4113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60ba581700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60ba581700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4114630 con 0x7f60b4102710 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60ba581700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b410da60 con 0x7f60b4101d70 2024-03-22T02:50:58.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60880/0 (socket says 172.21.15.103:60880) 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 -- 172.21.15.103:0/1859615737 learned_addr learned my addr 172.21.15.103:0/1859615737 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b3fff700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 0x7f60b4113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 -- 172.21.15.103:0/1859615737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 msgr2=0x7f60b4113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 0x7f60b4113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 -- 172.21.15.103:0/1859615737 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a8005040 con 0x7f60b4102710 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b37fe700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f60a800b0d0 tx=0x7f60a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.485+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a8021070 con 0x7f60b4102710 2024-03-22T02:50:58.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.486+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60a8013c00 con 0x7f60b4102710 2024-03-22T02:50:58.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.486+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b410dce0 con 0x7f60b4102710 2024-03-22T02:50:58.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.486+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60a800e070 con 0x7f60b4102710 2024-03-22T02:50:58.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.486+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b410e240 con 0x7f60b4102710 2024-03-22T02:50:58.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.486+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b40fd4c0 con 0x7f60b4102710 2024-03-22T02:50:58.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.487+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f60a801c9d0 con 0x7f60b4102710 2024-03-22T02:50:58.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.487+0000 7f60b0ff9700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 0x7f609c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:50:58.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.487+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f60a808f110 con 0x7f60b4102710 2024-03-22T02:50:58.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.487+0000 7f60b3fff700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 0x7f609c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:50:58.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.488+0000 7f60b3fff700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 0x7f609c070850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60a4004600 tx=0x7f60a400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:50:58.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.492+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60a805bf80 con 0x7f60b4102710 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.785+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f60b4108e50 con 0x7f60b4102710 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.785+0000 7f60b0ff9700 1 -- 172.21.15.103:0/1859615737 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f60a805bb10 con 0x7f60b4102710 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:50:58.787 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:50:58.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 msgr2=0x7f609c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 0x7f609c070850 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f60a4004600 tx=0x7f60a400b040 comp rx=0 tx=0).stop 2024-03-22T02:50:58.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 msgr2=0x7f60b4113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f60a800b0d0 tx=0x7f60a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 shutdown_connections 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f609c06e3a0 0x7f609c070850 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f60b4101d70 0x7f60b4113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 --2- 172.21.15.103:0/1859615737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60b4102710 0x7f60b4113fe0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 >> 172.21.15.103:0/1859615737 conn(0x7f60b4072bc0 msgr2=0x7f60b4065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.787+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 shutdown_connections 2024-03-22T02:50:58.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:50:58.788+0000 7f60ba581700 1 -- 172.21.15.103:0/1859615737 wait complete. 2024-03-22T02:50:59.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:58 smithi122 ceph-mon[83368]: pgmap v583: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:58 smithi122 ceph-mon[83368]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:58 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2707928532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:50:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:58 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/180893245' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:50:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:58 smithi103 ceph-mon[115211]: pgmap v583: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:50:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:58 smithi103 ceph-mon[115211]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:50:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:58 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2707928532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:50:59.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:58 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/180893245' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:51:00.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:59 smithi122 ceph-mon[83368]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:00.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:50:59 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1859615737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:51:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:59 smithi103 ceph-mon[115211]: from='client.35116 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:50:59 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1859615737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:51:00.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:00 smithi103 ceph-mon[115211]: pgmap v584: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:00 smithi122 ceph-mon[83368]: pgmap v584: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:02 smithi122 ceph-mon[83368]: pgmap v585: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:02 smithi103 ceph-mon[115211]: pgmap v585: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:04 smithi122 ceph-mon[83368]: pgmap v586: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:04 smithi103 ceph-mon[115211]: pgmap v586: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:07.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:06 smithi122 ceph-mon[83368]: pgmap v587: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:07.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:06 smithi103 ceph-mon[115211]: pgmap v587: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:09.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:08 smithi122 ceph-mon[83368]: pgmap v588: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:08 smithi103 ceph-mon[115211]: pgmap v588: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:10.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:10 smithi103 ceph-mon[115211]: pgmap v589: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:10 smithi122 ceph-mon[83368]: pgmap v589: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:12 smithi122 ceph-mon[83368]: pgmap v590: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:12 smithi103 ceph-mon[115211]: pgmap v590: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:15.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:14 smithi122 ceph-mon[83368]: pgmap v591: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:14 smithi103 ceph-mon[115211]: pgmap v591: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:16 smithi122 ceph-mon[83368]: pgmap v592: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:16 smithi103 ceph-mon[115211]: pgmap v592: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:18 smithi122 ceph-mon[83368]: pgmap v593: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:18 smithi103 ceph-mon[115211]: pgmap v593: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:20.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:20 smithi103 ceph-mon[115211]: pgmap v594: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:20 smithi122 ceph-mon[83368]: pgmap v594: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:22 smithi122 ceph-mon[83368]: pgmap v595: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:22 smithi103 ceph-mon[115211]: pgmap v595: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:24 smithi122 ceph-mon[83368]: pgmap v596: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:24 smithi103 ceph-mon[115211]: pgmap v596: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:26 smithi122 ceph-mon[83368]: pgmap v597: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:26 smithi103 ceph-mon[115211]: pgmap v597: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:28.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/51794728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac0fb650 msgr2=0x7f9bac0fbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:28.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/51794728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac0fb650 0x7f9bac0fbac0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c004940 tx=0x7f9b9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:28.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/51794728 shutdown_connections 2024-03-22T02:51:28.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/51794728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac0fb650 0x7f9bac0fbac0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:28.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/51794728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac1015c0 0x7f9bac101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/51794728 >> 172.21.15.103:0/51794728 conn(0x7f9bac071530 msgr2=0x7f9bac071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/51794728 shutdown_connections 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.903+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/51794728 wait complete. 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 Processor -- start 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 -- start start 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac0fb650 0x7f9bac115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1166f0 con 0x7f9bac1015c0 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb3b9d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac10fcd0 con 0x7f9bac0fb650 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:28.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34230/0 (socket says 172.21.15.103:34230) 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 -- 172.21.15.103:0/179401339 learned_addr learned my addr 172.21.15.103:0/179401339 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1939700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac0fb650 0x7f9bac115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 -- 172.21.15.103:0/179401339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac0fb650 msgr2=0x7f9bac115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac0fb650 0x7f9bac115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.904+0000 7f9bb1138700 1 -- 172.21.15.103:0/179401339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c005040 con 0x7f9bac1015c0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9bb1138700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b8b0 tx=0x7f9b9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9c00e040 con 0x7f9bac1015c0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac10ff50 con 0x7f9bac1015c0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b9c004290 con 0x7f9bac1015c0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9c01c3f0 con 0x7f9bac1015c0 2024-03-22T02:51:28.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.905+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac110490 con 0x7f9bac1015c0 2024-03-22T02:51:28.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.906+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac103a70 con 0x7f9bac1015c0 2024-03-22T02:51:28.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.906+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9b9c0137b0 con 0x7f9bac1015c0 2024-03-22T02:51:28.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.907+0000 7f9ba27fc700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 0x7f9b98070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:28.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.907+0000 7f9bb1939700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 0x7f9b98070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:28.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.907+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9b9c08f130 con 0x7f9bac1015c0 2024-03-22T02:51:28.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.907+0000 7f9bb1939700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 0x7f9b98070850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800afd0 tx=0x7f9ba8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:28.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:28.911+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b9c05bee0 con 0x7f9bac1015c0 2024-03-22T02:51:29.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:28 smithi122 ceph-mon[83368]: pgmap v598: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:28 smithi103 ceph-mon[115211]: pgmap v598: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:29.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.133+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bac111220 con 0x7f9b9806e3a0 2024-03-22T02:51:29.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.134+0000 7f9ba27fc700 1 -- 172.21.15.103:0/179401339 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9bac111220 con 0x7f9b9806e3a0 2024-03-22T02:51:29.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 msgr2=0x7f9b98070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 0x7f9b98070850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800afd0 tx=0x7f9ba8009040 comp rx=0 tx=0).stop 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 msgr2=0x7f9bac1160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b8b0 tx=0x7f9b9c009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 shutdown_connections 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9b9806e3a0 0x7f9b98070850 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9bac0fb650 0x7f9bac115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 --2- 172.21.15.103:0/179401339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9bac1015c0 0x7f9bac1160a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.136+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 >> 172.21.15.103:0/179401339 conn(0x7f9bac071530 msgr2=0x7f9bac0fa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.137+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 shutdown_connections 2024-03-22T02:51:29.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.137+0000 7f9bb3b9d700 1 -- 172.21.15.103:0/179401339 wait complete. 2024-03-22T02:51:29.147 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:51:29.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- 172.21.15.103:0/542834009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0103780 msgr2=0x7f42f00fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 --2- 172.21.15.103:0/542834009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0103780 0x7f42f00fcdf0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f42dc004940 tx=0x7f42dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- 172.21.15.103:0/542834009 shutdown_connections 2024-03-22T02:51:29.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 --2- 172.21.15.103:0/542834009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0103780 0x7f42f00fcdf0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 --2- 172.21.15.103:0/542834009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0102de0 0x7f42f01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- 172.21.15.103:0/542834009 >> 172.21.15.103:0/542834009 conn(0x7f42f00f8950 msgr2=0x7f42f00fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- 172.21.15.103:0/542834009 shutdown_connections 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- 172.21.15.103:0/542834009 wait complete. 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 Processor -- start 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.238+0000 7f42f6bee700 1 -- start start 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0103780 0x7f42f0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f00721f0 con 0x7f42f0102de0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0070d40 con 0x7f42f0103780 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34254/0 (socket says 172.21.15.103:34254) 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 -- 172.21.15.103:0/1530755003 learned_addr learned my addr 172.21.15.103:0/1530755003 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42effff700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0103780 0x7f42f0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 -- 172.21.15.103:0/1530755003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0103780 msgr2=0x7f42f0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0103780 0x7f42f0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 -- 172.21.15.103:0/1530755003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc005040 con 0x7f42f0102de0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f498a700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f42e4002a50 tx=0x7f42e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42e4007cc0 con 0x7f42f0102de0 2024-03-22T02:51:29.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f0071020 con 0x7f42f0102de0 2024-03-22T02:51:29.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42e400f040 con 0x7f42f0102de0 2024-03-22T02:51:29.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42e4009640 con 0x7f42f0102de0 2024-03-22T02:51:29.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.239+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f011a0f0 con 0x7f42f0102de0 2024-03-22T02:51:29.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.240+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f00fe530 con 0x7f42f0102de0 2024-03-22T02:51:29.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.241+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f42e40097c0 con 0x7f42f0102de0 2024-03-22T02:51:29.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.241+0000 7f42ed7fa700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 0x7f42e0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.241+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f42e408dba0 con 0x7f42f0102de0 2024-03-22T02:51:29.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.241+0000 7f42effff700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 0x7f42e0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.242+0000 7f42effff700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 0x7f42e0070850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00b3b0 tx=0x7f42dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.246+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42e4056da0 con 0x7f42f0102de0 2024-03-22T02:51:29.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.459+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f42f0109f60 con 0x7f42e006e3a0 2024-03-22T02:51:29.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.460+0000 7f42ed7fa700 1 -- 172.21.15.103:0/1530755003 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f42dc00b5c0 con 0x7f42e006e3a0 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 msgr2=0x7f42e0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 0x7f42e0070850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00b3b0 tx=0x7f42dc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 msgr2=0x7f42f00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f42e4002a50 tx=0x7f42e40046b0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 shutdown_connections 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42e006e3a0 0x7f42e0070850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42f0102de0 0x7f42f00702c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 --2- 172.21.15.103:0/1530755003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f0103780 0x7f42f0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 >> 172.21.15.103:0/1530755003 conn(0x7f42f00f8950 msgr2=0x7f42f00f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 shutdown_connections 2024-03-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.462+0000 7f42f6bee700 1 -- 172.21.15.103:0/1530755003 wait complete. 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 -- 172.21.15.103:0/3661587442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 msgr2=0x7f9e08102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/3661587442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e08102140 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc004940 tx=0x7f9dfc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 -- 172.21.15.103:0/3661587442 shutdown_connections 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/3661587442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 0x7f9e080fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/3661587442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e08102140 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 -- 172.21.15.103:0/3661587442 >> 172.21.15.103:0/3661587442 conn(0x7f9e08072bc0 msgr2=0x7f9e08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 -- 172.21.15.103:0/3661587442 shutdown_connections 2024-03-22T02:51:29.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.563+0000 7f9e0ea91700 1 -- 172.21.15.103:0/3661587442 wait complete. 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 Processor -- start 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 -- start start 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 0x7f9e0810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0810f720 con 0x7f9e08101d70 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0ea91700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0810f860 con 0x7f9e08102710 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0c82d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0c82d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34272/0 (socket says 172.21.15.103:34272) 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.564+0000 7f9e0c82d700 1 -- 172.21.15.103:0/1172609824 learned_addr learned my addr 172.21.15.103:0/1172609824 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e07fff700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 0x7f9e0810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0c82d700 1 -- 172.21.15.103:0/1172609824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 msgr2=0x7f9e0810f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0c82d700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 0x7f9e0810f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0c82d700 1 -- 172.21.15.103:0/1172609824 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc005040 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0c82d700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00b8b0 tx=0x7f9dfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dfc00e040 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08117b50 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dfc004290 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dfc01c3f0 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.565+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e081180e0 con 0x7f9e08101d70 2024-03-22T02:51:29.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.566+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e080ff080 con 0x7f9e08101d70 2024-03-22T02:51:29.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.567+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f9dfc0137b0 con 0x7f9e08101d70 2024-03-22T02:51:29.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.567+0000 7f9e057fa700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 0x7f9df8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.567+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9dfc08f110 con 0x7f9e08101d70 2024-03-22T02:51:29.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.567+0000 7f9e07fff700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 0x7f9df8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.567+0000 7f9e07fff700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 0x7f9df8070850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9df4007fd0 tx=0x7f9df4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.572+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9dfc058390 con 0x7f9e08101d70 2024-03-22T02:51:29.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.779+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e0804bed0 con 0x7f9df806e3a0 2024-03-22T02:51:29.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.789+0000 7f9e057fa700 1 -- 172.21.15.103:0/1172609824 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9e0804bed0 con 0x7f9df806e3a0 2024-03-22T02:51:29.790 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:51:29.790 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (20m) 13m ago 31m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:51:29.790 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (17m) 13m ago 31m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:51:29.790 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (17m) 13m ago 30m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (19m) 13m ago 30m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (14m) 13m ago 25m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (13m) 13m ago 25m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (13m) 13m ago 25m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (13m) 13m ago 25m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (21m) 13m ago 32m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (20m) 13m ago 30m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:51:29.791 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (18m) 13m ago 32m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (18m) 13m ago 30m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (20m) 13m ago 31m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (20m) 13m ago 30m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (17m) 13m ago 29m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (16m) 13m ago 28m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (16m) 13m ago 28m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (15m) 13m ago 28m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (14m) 13m ago 27m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (14m) 13m ago 27m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:51:29.792 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (20m) 13m ago 30m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:51:29.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 msgr2=0x7f9df8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 0x7f9df8070850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9df4007fd0 tx=0x7f9df4009040 comp rx=0 tx=0).stop 2024-03-22T02:51:29.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 msgr2=0x7f9e0810eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00b8b0 tx=0x7f9dfc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 shutdown_connections 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f9df806e3a0 0x7f9df8070850 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e08101d70 0x7f9e0810eca0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 --2- 172.21.15.103:0/1172609824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08102710 0x7f9e0810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 >> 172.21.15.103:0/1172609824 conn(0x7f9e08072bc0 msgr2=0x7f9e080fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 shutdown_connections 2024-03-22T02:51:29.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.791+0000 7f9e0ea91700 1 -- 172.21.15.103:0/1172609824 wait complete. 2024-03-22T02:51:29.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.890+0000 7f8268fa4700 1 -- 172.21.15.103:0/4062034356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 msgr2=0x7f82640fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.890+0000 7f8268fa4700 1 --2- 172.21.15.103:0/4062034356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f82640fb820 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8258004940 tx=0x7f8258009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- 172.21.15.103:0/4062034356 shutdown_connections 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 --2- 172.21.15.103:0/4062034356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f82640fb820 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 --2- 172.21.15.103:0/4062034356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 0x7f8264103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- 172.21.15.103:0/4062034356 >> 172.21.15.103:0/4062034356 conn(0x7f8264072310 msgr2=0x7f8264072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- 172.21.15.103:0/4062034356 shutdown_connections 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- 172.21.15.103:0/4062034356 wait complete. 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 Processor -- start 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- start start 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 0x7f8264113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82641144f0 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f8268fa4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826410db90 con 0x7f8264102f70 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.891+0000 7f826259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34292/0 (socket says 172.21.15.103:34292) 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 -- 172.21.15.103:0/2717448227 learned_addr learned my addr 172.21.15.103:0/2717448227 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8261d9b700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 0x7f8264113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 -- 172.21.15.103:0/2717448227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 msgr2=0x7f8264113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 0x7f8264113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 -- 172.21.15.103:0/2717448227 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258005040 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f826259c700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8254002a50 tx=0x7f82540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8254007cc0 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f825400f040 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f825400da70 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826410de70 con 0x7f82640fb3b0 2024-03-22T02:51:29.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.892+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826410e330 con 0x7f82640fb3b0 2024-03-22T02:51:29.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.893+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8264105350 con 0x7f82640fb3b0 2024-03-22T02:51:29.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.894+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f825400dc10 con 0x7f82640fb3b0 2024-03-22T02:51:29.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.894+0000 7f8252ffd700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 0x7f824c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:29.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.894+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f825408d420 con 0x7f82640fb3b0 2024-03-22T02:51:29.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.894+0000 7f8261d9b700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 0x7f824c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:29.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.894+0000 7f8261d9b700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 0x7f824c070850 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f826410f310 tx=0x7f8258009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:29.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:29.899+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8254056620 con 0x7f82640fb3b0 2024-03-22T02:51:30.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.180+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f826410a020 con 0x7f82640fb3b0 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.181+0000 7f8252ffd700 1 -- 172.21.15.103:0/2717448227 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8254059ce0 con 0x7f82640fb3b0 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:51:30.182 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:51:30.183 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 msgr2=0x7f824c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 0x7f824c070850 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f826410f310 tx=0x7f8258009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 msgr2=0x7f8264113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8254002a50 tx=0x7f82540046b0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 shutdown_connections 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f824c06e3a0 0x7f824c070850 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82640fb3b0 0x7f8264113a00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 --2- 172.21.15.103:0/2717448227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8264102f70 0x7f8264113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 >> 172.21.15.103:0/2717448227 conn(0x7f8264072310 msgr2=0x7f82640f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 shutdown_connections 2024-03-22T02:51:30.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.183+0000 7f8268fa4700 1 -- 172.21.15.103:0/2717448227 wait complete. 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 -- 172.21.15.103:0/2083185792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a78102e00 msgr2=0x7f2a781031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2083185792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a78102e00 0x7f2a781031d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2a74004940 tx=0x7f2a74009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 -- 172.21.15.103:0/2083185792 shutdown_connections 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2083185792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a781037a0 0x7f2a780fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2083185792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a78102e00 0x7f2a781031d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.286+0000 7f2a806b5700 1 -- 172.21.15.103:0/2083185792 >> 172.21.15.103:0/2083185792 conn(0x7f2a780f89b0 msgr2=0x7f2a780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 -- 172.21.15.103:0/2083185792 shutdown_connections 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 -- 172.21.15.103:0/2083185792 wait complete. 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 Processor -- start 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 -- start start 2024-03-22T02:51:30.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a78102e00 0x7f2a78113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78114520 con 0x7f2a781037a0 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a806b5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7810db70 con 0x7f2a78102e00 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a7dc50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a7dc50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34316/0 (socket says 172.21.15.103:34316) 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.287+0000 7f2a7dc50700 1 -- 172.21.15.103:0/2672794513 learned_addr learned my addr 172.21.15.103:0/2672794513 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a7e451700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a78102e00 0x7f2a78113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a7dc50700 1 -- 172.21.15.103:0/2672794513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a78102e00 msgr2=0x7f2a78113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a7dc50700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a78102e00 0x7f2a78113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a7dc50700 1 -- 172.21.15.103:0/2672794513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74005040 con 0x7f2a781037a0 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a7dc50700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c800 tx=0x7f2a68004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a680106f0 con 0x7f2a781037a0 2024-03-22T02:51:30.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a68010d20 con 0x7f2a781037a0 2024-03-22T02:51:30.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7810de50 con 0x7f2a781037a0 2024-03-22T02:51:30.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a680113a0 con 0x7f2a781037a0 2024-03-22T02:51:30.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.288+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7810e410 con 0x7f2a781037a0 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.289+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a78100110 con 0x7f2a781037a0 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.289+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2a68012400 con 0x7f2a781037a0 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.290+0000 7f2a6effd700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 0x7f2a64070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.290+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2a6808dd00 con 0x7f2a781037a0 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.290+0000 7f2a7e451700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 0x7f2a64070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.290+0000 7f2a7e451700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 0x7f2a64070850 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a74004f90 tx=0x7f2a74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.295+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a6805aab0 con 0x7f2a781037a0 2024-03-22T02:51:30.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.535+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a7810efe0 con 0x7f2a781037a0 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.536+0000 7f2a6effd700 1 -- 172.21.15.103:0/2672794513 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f2a6805a640 con 0x7f2a781037a0 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:51:30.537 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:51:30.538 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:51:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:51:30.540 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 msgr2=0x7f2a64070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 0x7f2a64070850 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a74004f90 tx=0x7f2a74009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 msgr2=0x7f2a78113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c800 tx=0x7f2a68004470 comp rx=0 tx=0).stop 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 shutdown_connections 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a6406e3a0 0x7f2a64070850 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a78102e00 0x7f2a78113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 --2- 172.21.15.103:0/2672794513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a781037a0 0x7f2a78113ed0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 >> 172.21.15.103:0/2672794513 conn(0x7f2a780f89b0 msgr2=0x7f2a781073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 shutdown_connections 2024-03-22T02:51:30.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.538+0000 7f2a806b5700 1 -- 172.21.15.103:0/2672794513 wait complete. 2024-03-22T02:51:30.542 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:51:30.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.639+0000 7f6294842700 1 -- 172.21.15.103:0/2310371671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101d20 msgr2=0x7f62901020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.639+0000 7f6294842700 1 --2- 172.21.15.103:0/2310371671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101d20 0x7f62901020f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6280004940 tx=0x7f6280009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- 172.21.15.103:0/2310371671 shutdown_connections 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 --2- 172.21.15.103:0/2310371671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62901026c0 0x7f62900fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 --2- 172.21.15.103:0/2310371671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101d20 0x7f62901020f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- 172.21.15.103:0/2310371671 >> 172.21.15.103:0/2310371671 conn(0x7f62900f78b0 msgr2=0x7f62900f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- 172.21.15.103:0/2310371671 shutdown_connections 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- 172.21.15.103:0/2310371671 wait complete. 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 Processor -- start 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- start start 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6290101d20 0x7f629010eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010f5c0 con 0x7f62901026c0 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f6294842700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010f700 con 0x7f6290101d20 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f628dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f628dd9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34334/0 (socket says 172.21.15.103:34334) 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.640+0000 7f628dd9b700 1 -- 172.21.15.103:0/1265761959 learned_addr learned my addr 172.21.15.103:0/1265761959 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f628e59c700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6290101d20 0x7f629010eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f628dd9b700 1 -- 172.21.15.103:0/1265761959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6290101d20 msgr2=0x7f629010eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f628dd9b700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6290101d20 0x7f629010eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f628dd9b700 1 -- 172.21.15.103:0/1265761959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280005040 con 0x7f62901026c0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f628dd9b700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f628400c800 tx=0x7f6284004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62840106f0 con 0x7f62901026c0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6284010d20 con 0x7f62901026c0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6290117bb0 con 0x7f62901026c0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62840113a0 con 0x7f62901026c0 2024-03-22T02:51:30.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.641+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6290117f80 con 0x7f62901026c0 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.642+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62900fd470 con 0x7f62901026c0 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.643+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6284012400 con 0x7f62901026c0 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.643+0000 7f627effd700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 0x7f6278070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.643+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f628408dc10 con 0x7f62901026c0 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.643+0000 7f628e59c700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 0x7f6278070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.644+0000 7f628e59c700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 0x7f6278070850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6280004f90 tx=0x7f6280009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.648+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6284056f50 con 0x7f62901026c0 2024-03-22T02:51:30.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.862+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62901055c0 con 0x7f627806e3a0 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: pgmap v599: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2717448227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:51:30.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:30 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2672794513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:51:30.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.864+0000 7f627effd700 1 -- 172.21.15.103:0/1265761959 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f62901055c0 con 0x7f627806e3a0 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:51:30.865 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:51:30.866 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:51:30.866 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 msgr2=0x7f6278070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 0x7f6278070850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6280004f90 tx=0x7f6280009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 msgr2=0x7f629010f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f628400c800 tx=0x7f6284004470 comp rx=0 tx=0).stop 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 shutdown_connections 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f627806e3a0 0x7f6278070850 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6290101d20 0x7f629010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 --2- 172.21.15.103:0/1265761959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62901026c0 0x7f629010f080 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 >> 172.21.15.103:0/1265761959 conn(0x7f62900f78b0 msgr2=0x7f6290104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 shutdown_connections 2024-03-22T02:51:30.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.866+0000 7f6294842700 1 -- 172.21.15.103:0/1265761959 wait complete. 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 -- 172.21.15.103:0/407463553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 msgr2=0x7facf40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 --2- 172.21.15.103:0/407463553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf40fce10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7facdc004940 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 -- 172.21.15.103:0/407463553 shutdown_connections 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 --2- 172.21.15.103:0/407463553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf40fce10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 --2- 172.21.15.103:0/407463553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 0x7facf41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 -- 172.21.15.103:0/407463553 >> 172.21.15.103:0/407463553 conn(0x7facf40f89b0 msgr2=0x7facf40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 -- 172.21.15.103:0/407463553 shutdown_connections 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 -- 172.21.15.103:0/407463553 wait complete. 2024-03-22T02:51:30.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.967+0000 7facfa117700 1 Processor -- start 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facfa117700 1 -- start start 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facfa117700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 0x7facf4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facfa117700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facfa117700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf4114680 con 0x7facf41037a0 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facfa117700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf410da70 con 0x7facf4102e00 2024-03-22T02:51:30.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34360/0 (socket says 172.21.15.103:34360) 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 -- 172.21.15.103:0/1007752635 learned_addr learned my addr 172.21.15.103:0/1007752635 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facf37fe700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 0x7facf4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 -- 172.21.15.103:0/1007752635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 msgr2=0x7facf4113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 0x7facf4113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 -- 172.21.15.103:0/1007752635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc005040 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7faceaffd700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7facdc009180 tx=0x7facdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.968+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc00e040 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.969+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf410dcf0 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.969+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facdc013420 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.969+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facdc01c610 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.969+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf410e2b0 con 0x7facf41037a0 2024-03-22T02:51:30.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.969+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf41052d0 con 0x7facf41037a0 2024-03-22T02:51:30.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.970+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7facdc013a00 con 0x7facf41037a0 2024-03-22T02:51:30.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.970+0000 7facf0ff9700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 0x7face0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:51:30.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.971+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7facdc08ef00 con 0x7facf41037a0 2024-03-22T02:51:30.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.971+0000 7facf37fe700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 0x7face0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:51:30.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.971+0000 7facf37fe700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 0x7face0070850 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7face4009fd0 tx=0x7face400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:51:30.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:30.975+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7facdc05bd60 con 0x7facf41037a0 2024-03-22T02:51:31.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:31.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: pgmap v599: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:31.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:31.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:31.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2717448227' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:51:31.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:30 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2672794513' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:51:31.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.257+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7facf410f140 con 0x7facf41037a0 2024-03-22T02:51:31.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.258+0000 7facf0ff9700 1 -- 172.21.15.103:0/1007752635 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7facdc05b8f0 con 0x7facf41037a0 2024-03-22T02:51:31.259 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:51:31.260 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:51:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:51:31.260 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:51:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:51:31.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.259+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 msgr2=0x7face0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:31.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.259+0000 7facfa117700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 0x7face0070850 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7face4009fd0 tx=0x7face400b040 comp rx=0 tx=0).stop 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.259+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 msgr2=0x7facf4114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.259+0000 7facfa117700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7facdc009180 tx=0x7facdc009e30 comp rx=0 tx=0).stop 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 shutdown_connections 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7face006e3a0 0x7face0070850 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facf4102e00 0x7facf4113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 --2- 172.21.15.103:0/1007752635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facf41037a0 0x7facf4114030 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 >> 172.21.15.103:0/1007752635 conn(0x7facf40f89b0 msgr2=0x7facf41021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 shutdown_connections 2024-03-22T02:51:31.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:51:31.260+0000 7facfa117700 1 -- 172.21.15.103:0/1007752635 wait complete. 2024-03-22T02:51:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:31 smithi122 ceph-mon[83368]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:32.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:31 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1007752635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:51:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:31 smithi103 ceph-mon[115211]: from='client.35144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:51:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:31 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1007752635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:51:33.000 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:32 smithi122 ceph-mon[83368]: pgmap v600: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:33.130 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:33.131 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:32 smithi103 ceph-mon[115211]: pgmap v600: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:33.132 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:35.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:34 smithi122 ceph-mon[83368]: pgmap v601: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:34 smithi103 ceph-mon[115211]: pgmap v601: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:36 smithi122 ceph-mon[83368]: pgmap v602: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:36 smithi103 ceph-mon[115211]: pgmap v602: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:38 smithi122 ceph-mon[83368]: pgmap v603: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:39.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:38 smithi103 ceph-mon[115211]: pgmap v603: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:40.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:40 smithi103 ceph-mon[115211]: pgmap v604: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:41.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:40 smithi122 ceph-mon[83368]: pgmap v604: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:43.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:42 smithi122 ceph-mon[83368]: pgmap v605: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:42 smithi103 ceph-mon[115211]: pgmap v605: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:45.084 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:44 smithi103 ceph-mon[115211]: pgmap v606: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:45.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:44 smithi122 ceph-mon[83368]: pgmap v606: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:47.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:46 smithi122 ceph-mon[83368]: pgmap v607: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:46 smithi103 ceph-mon[115211]: pgmap v607: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:51:49.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:48 smithi122 ceph-mon[83368]: pgmap v608: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:48 smithi103 ceph-mon[115211]: pgmap v608: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:50.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:50 smithi103 ceph-mon[115211]: pgmap v609: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:51.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:50 smithi122 ceph-mon[83368]: pgmap v609: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:53.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:52 smithi122 ceph-mon[83368]: pgmap v610: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:52 smithi103 ceph-mon[115211]: pgmap v610: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:55.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:54 smithi122 ceph-mon[83368]: pgmap v611: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:55.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:54 smithi103 ceph-mon[115211]: pgmap v611: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:57.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:56 smithi122 ceph-mon[83368]: pgmap v612: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:57.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:56 smithi103 ceph-mon[115211]: pgmap v612: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:59.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:51:58 smithi122 ceph-mon[83368]: pgmap v613: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:51:59.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:51:58 smithi103 ceph-mon[115211]: pgmap v613: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:00 smithi122 ceph-mon[83368]: pgmap v614: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:00 smithi103 ceph-mon[115211]: pgmap v614: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.374+0000 7f8dae41e700 1 -- 172.21.15.103:0/3984397087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 msgr2=0x7f8da8101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.374+0000 7f8dae41e700 1 --2- 172.21.15.103:0/3984397087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da8101900 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004940 tx=0x7f8d98009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.374+0000 7f8dae41e700 1 -- 172.21.15.103:0/3984397087 shutdown_connections 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.374+0000 7f8dae41e700 1 --2- 172.21.15.103:0/3984397087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 0x7f8da80fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.374+0000 7f8dae41e700 1 --2- 172.21.15.103:0/3984397087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da8101900 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.375+0000 7f8dae41e700 1 -- 172.21.15.103:0/3984397087 >> 172.21.15.103:0/3984397087 conn(0x7f8da80714c0 msgr2=0x7f8da80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.375+0000 7f8dae41e700 1 -- 172.21.15.103:0/3984397087 shutdown_connections 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.375+0000 7f8dae41e700 1 -- 172.21.15.103:0/3984397087 wait complete. 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.375+0000 7f8dae41e700 1 Processor -- start 2024-03-22T02:52:01.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.375+0000 7f8dae41e700 1 -- start start 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 0x7f8da80fd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80ff0a0 con 0x7f8da8101530 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80fdbd0 con 0x7f8da80fb570 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38258/0 (socket says 172.21.15.103:38258) 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 -- 172.21.15.103:0/877828768 learned_addr learned my addr 172.21.15.103:0/877828768 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da7fff700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 0x7f8da80fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 -- 172.21.15.103:0/877828768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 msgr2=0x7f8da80fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 0x7f8da80fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 -- 172.21.15.103:0/877828768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98005040 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da77fe700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c830 tx=0x7f8d9c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9c0106f0 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d9c010d20 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da80fdeb0 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9c0113a0 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.376+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da811a150 con 0x7f8da8101530 2024-03-22T02:52:01.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.377+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da80fcc90 con 0x7f8da8101530 2024-03-22T02:52:01.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.378+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8d9c012400 con 0x7f8da8101530 2024-03-22T02:52:01.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.378+0000 7f8da4ff9700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 0x7f8d90070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.379+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8d9c08dd70 con 0x7f8da8101530 2024-03-22T02:52:01.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.379+0000 7f8da7fff700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 0x7f8d90070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.379+0000 7f8da7fff700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 0x7f8d90070850 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004f90 tx=0x7f8d98009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:01.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.383+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d9c056ff0 con 0x7f8da8101530 2024-03-22T02:52:01.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.600+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8da80fa910 con 0x7f8d9006e3a0 2024-03-22T02:52:01.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.602+0000 7f8da4ff9700 1 -- 172.21.15.103:0/877828768 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8da80fa910 con 0x7f8d9006e3a0 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 msgr2=0x7f8d90070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 0x7f8d90070850 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004f90 tx=0x7f8d98009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 msgr2=0x7f8da80fd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c830 tx=0x7f8d9c004470 comp rx=0 tx=0).stop 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 shutdown_connections 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8d9006e3a0 0x7f8d90070850 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8da80fb570 0x7f8da80fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 --2- 172.21.15.103:0/877828768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8da8101530 0x7f8da80fd690 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 >> 172.21.15.103:0/877828768 conn(0x7f8da80714c0 msgr2=0x7f8da80fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 shutdown_connections 2024-03-22T02:52:01.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.604+0000 7f8dae41e700 1 -- 172.21.15.103:0/877828768 wait complete. 2024-03-22T02:52:01.615 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 -- 172.21.15.103:0/2752336525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 msgr2=0x7f8c44102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/2752336525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44102230 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c40004940 tx=0x7f8c40009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 -- 172.21.15.103:0/2752336525 shutdown_connections 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/2752336525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 0x7f8c440fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/2752336525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44102230 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 -- 172.21.15.103:0/2752336525 >> 172.21.15.103:0/2752336525 conn(0x7f8c44071530 msgr2=0x7f8c44071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 -- 172.21.15.103:0/2752336525 shutdown_connections 2024-03-22T02:52:01.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.706+0000 7f8c4c06c700 1 -- 172.21.15.103:0/2752336525 wait complete. 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 Processor -- start 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 -- start start 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 0x7f8c44113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44114520 con 0x7f8c44101e60 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c4c06c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4410db20 con 0x7f8c44102800 2024-03-22T02:52:01.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38276/0 (socket says 172.21.15.103:38276) 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 -- 172.21.15.103:0/694837520 learned_addr learned my addr 172.21.15.103:0/694837520 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49607700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 0x7f8c44113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 -- 172.21.15.103:0/694837520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 msgr2=0x7f8c44113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 0x7f8c44113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 -- 172.21.15.103:0/694837520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c40005040 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.707+0000 7f8c49e08700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b8b0 tx=0x7f8c40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.708+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4000e040 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.708+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4410dda0 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.708+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c40004290 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.708+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c4001c3f0 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.708+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4410e360 con 0x7f8c44101e60 2024-03-22T02:52:01.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.709+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c441001d0 con 0x7f8c44101e60 2024-03-22T02:52:01.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.709+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f8c400137b0 con 0x7f8c44101e60 2024-03-22T02:52:01.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.709+0000 7f8c3a7fc700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 0x7f8c30070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:01.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.710+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8c4008f050 con 0x7f8c44101e60 2024-03-22T02:52:01.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.710+0000 7f8c49607700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 0x7f8c30070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:01.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.710+0000 7f8c49607700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 0x7f8c30070850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c34007fd0 tx=0x7f8c34009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:01.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.714+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c4005bec0 con 0x7f8c44101e60 2024-03-22T02:52:01.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.943+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c4410efe0 con 0x7f8c3006e3a0 2024-03-22T02:52:01.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.944+0000 7f8c3a7fc700 1 -- 172.21.15.103:0/694837520 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8c4410efe0 con 0x7f8c3006e3a0 2024-03-22T02:52:01.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 msgr2=0x7f8c30070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 0x7f8c30070850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c34007fd0 tx=0x7f8c34009040 comp rx=0 tx=0).stop 2024-03-22T02:52:01.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 msgr2=0x7f8c44113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:01.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b8b0 tx=0x7f8c40009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 shutdown_connections 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f8c3006e3a0 0x7f8c30070850 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c44101e60 0x7f8c44113990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 --2- 172.21.15.103:0/694837520 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c44102800 0x7f8c44113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 >> 172.21.15.103:0/694837520 conn(0x7f8c44071530 msgr2=0x7f8c441052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 shutdown_connections 2024-03-22T02:52:01.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:01.946+0000 7f8c4c06c700 1 -- 172.21.15.103:0/694837520 wait complete. 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.047+0000 7f90c83f5700 1 -- 172.21.15.103:0/3399483224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 msgr2=0x7f90c00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.047+0000 7f90c83f5700 1 --2- 172.21.15.103:0/3399483224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c00fbd80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f90b0004940 tx=0x7f90b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.047+0000 7f90c83f5700 1 -- 172.21.15.103:0/3399483224 shutdown_connections 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.047+0000 7f90c83f5700 1 --2- 172.21.15.103:0/3399483224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c00fbd80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.047+0000 7f90c83f5700 1 --2- 172.21.15.103:0/3399483224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 0x7f90c0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- 172.21.15.103:0/3399483224 >> 172.21.15.103:0/3399483224 conn(0x7f90c0072bc0 msgr2=0x7f90c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- 172.21.15.103:0/3399483224 shutdown_connections 2024-03-22T02:52:02.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- 172.21.15.103:0/3399483224 wait complete. 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 Processor -- start 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- start start 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 0x7f90c0112a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c0113610 con 0x7f90c0102710 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.048+0000 7f90c83f5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c010ca00 con 0x7f90c0101d70 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38290/0 (socket says 172.21.15.103:38290) 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 -- 172.21.15.103:0/2697066950 learned_addr learned my addr 172.21.15.103:0/2697066950 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:02.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c6191700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 0x7f90c0112a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 -- 172.21.15.103:0/2697066950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 msgr2=0x7f90c0112a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 0x7f90c0112a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 -- 172.21.15.103:0/2697066950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0005040 con 0x7f90c0102710 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c5990700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b3b0 tx=0x7f90b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90b000e040 con 0x7f90c0102710 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c010cc80 con 0x7f90c0102710 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c010d1e0 con 0x7f90c0102710 2024-03-22T02:52:02.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.049+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90b000bd00 con 0x7f90c0102710 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.050+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90b0013660 con 0x7f90c0102710 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.051+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c00fd4c0 con 0x7f90c0102710 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.051+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f90b001e420 con 0x7f90c0102710 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.051+0000 7f90b6ffd700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 0x7f90ac070650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.051+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f90b0091c80 con 0x7f90c0102710 2024-03-22T02:52:02.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.051+0000 7f90c6191700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 0x7f90ac070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.052+0000 7f90c6191700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 0x7f90ac070650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f90bc004f60 tx=0x7f90bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.056+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90b005ea30 con 0x7f90c0102710 2024-03-22T02:52:02.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.262+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f90c0112750 con 0x7f90ac06e1a0 2024-03-22T02:52:02.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.272+0000 7f90b6ffd700 1 -- 172.21.15.103:0/2697066950 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f90c0112750 con 0x7f90ac06e1a0 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (20m) 14m ago 31m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (18m) 14m ago 31m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (18m) 13m ago 30m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (20m) 14m ago 31m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (14m) 14m ago 26m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (14m) 14m ago 26m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (13m) 13m ago 26m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (14m) 13m ago 26m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (22m) 14m ago 33m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:52:02.273 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (21m) 13m ago 30m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (19m) 14m ago 33m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (18m) 13m ago 30m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (21m) 14m ago 31m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (21m) 13m ago 30m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (17m) 14m ago 29m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (17m) 14m ago 29m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (16m) 14m ago 29m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (16m) 13m ago 28m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (15m) 13m ago 28m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (14m) 13m ago 27m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:52:02.274 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (20m) 14m ago 31m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:52:02.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.274+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 msgr2=0x7f90ac070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.274+0000 7f90c83f5700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 0x7f90ac070650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f90bc004f60 tx=0x7f90bc00b040 comp rx=0 tx=0).stop 2024-03-22T02:52:02.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.274+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 msgr2=0x7f90c0112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.274+0000 7f90c83f5700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f90b000b3b0 tx=0x7f90b0009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:02.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.274+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 shutdown_connections 2024-03-22T02:52:02.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f90ac06e1a0 0x7f90ac070650 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90c0101d70 0x7f90c0112a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 --2- 172.21.15.103:0/2697066950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90c0102710 0x7f90c0112fc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 >> 172.21.15.103:0/2697066950 conn(0x7f90c0072bc0 msgr2=0x7f90c00fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:02.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 shutdown_connections 2024-03-22T02:52:02.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.275+0000 7f90c83f5700 1 -- 172.21.15.103:0/2697066950 wait complete. 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- 172.21.15.103:0/3776370755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 msgr2=0x7f416c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 --2- 172.21.15.103:0/3776370755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c0fbd20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4158004940 tx=0x7f4158009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- 172.21.15.103:0/3776370755 shutdown_connections 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 --2- 172.21.15.103:0/3776370755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c0fbd20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 --2- 172.21.15.103:0/3776370755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 0x7f416c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- 172.21.15.103:0/3776370755 >> 172.21.15.103:0/3776370755 conn(0x7f416c0f7880 msgr2=0x7f416c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:02.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- 172.21.15.103:0/3776370755 shutdown_connections 2024-03-22T02:52:02.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- 172.21.15.103:0/3776370755 wait complete. 2024-03-22T02:52:02.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 Processor -- start 2024-03-22T02:52:02.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.374+0000 7f4172b88700 1 -- start start 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f4172b88700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 0x7f416c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f4172b88700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f4172b88700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1149b0 con 0x7f416c1026b0 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f4172b88700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c10dd80 con 0x7f416c101d10 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38310/0 (socket says 172.21.15.103:38310) 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 -- 172.21.15.103:0/3422979773 learned_addr learned my addr 172.21.15.103:0/3422979773 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:02.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f4170924700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 0x7f416c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 -- 172.21.15.103:0/3422979773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 msgr2=0x7f416c113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 0x7f416c113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 -- 172.21.15.103:0/3422979773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158005040 con 0x7f416c1026b0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.375+0000 7f416bfff700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4158000c00 tx=0x7f4158009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.376+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415800e070 con 0x7f416c1026b0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.376+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c10e000 con 0x7f416c1026b0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.376+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4158013c00 con 0x7f416c1026b0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.376+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415800b650 con 0x7f416c1026b0 2024-03-22T02:52:02.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.376+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c10e540 con 0x7f416c1026b0 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.377+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c0fd460 con 0x7f416c1026b0 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.377+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f415801d9c0 con 0x7f416c1026b0 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.378+0000 7f41697fa700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 0x7f415c070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.378+0000 7f4170924700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 0x7f415c070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.378+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f415808efc0 con 0x7f416c1026b0 2024-03-22T02:52:02.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.378+0000 7f4170924700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 0x7f415c070860 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4160009b80 tx=0x7f416000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.382+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41580c3740 con 0x7f416c1026b0 2024-03-22T02:52:02.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.659+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f416c04bed0 con 0x7f416c1026b0 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.660+0000 7f41697fa700 1 -- 172.21.15.103:0/3422979773 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4158022280 con 0x7f416c1026b0 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:52:02.661 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:52:02.662 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 msgr2=0x7f415c070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 0x7f415c070860 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4160009b80 tx=0x7f416000b040 comp rx=0 tx=0).stop 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 msgr2=0x7f416c114360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4158000c00 tx=0x7f4158009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 shutdown_connections 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f415c06e3b0 0x7f415c070860 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f416c101d10 0x7f416c113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 --2- 172.21.15.103:0/3422979773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f416c1026b0 0x7f416c114360 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 >> 172.21.15.103:0/3422979773 conn(0x7f416c0f7880 msgr2=0x7f416c065e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 shutdown_connections 2024-03-22T02:52:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.662+0000 7f4172b88700 1 -- 172.21.15.103:0/3422979773 wait complete. 2024-03-22T02:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 -- 172.21.15.103:0/1956287034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a0102f50 msgr2=0x7f17a0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 --2- 172.21.15.103:0/1956287034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a0102f50 0x7f17a0103320 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1790004940 tx=0x7f1790009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 -- 172.21.15.103:0/1956287034 shutdown_connections 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 --2- 172.21.15.103:0/1956287034 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a00fb4d0 0x7f17a00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 --2- 172.21.15.103:0/1956287034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a0102f50 0x7f17a0103320 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 -- 172.21.15.103:0/1956287034 >> 172.21.15.103:0/1956287034 conn(0x7f17a00714c0 msgr2=0x7f17a00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 -- 172.21.15.103:0/1956287034 shutdown_connections 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.763+0000 7f17a5f21700 1 -- 172.21.15.103:0/1956287034 wait complete. 2024-03-22T02:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 Processor -- start 2024-03-22T02:52:02.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 -- start start 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a0102f50 0x7f17a0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a01145b0 con 0x7f17a00fb4d0 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f17a5f21700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a010db90 con 0x7f17a0102f50 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f179f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f179f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38326/0 (socket says 172.21.15.103:38326) 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f179f7fe700 1 -- 172.21.15.103:0/2357730876 learned_addr learned my addr 172.21.15.103:0/2357730876 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.764+0000 7f179effd700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a0102f50 0x7f17a0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f179f7fe700 1 -- 172.21.15.103:0/2357730876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a0102f50 msgr2=0x7f17a0113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f179f7fe700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a0102f50 0x7f17a0113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f179f7fe700 1 -- 172.21.15.103:0/2357730876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790005040 con 0x7f17a00fb4d0 2024-03-22T02:52:02.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f179f7fe700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1790009180 tx=0x7f1790009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179000e040 con 0x7f17a00fb4d0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1790013420 con 0x7f17a00fb4d0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a010de10 con 0x7f17a00fb4d0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179001c610 con 0x7f17a00fb4d0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.765+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a010e350 con 0x7f17a00fb4d0 2024-03-22T02:52:02.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.766+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a0100200 con 0x7f17a00fb4d0 2024-03-22T02:52:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.767+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1790013a00 con 0x7f17a00fb4d0 2024-03-22T02:52:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.767+0000 7f1787fff700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 0x7f1788070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.767+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f179008ef00 con 0x7f17a00fb4d0 2024-03-22T02:52:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.767+0000 7f179effd700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 0x7f1788070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:02.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.767+0000 7f179effd700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 0x7f1788070850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f17a010f2c0 tx=0x7f179400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:02.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:02.772+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f179005bd60 con 0x7f17a00fb4d0 2024-03-22T02:52:03.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.013+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f17a00fe7f0 con 0x7f17a00fb4d0 2024-03-22T02:52:03.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:02 smithi103 ceph-mon[115211]: pgmap v615: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:03.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:02 smithi103 ceph-mon[115211]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:03.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:03.014 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:02 smithi103 ceph-mon[115211]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:03.015 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:02 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3422979773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.013+0000 7f1787fff700 1 -- 172.21.15.103:0/2357730876 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f179005b8f0 con 0x7f17a00fb4d0 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:03.015 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:52:03.016 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:52:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:03.018 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 msgr2=0x7f1788070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 0x7f1788070850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f17a010f2c0 tx=0x7f179400b040 comp rx=0 tx=0).stop 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 msgr2=0x7f17a0113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1790009180 tx=0x7f1790009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 shutdown_connections 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f178806e3a0 0x7f1788070850 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17a00fb4d0 0x7f17a0113a20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.015+0000 7f17a5f21700 1 --2- 172.21.15.103:0/2357730876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17a0102f50 0x7f17a0113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.016+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 >> 172.21.15.103:0/2357730876 conn(0x7f17a00714c0 msgr2=0x7f17a00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:03.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.016+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 shutdown_connections 2024-03-22T02:52:03.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.016+0000 7f17a5f21700 1 -- 172.21.15.103:0/2357730876 wait complete. 2024-03-22T02:52:03.020 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:52:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:02 smithi122 ceph-mon[83368]: pgmap v615: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:02 smithi122 ceph-mon[83368]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:02 smithi122 ceph-mon[83368]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:02 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3422979773' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:52:03.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/2514579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 msgr2=0x7fe1d81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/2514579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d81031f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8004940 tx=0x7fe1c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:03.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/2514579 shutdown_connections 2024-03-22T02:52:03.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/2514579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 0x7fe1d80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/2514579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d81031f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/2514579 >> 172.21.15.103:0/2514579 conn(0x7fe1d80f89b0 msgr2=0x7fe1d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:03.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/2514579 shutdown_connections 2024-03-22T02:52:03.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.118+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/2514579 wait complete. 2024-03-22T02:52:03.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 Processor -- start 2024-03-22T02:52:03.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 -- start start 2024-03-22T02:52:03.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 0x7fe1d8113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d81144a0 con 0x7fe1d8102e20 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1dd3f1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d810daf0 con 0x7fe1d81037c0 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1d6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1d6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38344/0 (socket says 172.21.15.103:38344) 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1d6ffd700 1 -- 172.21.15.103:0/3692440609 learned_addr learned my addr 172.21.15.103:0/3692440609 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.119+0000 7fe1d67fc700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 0x7fe1d8113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1d6ffd700 1 -- 172.21.15.103:0/3692440609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 msgr2=0x7fe1d8113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1d6ffd700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 0x7fe1d8113e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1d6ffd700 1 -- 172.21.15.103:0/3692440609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8005040 con 0x7fe1d8102e20 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1d6ffd700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009b30 tx=0x7fe1c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:03.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c8021070 con 0x7fe1d8102e20 2024-03-22T02:52:03.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1c8013c00 con 0x7fe1d8102e20 2024-03-22T02:52:03.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c800e070 con 0x7fe1d8102e20 2024-03-22T02:52:03.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d810dd70 con 0x7fe1d8102e20 2024-03-22T02:52:03.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.120+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d810e250 con 0x7fe1d8102e20 2024-03-22T02:52:03.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.121+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d8100130 con 0x7fe1d8102e20 2024-03-22T02:52:03.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.121+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe1c801cad0 con 0x7fe1d8102e20 2024-03-22T02:52:03.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.122+0000 7fe1bf7fe700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 0x7fe1c0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.122+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe1c808f270 con 0x7fe1d8102e20 2024-03-22T02:52:03.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.122+0000 7fe1d67fc700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 0x7fe1c0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.122+0000 7fe1d67fc700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 0x7fe1c0070850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1d810ec60 tx=0x7fe1cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:03.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.126+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe1c805c020 con 0x7fe1d8102e20 2024-03-22T02:52:03.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.342+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1d810efd0 con 0x7fe1c006e3a0 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.343+0000 7fe1bf7fe700 1 -- 172.21.15.103:0/3692440609 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe1d810efd0 con 0x7fe1c006e3a0 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:52:03.344 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:52:03.345 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:52:03.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 msgr2=0x7fe1c0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 0x7fe1c0070850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1d810ec60 tx=0x7fe1cc00b040 comp rx=0 tx=0).stop 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 msgr2=0x7fe1d8113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009b30 tx=0x7fe1c8009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 shutdown_connections 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe1c006e3a0 0x7fe1c0070850 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1d8102e20 0x7fe1d8113910 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 --2- 172.21.15.103:0/3692440609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81037c0 0x7fe1d8113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 >> 172.21.15.103:0/3692440609 conn(0x7fe1d80f89b0 msgr2=0x7fe1d80fa730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 shutdown_connections 2024-03-22T02:52:03.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.346+0000 7fe1dd3f1700 1 -- 172.21.15.103:0/3692440609 wait complete. 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.445+0000 7f19b4887700 1 -- 172.21.15.103:0/4000239848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 msgr2=0x7f19b00fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.445+0000 7f19b4887700 1 --2- 172.21.15.103:0/4000239848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b00fb820 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f19a4004940 tx=0x7f19a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 -- 172.21.15.103:0/4000239848 shutdown_connections 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 --2- 172.21.15.103:0/4000239848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b00fb820 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 --2- 172.21.15.103:0/4000239848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 0x7f19b0103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 -- 172.21.15.103:0/4000239848 >> 172.21.15.103:0/4000239848 conn(0x7f19b0072310 msgr2=0x7f19b0072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 -- 172.21.15.103:0/4000239848 shutdown_connections 2024-03-22T02:52:03.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 -- 172.21.15.103:0/4000239848 wait complete. 2024-03-22T02:52:03.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 Processor -- start 2024-03-22T02:52:03.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.446+0000 7f19b4887700 1 -- start start 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 0x7f19b0113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b01144f0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b010daf0 con 0x7f19b0102f70 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38354/0 (socket says 172.21.15.103:38354) 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 -- 172.21.15.103:0/4066887794 learned_addr learned my addr 172.21.15.103:0/4066887794 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19add9b700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 0x7f19b0113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 -- 172.21.15.103:0/4066887794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 msgr2=0x7f19b0113ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 0x7f19b0113ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 -- 172.21.15.103:0/4066887794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4005040 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19ae59c700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f19a0002a50 tx=0x7f19a00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a0007cc0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19a000f040 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b010ddd0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a0009640 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.447+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b010e390 con 0x7f19b00fb3b0 2024-03-22T02:52:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.449+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0105350 con 0x7f19b00fb3b0 2024-03-22T02:52:03.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.449+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f19a00097a0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.449+0000 7f199effd700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 0x7f1998070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:03.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.450+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19a008dbd0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.450+0000 7f19add9b700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 0x7f1998070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:03.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.450+0000 7f19add9b700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 0x7f1998070850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f19a400b3b0 tx=0x7f19a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:03.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.454+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19a005a900 con 0x7f19b00fb3b0 2024-03-22T02:52:03.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.739+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f19b010efb0 con 0x7f19b00fb3b0 2024-03-22T02:52:03.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.740+0000 7f199effd700 1 -- 172.21.15.103:0/4066887794 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f19a005a490 con 0x7f19b00fb3b0 2024-03-22T02:52:03.741 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:52:03.741 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:52:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:52:03.741 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:52:03.741 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 msgr2=0x7f1998070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 0x7f1998070850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f19a400b3b0 tx=0x7f19a4009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 msgr2=0x7f19b0113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f19a0002a50 tx=0x7f19a00046b0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 shutdown_connections 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f199806e3a0 0x7f1998070850 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b00fb3b0 0x7f19b0113960 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 --2- 172.21.15.103:0/4066887794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19b0102f70 0x7f19b0113ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 >> 172.21.15.103:0/4066887794 conn(0x7f19b0072310 msgr2=0x7f19b00f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 shutdown_connections 2024-03-22T02:52:03.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:03.742+0000 7f19b4887700 1 -- 172.21.15.103:0/4066887794 wait complete. 2024-03-22T02:52:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:03 smithi122 ceph-mon[83368]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:03 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2357730876' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:52:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:03 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4066887794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:52:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:03 smithi103 ceph-mon[115211]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:03 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2357730876' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:52:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:03 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4066887794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:52:05.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:04 smithi122 ceph-mon[83368]: pgmap v616: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:05.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:04 smithi122 ceph-mon[83368]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:04 smithi103 ceph-mon[115211]: pgmap v616: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:05.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:04 smithi103 ceph-mon[115211]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:07.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:06 smithi122 ceph-mon[83368]: pgmap v617: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:07.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:06 smithi103 ceph-mon[115211]: pgmap v617: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:09.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:08 smithi122 ceph-mon[83368]: pgmap v618: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:08 smithi103 ceph-mon[115211]: pgmap v618: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:11.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:10 smithi122 ceph-mon[83368]: pgmap v619: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:11.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:10 smithi103 ceph-mon[115211]: pgmap v619: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:13.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:12 smithi122 ceph-mon[83368]: pgmap v620: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:13.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:12 smithi103 ceph-mon[115211]: pgmap v620: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:15.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:14 smithi122 ceph-mon[83368]: pgmap v621: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:15.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:14 smithi103 ceph-mon[115211]: pgmap v621: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:17.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:16 smithi122 ceph-mon[83368]: pgmap v622: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:17.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:16 smithi103 ceph-mon[115211]: pgmap v622: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:19.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:18 smithi122 ceph-mon[83368]: pgmap v623: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:19.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:18 smithi103 ceph-mon[115211]: pgmap v623: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:21.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:20 smithi122 ceph-mon[83368]: pgmap v624: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:21.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:20 smithi103 ceph-mon[115211]: pgmap v624: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:23.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:22 smithi122 ceph-mon[83368]: pgmap v625: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:23.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:22 smithi103 ceph-mon[115211]: pgmap v625: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:25.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:24 smithi122 ceph-mon[83368]: pgmap v626: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:24 smithi103 ceph-mon[115211]: pgmap v626: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:26 smithi122 ceph-mon[83368]: pgmap v627: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:26 smithi103 ceph-mon[115211]: pgmap v627: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:29.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:28 smithi122 ceph-mon[83368]: pgmap v628: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:28 smithi103 ceph-mon[115211]: pgmap v628: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:31.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:30 smithi122 ceph-mon[83368]: pgmap v629: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:31.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:30 smithi103 ceph-mon[115211]: pgmap v629: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:32 smithi122 ceph-mon[83368]: pgmap v630: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:33.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:32 smithi103 ceph-mon[115211]: pgmap v630: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 -- 172.21.15.103:0/2963977534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f401026b0 msgr2=0x7f4f400fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2963977534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f401026b0 0x7f4f400fbd20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c004940 tx=0x7f4f3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 -- 172.21.15.103:0/2963977534 shutdown_connections 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2963977534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f401026b0 0x7f4f400fbd20 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2963977534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f40101d10 0x7f4f401020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.856+0000 7f4f4763c700 1 -- 172.21.15.103:0/2963977534 >> 172.21.15.103:0/2963977534 conn(0x7f4f400f7880 msgr2=0x7f4f400f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 -- 172.21.15.103:0/2963977534 shutdown_connections 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 -- 172.21.15.103:0/2963977534 wait complete. 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 Processor -- start 2024-03-22T02:52:33.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 -- start start 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f401026b0 0x7f4f401143d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40114a20 con 0x7f4f40101d10 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f4763c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4010ddf0 con 0x7f4f401026b0 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f453d8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.857+0000 7f4f453d8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36606/0 (socket says 172.21.15.103:36606) 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f453d8700 1 -- 172.21.15.103:0/2329897022 learned_addr learned my addr 172.21.15.103:0/2329897022 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f44bd7700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f401026b0 0x7f4f401143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f453d8700 1 -- 172.21.15.103:0/2329897022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f401026b0 msgr2=0x7f4f401143d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f453d8700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f401026b0 0x7f4f401143d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f453d8700 1 -- 172.21.15.103:0/2329897022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c005040 con 0x7f4f40101d10 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f453d8700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000c830 tx=0x7f4f30004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:33.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f300106f0 con 0x7f4f40101d10 2024-03-22T02:52:33.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f30010d20 con 0x7f4f40101d10 2024-03-22T02:52:33.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4010e0d0 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f300113a0 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.858+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4010e660 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.859+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f400fd460 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.860+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4f30012400 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.860+0000 7f4f35ffb700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 0x7f4f2c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.860+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f3008dd70 con 0x7f4f40101d10 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.860+0000 7f4f44bd7700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 0x7f4f2c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:33.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.861+0000 7f4f44bd7700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 0x7f4f2c070850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c004890 tx=0x7f4f3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:33.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:33.865+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f3005ab20 con 0x7f4f40101d10 2024-03-22T02:52:34.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.080+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f4004bf40 con 0x7f4f2c06e3a0 2024-03-22T02:52:34.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.081+0000 7f4f35ffb700 1 -- 172.21.15.103:0/2329897022 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4f4004bf40 con 0x7f4f2c06e3a0 2024-03-22T02:52:34.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 msgr2=0x7f4f2c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 0x7f4f2c070850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c004890 tx=0x7f4f3c009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 msgr2=0x7f4f40113e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.085 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000c830 tx=0x7f4f30004470 comp rx=0 tx=0).stop 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 shutdown_connections 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4f2c06e3a0 0x7f4f2c070850 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f40101d10 0x7f4f40113e90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 --2- 172.21.15.103:0/2329897022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f401026b0 0x7f4f401143d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 >> 172.21.15.103:0/2329897022 conn(0x7f4f400f7880 msgr2=0x7f4f40065e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 shutdown_connections 2024-03-22T02:52:34.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.083+0000 7f4f4763c700 1 -- 172.21.15.103:0/2329897022 wait complete. 2024-03-22T02:52:34.095 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 -- 172.21.15.103:0/3053526350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 msgr2=0x7f2c241020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 --2- 172.21.15.103:0/3053526350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c241020e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004940 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 -- 172.21.15.103:0/3053526350 shutdown_connections 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 --2- 172.21.15.103:0/3053526350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 0x7f2c240fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 --2- 172.21.15.103:0/3053526350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c241020e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 -- 172.21.15.103:0/3053526350 >> 172.21.15.103:0/3053526350 conn(0x7f2c240f7880 msgr2=0x7f2c240f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 -- 172.21.15.103:0/3053526350 shutdown_connections 2024-03-22T02:52:34.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.186+0000 7f2c289e8700 1 -- 172.21.15.103:0/3053526350 wait complete. 2024-03-22T02:52:34.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 Processor -- start 2024-03-22T02:52:34.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 -- start start 2024-03-22T02:52:34.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 0x7f2c2410f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410f9b0 con 0x7f2c24101d10 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c289e8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410faf0 con 0x7f2c241026b0 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36624/0 (socket says 172.21.15.103:36624) 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 -- 172.21.15.103:0/2552700859 learned_addr learned my addr 172.21.15.103:0/2552700859 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c21d9b700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 0x7f2c2410f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 -- 172.21.15.103:0/2552700859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 msgr2=0x7f2c2410f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 0x7f2c2410f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 -- 172.21.15.103:0/2552700859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18005040 con 0x7f2c24101d10 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c2259c700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b3b0 tx=0x7f2c18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.187+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1800e040 con 0x7f2c24101d10 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.188+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c1800bbc0 con 0x7f2c24101d10 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.188+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c24117e90 con 0x7f2c24101d10 2024-03-22T02:52:34.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.188+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1801c3f0 con 0x7f2c24101d10 2024-03-22T02:52:34.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.188+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c24118370 con 0x7f2c24101d10 2024-03-22T02:52:34.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.189+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c240fd460 con 0x7f2c24101d10 2024-03-22T02:52:34.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.189+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2c1800bd30 con 0x7f2c24101d10 2024-03-22T02:52:34.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.189+0000 7f2c12ffd700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 0x7f2c0c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.189+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2c1808f050 con 0x7f2c24101d10 2024-03-22T02:52:34.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.190+0000 7f2c21d9b700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 0x7f2c0c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.190+0000 7f2c21d9b700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 0x7f2c0c070850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2c14007fd0 tx=0x7f2c14009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.194+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c18058250 con 0x7f2c24101d10 2024-03-22T02:52:34.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.411+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c2404bed0 con 0x7f2c0c06e3a0 2024-03-22T02:52:34.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.413+0000 7f2c12ffd700 1 -- 172.21.15.103:0/2552700859 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2c2404bed0 con 0x7f2c0c06e3a0 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 msgr2=0x7f2c0c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 0x7f2c0c070850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2c14007fd0 tx=0x7f2c14009040 comp rx=0 tx=0).stop 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 msgr2=0x7f2c2410ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b3b0 tx=0x7f2c18009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 shutdown_connections 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2c0c06e3a0 0x7f2c0c070850 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c24101d10 0x7f2c2410ee20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 --2- 172.21.15.103:0/2552700859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c241026b0 0x7f2c2410f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 >> 172.21.15.103:0/2552700859 conn(0x7f2c240f7880 msgr2=0x7f2c240f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 shutdown_connections 2024-03-22T02:52:34.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.415+0000 7f2c289e8700 1 -- 172.21.15.103:0/2552700859 wait complete. 2024-03-22T02:52:34.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 -- 172.21.15.103:0/978962332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0101d70 msgr2=0x7f4af0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 --2- 172.21.15.103:0/978962332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0101d70 0x7f4af0102140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004940 tx=0x7f4ae0009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 -- 172.21.15.103:0/978962332 shutdown_connections 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 --2- 172.21.15.103:0/978962332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0102710 0x7f4af00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 --2- 172.21.15.103:0/978962332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0101d70 0x7f4af0102140 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.515+0000 7f4af6225700 1 -- 172.21.15.103:0/978962332 >> 172.21.15.103:0/978962332 conn(0x7f4af0072bc0 msgr2=0x7f4af0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.516+0000 7f4af6225700 1 -- 172.21.15.103:0/978962332 shutdown_connections 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.516+0000 7f4af6225700 1 -- 172.21.15.103:0/978962332 wait complete. 2024-03-22T02:52:34.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.516+0000 7f4af6225700 1 Processor -- start 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.516+0000 7f4af6225700 1 -- start start 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4af6225700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 0x7f4af01032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4af6225700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4af6225700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0103da0 con 0x7f4af0102710 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4af6225700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0103ee0 con 0x7f4af0101d70 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aef7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 0x7f4af01032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36640/0 (socket says 172.21.15.103:36640) 2024-03-22T02:52:34.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 -- 172.21.15.103:0/3145616655 learned_addr learned my addr 172.21.15.103:0/3145616655 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aef7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 0x7f4af01032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:36738/0 (socket says 172.21.15.103:36738) 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 -- 172.21.15.103:0/3145616655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 msgr2=0x7f4af01032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 0x7f4af01032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 -- 172.21.15.103:0/3145616655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae0005040 con 0x7f4af0102710 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.517+0000 7f4aeeffd700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400c7e0 tx=0x7f4ae4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.518+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae4012810 con 0x7f4af0102710 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.518+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae4012e40 con 0x7f4af0102710 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.518+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0119e40 con 0x7f4af0102710 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.518+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af011a2c0 con 0x7f4af0102710 2024-03-22T02:52:34.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.518+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae40102e0 con 0x7f4af0102710 2024-03-22T02:52:34.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.519+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af00fd4c0 con 0x7f4af0102710 2024-03-22T02:52:34.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.520+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4ae4012970 con 0x7f4af0102710 2024-03-22T02:52:34.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.520+0000 7f4ad7fff700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 0x7f4ad8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.520+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4ae408e0a0 con 0x7f4af0102710 2024-03-22T02:52:34.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.520+0000 7f4aef7fe700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 0x7f4ad8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.521+0000 7f4aef7fe700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 0x7f4ad8070850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004f90 tx=0x7f4ae0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.525+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4ae40572a0 con 0x7f4af0102710 2024-03-22T02:52:34.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.731+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af004bed0 con 0x7f4ad806e3a0 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.741+0000 7f4ad7fff700 1 -- 172.21.15.103:0/3145616655 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4af004bed0 con 0x7f4ad806e3a0 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (21m) 14m ago 32m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (18m) 14m ago 32m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (18m) 14m ago 31m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (20m) 14m ago 32m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (15m) 14m ago 26m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (14m) 14m ago 26m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:52:34.743 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (14m) 14m ago 26m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (14m) 14m ago 26m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (22m) 14m ago 33m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (22m) 14m ago 31m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (19m) 14m ago 33m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (19m) 14m ago 31m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (21m) 14m ago 32m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (21m) 14m ago 31m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (18m) 14m ago 30m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (17m) 14m ago 30m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (17m) 14m ago 29m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (16m) 14m ago 29m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (16m) 14m ago 28m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (15m) 14m ago 28m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:52:34.744 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (21m) 14m ago 31m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:52:34.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 msgr2=0x7f4ad8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 0x7f4ad8070850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0004f90 tx=0x7f4ae0009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 msgr2=0x7f4af0103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400c7e0 tx=0x7f4ae4007f90 comp rx=0 tx=0).stop 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 shutdown_connections 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ad806e3a0 0x7f4ad8070850 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af0101d70 0x7f4af01032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 --2- 172.21.15.103:0/3145616655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af0102710 0x7f4af0103810 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 >> 172.21.15.103:0/3145616655 conn(0x7f4af0072bc0 msgr2=0x7f4af0105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 shutdown_connections 2024-03-22T02:52:34.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.743+0000 7f4af6225700 1 -- 172.21.15.103:0/3145616655 wait complete. 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 -- 172.21.15.103:0/1585676346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac81026b0 msgr2=0x7f2ac80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/1585676346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac81026b0 0x7f2ac80fbd20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2abc004940 tx=0x7f2abc009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 -- 172.21.15.103:0/1585676346 shutdown_connections 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/1585676346 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac81026b0 0x7f2ac80fbd20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/1585676346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac8101d10 0x7f2ac81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 -- 172.21.15.103:0/1585676346 >> 172.21.15.103:0/1585676346 conn(0x7f2ac80f7880 msgr2=0x7f2ac80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:34.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 -- 172.21.15.103:0/1585676346 shutdown_connections 2024-03-22T02:52:34.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 -- 172.21.15.103:0/1585676346 wait complete. 2024-03-22T02:52:34.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.843+0000 7f2ace3e7700 1 Processor -- start 2024-03-22T02:52:34.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ace3e7700 1 -- start start 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ace3e7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ace3e7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac81026b0 0x7f2ac810f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ace3e7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac810f9e0 con 0x7f2ac8101d10 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ace3e7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac810fb20 con 0x7f2ac81026b0 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36656/0 (socket says 172.21.15.103:36656) 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 -- 172.21.15.103:0/2157607843 learned_addr learned my addr 172.21.15.103:0/2157607843 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac77fe700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac81026b0 0x7f2ac810f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 -- 172.21.15.103:0/2157607843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac81026b0 msgr2=0x7f2ac810f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac81026b0 0x7f2ac810f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 -- 172.21.15.103:0/2157607843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc005040 con 0x7f2ac8101d10 2024-03-22T02:52:34.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac7fff700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800eb40 tx=0x7f2ab800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.844+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab8007cc0 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.845+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac8117e10 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.845+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ab800f040 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.845+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab8009620 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.845+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81182c0 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.846+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80fd460 con 0x7f2ac8101d10 2024-03-22T02:52:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.846+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2ab8009780 con 0x7f2ac8101d10 2024-03-22T02:52:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.847+0000 7f2ac4ff9700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 0x7f2ab0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.847+0000 7f2ac77fe700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 0x7f2ab0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.847+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ab808d9f0 con 0x7f2ac8101d10 2024-03-22T02:52:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.847+0000 7f2ac77fe700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 0x7f2ab0070850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009b30 tx=0x7f2abc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:34.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:34.851+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ab8056ca0 con 0x7f2ac8101d10 2024-03-22T02:52:35.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:34 smithi122 ceph-mon[83368]: pgmap v631: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:35.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:34 smithi122 ceph-mon[83368]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:34 smithi103 ceph-mon[115211]: pgmap v631: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:35.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:34 smithi103 ceph-mon[115211]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:35.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.134+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2ac804bf60 con 0x7f2ac8101d10 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.134+0000 7f2ac4ff9700 1 -- 172.21.15.103:0/2157607843 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2ab805a360 con 0x7f2ac8101d10 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:52:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:52:35.137 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:52:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 msgr2=0x7f2ab0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 0x7f2ab0070850 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009b30 tx=0x7f2abc009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 msgr2=0x7f2ac810ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2ab800eb40 tx=0x7f2ab800b040 comp rx=0 tx=0).stop 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 shutdown_connections 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2ab006e3a0 0x7f2ab0070850 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ac8101d10 0x7f2ac810ef60 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 --2- 172.21.15.103:0/2157607843 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ac81026b0 0x7f2ac810f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 >> 172.21.15.103:0/2157607843 conn(0x7f2ac80f7880 msgr2=0x7f2ac80f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 shutdown_connections 2024-03-22T02:52:35.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.136+0000 7f2ace3e7700 1 -- 172.21.15.103:0/2157607843 wait complete. 2024-03-22T02:52:35.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 -- 172.21.15.103:0/263667822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fde90 msgr2=0x7f430c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 --2- 172.21.15.103:0/263667822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fde90 0x7f430c0fe260 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f42fc004970 tx=0x7f42fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 -- 172.21.15.103:0/263667822 shutdown_connections 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 --2- 172.21.15.103:0/263667822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fe830 0x7f430c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 --2- 172.21.15.103:0/263667822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fde90 0x7f430c0fe260 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 -- 172.21.15.103:0/263667822 >> 172.21.15.103:0/263667822 conn(0x7f430c0f78f0 msgr2=0x7f430c0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 -- 172.21.15.103:0/263667822 shutdown_connections 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 -- 172.21.15.103:0/263667822 wait complete. 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.237+0000 7f43114ef700 1 Processor -- start 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43114ef700 1 -- start start 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43114ef700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 0x7f430c1138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43114ef700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43114ef700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c114480 con 0x7f430c0fe830 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43114ef700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c10dad0 con 0x7f430c0fde90 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36662/0 (socket says 172.21.15.103:36662) 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 -- 172.21.15.103:0/179448094 learned_addr learned my addr 172.21.15.103:0/179448094 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f430affd700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 0x7f430c1138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 -- 172.21.15.103:0/179448094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 msgr2=0x7f430c1138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 0x7f430c1138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 -- 172.21.15.103:0/179448094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc005040 con 0x7f430c0fe830 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f430affd700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 0x7f430c1138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.238+0000 7f43027fc700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c8c0 tx=0x7f42f40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.239+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f400a980 con 0x7f430c0fe830 2024-03-22T02:52:35.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.239+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42f400ce60 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.239+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f40115e0 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.239+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c10ddb0 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.239+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c10e2f0 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.240+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c066a50 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.240+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f42f4012400 con 0x7f430c0fe830 2024-03-22T02:52:35.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.240+0000 7f4303fff700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 0x7f42f8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.241+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f42f408ded0 con 0x7f430c0fe830 2024-03-22T02:52:35.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.241+0000 7f430affd700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 0x7f42f8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.241+0000 7f430affd700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 0x7f42f8070850 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f42fc004560 tx=0x7f42fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.245+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42f405ac80 con 0x7f430c0fe830 2024-03-22T02:52:35.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.484+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f430c109f80 con 0x7f430c0fe830 2024-03-22T02:52:35.486 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.485+0000 7f4303fff700 1 -- 172.21.15.103:0/179448094 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f42f40126b0 con 0x7f430c0fe830 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:52:35.487 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:52:35.488 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:52:35.489 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:35.490 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 msgr2=0x7f42f8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 0x7f42f8070850 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f42fc004560 tx=0x7f42fc009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 msgr2=0x7f430c113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c8c0 tx=0x7f42f40044c0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 shutdown_connections 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f42f806e3a0 0x7f42f8070850 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f430c0fde90 0x7f430c1138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 --2- 172.21.15.103:0/179448094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f430c0fe830 0x7f430c113e30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 >> 172.21.15.103:0/179448094 conn(0x7f430c0f78f0 msgr2=0x7f430c103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 shutdown_connections 2024-03-22T02:52:35.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.487+0000 7f43114ef700 1 -- 172.21.15.103:0/179448094 wait complete. 2024-03-22T02:52:35.492 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1331765753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 msgr2=0x7f2fa40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1331765753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa40fce30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2f98004940 tx=0x7f2f98009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1331765753 shutdown_connections 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1331765753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa40fce30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1331765753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 0x7f2fa41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1331765753 >> 172.21.15.103:0/1331765753 conn(0x7f2fa40f89d0 msgr2=0x7f2fa40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1331765753 shutdown_connections 2024-03-22T02:52:35.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.590+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1331765753 wait complete. 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 Processor -- start 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 -- start start 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 0x7f2fa4115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa4116930 con 0x7f2fa41037c0 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa8caa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa410fd00 con 0x7f2fa4102e20 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36678/0 (socket says 172.21.15.103:36678) 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 -- 172.21.15.103:0/1180678371 learned_addr learned my addr 172.21.15.103:0/1180678371 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa259c700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 0x7f2fa4115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 -- 172.21.15.103:0/1180678371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 msgr2=0x7f2fa4115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 0x7f2fa4115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.591+0000 7f2fa1d9b700 1 -- 172.21.15.103:0/1180678371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98005040 con 0x7f2fa41037c0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2fa1d9b700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009180 tx=0x7f2f98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f9800e040 con 0x7f2fa41037c0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f98013420 con 0x7f2fa41037c0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa410ff80 con 0x7f2fa41037c0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f9801c610 con 0x7f2fa41037c0 2024-03-22T02:52:35.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.592+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa4110540 con 0x7f2fa41037c0 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.593+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4102140 con 0x7f2fa41037c0 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.593+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2f98013a00 con 0x7f2fa41037c0 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.594+0000 7f2f92ffd700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 0x7f2f8c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.594+0000 7f2fa259c700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 0x7f2f8c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.594+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2f9808ef30 con 0x7f2fa41037c0 2024-03-22T02:52:35.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.595+0000 7f2fa259c700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 0x7f2f8c070850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f94004f40 tx=0x7f2f9400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.599+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f9805bd90 con 0x7f2fa41037c0 2024-03-22T02:52:35.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.815+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fa4111460 con 0x7f2f8c06e3a0 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.816+0000 7f2f92ffd700 1 -- 172.21.15.103:0/1180678371 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2fa4111460 con 0x7f2f8c06e3a0 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:52:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:52:35.819 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:52:35.819 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:52:35.819 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:52:35.819 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:52:35.819 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:52:35.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.818+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 msgr2=0x7f2f8c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 0x7f2f8c070850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2f94004f40 tx=0x7f2f9400b040 comp rx=0 tx=0).stop 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 msgr2=0x7f2fa41162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009180 tx=0x7f2f98009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 shutdown_connections 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2f8c06e3a0 0x7f2f8c070850 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2fa4102e20 0x7f2fa4115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 --2- 172.21.15.103:0/1180678371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fa41037c0 0x7f2fa41162e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 >> 172.21.15.103:0/1180678371 conn(0x7f2fa40f89d0 msgr2=0x7f2fa40fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 shutdown_connections 2024-03-22T02:52:35.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.819+0000 7f2fa8caa700 1 -- 172.21.15.103:0/1180678371 wait complete. 2024-03-22T02:52:35.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 -- 172.21.15.103:0/4187766877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56401037a0 msgr2=0x7f56400fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 --2- 172.21.15.103:0/4187766877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56401037a0 0x7f56400fce10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f562c004940 tx=0x7f562c009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:35.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 -- 172.21.15.103:0/4187766877 shutdown_connections 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 --2- 172.21.15.103:0/4187766877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56401037a0 0x7f56400fce10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 --2- 172.21.15.103:0/4187766877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5640102e00 0x7f56401031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 -- 172.21.15.103:0/4187766877 >> 172.21.15.103:0/4187766877 conn(0x7f56400f89b0 msgr2=0x7f56400fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 -- 172.21.15.103:0/4187766877 shutdown_connections 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.922+0000 7f5647263700 1 -- 172.21.15.103:0/4187766877 wait complete. 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 Processor -- start 2024-03-22T02:52:35.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 -- start start 2024-03-22T02:52:35.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 0x7f5640113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401145b0 con 0x7f5640102e00 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5647263700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564010db40 con 0x7f56401037a0 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36696/0 (socket says 172.21.15.103:36696) 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 -- 172.21.15.103:0/2416000382 learned_addr learned my addr 172.21.15.103:0/2416000382 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f563ffff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 0x7f5640113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 -- 172.21.15.103:0/2416000382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 msgr2=0x7f5640113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 0x7f5640113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f5644fff700 1 -- 172.21.15.103:0/2416000382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c005040 con 0x7f5640102e00 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.923+0000 7f563ffff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 0x7f5640113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f5644fff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f563400b790 tx=0x7f5634009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563400f770 con 0x7f5640102e00 2024-03-22T02:52:35.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f563400fda0 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564010de20 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563400d310 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.924+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564010e3e0 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.925+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640100110 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.925+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f563400f8d0 con 0x7f5640102e00 2024-03-22T02:52:35.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.926+0000 7f563d7fa700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 0x7f5630070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:52:35.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.926+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f563408dd00 con 0x7f5640102e00 2024-03-22T02:52:35.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.926+0000 7f563ffff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 0x7f5630070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:52:35.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.926+0000 7f563ffff700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 0x7f5630070850 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f562c00b3b0 tx=0x7f562c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:52:35.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:35.930+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5634091030 con 0x7f5640102e00 2024-03-22T02:52:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:35 smithi122 ceph-mon[83368]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:36.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:35 smithi122 ceph-mon[83368]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:36.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:35 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2157607843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:52:36.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:35 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/179448094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:52:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:35 smithi103 ceph-mon[115211]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:35 smithi103 ceph-mon[115211]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:35 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2157607843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:52:36.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:35 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/179448094' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.216+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5640109f50 con 0x7f5640102e00 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.216+0000 7f563d7fa700 1 -- 172.21.15.103:0/2416000382 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5634057040 con 0x7f5640102e00 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:52:36.218 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 msgr2=0x7f5630070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 0x7f5630070850 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f562c00b3b0 tx=0x7f562c009e30 comp rx=0 tx=0).stop 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 msgr2=0x7f5640113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f563400b790 tx=0x7f5634009250 comp rx=0 tx=0).stop 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 shutdown_connections 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f563006e3a0 0x7f5630070850 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5640102e00 0x7f5640113990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 --2- 172.21.15.103:0/2416000382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f56401037a0 0x7f5640113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:52:36.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 >> 172.21.15.103:0/2416000382 conn(0x7f56400f89b0 msgr2=0x7f56400fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:52:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 shutdown_connections 2024-03-22T02:52:36.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:52:36.218+0000 7f5647263700 1 -- 172.21.15.103:0/2416000382 wait complete. 2024-03-22T02:52:37.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:36 smithi122 ceph-mon[83368]: pgmap v632: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:37.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:36 smithi122 ceph-mon[83368]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:37.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:36 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2416000382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:52:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:36 smithi103 ceph-mon[115211]: pgmap v632: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:36 smithi103 ceph-mon[115211]: from='client.35200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:52:37.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:36 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2416000382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:52:39.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:38 smithi122 ceph-mon[83368]: pgmap v633: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:39.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:38 smithi103 ceph-mon[115211]: pgmap v633: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:41.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:40 smithi122 ceph-mon[83368]: pgmap v634: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:41.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:40 smithi103 ceph-mon[115211]: pgmap v634: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:43.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:42 smithi103 ceph-mon[115211]: pgmap v635: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:42 smithi122 ceph-mon[83368]: pgmap v635: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:45.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:44 smithi103 ceph-mon[115211]: pgmap v636: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:45.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:44 smithi122 ceph-mon[83368]: pgmap v636: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:47.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:46 smithi103 ceph-mon[115211]: pgmap v637: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:46 smithi122 ceph-mon[83368]: pgmap v637: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:52:49.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:48 smithi103 ceph-mon[115211]: pgmap v638: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:49.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:48 smithi122 ceph-mon[83368]: pgmap v638: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:51.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:50 smithi103 ceph-mon[115211]: pgmap v639: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:51.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:50 smithi122 ceph-mon[83368]: pgmap v639: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:53.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:52 smithi103 ceph-mon[115211]: pgmap v640: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:53.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:52 smithi122 ceph-mon[83368]: pgmap v640: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:55.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:54 smithi122 ceph-mon[83368]: pgmap v641: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:55.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:54 smithi103 ceph-mon[115211]: pgmap v641: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:57.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:56 smithi122 ceph-mon[83368]: pgmap v642: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:57.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:56 smithi103 ceph-mon[115211]: pgmap v642: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:59.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:52:58 smithi122 ceph-mon[83368]: pgmap v643: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:52:59.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:52:58 smithi103 ceph-mon[115211]: pgmap v643: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:00 smithi122 ceph-mon[83368]: pgmap v644: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:01.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:00 smithi103 ceph-mon[115211]: pgmap v644: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:03.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:02 smithi122 ceph-mon[83368]: pgmap v645: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:03.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:02 smithi103 ceph-mon[115211]: pgmap v645: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:03.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:05.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:04 smithi122 ceph-mon[83368]: pgmap v646: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:05.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:04 smithi103 ceph-mon[115211]: pgmap v646: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 -- 172.21.15.103:0/668269695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c101d20 msgr2=0x7f349c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 --2- 172.21.15.103:0/668269695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c101d20 0x7f349c1020f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3498004940 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 -- 172.21.15.103:0/668269695 shutdown_connections 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 --2- 172.21.15.103:0/668269695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c1026c0 0x7f349c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 --2- 172.21.15.103:0/668269695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c101d20 0x7f349c1020f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 -- 172.21.15.103:0/668269695 >> 172.21.15.103:0/668269695 conn(0x7f349c0f7870 msgr2=0x7f349c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 -- 172.21.15.103:0/668269695 shutdown_connections 2024-03-22T02:53:06.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 -- 172.21.15.103:0/668269695 wait complete. 2024-03-22T02:53:06.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.342+0000 7f34a3e08700 1 Processor -- start 2024-03-22T02:53:06.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a3e08700 1 -- start start 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a3e08700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c101d20 0x7f349c10eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a3e08700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a3e08700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c10f560 con 0x7f349c1026c0 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a3e08700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c10f6a0 con 0x7f349c101d20 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45640/0 (socket says 172.21.15.103:45640) 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 -- 172.21.15.103:0/1983739193 learned_addr learned my addr 172.21.15.103:0/1983739193 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a1ba4700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c101d20 0x7f349c10eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 -- 172.21.15.103:0/1983739193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c101d20 msgr2=0x7f349c10eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c101d20 0x7f349c10eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 -- 172.21.15.103:0/1983739193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005040 con 0x7f349c1026c0 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.343+0000 7f34a13a3700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f348c00eb40 tx=0x7f348c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:06.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.344+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f348c007cc0 con 0x7f349c1026c0 2024-03-22T02:53:06.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.344+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c10f920 con 0x7f349c1026c0 2024-03-22T02:53:06.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.344+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f348c00f040 con 0x7f349c1026c0 2024-03-22T02:53:06.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.344+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f348c0095e0 con 0x7f349c1026c0 2024-03-22T02:53:06.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.344+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c117dd0 con 0x7f349c1026c0 2024-03-22T02:53:06.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.345+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c0fd470 con 0x7f349c1026c0 2024-03-22T02:53:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.345+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f348c009740 con 0x7f349c1026c0 2024-03-22T02:53:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.345+0000 7f34927fc700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 0x7f3488070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.346+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f348c08d880 con 0x7f349c1026c0 2024-03-22T02:53:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.346+0000 7f34a1ba4700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 0x7f3488070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.346+0000 7f34a1ba4700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 0x7f3488070850 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f349800b0d0 tx=0x7f3498009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:06.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.350+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f348c056b30 con 0x7f349c1026c0 2024-03-22T02:53:06.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.570+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f349c0f8ef0 con 0x7f348806e3a0 2024-03-22T02:53:06.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.571+0000 7f34927fc700 1 -- 172.21.15.103:0/1983739193 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f349c0f8ef0 con 0x7f348806e3a0 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 msgr2=0x7f3488070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 0x7f3488070850 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f349800b0d0 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 msgr2=0x7f349c10f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f348c00eb40 tx=0x7f348c00b040 comp rx=0 tx=0).stop 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 shutdown_connections 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f348806e3a0 0x7f3488070850 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349c101d20 0x7f349c10eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 --2- 172.21.15.103:0/1983739193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f349c1026c0 0x7f349c10f020 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 >> 172.21.15.103:0/1983739193 conn(0x7f349c0f7870 msgr2=0x7f349c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:06.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 shutdown_connections 2024-03-22T02:53:06.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.573+0000 7f34a3e08700 1 -- 172.21.15.103:0/1983739193 wait complete. 2024-03-22T02:53:06.584 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 -- 172.21.15.103:0/3337256076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8640fb4c0 msgr2=0x7fe8640fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3337256076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8640fb4c0 0x7fe8640fb930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe858004940 tx=0x7fe858009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 -- 172.21.15.103:0/3337256076 shutdown_connections 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3337256076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8640fb4c0 0x7fe8640fb930 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3337256076 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe864101520 0x7fe8641018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 -- 172.21.15.103:0/3337256076 >> 172.21.15.103:0/3337256076 conn(0x7fe864072310 msgr2=0x7fe864072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:06.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 -- 172.21.15.103:0/3337256076 shutdown_connections 2024-03-22T02:53:06.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 -- 172.21.15.103:0/3337256076 wait complete. 2024-03-22T02:53:06.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.675+0000 7fe86a63d700 1 Processor -- start 2024-03-22T02:53:06.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe86a63d700 1 -- start start 2024-03-22T02:53:06.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe86a63d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8640fb4c0 0x7fe864119f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe86a63d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe86a63d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86411ab00 con 0x7fe864101520 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe86a63d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641140e0 con 0x7fe8640fb4c0 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45666/0 (socket says 172.21.15.103:45666) 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 -- 172.21.15.103:0/3838393301 learned_addr learned my addr 172.21.15.103:0/3838393301 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe863fff700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8640fb4c0 0x7fe864119f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 -- 172.21.15.103:0/3838393301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8640fb4c0 msgr2=0x7fe864119f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8640fb4c0 0x7fe864119f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 -- 172.21.15.103:0/3838393301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe858005040 con 0x7fe864101520 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.676+0000 7fe8637fe700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe85800b3b0 tx=0x7fe858009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.677+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe85800e040 con 0x7fe864101520 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.677+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe858013420 con 0x7fe864101520 2024-03-22T02:53:06.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.677+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe864114360 con 0x7fe864101520 2024-03-22T02:53:06.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.677+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe85801c630 con 0x7fe864101520 2024-03-22T02:53:06.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.677+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe864114920 con 0x7fe864101520 2024-03-22T02:53:06.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.678+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8640fcc00 con 0x7fe864101520 2024-03-22T02:53:06.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.678+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe858013590 con 0x7fe864101520 2024-03-22T02:53:06.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.679+0000 7fe860ff9700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 0x7fe84c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:06.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.679+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe85808efe0 con 0x7fe864101520 2024-03-22T02:53:06.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.679+0000 7fe863fff700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 0x7fe84c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:06.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.679+0000 7fe863fff700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 0x7fe84c070850 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8540041d0 tx=0x7fe8540043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:06.684 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.683+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe85805bd90 con 0x7fe864101520 2024-03-22T02:53:06.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.898+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe864108e80 con 0x7fe84c06e3a0 2024-03-22T02:53:06.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.900+0000 7fe860ff9700 1 -- 172.21.15.103:0/3838393301 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe864108e80 con 0x7fe84c06e3a0 2024-03-22T02:53:06.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 msgr2=0x7fe84c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 0x7fe84c070850 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8540041d0 tx=0x7fe8540043c0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 msgr2=0x7fe86411a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe85800b3b0 tx=0x7fe858009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 shutdown_connections 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe84c06e3a0 0x7fe84c070850 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8640fb4c0 0x7fe864119f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 --2- 172.21.15.103:0/3838393301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe864101520 0x7fe86411a4b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 >> 172.21.15.103:0/3838393301 conn(0x7fe864072310 msgr2=0x7fe8640fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 shutdown_connections 2024-03-22T02:53:06.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:06.902+0000 7fe86a63d700 1 -- 172.21.15.103:0/3838393301 wait complete. 2024-03-22T02:53:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 -- 172.21.15.103:0/3791225497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 msgr2=0x7ff7c80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/3791225497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c80fce10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc004940 tx=0x7ff7bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 -- 172.21.15.103:0/3791225497 shutdown_connections 2024-03-22T02:53:07.004 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:06 smithi103 ceph-mon[115211]: pgmap v647: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:07.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/3791225497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c80fce10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/3791225497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 0x7ff7c81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 -- 172.21.15.103:0/3791225497 >> 172.21.15.103:0/3791225497 conn(0x7ff7c80f89b0 msgr2=0x7ff7c80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 -- 172.21.15.103:0/3791225497 shutdown_connections 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.002+0000 7ff7cdab1700 1 -- 172.21.15.103:0/3791225497 wait complete. 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 Processor -- start 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 -- start start 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c80722a0 con 0x7ff7c81037a0 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7cdab1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8070d40 con 0x7ff7c8102e00 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7c6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45686/0 (socket says 172.21.15.103:45686) 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.003+0000 7ff7c6ffd700 1 -- 172.21.15.103:0/238452447 learned_addr learned my addr 172.21.15.103:0/238452447 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7c77fe700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7c6ffd700 1 -- 172.21.15.103:0/238452447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 msgr2=0x7ff7c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7c6ffd700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7c6ffd700 1 -- 172.21.15.103:0/238452447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc005040 con 0x7ff7c81037a0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7c6ffd700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009180 tx=0x7ff7bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7bc00e040 con 0x7ff7c81037a0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8070fc0 con 0x7ff7c81037a0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7bc013420 con 0x7ff7c81037a0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c811a1d0 con 0x7ff7c81037a0 2024-03-22T02:53:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.004+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7bc01c610 con 0x7ff7c81037a0 2024-03-22T02:53:07.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.005+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8100110 con 0x7ff7c81037a0 2024-03-22T02:53:07.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.006+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7ff7bc013a00 con 0x7ff7c81037a0 2024-03-22T02:53:07.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.006+0000 7ff7affff700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 0x7ff7b0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.006+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff7bc08ee00 con 0x7ff7c81037a0 2024-03-22T02:53:07.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.006+0000 7ff7c77fe700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 0x7ff7b0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.007+0000 7ff7c77fe700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 0x7ff7b0070850 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009bf0 tx=0x7ff7b80091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.011+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff7bc058140 con 0x7ff7c81037a0 2024-03-22T02:53:07.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.220+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7c8071810 con 0x7ff7b006e3a0 2024-03-22T02:53:07.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.230+0000 7ff7affff700 1 -- 172.21.15.103:0/238452447 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff7c8071810 con 0x7ff7b006e3a0 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (21m) 15m ago 32m 25.9M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (19m) 15m ago 33m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (19m) 14m ago 32m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (21m) 15m ago 32m 80.2M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (15m) 15m ago 27m 12.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (15m) 15m ago 27m 11.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (14m) 14m ago 27m 13.9M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (15m) 14m ago 27m 14.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (23m) 15m ago 34m 509M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (22m) 14m ago 31m 451M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:53:07.231 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (20m) 15m ago 34m 52.2M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (20m) 14m ago 31m 45.3M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (22m) 15m ago 32m 15.4M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (22m) 14m ago 31m 18.4M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (18m) 15m ago 31m 91.2M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (18m) 15m ago 30m 102M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (17m) 15m ago 30m 83.0M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (17m) 14m ago 29m 98.9M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (16m) 14m ago 29m 91.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (15m) 14m ago 28m 85.7M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:53:07.232 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (21m) 15m ago 32m 69.3M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 msgr2=0x7ff7b0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 0x7ff7b0070850 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009bf0 tx=0x7ff7b80091a0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 msgr2=0x7ff7c8070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009180 tx=0x7ff7bc009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 shutdown_connections 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7ff7b006e3a0 0x7ff7b0070850 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7c8102e00 0x7ff7c80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 --2- 172.21.15.103:0/238452447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7c81037a0 0x7ff7c8070800 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 >> 172.21.15.103:0/238452447 conn(0x7ff7c80f89b0 msgr2=0x7ff7c80fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 shutdown_connections 2024-03-22T02:53:07.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.232+0000 7ff7cdab1700 1 -- 172.21.15.103:0/238452447 wait complete. 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 -- 172.21.15.103:0/2710689368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68101e60 msgr2=0x7f7f68102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/2710689368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68101e60 0x7f7f68102230 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004940 tx=0x7f7f58009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 -- 172.21.15.103:0/2710689368 shutdown_connections 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/2710689368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68102800 0x7f7f680fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/2710689368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68101e60 0x7f7f68102230 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 -- 172.21.15.103:0/2710689368 >> 172.21.15.103:0/2710689368 conn(0x7f7f68071530 msgr2=0x7f7f68071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 -- 172.21.15.103:0/2710689368 shutdown_connections 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.332+0000 7f7f6d07d700 1 -- 172.21.15.103:0/2710689368 wait complete. 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 Processor -- start 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 -- start start 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 0x7f7f6810eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6810f530 con 0x7f7f68102800 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6d07d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6810f670 con 0x7f7f68101e60 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6659c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45702/0 (socket says 172.21.15.103:45702) 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f6659c700 1 -- 172.21.15.103:0/24642434 learned_addr learned my addr 172.21.15.103:0/24642434 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.333+0000 7f7f66d9d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 0x7f7f6810eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6659c700 1 -- 172.21.15.103:0/24642434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 msgr2=0x7f7f6810eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6659c700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 0x7f7f6810eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6659c700 1 -- 172.21.15.103:0/24642434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58005040 con 0x7f7f68102800 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f66d9d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 0x7f7f6810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6659c700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c002a50 tx=0x7f7f5c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c007cc0 con 0x7f7f68102800 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f5c00f040 con 0x7f7f68102800 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f68117b20 con 0x7f7f68102800 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5c00d9c0 con 0x7f7f68102800 2024-03-22T02:53:07.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.334+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f68117f40 con 0x7f7f68102800 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.335+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f680ff1c0 con 0x7f7f68102800 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.336+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7f5c009730 con 0x7f7f68102800 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.336+0000 7f7f4f7fe700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 0x7f7f50070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.336+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f5c08da60 con 0x7f7f68102800 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.336+0000 7f7f66d9d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 0x7f7f50070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.337+0000 7f7f66d9d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 0x7f7f50070850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004560 tx=0x7f7f58009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.342+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f5c056c60 con 0x7f7f68102800 2024-03-22T02:53:07.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:06 smithi122 ceph-mon[83368]: pgmap v647: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:07.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.621+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7f680fbb40 con 0x7f7f68102800 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.622+0000 7f7f4f7fe700 1 -- 172.21.15.103:0/24642434 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7f5c05a320 con 0x7f7f68102800 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:53:07.623 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:53:07.624 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:53:07.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 msgr2=0x7f7f50070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 0x7f7f50070850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004560 tx=0x7f7f58009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 msgr2=0x7f7f6810eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c002a50 tx=0x7f7f5c004600 comp rx=0 tx=0).stop 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 shutdown_connections 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7f5006e3a0 0x7f7f50070850 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f68101e60 0x7f7f6810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 --2- 172.21.15.103:0/24642434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102800 0x7f7f6810eff0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.624+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 >> 172.21.15.103:0/24642434 conn(0x7f7f68071530 msgr2=0x7f7f680fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.625+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 shutdown_connections 2024-03-22T02:53:07.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.625+0000 7f7f6d07d700 1 -- 172.21.15.103:0/24642434 wait complete. 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 -- 172.21.15.103:0/2498973504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 msgr2=0x7f6f100fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/2498973504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f100fb990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c004940 tx=0x7f6f0c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 -- 172.21.15.103:0/2498973504 shutdown_connections 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/2498973504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f100fb990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/2498973504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 0x7f6f101033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 -- 172.21.15.103:0/2498973504 >> 172.21.15.103:0/2498973504 conn(0x7f6f10071530 msgr2=0x7f6f10071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 -- 172.21.15.103:0/2498973504 shutdown_connections 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.725+0000 7f6f17dcd700 1 -- 172.21.15.103:0/2498973504 wait complete. 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 Processor -- start 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 -- start start 2024-03-22T02:53:07.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 0x7f6f10113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f101144e0 con 0x7f6f100fb540 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f17dcd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1010dae0 con 0x7f6f10102fd0 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45710/0 (socket says 172.21.15.103:45710) 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 -- 172.21.15.103:0/3391966075 learned_addr learned my addr 172.21.15.103:0/3391966075 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15368700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 0x7f6f10113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 -- 172.21.15.103:0/3391966075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 msgr2=0x7f6f10113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 0x7f6f10113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 -- 172.21.15.103:0/3391966075 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c005040 con 0x7f6f100fb540 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.726+0000 7f6f15b69700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c8c0 tx=0x7f6f000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.727+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f0000a980 con 0x7f6f100fb540 2024-03-22T02:53:07.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.727+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f0000ce60 con 0x7f6f100fb540 2024-03-22T02:53:07.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.727+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1010ddc0 con 0x7f6f100fb540 2024-03-22T02:53:07.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.727+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f000115e0 con 0x7f6f100fb540 2024-03-22T02:53:07.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.727+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1010e380 con 0x7f6f100fb540 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.728+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f101053b0 con 0x7f6f100fb540 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.728+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6f00012400 con 0x7f6f100fb540 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.729+0000 7f6f067fc700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 0x7f6efc070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.729+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6f0008ded0 con 0x7f6f100fb540 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.729+0000 7f6f15368700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 0x7f6efc070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:07.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.729+0000 7f6f15368700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 0x7f6efc070850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c000c00 tx=0x7f6f0c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:07.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.733+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f0005ac80 con 0x7f6f100fb540 2024-03-22T02:53:07.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.977+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6f1010efa0 con 0x7f6f100fb540 2024-03-22T02:53:07.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.978+0000 7f6f067fc700 1 -- 172.21.15.103:0/3391966075 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1881 (secure 0 0 0) 0x7f6f000126b0 con 0x7f6f100fb540 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:e31 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 22 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:37:16.024479+0000 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:53:07.979 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:up {} 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:failed 0,1 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:53:07.980 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 0 members: 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:53:07.981 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:53:07.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 msgr2=0x7f6efc070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 0x7f6efc070850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c000c00 tx=0x7f6f0c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 msgr2=0x7f6f10113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c8c0 tx=0x7f6f000044c0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 shutdown_connections 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6efc06e3a0 0x7f6efc070850 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f100fb540 0x7f6f10113950 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 --2- 172.21.15.103:0/3391966075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f10102fd0 0x7f6f10113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 >> 172.21.15.103:0/3391966075 conn(0x7f6f10071530 msgr2=0x7f6f100fa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 shutdown_connections 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:07.980+0000 7f6f17dcd700 1 -- 172.21.15.103:0/3391966075 wait complete. 2024-03-22T02:53:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 -- 172.21.15.103:0/1537347744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fde20 msgr2=0x7f241c0fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 --2- 172.21.15.103:0/1537347744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fde20 0x7f241c0fe1f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2418004940 tx=0x7f2418009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 -- 172.21.15.103:0/1537347744 shutdown_connections 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 --2- 172.21.15.103:0/1537347744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fe7c0 0x7f241c106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 --2- 172.21.15.103:0/1537347744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fde20 0x7f241c0fe1f0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.082+0000 7f242371c700 1 -- 172.21.15.103:0/1537347744 >> 172.21.15.103:0/1537347744 conn(0x7f241c0f7870 msgr2=0x7f241c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 -- 172.21.15.103:0/1537347744 shutdown_connections 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 -- 172.21.15.103:0/1537347744 wait complete. 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 Processor -- start 2024-03-22T02:53:08.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 -- start start 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 0x7f241c113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1144f0 con 0x7f241c0fe7c0 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f242371c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c10daf0 con 0x7f241c0fde20 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f2420cb7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f2420cb7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45724/0 (socket says 172.21.15.103:45724) 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.083+0000 7f2420cb7700 1 -- 172.21.15.103:0/256128060 learned_addr learned my addr 172.21.15.103:0/256128060 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f24214b8700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 0x7f241c113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2420cb7700 1 -- 172.21.15.103:0/256128060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 msgr2=0x7f241c113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2420cb7700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 0x7f241c113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2420cb7700 1 -- 172.21.15.103:0/256128060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418005040 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f24214b8700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 0x7f241c113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2420cb7700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f240c002a50 tx=0x7f240c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240c007cc0 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f240c00f040 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240c009640 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c10ddd0 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.084+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c10e310 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.085+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c066b50 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.086+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f240c0097a0 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.086+0000 7f2411ffb700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 0x7f2408070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.086+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f240c08dc50 con 0x7f241c0fe7c0 2024-03-22T02:53:08.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.086+0000 7f24214b8700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 0x7f2408070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.087+0000 7f24214b8700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 0x7f2408070850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:08.091 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.091+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f240c05a900 con 0x7f241c0fe7c0 2024-03-22T02:53:08.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.308+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f241c113630 con 0x7f240806e3a0 2024-03-22T02:53:08.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:07 smithi103 ceph-mon[115211]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:08.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:07 smithi103 ceph-mon[115211]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:08.310 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:07 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/24642434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:08.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.310+0000 7f2411ffb700 1 -- 172.21.15.103:0/256128060 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f241c113630 con 0x7f240806e3a0 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": true, 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [ 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "osd", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "mds", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "mon" 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "Currently upgrading mds daemons", 2024-03-22T02:53:08.311 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:53:08.312 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 msgr2=0x7f2408070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 0x7f2408070850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2418000c00 tx=0x7f2418009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 msgr2=0x7f241c113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f240c002a50 tx=0x7f240c0046b0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 shutdown_connections 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f240806e3a0 0x7f2408070850 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f241c0fde20 0x7f241c113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 --2- 172.21.15.103:0/256128060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f241c0fe7c0 0x7f241c113ea0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 >> 172.21.15.103:0/256128060 conn(0x7f241c0f7870 msgr2=0x7f241c0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:08.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 shutdown_connections 2024-03-22T02:53:08.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.312+0000 7f242371c700 1 -- 172.21.15.103:0/256128060 wait complete. 2024-03-22T02:53:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:07 smithi122 ceph-mon[83368]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:07 smithi122 ceph-mon[83368]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:07 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/24642434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- 172.21.15.103:0/3613999183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fde50 msgr2=0x7f2bb40fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/3613999183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fde50 0x7f2bb40fe220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2bac004940 tx=0x7f2bac009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- 172.21.15.103:0/3613999183 shutdown_connections 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/3613999183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/3613999183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fde50 0x7f2bb40fe220 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- 172.21.15.103:0/3613999183 >> 172.21.15.103:0/3613999183 conn(0x7f2bb40f7870 msgr2=0x7f2bb40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- 172.21.15.103:0/3613999183 shutdown_connections 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- 172.21.15.103:0/3613999183 wait complete. 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 Processor -- start 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.413+0000 7f2bb8d86700 1 -- start start 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fde50 0x7f2bb4113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4114630 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb410dc10 con 0x7f2bb40fde50 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45734/0 (socket says 172.21.15.103:45734) 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 -- 172.21.15.103:0/1159948641 learned_addr learned my addr 172.21.15.103:0/1159948641 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb259c700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fde50 0x7f2bb4113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 -- 172.21.15.103:0/1159948641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fde50 msgr2=0x7f2bb4113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fde50 0x7f2bb4113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 -- 172.21.15.103:0/1159948641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac005040 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb1d9b700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800cb10 tx=0x7f2ba8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ba8004200 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ba800ac70 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ba8003d30 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb410def0 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.414+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb410e430 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.415+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb4066a90 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.416+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2ba800a7a0 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.416+0000 7f2ba2ffd700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 0x7f2b9c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.416+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2ba808e2a0 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.416+0000 7f2bb259c700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 0x7f2b9c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:08.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.417+0000 7f2bb259c700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 0x7f2b9c070850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b3b0 tx=0x7f2bac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:08.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.421+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ba805afd0 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.703+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2bb404c710 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.703+0000 7f2ba2ffd700 1 -- 172.21.15.103:0/1159948641 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2ba8004360 con 0x7f2bb40fe7f0 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is degraded 2024-03-22T02:53:08.704 INFO:teuthology.orchestra.run.smithi103.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-22T02:53:08.705 INFO:teuthology.orchestra.run.smithi103.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 msgr2=0x7f2b9c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 0x7f2b9c070850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b3b0 tx=0x7f2bac009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 msgr2=0x7f2bb4113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800cb10 tx=0x7f2ba8007f90 comp rx=0 tx=0).stop 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 shutdown_connections 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2b9c06e3a0 0x7f2b9c070850 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bb40fde50 0x7f2bb4113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 --2- 172.21.15.103:0/1159948641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2bb40fe7f0 0x7f2bb4113fe0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 >> 172.21.15.103:0/1159948641 conn(0x7f2bb40f7870 msgr2=0x7f2bb40f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 shutdown_connections 2024-03-22T02:53:08.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:08.706+0000 7f2bb8d86700 1 -- 172.21.15.103:0/1159948641 wait complete. 2024-03-22T02:53:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:08 smithi103 ceph-mon[115211]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:08 smithi103 ceph-mon[115211]: pgmap v648: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:08 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3391966075' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:53:09.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:08 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1159948641' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:53:09.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:08 smithi122 ceph-mon[83368]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:09.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:08 smithi122 ceph-mon[83368]: pgmap v648: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:09.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:08 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3391966075' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:53:09.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:08 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1159948641' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:53:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:09 smithi122 ceph-mon[83368]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:10.440 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:09 smithi103 ceph-mon[115211]: from='client.35228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:11.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:10 smithi122 ceph-mon[83368]: pgmap v649: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:11.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:10 smithi103 ceph-mon[115211]: pgmap v649: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:12 smithi122 ceph-mon[83368]: pgmap v650: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:12 smithi103 ceph-mon[115211]: pgmap v650: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: pgmap v651: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.gtuatp"}]': finished 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.xhmfjp"}]': finished 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lvsegg"}]': finished 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.ztrkgp"}]': finished 2024-03-22T02:53:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:14 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-22T02:53:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: pgmap v651: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.gtuatp"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.gtuatp"}]': finished 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.xhmfjp"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi103.xhmfjp"}]': finished 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lvsegg"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lvsegg"}]': finished 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.ztrkgp"}]: dispatch 2024-03-22T02:53:15.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.ztrkgp"}]': finished 2024-03-22T02:53:15.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:14 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: pgmap v652: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all mds 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Upgrade: Setting filesystem cephfs Joinable 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-22T02:53:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Standby daemon mds.cephfs.smithi103.gtuatp assigned to filesystem cephfs as rank 0 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Standby daemon mds.cephfs.smithi103.xhmfjp assigned to filesystem cephfs as rank 1 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:replay,1=cephfs.smithi103.xhmfjp=up:replay} 2 up:standby 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.351 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T02:53:16.352 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.353 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T02:53:16.354 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:15 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: pgmap v652: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all mds 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Upgrade: Setting filesystem cephfs Joinable 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Standby daemon mds.cephfs.smithi103.gtuatp assigned to filesystem cephfs as rank 0 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Standby daemon mds.cephfs.smithi103.xhmfjp assigned to filesystem cephfs as rank 1 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:53:16.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:replay,1=cephfs.smithi103.xhmfjp=up:replay} 2 up:standby 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.377 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T02:53:16.378 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T02:53:16.379 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T02:53:16.380 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:15 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:17.246 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all rgw 2024-03-22T02:53:17.246 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all iscsi 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all nfs 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Setting container_image for all nvmeof 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Finalizing container_image settings 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: Upgrade: Complete! 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.247 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all rgw 2024-03-22T02:53:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all iscsi 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all nfs 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Setting container_image for all nvmeof 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Finalizing container_image settings 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: Upgrade: Complete! 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:17 smithi122 ceph-mon[83368]: pgmap v653: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s rd, 1 op/s 2024-03-22T02:53:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:18.298 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:17 smithi103 ceph-mon[115211]: pgmap v653: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s rd, 1 op/s 2024-03-22T02:53:18.298 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:19 smithi103 ceph-mon[115211]: pgmap v654: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 5.0 MiB/s rd, 1 op/s 2024-03-22T02:53:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:19 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:19 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:19 smithi122 ceph-mon[83368]: pgmap v654: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 5.0 MiB/s rd, 1 op/s 2024-03-22T02:53:20.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:19 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:20.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:19 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:21.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:21 smithi103 ceph-mon[115211]: pgmap v655: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 5 op/s 2024-03-22T02:53:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:21 smithi122 ceph-mon[83368]: pgmap v655: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 5 op/s 2024-03-22T02:53:23.616 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:23 smithi103 ceph-mon[115211]: pgmap v656: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 7 op/s 2024-03-22T02:53:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:23 smithi122 ceph-mon[83368]: pgmap v656: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 7 op/s 2024-03-22T02:53:24.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:resolve 2024-03-22T02:53:24.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:resolve 2024-03-22T02:53:24.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:resolve,1=cephfs.smithi103.xhmfjp=up:resolve} 2 up:standby 2024-03-22T02:53:24.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:24.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:resolve 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:resolve 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:resolve,1=cephfs.smithi103.xhmfjp=up:resolve} 2 up:standby 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:53:25.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: pgmap v657: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: reconnect by client.14526 192.168.0.1:0/4037363279 after 0.000999981 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: reconnect by client.14526 192.168.0.1:0/4037363279 after 0.000999981 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: reconnect by client.24329 192.168.0.1:0/1177476251 after 0.000999981 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: reconnect by client.24329 192.168.0.1:0/1177476251 after 0.000999981 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:reconnect 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:reconnect 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:reconnect,1=cephfs.smithi103.xhmfjp=up:reconnect} 2 up:standby 2024-03-22T02:53:26.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:25 smithi122 ceph-mon[83368]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-22T02:53:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: pgmap v657: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-03-22T02:53:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: reconnect by client.14526 192.168.0.1:0/4037363279 after 0.000999981 2024-03-22T02:53:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: reconnect by client.14526 192.168.0.1:0/4037363279 after 0.000999981 2024-03-22T02:53:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: reconnect by client.24329 192.168.0.1:0/1177476251 after 0.000999981 2024-03-22T02:53:26.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: reconnect by client.24329 192.168.0.1:0/1177476251 after 0.000999981 2024-03-22T02:53:26.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:reconnect 2024-03-22T02:53:26.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:reconnect 2024-03-22T02:53:26.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:reconnect,1=cephfs.smithi103.xhmfjp=up:reconnect} 2 up:standby 2024-03-22T02:53:26.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:25 smithi103 ceph-mon[115211]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-22T02:53:27.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:26 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:rejoin 2024-03-22T02:53:27.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:26 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:rejoin 2024-03-22T02:53:27.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:26 smithi122 ceph-mon[83368]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:rejoin,1=cephfs.smithi103.xhmfjp=up:rejoin} 2 up:standby 2024-03-22T02:53:27.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:26 smithi122 ceph-mon[83368]: daemon mds.cephfs.smithi103.gtuatp is now active in filesystem cephfs as rank 0 2024-03-22T02:53:27.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:26 smithi122 ceph-mon[83368]: daemon mds.cephfs.smithi103.xhmfjp is now active in filesystem cephfs as rank 1 2024-03-22T02:53:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:26 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:rejoin 2024-03-22T02:53:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:26 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:rejoin 2024-03-22T02:53:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:26 smithi103 ceph-mon[115211]: fsmap cephfs:2/2 {0=cephfs.smithi103.gtuatp=up:rejoin,1=cephfs.smithi103.xhmfjp=up:rejoin} 2 up:standby 2024-03-22T02:53:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:26 smithi103 ceph-mon[115211]: daemon mds.cephfs.smithi103.gtuatp is now active in filesystem cephfs as rank 0 2024-03-22T02:53:27.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:26 smithi103 ceph-mon[115211]: daemon mds.cephfs.smithi103.xhmfjp is now active in filesystem cephfs as rank 1 2024-03-22T02:53:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: pgmap v658: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 86 B/s wr, 7 op/s 2024-03-22T02:53:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:53:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: Cluster is now healthy 2024-03-22T02:53:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:active 2024-03-22T02:53:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:active 2024-03-22T02:53:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:27 smithi122 ceph-mon[83368]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi103.xhmfjp=up:active} 2 up:standby 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: pgmap v658: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 86 B/s wr, 7 op/s 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: Cluster is now healthy 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] up:active 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: mds.? [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] up:active 2024-03-22T02:53:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:27 smithi103 ceph-mon[115211]: fsmap cephfs:2 {0=cephfs.smithi103.gtuatp=up:active,1=cephfs.smithi103.xhmfjp=up:active} 2 up:standby 2024-03-22T02:53:30.032 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:29 smithi103 ceph-mon[115211]: pgmap v659: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 86 B/s wr, 6 op/s 2024-03-22T02:53:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:29 smithi122 ceph-mon[83368]: pgmap v659: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 86 B/s wr, 6 op/s 2024-03-22T02:53:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:31 smithi122 ceph-mon[83368]: pgmap v660: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-03-22T02:53:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:31 smithi103 ceph-mon[115211]: pgmap v660: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 17 MiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-03-22T02:53:33.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:33.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:53:33.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:34.250 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:33 smithi122 ceph-mon[83368]: pgmap v661: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 7.3 MiB/s rd, 4.9 KiB/s wr, 10 op/s 2024-03-22T02:53:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:33 smithi103 ceph-mon[115211]: pgmap v661: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 7.3 MiB/s rd, 4.9 KiB/s wr, 10 op/s 2024-03-22T02:53:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:35 smithi122 ceph-mon[83368]: pgmap v662: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-03-22T02:53:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:35 smithi103 ceph-mon[115211]: pgmap v662: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.9 KiB/s wr, 9 op/s 2024-03-22T02:53:38.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:37 smithi122 ceph-mon[83368]: pgmap v663: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-03-22T02:53:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:37 smithi103 ceph-mon[115211]: pgmap v663: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-03-22T02:53:38.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.811+0000 7fd648053700 1 -- 172.21.15.103:0/544424100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 msgr2=0x7fd6400fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.811+0000 7fd648053700 1 --2- 172.21.15.103:0/544424100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fe2a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd63c004940 tx=0x7fd63c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- 172.21.15.103:0/544424100 shutdown_connections 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 --2- 172.21.15.103:0/544424100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 0x7fd640106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 --2- 172.21.15.103:0/544424100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fe2a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- 172.21.15.103:0/544424100 >> 172.21.15.103:0/544424100 conn(0x7fd640072bc0 msgr2=0x7fd640072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- 172.21.15.103:0/544424100 shutdown_connections 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- 172.21.15.103:0/544424100 wait complete. 2024-03-22T02:53:38.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 Processor -- start 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- start start 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 0x7fd6400fc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400fc8a0 con 0x7fd6400fded0 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd648053700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6400fc9e0 con 0x7fd6400fe870 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.812+0000 7fd645def700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44718/0 (socket says 172.21.15.103:44718) 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 -- 172.21.15.103:0/2056449802 learned_addr learned my addr 172.21.15.103:0/2056449802 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd6455ee700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 0x7fd6400fc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 -- 172.21.15.103:0/2056449802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 msgr2=0x7fd6400fc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 0x7fd6400fc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 -- 172.21.15.103:0/2056449802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c005040 con 0x7fd6400fded0 2024-03-22T02:53:38.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd645def700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009380 tx=0x7fd63c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63c00e040 con 0x7fd6400fded0 2024-03-22T02:53:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd63c009af0 con 0x7fd6400fded0 2024-03-22T02:53:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd640119e20 con 0x7fd6400fded0 2024-03-22T02:53:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63c01c3f0 con 0x7fd6400fded0 2024-03-22T02:53:38.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.813+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64011a280 con 0x7fd6400fded0 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.814+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd640065df0 con 0x7fd6400fded0 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.815+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fd63c013420 con 0x7fd6400fded0 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.816+0000 7fd6367fc700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 0x7fd62c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.816+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd63c08fbe0 con 0x7fd6400fded0 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.816+0000 7fd6455ee700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 0x7fd62c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:38.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.816+0000 7fd6455ee700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 0x7fd62c070850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6400fd680 tx=0x7fd6300091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:38.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:38.823+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd63c058e60 con 0x7fd6400fded0 2024-03-22T02:53:39.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.046+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6400fd430 con 0x7fd62c06e3a0 2024-03-22T02:53:39.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.047+0000 7fd6367fc700 1 -- 172.21.15.103:0/2056449802 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fd6400fd430 con 0x7fd62c06e3a0 2024-03-22T02:53:39.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 msgr2=0x7fd62c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:39.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 0x7fd62c070850 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6400fd680 tx=0x7fd6300091a0 comp rx=0 tx=0).stop 2024-03-22T02:53:39.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 msgr2=0x7fd6400fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:39.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009380 tx=0x7fd63c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 shutdown_connections 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fd62c06e3a0 0x7fd62c070850 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6400fded0 0x7fd6400fbd90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 --2- 172.21.15.103:0/2056449802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6400fe870 0x7fd6400fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 >> 172.21.15.103:0/2056449802 conn(0x7fd640072bc0 msgr2=0x7fd6401037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 shutdown_connections 2024-03-22T02:53:39.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:39.049+0000 7fd648053700 1 -- 172.21.15.103:0/2056449802 wait complete. 2024-03-22T02:53:39.741 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch ps' 2024-03-22T02:53:40.069 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:39 smithi103 ceph-mon[115211]: pgmap v664: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 8.7 KiB/s wr, 10 op/s 2024-03-22T02:53:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:39 smithi103 ceph-mon[115211]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:39 smithi122 ceph-mon[83368]: pgmap v664: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 8.7 KiB/s wr, 10 op/s 2024-03-22T02:53:40.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:39 smithi122 ceph-mon[83368]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- 172.21.15.103:0/3958366345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fde90 msgr2=0x7efd440fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 --2- 172.21.15.103:0/3958366345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fde90 0x7efd440fe260 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efd34004940 tx=0x7efd34009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- 172.21.15.103:0/3958366345 shutdown_connections 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 --2- 172.21.15.103:0/3958366345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fe830 0x7efd44106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 --2- 172.21.15.103:0/3958366345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fde90 0x7efd440fe260 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- 172.21.15.103:0/3958366345 >> 172.21.15.103:0/3958366345 conn(0x7efd44072bc0 msgr2=0x7efd44072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- 172.21.15.103:0/3958366345 shutdown_connections 2024-03-22T02:53:41.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- 172.21.15.103:0/3958366345 wait complete. 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 Processor -- start 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.671+0000 7efd4955b700 1 -- start start 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd4955b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fde90 0x7efd44113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd4955b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd4955b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd44114720 con 0x7efd440fe830 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd4955b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4410db30 con 0x7efd440fde90 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44744/0 (socket says 172.21.15.103:44744) 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 -- 172.21.15.103:0/360868545 learned_addr learned my addr 172.21.15.103:0/360868545 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:41.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd42ffd700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fde90 0x7efd44113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 -- 172.21.15.103:0/360868545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fde90 msgr2=0x7efd44113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fde90 0x7efd44113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 -- 172.21.15.103:0/360868545 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd34005040 con 0x7efd440fe830 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd427fc700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c8c0 tx=0x7efd380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.672+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3800a760 con 0x7efd440fe830 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.673+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd3800ad90 con 0x7efd440fe830 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.673+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4410de10 con 0x7efd440fe830 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.673+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd380113a0 con 0x7efd440fe830 2024-03-22T02:53:41.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.673+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4410e350 con 0x7efd440fe830 2024-03-22T02:53:41.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.673+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd44101980 con 0x7efd440fe830 2024-03-22T02:53:41.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.674+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7efd38012400 con 0x7efd440fe830 2024-03-22T02:53:41.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.674+0000 7efd2b7fe700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 0x7efd2c070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:41.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.675+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7efd3808dde0 con 0x7efd440fe830 2024-03-22T02:53:41.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.675+0000 7efd42ffd700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 0x7efd2c070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:41.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.675+0000 7efd42ffd700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 0x7efd2c070850 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b3b0 tx=0x7efd34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:41.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.680+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd38057060 con 0x7efd440fe830 2024-03-22T02:53:41.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.897+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efd4410f1e0 con 0x7efd2c06e3a0 2024-03-22T02:53:41.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.907+0000 7efd2b7fe700 1 -- 172.21.15.103:0/360868545 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7efd4410f1e0 con 0x7efd2c06e3a0 2024-03-22T02:53:41.907 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (22m) 18s ago 33m 25.1M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (19m) 18s ago 33m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (19m) 23s ago 32m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (21m) 18s ago 33m 79.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (16m) 18s ago 27m 98.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (16m) 18s ago 27m 104M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (15m) 23s ago 27m 19.4M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (15m) 23s ago 27m 19.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (23m) 18s ago 34m 512M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (23m) 23s ago 32m 457M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (21m) 18s ago 35m 70.1M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (20m) 23s ago 32m 52.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:53:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (22m) 18s ago 33m 15.6M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (22m) 23s ago 32m 19.0M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (19m) 18s ago 31m 96.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (18m) 18s ago 31m 112M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (18m) 18s ago 30m 88.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (17m) 23s ago 30m 110M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (17m) 23s ago 29m 98.5M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (16m) 23s ago 29m 94.5M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:53:41.909 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (22m) 18s ago 32m 73.8M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 msgr2=0x7efd2c070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 0x7efd2c070850 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd3400b3b0 tx=0x7efd34009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 msgr2=0x7efd441140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efd3800c8c0 tx=0x7efd380044c0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 shutdown_connections 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7efd2c06e3a0 0x7efd2c070850 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efd440fde90 0x7efd44113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 --2- 172.21.15.103:0/360868545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd440fe830 0x7efd441140d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 >> 172.21.15.103:0/360868545 conn(0x7efd44072bc0 msgr2=0x7efd440ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 shutdown_connections 2024-03-22T02:53:41.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:41.909+0000 7efd4955b700 1 -- 172.21.15.103:0/360868545 wait complete. 2024-03-22T02:53:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:41 smithi122 ceph-mon[83368]: pgmap v665: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 5.3 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-03-22T02:53:42.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:41 smithi103 ceph-mon[115211]: pgmap v665: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 5.3 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-03-22T02:53:42.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch upgrade status' 2024-03-22T02:53:42.860 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:43.120 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:42 smithi103 ceph-mon[115211]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:43.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:42 smithi122 ceph-mon[83368]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:44.010 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:43 smithi103 ceph-mon[115211]: pgmap v666: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-22T02:53:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:43 smithi122 ceph-mon[83368]: pgmap v666: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 -- 172.21.15.103:0/475870289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fded0 msgr2=0x7f0a600fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/475870289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fded0 0x7f0a600fe2a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0a54004940 tx=0x7f0a54009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 -- 172.21.15.103:0/475870289 shutdown_connections 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/475870289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fe870 0x7f0a60106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/475870289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fded0 0x7f0a600fe2a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 -- 172.21.15.103:0/475870289 >> 172.21.15.103:0/475870289 conn(0x7f0a60072bc0 msgr2=0x7f0a60072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 -- 172.21.15.103:0/475870289 shutdown_connections 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.590+0000 7f0a65cf5700 1 -- 172.21.15.103:0/475870289 wait complete. 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 Processor -- start 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 -- start start 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fded0 0x7f0a60113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:44.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60114600 con 0x7f0a600fe870 2024-03-22T02:53:44.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a65cf5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6010da40 con 0x7f0a600fded0 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a5effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a5effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59578/0 (socket says 172.21.15.103:59578) 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a5effd700 1 -- 172.21.15.103:0/3192954616 learned_addr learned my addr 172.21.15.103:0/3192954616 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.591+0000 7f0a5f7fe700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fded0 0x7f0a60113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a5effd700 1 -- 172.21.15.103:0/3192954616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fded0 msgr2=0x7f0a60113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a5effd700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fded0 0x7f0a60113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a5effd700 1 -- 172.21.15.103:0/3192954616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a54005040 con 0x7f0a600fe870 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a5effd700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000eb70 tx=0x7f0a5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a50007cc0 con 0x7f0a600fe870 2024-03-22T02:53:44.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5000f040 con 0x7f0a600fe870 2024-03-22T02:53:44.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.592+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a50009640 con 0x7f0a600fe870 2024-03-22T02:53:44.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.593+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6010dcc0 con 0x7f0a600fe870 2024-03-22T02:53:44.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.593+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6010e180 con 0x7f0a600fe870 2024-03-22T02:53:44.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.594+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60065df0 con 0x7f0a600fe870 2024-03-22T02:53:44.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.595+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f0a500097a0 con 0x7f0a600fe870 2024-03-22T02:53:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.595+0000 7f0a47fff700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 0x7f0a48070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.595+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0a5008d750 con 0x7f0a600fe870 2024-03-22T02:53:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.595+0000 7f0a5f7fe700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 0x7f0a48070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.596+0000 7f0a5f7fe700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 0x7f0a48070850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b0d0 tx=0x7f0a54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.600+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a5005a480 con 0x7f0a600fe870 2024-03-22T02:53:44.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.915+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a60113740 con 0x7f0a4806e3a0 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.916+0000 7f0a47fff700 1 -- 172.21.15.103:0/3192954616 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0a60113740 con 0x7f0a4806e3a0 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "target_image": null, 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "in_progress": false, 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "which": "", 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "services_complete": [], 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "progress": null, 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "message": "", 2024-03-22T02:53:44.918 INFO:teuthology.orchestra.run.smithi103.stdout: "is_paused": false 2024-03-22T02:53:44.919 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:53:44.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 msgr2=0x7f0a48070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:44.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 0x7f0a48070850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400b0d0 tx=0x7f0a54009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:44.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 msgr2=0x7f0a60113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:44.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000eb70 tx=0x7f0a5000b040 comp rx=0 tx=0).stop 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 shutdown_connections 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f0a4806e3a0 0x7f0a48070850 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a600fded0 0x7f0a60113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 --2- 172.21.15.103:0/3192954616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a600fe870 0x7f0a60113fb0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 >> 172.21.15.103:0/3192954616 conn(0x7f0a60072bc0 msgr2=0x7f0a601037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 shutdown_connections 2024-03-22T02:53:44.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:44.918+0000 7f0a65cf5700 1 -- 172.21.15.103:0/3192954616 wait complete. 2024-03-22T02:53:45.576 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph health detail' 2024-03-22T02:53:45.858 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:46.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:45 smithi103 ceph-mon[115211]: pgmap v667: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s wr, 1 op/s 2024-03-22T02:53:46.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:45 smithi103 ceph-mon[115211]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:45 smithi122 ceph-mon[83368]: pgmap v667: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s wr, 1 op/s 2024-03-22T02:53:46.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:45 smithi122 ceph-mon[83368]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:53:47.183 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:46 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:46 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:53:47.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 -- 172.21.15.103:0/518783282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fde90 msgr2=0x7f546c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:47.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 --2- 172.21.15.103:0/518783282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fde90 0x7f546c0fe260 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f545c004940 tx=0x7f545c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 -- 172.21.15.103:0/518783282 shutdown_connections 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 --2- 172.21.15.103:0/518783282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fe830 0x7f546c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 --2- 172.21.15.103:0/518783282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fde90 0x7f546c0fe260 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 -- 172.21.15.103:0/518783282 >> 172.21.15.103:0/518783282 conn(0x7f546c072bc0 msgr2=0x7f546c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 -- 172.21.15.103:0/518783282 shutdown_connections 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.537+0000 7f5471ced700 1 -- 172.21.15.103:0/518783282 wait complete. 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 Processor -- start 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 -- start start 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fde90 0x7f546c113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c114720 con 0x7f546c0fe830 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f5471ced700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c10db30 con 0x7f546c0fde90 2024-03-22T02:53:47.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546affd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59584/0 (socket says 172.21.15.103:59584) 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546affd700 1 -- 172.21.15.103:0/1090958239 learned_addr learned my addr 172.21.15.103:0/1090958239 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546b7fe700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fde90 0x7f546c113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546affd700 1 -- 172.21.15.103:0/1090958239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fde90 msgr2=0x7f546c113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.538+0000 7f546affd700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fde90 0x7f546c113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f546affd700 1 -- 172.21.15.103:0/1090958239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545c005040 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f546affd700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f546000c8c0 tx=0x7f54600044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f546000a980 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f546000ce60 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54600115e0 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c10de10 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.539+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c10e350 con 0x7f546c0fe830 2024-03-22T02:53:47.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.540+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c101980 con 0x7f546c0fe830 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.541+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5460012400 con 0x7f546c0fe830 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.541+0000 7f5453fff700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 0x7f5454070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.542+0000 7f546b7fe700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 0x7f5454070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.542+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f546008df10 con 0x7f546c0fe830 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.542+0000 7f546b7fe700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 0x7f5454070850 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f545c00b3b0 tx=0x7f545c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:47.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.548+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5460057150 con 0x7f546c0fe830 2024-03-22T02:53:47.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.901+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f546c10f1e0 con 0x7f546c0fe830 2024-03-22T02:53:47.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.902+0000 7f5453fff700 1 -- 172.21.15.103:0/1090958239 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f54600126b0 con 0x7f546c0fe830 2024-03-22T02:53:47.903 INFO:teuthology.orchestra.run.smithi103.stdout:HEALTH_OK 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 msgr2=0x7f5454070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 0x7f5454070850 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f545c00b3b0 tx=0x7f545c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 msgr2=0x7f546c1140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f546000c8c0 tx=0x7f54600044c0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 shutdown_connections 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f545406e3a0 0x7f5454070850 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f546c0fde90 0x7f546c113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 --2- 172.21.15.103:0/1090958239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f546c0fe830 0x7f546c1140d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:47.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 >> 172.21.15.103:0/1090958239 conn(0x7f546c072bc0 msgr2=0x7f546c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:47.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 shutdown_connections 2024-03-22T02:53:47.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:47.904+0000 7f5471ced700 1 -- 172.21.15.103:0/1090958239 wait complete. 2024-03-22T02:53:48.218 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:47 smithi103 ceph-mon[115211]: pgmap v668: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s wr, 1 op/s 2024-03-22T02:53:48.218 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:47 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1090958239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:53:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:47 smithi122 ceph-mon[83368]: pgmap v668: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s wr, 1 op/s 2024-03-22T02:53:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:47 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1090958239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T02:53:48.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions' 2024-03-22T02:53:48.851 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:50.002 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:49 smithi103 ceph-mon[115211]: pgmap v669: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:49 smithi122 ceph-mon[83368]: pgmap v669: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:50.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- 172.21.15.103:0/2723477630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c102710 msgr2=0x7f728c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:50.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 --2- 172.21.15.103:0/2723477630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c102710 0x7f728c0fbd80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f727c004940 tx=0x7f727c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:50.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- 172.21.15.103:0/2723477630 shutdown_connections 2024-03-22T02:53:50.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 --2- 172.21.15.103:0/2723477630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c102710 0x7f728c0fbd80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:50.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 --2- 172.21.15.103:0/2723477630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c101d70 0x7f728c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- 172.21.15.103:0/2723477630 >> 172.21.15.103:0/2723477630 conn(0x7f728c072bc0 msgr2=0x7f728c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- 172.21.15.103:0/2723477630 shutdown_connections 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- 172.21.15.103:0/2723477630 wait complete. 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 Processor -- start 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.756+0000 7f7294011700 1 -- start start 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7294011700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7294011700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c102710 0x7f728c10cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7294011700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c10d2f0 con 0x7f728c101d70 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7294011700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c10d460 con 0x7f728c102710 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:59610/0 (socket says 172.21.15.103:59610) 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 -- 172.21.15.103:0/1135247864 learned_addr learned my addr 172.21.15.103:0/1135247864 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f72915ac700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c102710 0x7f728c10cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:50.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 -- 172.21.15.103:0/1135247864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c102710 msgr2=0x7f728c10cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:50.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c102710 0x7f728c10cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:50.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 -- 172.21.15.103:0/1135247864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c005040 con 0x7f728c101d70 2024-03-22T02:53:50.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f7291dad700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f728800eb40 tx=0x7f728800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:50.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.757+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7288007cc0 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.758+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c10d6e0 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.758+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f728800f040 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.758+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7288009620 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.758+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c11a1b0 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.759+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c104240 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.759+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7288009780 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.759+0000 7f72827fc700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 0x7f7278070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.760+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f728808d810 con 0x7f728c101d70 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.760+0000 7f72915ac700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 0x7f7278070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:53:50.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.760+0000 7f72915ac700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 0x7f7278070850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f727c009b30 tx=0x7f727c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:53:50.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:50.765+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f728805a5f0 con 0x7f728c101d70 2024-03-22T02:53:51.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.094+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f728c0fa490 con 0x7f728c101d70 2024-03-22T02:53:51.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.094+0000 7f72827fc700 1 -- 172.21.15.103:0/1135247864 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f728805a180 con 0x7f728c101d70 2024-03-22T02:53:51.095 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:53:51.095 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:53:51.096 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:53:51.097 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:53:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 msgr2=0x7f7278070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 0x7f7278070850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f727c009b30 tx=0x7f727c009e30 comp rx=0 tx=0).stop 2024-03-22T02:53:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 msgr2=0x7f728c10c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:53:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f728800eb40 tx=0x7f728800b040 comp rx=0 tx=0).stop 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 shutdown_connections 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f727806e3a0 0x7f7278070850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728c101d70 0x7f728c10c870 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 --2- 172.21.15.103:0/1135247864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f728c102710 0x7f728c10cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 >> 172.21.15.103:0/1135247864 conn(0x7f728c072bc0 msgr2=0x7f728c0faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 shutdown_connections 2024-03-22T02:53:51.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:53:51.097+0000 7f7294011700 1 -- 172.21.15.103:0/1135247864 wait complete. 2024-03-22T02:53:51.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-22T02:53:52.044 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:51 smithi103 ceph-mon[115211]: pgmap v670: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:51 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1135247864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:51 smithi122 ceph-mon[83368]: pgmap v670: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:52.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:51 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1135247864' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:53:54.100 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:53 smithi103 ceph-mon[115211]: pgmap v671: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:53 smithi122 ceph-mon[83368]: pgmap v671: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:54.373 INFO:teuthology.orchestra.run.smithi103.stdout:wait for servicemap items w/ changing names to refresh 2024-03-22T02:53:54.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'sleep 60' 2024-03-22T02:53:55.262 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:53:56.067 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:55 smithi103 ceph-mon[115211]: pgmap v672: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:55 smithi122 ceph-mon[83368]: pgmap v672: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:57 smithi122 ceph-mon[83368]: pgmap v673: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:53:58.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:57 smithi103 ceph-mon[115211]: pgmap v673: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:53:59 smithi122 ceph-mon[83368]: pgmap v674: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:00.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:53:59 smithi103 ceph-mon[115211]: pgmap v674: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:01 smithi122 ceph-mon[83368]: pgmap v675: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:02.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:01 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:01 smithi103 ceph-mon[115211]: pgmap v675: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:02.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:01 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:03 smithi122 ceph-mon[83368]: pgmap v676: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:04.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:03 smithi103 ceph-mon[115211]: pgmap v676: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:05 smithi122 ceph-mon[83368]: pgmap v677: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:05 smithi103 ceph-mon[115211]: pgmap v677: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:07 smithi122 ceph-mon[83368]: pgmap v678: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:08.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:07 smithi103 ceph-mon[115211]: pgmap v678: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:09 smithi122 ceph-mon[83368]: pgmap v679: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:09 smithi103 ceph-mon[115211]: pgmap v679: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:11 smithi122 ceph-mon[83368]: pgmap v680: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:11 smithi103 ceph-mon[115211]: pgmap v680: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:13 smithi122 ceph-mon[83368]: pgmap v681: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:13 smithi103 ceph-mon[115211]: pgmap v681: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:15 smithi122 ceph-mon[83368]: pgmap v682: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:16.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:15 smithi103 ceph-mon[115211]: pgmap v682: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:16 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:16 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:18.275 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:17 smithi103 ceph-mon[115211]: pgmap v683: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:17 smithi122 ceph-mon[83368]: pgmap v683: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:19 smithi122 ceph-mon[83368]: pgmap v684: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:19 smithi103 ceph-mon[115211]: pgmap v684: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:21 smithi122 ceph-mon[83368]: pgmap v685: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:21 smithi103 ceph-mon[115211]: pgmap v685: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:23 smithi122 ceph-mon[83368]: pgmap v686: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:24.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:23 smithi103 ceph-mon[115211]: pgmap v686: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:25.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:24 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:54:25.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:24 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:54:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:25 smithi122 ceph-mon[83368]: pgmap v687: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:54:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:54:26.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:25 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:54:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:25 smithi103 ceph-mon[115211]: pgmap v687: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:54:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:54:26.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:25 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:54:28.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:27 smithi122 ceph-mon[83368]: pgmap v688: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:27 smithi103 ceph-mon[115211]: pgmap v688: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:30.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:29 smithi122 ceph-mon[83368]: pgmap v689: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:29 smithi103 ceph-mon[115211]: pgmap v689: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:32.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:31 smithi122 ceph-mon[83368]: pgmap v690: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:32.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:31 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:31 smithi103 ceph-mon[115211]: pgmap v690: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:31 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:34.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:33 smithi122 ceph-mon[83368]: pgmap v691: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:33 smithi103 ceph-mon[115211]: pgmap v691: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:36.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:35 smithi122 ceph-mon[83368]: pgmap v692: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:35 smithi103 ceph-mon[115211]: pgmap v692: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:38.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:37 smithi122 ceph-mon[83368]: pgmap v693: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:38.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:37 smithi103 ceph-mon[115211]: pgmap v693: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:40.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:39 smithi122 ceph-mon[83368]: pgmap v694: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:40.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:39 smithi103 ceph-mon[115211]: pgmap v694: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:42.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:41 smithi122 ceph-mon[83368]: pgmap v695: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:42.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:41 smithi103 ceph-mon[115211]: pgmap v695: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:44.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:44 smithi122 ceph-mon[83368]: pgmap v696: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:44.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:44 smithi103 ceph-mon[115211]: pgmap v696: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:46.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:46 smithi122 ceph-mon[83368]: pgmap v697: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:46.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:46 smithi103 ceph-mon[115211]: pgmap v697: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:47.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:47.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:54:48.275 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:48 smithi103 ceph-mon[115211]: pgmap v698: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:48.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:48 smithi122 ceph-mon[83368]: pgmap v698: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:50.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:50 smithi122 ceph-mon[83368]: pgmap v699: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:50.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:50 smithi103 ceph-mon[115211]: pgmap v699: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:52.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:52 smithi122 ceph-mon[83368]: pgmap v700: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:52.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:52 smithi103 ceph-mon[115211]: pgmap v700: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:54.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:54 smithi122 ceph-mon[83368]: pgmap v701: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:54.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:54 smithi103 ceph-mon[115211]: pgmap v701: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:56.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:56 smithi122 ceph-mon[83368]: pgmap v702: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:56.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:56 smithi103 ceph-mon[115211]: pgmap v702: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:57.484 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph orch ps' 2024-03-22T02:54:57.759 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:54:58.162 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:54:58 smithi103 ceph-mon[115211]: pgmap v703: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:58.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:54:58 smithi122 ceph-mon[83368]: pgmap v703: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:54:59.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 -- 172.21.15.103:0/3569992056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fde00 msgr2=0x7f1cd00fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 --2- 172.21.15.103:0/3569992056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fde00 0x7f1cd00fe1d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004940 tx=0x7f1cc0009e30 comp rx=0 tx=0).stop 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 -- 172.21.15.103:0/3569992056 shutdown_connections 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 --2- 172.21.15.103:0/3569992056 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd0106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 --2- 172.21.15.103:0/3569992056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fde00 0x7f1cd00fe1d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 -- 172.21.15.103:0/3569992056 >> 172.21.15.103:0/3569992056 conn(0x7f1cd00f7830 msgr2=0x7f1cd00f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 -- 172.21.15.103:0/3569992056 shutdown_connections 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.546+0000 7f1cd58df700 1 -- 172.21.15.103:0/3569992056 wait complete. 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 Processor -- start 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 -- start start 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fde00 0x7f1cd0115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01167a0 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.547+0000 7f1cd58df700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd010fcf0 con 0x7f1cd00fde00 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41252/0 (socket says 172.21.15.103:41252) 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 -- 172.21.15.103:0/2521053841 learned_addr learned my addr 172.21.15.103:0/2521053841 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cceffd700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fde00 0x7f1cd0115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 -- 172.21.15.103:0/2521053841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fde00 msgr2=0x7f1cd0115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fde00 0x7f1cd0115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 -- 172.21.15.103:0/2521053841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc0005040 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cce7fc700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400b790 tx=0x7f1cc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc400f6f0 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cc400fd20 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cc40103a0 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd010ff70 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.548+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd0110450 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.549+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0106850 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.550+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1cc4011400 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.550+0000 7f1cb77fe700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 0x7f1cb8070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:54:59.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.551+0000 7f1cceffd700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 0x7f1cb8070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:54:59.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.551+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1cc408dbe0 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.551+0000 7f1cceffd700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 0x7f1cb8070850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004f90 tx=0x7f1cc0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:54:59.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.555+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1cc405a990 con 0x7f1cd00fe7a0 2024-03-22T02:54:59.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.817+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 --> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cd0109f60 con 0x7f1cb806e3a0 2024-03-22T02:54:59.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.828+0000 7f1cb77fe700 1 -- 172.21.15.103:0/2521053841 <== mgr.34104 v2:172.21.15.103:6800/1355893550 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1cd0109f60 con 0x7f1cb806e3a0 2024-03-22T02:54:59.829 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:alertmanager.smithi103 smithi103 *:9093,9094 running (23m) 96s ago 34m 25.1M - 0.25.0 c8568f914cd2 817735a123d7 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi103 smithi103 running (21m) 96s ago 34m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed aaab0a8a63df 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:crash.smithi122 smithi122 running (21m) 101s ago 33m 6665k - 19.0.0-2344-gc3a04c0e 9d92c22378ed 16df01016c8c 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:grafana.smithi103 smithi103 *:3000 running (23m) 96s ago 34m 79.5M - 9.4.12 2bacad6d85d8 700d584c4b99 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.gtuatp smithi103 running (17m) 96s ago 29m 98.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed acdec019b473 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi103.xhmfjp smithi103 running (17m) 96s ago 29m 104M - 19.0.0-2344-gc3a04c0e 9d92c22378ed df5385f9ef2e 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.lvsegg smithi122 running (16m) 101s ago 29m 19.4M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 5ca549110865 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mds.cephfs.smithi122.ztrkgp smithi122 running (17m) 101s ago 29m 19.5M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 3069173ceb1c 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi103.pfvmui smithi103 *:8443,9283,8765 running (25m) 96s ago 36m 512M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 026658f6f34a 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.smithi122.prgtti smithi122 *:8443,9283,8765 running (24m) 101s ago 33m 457M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ce984ed8980f 2024-03-22T02:54:59.830 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi103 smithi103 running (22m) 96s ago 36m 70.1M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 73b5f7067b0e 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:mon.smithi122 smithi122 running (21m) 101s ago 33m 52.4M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed b334dc9d4e9f 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi103 smithi103 *:9100 running (24m) 96s ago 34m 15.6M - 1.5.0 0da6a335fe13 6bc02af39fb0 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:node-exporter.smithi122 smithi122 *:9100 running (24m) 101s ago 33m 19.0M - 1.5.0 0da6a335fe13 c96c2e11aa4b 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (20m) 96s ago 32m 96.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed fc16d3339890 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi103 running (20m) 96s ago 32m 112M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 6e2323777383 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi103 running (19m) 96s ago 32m 88.9M 1557M 19.0.0-2344-gc3a04c0e 9d92c22378ed 2be02d5e6564 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi122 running (19m) 101s ago 31m 110M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed 4357b0d26181 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi122 running (18m) 101s ago 31m 98.5M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed d25f12ea0232 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi122 running (17m) 101s ago 30m 94.5M 2581M 19.0.0-2344-gc3a04c0e 9d92c22378ed e61a4141f08d 2024-03-22T02:54:59.831 INFO:teuthology.orchestra.run.smithi103.stdout:prometheus.smithi103 smithi103 *:9095 running (23m) 96s ago 34m 73.8M - 2.43.0 a07b618ecd1d 1e7ac756e673 2024-03-22T02:54:59.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 msgr2=0x7f1cb8070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:54:59.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 0x7f1cb8070850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0004f90 tx=0x7f1cc0009e30 comp rx=0 tx=0).stop 2024-03-22T02:54:59.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 msgr2=0x7f1cd01160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400b790 tx=0x7f1cc4009250 comp rx=0 tx=0).stop 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 shutdown_connections 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1cb806e3a0 0x7f1cb8070850 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cd00fde00 0x7f1cd0115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 --2- 172.21.15.103:0/2521053841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cd00fe7a0 0x7f1cd01160c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.830+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 >> 172.21.15.103:0/2521053841 conn(0x7f1cd00f7830 msgr2=0x7f1cd00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.831+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 shutdown_connections 2024-03-22T02:54:59.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:54:59.831+0000 7f1cd58df700 1 -- 172.21.15.103:0/2521053841 wait complete. 2024-03-22T02:55:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:00 smithi103 ceph-mon[115211]: pgmap v704: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:00.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:00 smithi122 ceph-mon[83368]: pgmap v704: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:00.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions' 2024-03-22T02:55:00.931 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:55:01.280 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:01 smithi103 ceph-mon[115211]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:55:01.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:01 smithi122 ceph-mon[83368]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:55:02.269 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:02 smithi103 ceph-mon[115211]: pgmap v705: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:02.269 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:02 smithi122 ceph-mon[83368]: pgmap v705: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:02.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:02.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 -- 172.21.15.103:0/2796367872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8102fd0 msgr2=0x7f69b81033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/2796367872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8102fd0 0x7f69b81033a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f69a8004940 tx=0x7f69a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 -- 172.21.15.103:0/2796367872 shutdown_connections 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/2796367872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b80fb540 0x7f69b80fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/2796367872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b8102fd0 0x7f69b81033a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 -- 172.21.15.103:0/2796367872 >> 172.21.15.103:0/2796367872 conn(0x7f69b8071530 msgr2=0x7f69b8071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 -- 172.21.15.103:0/2796367872 shutdown_connections 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.656+0000 7f69bd6ed700 1 -- 172.21.15.103:0/2796367872 wait complete. 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.657+0000 7f69bd6ed700 1 Processor -- start 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.657+0000 7f69bd6ed700 1 -- start start 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 0x7f69b8113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8114500 con 0x7f69b80fb540 2024-03-22T02:55:02.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b810db00 con 0x7f69b8102fd0 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41260/0 (socket says 172.21.15.103:41260) 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 -- 172.21.15.103:0/783332247 learned_addr learned my addr 172.21.15.103:0/783332247 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b67fc700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 0x7f69b8113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 -- 172.21.15.103:0/783332247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 msgr2=0x7f69b8113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 0x7f69b8113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 -- 172.21.15.103:0/783332247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8005040 con 0x7f69b80fb540 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b67fc700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 0x7f69b8113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69b6ffd700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009150 tx=0x7f69a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a800e040 con 0x7f69b80fb540 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b810dd80 con 0x7f69b80fb540 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.658+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b810e340 con 0x7f69b80fb540 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.659+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69a8013420 con 0x7f69b80fb540 2024-03-22T02:55:02.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.659+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a801c630 con 0x7f69b80fb540 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.660+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f69a801c860 con 0x7f69b80fb540 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.660+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b8100270 con 0x7f69b80fb540 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.660+0000 7f699f7fe700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 0x7f69a0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.661+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f69a808ffa0 con 0x7f69b80fb540 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.661+0000 7f69b67fc700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 0x7f69a0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:02.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.661+0000 7f69b67fc700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 0x7f69a0070850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69ac007fd0 tx=0x7f69ac009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:02.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.665+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69a805ce10 con 0x7f69b80fb540 2024-03-22T02:55:02.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.957+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f69b804c770 con 0x7f69b80fb540 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.958+0000 7f699f7fe700 1 -- 172.21.15.103:0/783332247 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f69a805c9a0 con 0x7f69b80fb540 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: "mon": { 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr": { 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 2 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:55:02.960 INFO:teuthology.orchestra.run.smithi103.stdout: "osd": { 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 6 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 4 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "overall": { 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)": 14 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-22T02:55:02.961 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 msgr2=0x7f69a0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 0x7f69a0070850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69ac007fd0 tx=0x7f69ac009040 comp rx=0 tx=0).stop 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 msgr2=0x7f69b8113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009150 tx=0x7f69a8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 shutdown_connections 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f69a006e3a0 0x7f69a0070850 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69b80fb540 0x7f69b8113970 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 --2- 172.21.15.103:0/783332247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69b8102fd0 0x7f69b8113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:02.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 >> 172.21.15.103:0/783332247 conn(0x7f69b8071530 msgr2=0x7f69b80fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:02.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 shutdown_connections 2024-03-22T02:55:02.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:02.960+0000 7f69bd6ed700 1 -- 172.21.15.103:0/783332247 wait complete. 2024-03-22T02:55:03.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:03 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/783332247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:03.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:03 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/783332247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:03.678 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-22T02:55:03.956 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:55:04.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:04 smithi122 ceph-mon[83368]: pgmap v706: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:04.355 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:04 smithi103 ceph-mon[115211]: pgmap v706: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/2771160960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fc4e0 msgr2=0x7fa4ac109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/2771160960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac109240 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0004970 tx=0x7fa4a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/2771160960 shutdown_connections 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/2771160960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac109240 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/2771160960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac0fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/2771160960 >> 172.21.15.103:0/2771160960 conn(0x7fa4ac0f7860 msgr2=0x7fa4ac0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/2771160960 shutdown_connections 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.654+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/2771160960 wait complete. 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 Processor -- start 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 -- start start 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac116860 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4b0d8a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac10fc70 con 0x7fa4ac0fc4e0 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53164/0 (socket says 172.21.15.103:53164) 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 -- 172.21.15.103:0/1483123269 learned_addr learned my addr 172.21.15.103:0/1483123269 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4a9d9b700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:05.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 -- 172.21.15.103:0/1483123269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc4e0 msgr2=0x7fa4ac116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:05.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.655+0000 7fa4aa59c700 1 -- 172.21.15.103:0/1483123269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a0005040 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa4aa59c700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c830 tx=0x7fa49c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49c00a760 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac10ff50 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa49c00ad90 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49c0113a0 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.656+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac1104e0 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.657+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac066710 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.657+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa49c012400 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.658+0000 7fa49affd700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 0x7fa494070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:05.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.658+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa49c08de20 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.658+0000 7fa4a9d9b700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 0x7fa494070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:05.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.658+0000 7fa4a9d9b700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 0x7fa494070850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0004f50 tx=0x7fa4a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:05.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.663+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa49c0570a0 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.985+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa4ac04bed0 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.985+0000 7fa49affd700 1 -- 172.21.15.103:0/1483123269 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa49c05a760 con 0x7fa4ac0fbbd0 2024-03-22T02:55:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 msgr2=0x7fa494070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 0x7fa494070850 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0004f50 tx=0x7fa4a0009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 msgr2=0x7fa4ac115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c830 tx=0x7fa49c004470 comp rx=0 tx=0).stop 2024-03-22T02:55:05.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 shutdown_connections 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa49406e3a0 0x7fa494070850 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4ac0fbbd0 0x7fa4ac115cd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 --2- 172.21.15.103:0/1483123269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4ac0fc4e0 0x7fa4ac116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 >> 172.21.15.103:0/1483123269 conn(0x7fa4ac0f7860 msgr2=0x7fa4ac0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 shutdown_connections 2024-03-22T02:55:05.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:05.988+0000 7fa4b0d8a700 1 -- 172.21.15.103:0/1483123269 wait complete. 2024-03-22T02:55:06.000 INFO:teuthology.orchestra.run.smithi103.stdout:true 2024-03-22T02:55:06.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:06 smithi122 ceph-mon[83368]: pgmap v707: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:06.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:06 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1483123269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:06 smithi103 ceph-mon[115211]: pgmap v707: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:06.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:06 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1483123269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:06.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -e sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-22T02:55:06.919 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:55:08.092 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:08 smithi103 ceph-mon[115211]: pgmap v708: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:08.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:08 smithi122 ceph-mon[83368]: pgmap v708: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 -- 172.21.15.103:0/4046568889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 msgr2=0x7f33080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 --2- 172.21.15.103:0/4046568889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f33080fe260 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f32f8004940 tx=0x7f32f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 -- 172.21.15.103:0/4046568889 shutdown_connections 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 --2- 172.21.15.103:0/4046568889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 0x7f3308106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 --2- 172.21.15.103:0/4046568889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f33080fe260 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 -- 172.21.15.103:0/4046568889 >> 172.21.15.103:0/4046568889 conn(0x7f3308072bc0 msgr2=0x7f3308072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:08.655 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 -- 172.21.15.103:0/4046568889 shutdown_connections 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.644+0000 7f330d8dd700 1 -- 172.21.15.103:0/4046568889 wait complete. 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 Processor -- start 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 -- start start 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 0x7f33081140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3308114720 con 0x7f33080fde90 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f330d8dd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330810db30 con 0x7f33080fe830 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.645+0000 7f3306ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53180/0 (socket says 172.21.15.103:53180) 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 -- 172.21.15.103:0/2340711619 learned_addr learned my addr 172.21.15.103:0/2340711619 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f33067fc700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 0x7f33081140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 -- 172.21.15.103:0/2340711619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 msgr2=0x7f33081140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 0x7f33081140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.656 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 -- 172.21.15.103:0/2340711619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f8005040 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f3306ffd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32f800b3b0 tx=0x7f32f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f800e040 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32f8013420 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330810ddb0 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32f801c5f0 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.646+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330810e2f0 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.647+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3308101980 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.648+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f32f8013590 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.648+0000 7f32ef7fe700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 0x7f32f0070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.648+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f32f808efa0 con 0x7f33080fde90 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.648+0000 7f33067fc700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 0x7f32f0070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.649+0000 7f33067fc700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 0x7f32f0070850 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f330810f430 tx=0x7f32fc0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:08.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.653+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f32f80c3740 con 0x7f33080fde90 2024-03-22T02:55:08.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.989+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3308109f00 con 0x7f33080fde90 2024-03-22T02:55:08.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.989+0000 7f32ef7fe700 1 -- 172.21.15.103:0/2340711619 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f32f8058220 con 0x7f33080fde90 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 msgr2=0x7f32f0070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 0x7f32f0070850 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f330810f430 tx=0x7f32fc0043c0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 msgr2=0x7f3308113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32f800b3b0 tx=0x7f32f8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 shutdown_connections 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f32f006e3a0 0x7f32f0070850 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33080fde90 0x7f3308113b90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 --2- 172.21.15.103:0/2340711619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33080fe830 0x7f33081140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:08.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 >> 172.21.15.103:0/2340711619 conn(0x7f3308072bc0 msgr2=0x7f33080ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:08.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 shutdown_connections 2024-03-22T02:55:08.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:08.991+0000 7f330d8dd700 1 -- 172.21.15.103:0/2340711619 wait complete. 2024-03-22T02:55:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)" 2024-03-22T02:55:09.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:09 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2340711619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:09.350 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:09 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2340711619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T02:55:09.668 DEBUG:teuthology.parallel:result is None 2024-03-22T02:55:09.668 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:55:09.683 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T02:55:09.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-22T02:55:10.009 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/mon.smithi103/config 2024-03-22T02:55:10.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:10 smithi122 ceph-mon[83368]: pgmap v709: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:10.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:10 smithi103 ceph-mon[115211]: pgmap v709: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 -- 172.21.15.103:0/1239853516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fde90 msgr2=0x7f1eac0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 --2- 172.21.15.103:0/1239853516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fe260 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8004940 tx=0x7f1ea8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 -- 172.21.15.103:0/1239853516 shutdown_connections 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 --2- 172.21.15.103:0/1239853516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fe830 0x7f1eac106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 --2- 172.21.15.103:0/1239853516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fe260 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 -- 172.21.15.103:0/1239853516 >> 172.21.15.103:0/1239853516 conn(0x7f1eac072bc0 msgr2=0x7f1eac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 -- 172.21.15.103:0/1239853516 shutdown_connections 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 -- 172.21.15.103:0/1239853516 wait complete. 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.889+0000 7f1eb383d700 1 Processor -- start 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb383d700 1 -- start start 2024-03-22T02:55:11.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb383d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb383d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb383d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0fc820 con 0x7f1eac0fe830 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb383d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0fc960 con 0x7f1eac0fde90 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53204/0 (socket says 172.21.15.103:53204) 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 -- 172.21.15.103:0/2357389334 learned_addr learned my addr 172.21.15.103:0/2357389334 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb15d9700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 -- 172.21.15.103:0/2357389334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fde90 msgr2=0x7f1eac0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 -- 172.21.15.103:0/2357389334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1eac0fe830 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.890+0000 7f1eb0dd8700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00eb40 tx=0x7f1e9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.891+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c007cc0 con 0x7f1eac0fe830 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.891+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac0fcbb0 con 0x7f1eac0fe830 2024-03-22T02:55:11.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.891+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e9c00f040 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.891+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c00ca10 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.891+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac11a1e0 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.892+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac065e30 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.893+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f1e9c00cc80 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.893+0000 7f1ea1ffb700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 0x7f1e98070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.893+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1e9c08d9a0 con 0x7f1eac0fe830 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.893+0000 7f1eb15d9700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 0x7f1e98070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.893+0000 7f1eb15d9700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 0x7f1e98070850 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800b0d0 tx=0x7f1ea8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:11.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:11.898+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e9c056c50 con 0x7f1eac0fe830 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.161+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1eac04bed0 con 0x7f1eac0fe830 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.162+0000 7f1ea1ffb700 1 -- 172.21.15.103:0/2357389334 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 37 v37) v1 ==== 76+0+1961 (secure 0 0 0) 0x7f1e9c05a310 con 0x7f1eac0fe830 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:e37 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:55:12.164 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 37 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-03-22T02:25:42.837872+0000 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-03-22T02:53:26.631347+0000 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 60 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 80 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:55:12.165 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 2 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=34380,1=34412} 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3] 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 1 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 34380 members: 34380,34412 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.gtuatp{0:34380} state up:active seq 244 join_fscid=1 addr [v2:172.21.15.103:6826/4019539425,v1:172.21.15.103:6827/4019539425] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:55:12.166 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi103.xhmfjp{1:34412} state up:active seq 240 join_fscid=1 addr [v2:172.21.15.103:6828/3490746036,v1:172.21.15.103:6829/3490746036] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.ztrkgp{-1:44291} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/2991160252,v1:172.21.15.122:6825/2991160252] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:55:12.167 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.cephfs.smithi122.lvsegg{-1:44311} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/3038924550,v1:172.21.15.122:6827/3038924550] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 msgr2=0x7f1e98070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 0x7f1e98070850 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800b0d0 tx=0x7f1ea8009e30 comp rx=0 tx=0).stop 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 msgr2=0x7f1eac0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00eb40 tx=0x7f1e9c00b040 comp rx=0 tx=0).stop 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 shutdown_connections 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f1e9806e3a0 0x7f1e98070850 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1eac0fde90 0x7f1eac0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 --2- 172.21.15.103:0/2357389334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eac0fe830 0x7f1eac0fc290 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 >> 172.21.15.103:0/2357389334 conn(0x7f1eac072bc0 msgr2=0x7f1eac1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 shutdown_connections 2024-03-22T02:55:12.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.165+0000 7f1eb383d700 1 -- 172.21.15.103:0/2357389334 wait complete. 2024-03-22T02:55:12.169 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 37 2024-03-22T02:55:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:12 smithi122 ceph-mon[83368]: pgmap v710: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:12.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:12 smithi103 ceph-mon[115211]: pgmap v710: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:12.750 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-22T02:55:12.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:55:12.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 -- 172.21.15.103:0/2120946607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fd1e0 msgr2=0x7f50a40fd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2120946607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fd1e0 0x7f50a40fd5b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f50a00048d0 tx=0x7f50a0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 -- 172.21.15.103:0/2120946607 shutdown_connections 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2120946607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fdb80 0x7f50a41061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2120946607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fd1e0 0x7f50a40fd5b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 -- 172.21.15.103:0/2120946607 >> 172.21.15.103:0/2120946607 conn(0x7f50a40f6c70 msgr2=0x7f50a40f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.859+0000 7f50ab26e640 1 -- 172.21.15.103:0/2120946607 shutdown_connections 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 -- 172.21.15.103:0/2120946607 wait complete. 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 Processor -- start 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 -- start start 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fd1e0 0x7f50a4112d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:12.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a4113960 con 0x7f50a40fdb80 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50ab26e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a410cf20 con 0x7f50a40fd1e0 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50a8809640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50a8809640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53216/0 (socket says 172.21.15.103:53216) 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50a8809640 1 -- 172.21.15.103:0/2161695018 learned_addr learned my addr 172.21.15.103:0/2161695018 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.860+0000 7f50a900a640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fd1e0 0x7f50a4112d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50a8809640 1 -- 172.21.15.103:0/2161695018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fd1e0 msgr2=0x7f50a4112d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50a8809640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fd1e0 0x7f50a4112d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50a8809640 1 -- 172.21.15.103:0/2161695018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0005040 con 0x7f50a40fdb80 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50a8809640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f509400b790 tx=0x7f50940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509400c700 con 0x7f50a40fdb80 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a410d200 con 0x7f50a40fdb80 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f509400cd30 con 0x7f50a40fdb80 2024-03-22T02:55:12.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5094013370 con 0x7f50a40fdb80 2024-03-22T02:55:12.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.861+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a410d720 con 0x7f50a40fdb80 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.862+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a4065980 con 0x7f50a40fdb80 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.862+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f5094012070 con 0x7f50a40fdb80 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.863+0000 7f5099ffb640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 0x7f50900707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.863+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f509408d890 con 0x7f50a40fdb80 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.863+0000 7f50a900a640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 0x7f50900707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:12.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.863+0000 7f50a900a640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 0x7f50900707c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003080 tx=0x7f50a0009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:12.869 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:12.868+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5094056b10 con 0x7f50a40fdb80 2024-03-22T02:55:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.098+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50a404bcc0 con 0x7f50a40fdb80 2024-03-22T02:55:13.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.100+0000 7f5099ffb640 1 -- 172.21.15.103:0/2161695018 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f509405a1d0 con 0x7f50a40fdb80 2024-03-22T02:55:13.103 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:13.103 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":84,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:38:09.183439+0000","last_up_change":"2024-03-22T02:37:15.002547+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-22T02:25:41.764005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-22T02:25:42.064180+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6803","nonce":1332641876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6805","nonce":1332641876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6809","nonce":1332641876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6807","nonce":1332641876}]},"public_addr":"172.21.15.103:6803/1332641876","cluster_addr":"172.21.15.103:6805/1332641876","heartbeat_back_addr":"172.21.15.103:6809/1332641876","heartbeat_front_addr":"172.21.15.103:6807/1332641876","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6811","nonce":292218030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6813","nonce":292218030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6817","nonce":292218030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6815","nonce":292218030}]},"public_addr":"172.21.15.103:6811/292218030","cluster_addr":"172.21.15.103:6813/292218030","heartbeat_back_addr":"172.21.15.103:6817/292218030","heartbeat_front_addr":"172.21.15.103:6815/292218030","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6819","nonce":117269184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6821","nonce":117269184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6825","nonce":117269184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6823","nonce":117269184}]},"public_addr":"172.21.15.103:6819/117269184","cluster_addr":"172.21.15.103:6821/117269184","heartbeat_back_addr":"172.21.15.103:6825/117269184","heartbeat_front_addr":"172.21.15.103:6823/117269184","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6801","nonce":2866011991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6803","nonce":2866011991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6807","nonce":2866011991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6805","nonce":2866011991}]},"public_addr":"172.21.15.122:6801/2866011991","cluster_addr":"172.21.15.122:6803/2866011991","heartbeat_back_addr":"172.21.15.122:6807/2866011991","heartbeat_front_addr":"172.21.15.122:6805/2866011991","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6809","nonce":1009596471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6811","nonce":1009596471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6815","nonce":1009596471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6813","nonce":1009596471}]},"public_addr":"172.21.15.122:6809/1009596471","cluster_addr":"172.21.15.122:6811/1009596471","heartbeat_back_addr":"172.21.15.122:6815/1009596471","heartbeat_front_addr":"172.21.15.122:6813/1009596471","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6817","nonce":1621369093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6819","nonce":1621369093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6823","nonce":1621369093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6821","nonce":1621369093}]},"public_addr":"172.21.15.122:6817/1621369093","cluster_addr":"172.21.15.122:6819/1621369093","heartbeat_back_addr":"172.21.15.122:6823/1621369093","heartbeat_front_addr":"172.21.15.122:6821/1621369093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.122:6824/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.103:6829/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.122:0/2086230685":"2024-03-23T02:30:07.935727+0000","172.21.15.103:6827/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6828/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6826/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.122:0/1988149757":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.122:6829/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3955856929":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6800/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.122:0/3639247922":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.122:6828/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3724441621":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6801/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2509522170":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/385301047":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/1309215164":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3243311702":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2788707715":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.122:0/867576006":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/1085098758":"2024-03-23T02:32:46.783649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:55:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 msgr2=0x7f50900707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 0x7f50900707c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003080 tx=0x7f50a0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 msgr2=0x7f50a41132d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f509400b790 tx=0x7f50940091d0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 shutdown_connections 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f509006e310 0x7f50900707c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50a40fdb80 0x7f50a41132d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 --2- 172.21.15.103:0/2161695018 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50a40fd1e0 0x7f50a4112d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 >> 172.21.15.103:0/2161695018 conn(0x7f50a40f6c70 msgr2=0x7f50a40f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 shutdown_connections 2024-03-22T02:55:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.103+0000 7f50ab26e640 1 -- 172.21.15.103:0/2161695018 wait complete. 2024-03-22T02:55:13.119 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:23:06.693135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-22T02:25:41.764005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-22T02:25:42.064180+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:55:13.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:55:13.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- 172.21.15.103:0/3343407658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a10101c70 msgr2=0x7f2a100fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 --2- 172.21.15.103:0/3343407658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a10101c70 0x7f2a100fb300 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f29fc0048d0 tx=0x7f29fc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- 172.21.15.103:0/3343407658 shutdown_connections 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 --2- 172.21.15.103:0/3343407658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a10101c70 0x7f2a100fb300 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 --2- 172.21.15.103:0/3343407658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a101012d0 0x7f2a101016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- 172.21.15.103:0/3343407658 >> 172.21.15.103:0/3343407658 conn(0x7f2a100f6ea0 msgr2=0x7f2a100f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- 172.21.15.103:0/3343407658 shutdown_connections 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- 172.21.15.103:0/3343407658 wait complete. 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 Processor -- start 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.220+0000 7f2a16aae640 1 -- start start 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a16aae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a16aae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a10101c70 0x7f2a1010e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a16aae640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a10066490 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a16aae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1010eb60 con 0x7f2a10101c70 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53234/0 (socket says 172.21.15.103:53234) 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 -- 172.21.15.103:0/2648304274 learned_addr learned my addr 172.21.15.103:0/2648304274 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 -- 172.21.15.103:0/2648304274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a10101c70 msgr2=0x7f2a1010e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a10101c70 0x7f2a1010e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 -- 172.21.15.103:0/2648304274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc005040 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a1484a640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400eb70 tx=0x7f2a0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a04007c30 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a0400f040 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1010ede0 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.221+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a040095b0 con 0x7f2a101012d0 2024-03-22T02:55:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.222+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a101173a0 con 0x7f2a101012d0 2024-03-22T02:55:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.223+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a100fca40 con 0x7f2a101012d0 2024-03-22T02:55:13.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.223+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f2a04009730 con 0x7f2a101012d0 2024-03-22T02:55:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.223+0000 7f2a0d7fa640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 0x7f2a000707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.224+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2a0408db50 con 0x7f2a101012d0 2024-03-22T02:55:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.224+0000 7f2a0ffff640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 0x7f2a000707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.224+0000 7f2a0ffff640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 0x7f2a000707d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f29fc003080 tx=0x7f29fc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.228+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a04056d50 con 0x7f2a101012d0 2024-03-22T02:55:13.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:13 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2357389334' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:55:13.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:13 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2357389334' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:55:13.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.422+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a100638f0 con 0x7f2a101012d0 2024-03-22T02:55:13.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.423+0000 7f2a0d7fa640 1 -- 172.21.15.103:0/2648304274 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a0405a410 con 0x7f2a101012d0 2024-03-22T02:55:13.425 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 msgr2=0x7f2a000707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 0x7f2a000707d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f29fc003080 tx=0x7f29fc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 msgr2=0x7f2a1010e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400eb70 tx=0x7f2a0400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 shutdown_connections 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f2a0006e320 0x7f2a000707d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a10101c70 0x7f2a1010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 --2- 172.21.15.103:0/2648304274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a101012d0 0x7f2a1010e0e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 >> 172.21.15.103:0/2648304274 conn(0x7f2a100f6ea0 msgr2=0x7f2a100f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 shutdown_connections 2024-03-22T02:55:13.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.425+0000 7f2a16aae640 1 -- 172.21.15.103:0/2648304274 wait complete. 2024-03-22T02:55:13.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-22T02:55:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 -- 172.21.15.103:0/1297371823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 msgr2=0x7fcd481014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/1297371823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd481014d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcd380048d0 tx=0x7fcd38009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 -- 172.21.15.103:0/1297371823 shutdown_connections 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/1297371823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 0x7fcd480fb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/1297371823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd481014d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 -- 172.21.15.103:0/1297371823 >> 172.21.15.103:0/1297371823 conn(0x7fcd480f6cf0 msgr2=0x7fcd480f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 -- 172.21.15.103:0/1297371823 shutdown_connections 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.541+0000 7fcd4ea07640 1 -- 172.21.15.103:0/1297371823 wait complete. 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 Processor -- start 2024-03-22T02:55:13.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 -- start start 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 0x7fcd4810e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48066510 con 0x7fcd48101100 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd4ea07640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4810e960 con 0x7fcd48101aa0 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53258/0 (socket says 172.21.15.103:53258) 2024-03-22T02:55:13.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 -- 172.21.15.103:0/2223580519 learned_addr learned my addr 172.21.15.103:0/2223580519 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd477fe640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 0x7fcd4810e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 -- 172.21.15.103:0/2223580519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 msgr2=0x7fcd4810e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 0x7fcd4810e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.542+0000 7fcd47fff640 1 -- 172.21.15.103:0/2223580519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38005040 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd47fff640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcd38004930 tx=0x7fcd38009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd380094c0 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd38032e50 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4810ebe0 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd380433b0 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.543+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd48117110 con 0x7fcd48101100 2024-03-22T02:55:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.544+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd480fc870 con 0x7fcd48101100 2024-03-22T02:55:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.545+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcd3802f070 con 0x7fcd48101100 2024-03-22T02:55:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.545+0000 7fcd44ff9640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 0x7fcd300707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.545+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcd380afbf0 con 0x7fcd48101100 2024-03-22T02:55:13.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.545+0000 7fcd477fe640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 0x7fcd300707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.546+0000 7fcd477fe640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 0x7fcd300707c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcd48065d60 tx=0x7fcd3c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.552+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd380b6080 con 0x7fcd48101100 2024-03-22T02:55:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.734+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fcd4804bcc0 con 0x7fcd48101100 2024-03-22T02:55:13.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.734+0000 7fcd44ff9640 1 -- 172.21.15.103:0/2223580519 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcd38078ea0 con 0x7fcd48101100 2024-03-22T02:55:13.736 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:13.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 msgr2=0x7fcd300707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 0x7fcd300707c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcd48065d60 tx=0x7fcd3c00b040 comp rx=0 tx=0).stop 2024-03-22T02:55:13.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 msgr2=0x7fcd4810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fcd38004930 tx=0x7fcd38009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 shutdown_connections 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcd3006e310 0x7fcd300707c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd48101aa0 0x7fcd4810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 --2- 172.21.15.103:0/2223580519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd48101100 0x7fcd4810deb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 >> 172.21.15.103:0/2223580519 conn(0x7fcd480f6cf0 msgr2=0x7fcd480f7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 shutdown_connections 2024-03-22T02:55:13.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.736+0000 7fcd4ea07640 1 -- 172.21.15.103:0/2223580519 wait complete. 2024-03-22T02:55:13.752 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 -- 172.21.15.103:0/4219867729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 msgr2=0x7f549c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 --2- 172.21.15.103:0/4219867729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c108780 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f548c0048d0 tx=0x7f548c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 -- 172.21.15.103:0/4219867729 shutdown_connections 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 --2- 172.21.15.103:0/4219867729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c108780 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 --2- 172.21.15.103:0/4219867729 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f549c0643a0 0x7f549c064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 -- 172.21.15.103:0/4219867729 >> 172.21.15.103:0/4219867729 conn(0x7f549c0710b0 msgr2=0x7f549c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 -- 172.21.15.103:0/4219867729 shutdown_connections 2024-03-22T02:55:13.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.859+0000 7f54a32ae640 1 -- 172.21.15.103:0/4219867729 wait complete. 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 Processor -- start 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 -- start start 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f549c0643a0 0x7f549c114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c115a20 con 0x7f549c064cb0 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a32ae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c10f070 con 0x7f549c0643a0 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:32998/0 (socket says 172.21.15.103:32998) 2024-03-22T02:55:13.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 -- 172.21.15.103:0/1936564257 learned_addr learned my addr 172.21.15.103:0/1936564257 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:13.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 -- 172.21.15.103:0/1936564257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f549c0643a0 msgr2=0x7f549c114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:13.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f549c0643a0 0x7f549c114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.860+0000 7f54a0849640 1 -- 172.21.15.103:0/1936564257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c005040 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f54a0849640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f548c034040 tx=0x7f548c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f548c0092b0 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c10f2f0 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f548c03b400 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f548c0435d0 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.861+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c10f7b0 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.862+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c0fb730 con 0x7f549c064cb0 2024-03-22T02:55:13.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.863+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f548c02f070 con 0x7f549c064cb0 2024-03-22T02:55:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.863+0000 7f5491ffb640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 0x7f54880707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.863+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f548c0afb90 con 0x7f549c064cb0 2024-03-22T02:55:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.863+0000 7f54a104a640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 0x7f54880707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:13.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.863+0000 7f54a104a640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 0x7f54880707c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5498004d40 tx=0x7f549800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:13.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:13.872+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f548c078d90 con 0x7f549c064cb0 2024-03-22T02:55:14.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.080+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f549c04bcc0 con 0x7f549c064cb0 2024-03-22T02:55:14.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.080+0000 7f5491ffb640 1 -- 172.21.15.103:0/1936564257 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f548c07c450 con 0x7f549c064cb0 2024-03-22T02:55:14.082 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 msgr2=0x7f54880707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 0x7f54880707c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5498004d40 tx=0x7f549800b040 comp rx=0 tx=0).stop 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 msgr2=0x7f549c115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f548c034040 tx=0x7f548c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 shutdown_connections 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f548806e310 0x7f54880707c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c064cb0 0x7f549c115420 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 --2- 172.21.15.103:0/1936564257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f549c0643a0 0x7f549c114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.083+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 >> 172.21.15.103:0/1936564257 conn(0x7f549c0710b0 msgr2=0x7f549c0f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.084+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 shutdown_connections 2024-03-22T02:55:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.084+0000 7f54a32ae640 1 -- 172.21.15.103:0/1936564257 wait complete. 2024-03-22T02:55:14.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:55:14.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 -- 172.21.15.103:0/1105859755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc101a60 msgr2=0x7fefdc0fb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/1105859755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc101a60 0x7fefdc0fb0f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fefc40048d0 tx=0x7fefc4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 -- 172.21.15.103:0/1105859755 shutdown_connections 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/1105859755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc101a60 0x7fefdc0fb0f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/1105859755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc1010c0 0x7fefdc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 -- 172.21.15.103:0/1105859755 >> 172.21.15.103:0/1105859755 conn(0x7fefdc0f6c70 msgr2=0x7fefdc0f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 -- 172.21.15.103:0/1105859755 shutdown_connections 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.196+0000 7fefe0f5d640 1 -- 172.21.15.103:0/1105859755 wait complete. 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 Processor -- start 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 -- start start 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc101a60 0x7fefdc112150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc112750 con 0x7fefdc1010c0 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefe0f5d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc10bda0 con 0x7fefdc101a60 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33016/0 (socket says 172.21.15.103:33016) 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 -- 172.21.15.103:0/4130131371 learned_addr learned my addr 172.21.15.103:0/4130131371 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefd1d9b640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc101a60 0x7fefdc112150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 -- 172.21.15.103:0/4130131371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc101a60 msgr2=0x7fefdc112150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc101a60 0x7fefdc112150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 -- 172.21.15.103:0/4130131371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4005040 con 0x7fefdc1010c0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefda59c640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c790 tx=0x7fefcc0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.197+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefcc00a820 con 0x7fefdc1010c0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.198+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc10c080 con 0x7fefdc1010c0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.198+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefcc00ae50 con 0x7fefdc1010c0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.198+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefcc0115e0 con 0x7fefdc1010c0 2024-03-22T02:55:14.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.198+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc10c570 con 0x7fefdc1010c0 2024-03-22T02:55:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.199+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc0fc830 con 0x7fefdc1010c0 2024-03-22T02:55:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.199+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fefcc012400 con 0x7fefdc1010c0 2024-03-22T02:55:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.199+0000 7fefd37fe640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 0x7fefc80707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.200+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fefcc08dfb0 con 0x7fefdc1010c0 2024-03-22T02:55:14.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.200+0000 7fefd1d9b640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 0x7fefc80707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.200+0000 7fefd1d9b640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 0x7fefc80707c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f96b0 tx=0x7fefc4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.204+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefcc0c2740 con 0x7fefdc1010c0 2024-03-22T02:55:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:14 smithi122 ceph-mon[83368]: pgmap v711: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:14 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2161695018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:14 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2648304274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:14.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:14 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2223580519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:14 smithi103 ceph-mon[115211]: pgmap v711: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:14 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2161695018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:14 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2648304274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:14.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:14 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2223580519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.434+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefdc108220 con 0x7fefdc1010c0 2024-03-22T02:55:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.435+0000 7fefd37fe640 1 -- 172.21.15.103:0/4130131371 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5244 (secure 0 0 0) 0x7fefcc057230 con 0x7fefdc1010c0 2024-03-22T02:55:14.438 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:14.439 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:26.631347+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34380,"mds_1":34412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34380":{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":0,"incarnation":33,"state":"up:active","state_seq":244,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":1,"incarnation":33,"state":"up:active","state_seq":240,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34380,"qdb_cluster":[34380,34412]},"id":1}]} 2024-03-22T02:55:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 37 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 msgr2=0x7fefc80707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 0x7fefc80707c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f96b0 tx=0x7fefc4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 msgr2=0x7fefdc111c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c790 tx=0x7fefcc0043c0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 shutdown_connections 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fefc806e310 0x7fefc80707c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefdc101a60 0x7fefdc112150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 --2- 172.21.15.103:0/4130131371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefdc1010c0 0x7fefdc111c10 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 >> 172.21.15.103:0/4130131371 conn(0x7fefdc0f6c70 msgr2=0x7fefdc0f7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 shutdown_connections 2024-03-22T02:55:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.437+0000 7fefe0f5d640 1 -- 172.21.15.103:0/4130131371 wait complete. 2024-03-22T02:55:14.454 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 17, 'max_mds': 2, 'flags': 18} 2024-03-22T02:55:14.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- 172.21.15.103:0/2254518287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c0643a0 msgr2=0x7f620c0647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 --2- 172.21.15.103:0/2254518287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c0643a0 0x7f620c0647d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f61f80048d0 tx=0x7f61f8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- 172.21.15.103:0/2254518287 shutdown_connections 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 --2- 172.21.15.103:0/2254518287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c0643a0 0x7f620c0647d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 --2- 172.21.15.103:0/2254518287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c102240 0x7f620c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- 172.21.15.103:0/2254518287 >> 172.21.15.103:0/2254518287 conn(0x7f620c0710b0 msgr2=0x7f620c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- 172.21.15.103:0/2254518287 shutdown_connections 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- 172.21.15.103:0/2254518287 wait complete. 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 Processor -- start 2024-03-22T02:55:14.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.564+0000 7f6213114640 1 -- start start 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6213114640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c0643a0 0x7f620c112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6213114640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6213114640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c113810 con 0x7f620c102240 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6213114640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c10ce60 con 0x7f620c0643a0 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33030/0 (socket says 172.21.15.103:33030) 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 -- 172.21.15.103:0/187209400 learned_addr learned my addr 172.21.15.103:0/187209400 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6210eb0640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c0643a0 0x7f620c112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 -- 172.21.15.103:0/187209400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c0643a0 msgr2=0x7f620c112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c0643a0 0x7f620c112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 -- 172.21.15.103:0/187209400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8005040 con 0x7f620c102240 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f620bfff640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f61f8004930 tx=0x7f61f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f80094c0 con 0x7f620c102240 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.565+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c10d0e0 con 0x7f620c102240 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.566+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61f8032e50 con 0x7f620c102240 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.566+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f80433b0 con 0x7f620c102240 2024-03-22T02:55:14.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.566+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c10d5a0 con 0x7f620c102240 2024-03-22T02:55:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.567+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c0fd940 con 0x7f620c102240 2024-03-22T02:55:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.567+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f61f802f070 con 0x7f620c102240 2024-03-22T02:55:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.567+0000 7f62097fa640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 0x7f61fc0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.568+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f61f80afc20 con 0x7f620c102240 2024-03-22T02:55:14.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.568+0000 7f6210eb0640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 0x7f61fc0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.568+0000 7f6210eb0640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 0x7f61fc0707c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6200004ed0 tx=0x7f620000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.572+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f61f8078ed0 con 0x7f620c102240 2024-03-22T02:55:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.791+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7f620c109320 con 0x7f620c102240 2024-03-22T02:55:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.792+0000 7f62097fa640 1 -- 172.21.15.103:0/187209400 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v37) v1 ==== 107+0+4885 (secure 0 0 0) 0x7f61f807c590 con 0x7f620c102240 2024-03-22T02:55:14.794 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:14.794 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:27:58.738953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14472,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi103.xhmfjp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6829/2193581075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2193581075},{"type":"v1","addr":"172.21.15.103:6829","nonce":2193581075}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi122.ztrkgp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6825/1237228570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1237228570},{"type":"v1","addr":"172.21.15.122:6825","nonce":1237228570}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:14.794 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 18 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 msgr2=0x7f61fc0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 0x7f61fc0707c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6200004ed0 tx=0x7f620000b040 comp rx=0 tx=0).stop 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 msgr2=0x7f620c113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f61f8004930 tx=0x7f61f8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 shutdown_connections 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f61fc06e310 0x7f61fc0707c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c102240 0x7f620c113210 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 --2- 172.21.15.103:0/187209400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f620c0643a0 0x7f620c112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 >> 172.21.15.103:0/187209400 conn(0x7f620c0710b0 msgr2=0x7f620c0f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 shutdown_connections 2024-03-22T02:55:14.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.794+0000 7f6213114640 1 -- 172.21.15.103:0/187209400 wait complete. 2024-03-22T02:55:14.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 -- 172.21.15.103:0/2235853720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 msgr2=0x7f86c4101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 --2- 172.21.15.103:0/2235853720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c4101510 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f86b40048d0 tx=0x7f86b4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 -- 172.21.15.103:0/2235853720 shutdown_connections 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 --2- 172.21.15.103:0/2235853720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 0x7f86c40fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 --2- 172.21.15.103:0/2235853720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c4101510 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 -- 172.21.15.103:0/2235853720 >> 172.21.15.103:0/2235853720 conn(0x7f86c40710b0 msgr2=0x7f86c40714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:14.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 -- 172.21.15.103:0/2235853720 shutdown_connections 2024-03-22T02:55:14.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.909+0000 7f86c911d640 1 -- 172.21.15.103:0/2235853720 wait complete. 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 Processor -- start 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 -- start start 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 0x7f86c4111bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c41126f0 con 0x7f86c4101140 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c911d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c410bd40 con 0x7f86c40fb110 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33052/0 (socket says 172.21.15.103:33052) 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 -- 172.21.15.103:0/4183151422 learned_addr learned my addr 172.21.15.103:0/4183151422 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c2d9d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 0x7f86c4111bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 -- 172.21.15.103:0/4183151422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 msgr2=0x7f86c4111bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 0x7f86c4111bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.910+0000 7f86c259c640 1 -- 172.21.15.103:0/4183151422 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b4005040 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86c259c640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f86b8002a50 tx=0x7f86b8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b8007c30 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c410c020 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86b800f040 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86b800db50 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.911+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c410c540 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.912+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c40fc800 con 0x7f86c4101140 2024-03-22T02:55:14.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.912+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f86b801c400 con 0x7f86c4101140 2024-03-22T02:55:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.913+0000 7f86ab7fe640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 0x7f86ac0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.913+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86b808d7a0 con 0x7f86c4101140 2024-03-22T02:55:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.913+0000 7f86c2d9d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 0x7f86ac0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:14.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.913+0000 7f86c2d9d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 0x7f86ac0707c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009990 tx=0x7f86b4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:14.919 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:14.917+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f86b8056920 con 0x7f86c4101140 2024-03-22T02:55:15.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.199+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f86c4108220 con 0x7f86c4101140 2024-03-22T02:55:15.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.200+0000 7f86ab7fe640 1 -- 172.21.15.103:0/4183151422 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v37) v1 ==== 107+0+4886 (secure 0 0 0) 0x7f86b8059fe0 con 0x7f86c4101140 2024-03-22T02:55:15.202 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:15.202 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:29:14.569041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14472,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi103.xhmfjp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6829/2193581075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2193581075},{"type":"v1","addr":"172.21.15.103:6829","nonce":2193581075}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi122.ztrkgp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6825/1237228570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1237228570},{"type":"v1","addr":"172.21.15.122:6825","nonce":1237228570}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:15.202 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 19 2024-03-22T02:55:15.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.202+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 msgr2=0x7f86ac0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.202+0000 7f86c911d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 0x7f86ac0707c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009990 tx=0x7f86b4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.202+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 msgr2=0x7f86c41120f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.202+0000 7f86c911d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f86b8002a50 tx=0x7f86b8004600 comp rx=0 tx=0).stop 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 shutdown_connections 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f86ac06e310 0x7f86ac0707c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f86c4101140 0x7f86c41120f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 --2- 172.21.15.103:0/4183151422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86c40fb110 0x7f86c4111bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 >> 172.21.15.103:0/4183151422 conn(0x7f86c40710b0 msgr2=0x7f86c40f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 shutdown_connections 2024-03-22T02:55:15.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.203+0000 7f86c911d640 1 -- 172.21.15.103:0/4183151422 wait complete. 2024-03-22T02:55:15.215 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-03-22T02:55:15.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/500359811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a941022d0 msgr2=0x7f6a941026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/500359811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a941022d0 0x7f6a941026a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6a900048d0 tx=0x7f6a90009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/500359811 shutdown_connections 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/500359811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a94102c70 0x7f6a940fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/500359811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a941022d0 0x7f6a941026a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/500359811 >> 172.21.15.103:0/500359811 conn(0x7f6a940f7d90 msgr2=0x7f6a940fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/500359811 shutdown_connections 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.314+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/500359811 wait complete. 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 Processor -- start 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 -- start start 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a941022d0 0x7f6a94112d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a941138d0 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a9bb9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9410cf20 con 0x7f6a941022d0 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33068/0 (socket says 172.21.15.103:33068) 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 -- 172.21.15.103:0/1391671872 learned_addr learned my addr 172.21.15.103:0/1391671872 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99939640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a941022d0 0x7f6a94112d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 -- 172.21.15.103:0/1391671872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a941022d0 msgr2=0x7f6a94112d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a941022d0 0x7f6a94112d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 -- 172.21.15.103:0/1391671872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a90005040 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a99138640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400c830 tx=0x7f6a84004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.315+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a8400a850 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.316+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a8400ae80 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.316+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9410d200 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.316+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a84012ea0 con 0x7f6a94102c70 2024-03-22T02:55:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.316+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9410d720 con 0x7f6a94102c70 2024-03-22T02:55:15.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.317+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9404bcc0 con 0x7f6a94102c70 2024-03-22T02:55:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.317+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6a84012400 con 0x7f6a94102c70 2024-03-22T02:55:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.317+0000 7f6a8a7fc640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 0x7f6a800707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.317+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6a8408e090 con 0x7f6a94102c70 2024-03-22T02:55:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.317+0000 7f6a99939640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 0x7f6a800707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.318+0000 7f6a99939640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 0x7f6a800707c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a90003080 tx=0x7f6a90009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:15.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.324+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a84057310 con 0x7f6a94102c70 2024-03-22T02:55:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:15 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1936564257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:15 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4130131371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:55:15.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:15 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/187209400' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-22T02:55:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:15 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1936564257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:15 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4130131371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:55:15.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:15 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/187209400' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-03-22T02:55:15.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.543+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f6a9410e400 con 0x7f6a94102c70 2024-03-22T02:55:15.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.543+0000 7f6a8a7fc640 1 -- 172.21.15.103:0/1391671872 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v37) v1 ==== 107+0+4900 (secure 0 0 0) 0x7f6a840126b0 con 0x7f6a94102c70 2024-03-22T02:55:15.545 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:15.545 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:32:48.794547+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14472,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi103.xhmfjp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6829/2193581075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2193581075},{"type":"v1","addr":"172.21.15.103:6829","nonce":2193581075}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi122.ztrkgp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6825/1237228570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1237228570},{"type":"v1","addr":"172.21.15.122:6825","nonce":1237228570}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14472,"qdb_cluster":[14472,24271]},"id":1}]} 2024-03-22T02:55:15.546 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 20 2024-03-22T02:55:15.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 msgr2=0x7f6a800707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 0x7f6a800707c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a90003080 tx=0x7f6a90009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 msgr2=0x7f6a941132d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400c830 tx=0x7f6a84004430 comp rx=0 tx=0).stop 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 shutdown_connections 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6a8006e310 0x7f6a800707c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a94102c70 0x7f6a941132d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 --2- 172.21.15.103:0/1391671872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a941022d0 0x7f6a94112d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 >> 172.21.15.103:0/1391671872 conn(0x7f6a940f7d90 msgr2=0x7f6a940f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.545+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 shutdown_connections 2024-03-22T02:55:15.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.546+0000 7f6a9bb9d640 1 -- 172.21.15.103:0/1391671872 wait complete. 2024-03-22T02:55:15.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-22T02:55:15.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 -- 172.21.15.103:0/3822976662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 msgr2=0x7effa00fb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.657 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 --2- 172.21.15.103:0/3822976662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa00fb110 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0048d0 tx=0x7eff8c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 -- 172.21.15.103:0/3822976662 shutdown_connections 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 --2- 172.21.15.103:0/3822976662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa00fb110 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 --2- 172.21.15.103:0/3822976662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 0x7effa01014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 -- 172.21.15.103:0/3822976662 >> 172.21.15.103:0/3822976662 conn(0x7effa00f6cb0 msgr2=0x7effa00f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 -- 172.21.15.103:0/3822976662 shutdown_connections 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.655+0000 7effa6cae640 1 -- 172.21.15.103:0/3822976662 wait complete. 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 Processor -- start 2024-03-22T02:55:15.658 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 -- start start 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 0x7effa010ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa0066510 con 0x7effa0101a80 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7effa6cae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa010e980 con 0x7effa01010e0 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33076/0 (socket says 172.21.15.103:33076) 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.656+0000 7eff9ffff640 1 -- 172.21.15.103:0/2936337663 learned_addr learned my addr 172.21.15.103:0/2936337663 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7effa4a4a640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 0x7effa010ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9ffff640 1 -- 172.21.15.103:0/2936337663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 msgr2=0x7effa010ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9ffff640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 0x7effa010ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9ffff640 1 -- 172.21.15.103:0/2936337663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c005040 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9ffff640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7eff8c004930 tx=0x7eff8c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff8c0094c0 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa010ec00 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff8c032e50 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff8c0433b0 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.657+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa0117130 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.658+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa00fc850 con 0x7effa0101a80 2024-03-22T02:55:15.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.659+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7eff8c02f070 con 0x7effa0101a80 2024-03-22T02:55:15.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.659+0000 7eff9d7fa640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 0x7eff900707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:15.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.659+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff8c0afbf0 con 0x7effa0101a80 2024-03-22T02:55:15.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.659+0000 7effa4a4a640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 0x7eff900707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:15.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.660+0000 7effa4a4a640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 0x7eff900707c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff94004ed0 tx=0x7eff9400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:15.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.665+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff8c0b6080 con 0x7effa0101a80 2024-03-22T02:55:15.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.885+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7effa01082a0 con 0x7effa0101a80 2024-03-22T02:55:15.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.886+0000 7eff9d7fa640 1 -- 172.21.15.103:0/2936337663 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v37) v1 ==== 107+0+4899 (secure 0 0 0) 0x7eff8c078ea0 con 0x7effa0101a80 2024-03-22T02:55:15.888 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:15.888 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:32:54.641310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14472,"mds_1":24271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi103.xhmfjp","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6829/2193581075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2193581075},{"type":"v1","addr":"172.21.15.103:6829","nonce":2193581075}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi122.ztrkgp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.122:6825/1237228570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1237228570},{"type":"v1","addr":"172.21.15.122:6825","nonce":1237228570}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14472,"qdb_cluster":[24271,14472]},"id":1}]} 2024-03-22T02:55:15.888 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 21 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 msgr2=0x7eff900707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 0x7eff900707c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff94004ed0 tx=0x7eff9400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 msgr2=0x7effa010e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7eff8c004930 tx=0x7eff8c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 shutdown_connections 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7eff9006e310 0x7eff900707c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effa0101a80 0x7effa010e410 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 --2- 172.21.15.103:0/2936337663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effa01010e0 0x7effa010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 >> 172.21.15.103:0/2936337663 conn(0x7effa00f6cb0 msgr2=0x7effa00f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:15.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 shutdown_connections 2024-03-22T02:55:15.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:15.888+0000 7effa6cae640 1 -- 172.21.15.103:0/2936337663 wait complete. 2024-03-22T02:55:15.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-22T02:55:16.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 -- 172.21.15.103:0/975742289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 msgr2=0x7fa2740fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 --2- 172.21.15.103:0/975742289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa2740fc1e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa2640048d0 tx=0x7fa264009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 -- 172.21.15.103:0/975742289 shutdown_connections 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 --2- 172.21.15.103:0/975742289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa2740fc1e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 --2- 172.21.15.103:0/975742289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 0x7fa274102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 -- 172.21.15.103:0/975742289 >> 172.21.15.103:0/975742289 conn(0x7fa2740f7dc0 msgr2=0x7fa2740fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 -- 172.21.15.103:0/975742289 shutdown_connections 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.003+0000 7fa27b82e640 1 -- 172.21.15.103:0/975742289 wait complete. 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 Processor -- start 2024-03-22T02:55:16.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 -- start start 2024-03-22T02:55:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 0x7fa274114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274115a20 con 0x7fa274102b50 2024-03-22T02:55:16.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa27b82e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27410f070 con 0x7fa2741021b0 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa278dc9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa278dc9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33098/0 (socket says 172.21.15.103:33098) 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.004+0000 7fa278dc9640 1 -- 172.21.15.103:0/3025984978 learned_addr learned my addr 172.21.15.103:0/3025984978 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa2795ca640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 0x7fa274114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa278dc9640 1 -- 172.21.15.103:0/3025984978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 msgr2=0x7fa274114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa278dc9640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 0x7fa274114ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa278dc9640 1 -- 172.21.15.103:0/3025984978 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264005040 con 0x7fa274102b50 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa278dc9640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa264004930 tx=0x7fa264009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2640094c0 con 0x7fa274102b50 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa264032e50 con 0x7fa274102b50 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27410f2f0 con 0x7fa274102b50 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2640433b0 con 0x7fa274102b50 2024-03-22T02:55:16.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.005+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27410f7b0 con 0x7fa274102b50 2024-03-22T02:55:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.007+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2740fd920 con 0x7fa274102b50 2024-03-22T02:55:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.007+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa26402f070 con 0x7fa274102b50 2024-03-22T02:55:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.007+0000 7fa269ffb640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 0x7fa2600707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.007+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa2640afaa0 con 0x7fa274102b50 2024-03-22T02:55:16.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.007+0000 7fa2795ca640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 0x7fa2600707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.008+0000 7fa2795ca640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 0x7fa2600707c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa270009fd0 tx=0x7fa27000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.013+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa264078d50 con 0x7fa274102b50 2024-03-22T02:55:16.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.244+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7fa2741092f0 con 0x7fa274102b50 2024-03-22T02:55:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.245+0000 7fa269ffb640 1 -- 172.21.15.103:0/3025984978 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v37) v1 ==== 107+0+3299 (secure 0 0 0) 0x7fa26403fd00 con 0x7fa274102b50 2024-03-22T02:55:16.247 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:16.247 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 22 2024-03-22T02:55:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 msgr2=0x7fa2600707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 0x7fa2600707c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa270009fd0 tx=0x7fa27000b040 comp rx=0 tx=0).stop 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 msgr2=0x7fa274115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa264004930 tx=0x7fa264009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 shutdown_connections 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa26006e310 0x7fa2600707c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa274102b50 0x7fa274115420 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 --2- 172.21.15.103:0/3025984978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2741021b0 0x7fa274114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.247+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 >> 172.21.15.103:0/3025984978 conn(0x7fa2740f7dc0 msgr2=0x7fa274071050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.248+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 shutdown_connections 2024-03-22T02:55:16.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.248+0000 7fa27b82e640 1 -- 172.21.15.103:0/3025984978 wait complete. 2024-03-22T02:55:16.263 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:16 smithi103 ceph-mon[115211]: pgmap v712: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:16.263 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4183151422' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-22T02:55:16.263 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1391671872' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-22T02:55:16.263 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:16 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2936337663' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-22T02:55:16.264 DEBUG:tasks.fs:max_mds not reduced in epoch 22 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:16.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-22T02:55:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:16 smithi122 ceph-mon[83368]: pgmap v712: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4183151422' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-22T02:55:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1391671872' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-22T02:55:16.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:16 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2936337663' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-22T02:55:16.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 -- 172.21.15.103:0/283230261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 msgr2=0x7f6da0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 --2- 172.21.15.103:0/283230261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0108780 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6d900048d0 tx=0x7f6d90009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 -- 172.21.15.103:0/283230261 shutdown_connections 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 --2- 172.21.15.103:0/283230261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0108780 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 --2- 172.21.15.103:0/283230261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 0x7f6da0064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 -- 172.21.15.103:0/283230261 >> 172.21.15.103:0/283230261 conn(0x7f6da00710b0 msgr2=0x7f6da00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 -- 172.21.15.103:0/283230261 shutdown_connections 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.363+0000 7f6da762e640 1 -- 172.21.15.103:0/283230261 wait complete. 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 Processor -- start 2024-03-22T02:55:16.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 -- start start 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 0x7f6da0114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da0115a20 con 0x7f6da0064cb0 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da762e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da010f070 con 0x7f6da00643a0 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33120/0 (socket says 172.21.15.103:33120) 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 -- 172.21.15.103:0/2377431291 learned_addr learned my addr 172.21.15.103:0/2377431291 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da53ca640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 0x7f6da0114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 -- 172.21.15.103:0/2377431291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 msgr2=0x7f6da0114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 0x7f6da0114ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.364+0000 7f6da4bc9640 1 -- 172.21.15.103:0/2377431291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d90005040 con 0x7f6da0064cb0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6da4bc9640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6d90004930 tx=0x7f6d90009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d900094c0 con 0x7f6da0064cb0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da010f2f0 con 0x7f6da0064cb0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d90032e50 con 0x7f6da0064cb0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d900433b0 con 0x7f6da0064cb0 2024-03-22T02:55:16.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.365+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da010f7b0 con 0x7f6da0064cb0 2024-03-22T02:55:16.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.366+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da00fb730 con 0x7f6da0064cb0 2024-03-22T02:55:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.367+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f6d9002f070 con 0x7f6da0064cb0 2024-03-22T02:55:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.367+0000 7f6d95ffb640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 0x7f6d8c0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.367+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6d900afcd0 con 0x7f6da0064cb0 2024-03-22T02:55:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.367+0000 7f6da53ca640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 0x7f6d8c0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.368+0000 7f6da53ca640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 0x7f6d8c0707c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009fd0 tx=0x7f6d9c009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.372+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d90078ed0 con 0x7f6da0064cb0 2024-03-22T02:55:16.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.592+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f6da010b4f0 con 0x7f6da0064cb0 2024-03-22T02:55:16.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.592+0000 7f6d95ffb640 1 -- 172.21.15.103:0/2377431291 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v37) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f6d9007c590 con 0x7f6da0064cb0 2024-03-22T02:55:16.593 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:16.593 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/3867073217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3867073217},{"type":"v1","addr":"172.21.15.103:6827","nonce":3867073217}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14},{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":44253,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/1635524194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1635524194},{"type":"v1","addr":"172.21.15.103:6829","nonce":1635524194}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/615549338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":615549338},{"type":"v1","addr":"172.21.15.122:6825","nonce":615549338}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:16.594 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 23 2024-03-22T02:55:16.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 msgr2=0x7f6d8c0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 0x7f6d8c0707c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009fd0 tx=0x7f6d9c009e50 comp rx=0 tx=0).stop 2024-03-22T02:55:16.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 msgr2=0x7f6da0115420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6d90004930 tx=0x7f6d90009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 shutdown_connections 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f6d8c06e310 0x7f6d8c0707c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6da0064cb0 0x7f6da0115420 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.594+0000 7f6da762e640 1 --2- 172.21.15.103:0/2377431291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6da00643a0 0x7f6da0114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.595+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 >> 172.21.15.103:0/2377431291 conn(0x7f6da00710b0 msgr2=0x7f6da00f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.595+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 shutdown_connections 2024-03-22T02:55:16.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.595+0000 7f6da762e640 1 -- 172.21.15.103:0/2377431291 wait complete. 2024-03-22T02:55:16.607 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:16.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-22T02:55:16.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/2207024130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf80643a0 msgr2=0x7f4cf8064770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/2207024130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf80643a0 0x7f4cf8064770 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80048d0 tx=0x7f4ce8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/2207024130 shutdown_connections 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/2207024130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/2207024130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf80643a0 0x7f4cf8064770 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/2207024130 >> 172.21.15.103:0/2207024130 conn(0x7f4cf80710b0 msgr2=0x7f4cf80714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/2207024130 shutdown_connections 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/2207024130 wait complete. 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 Processor -- start 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.715+0000 7f4cfcb5d640 1 -- start start 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 0x7f4cf8114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf8115a50 con 0x7f4cf8064cb0 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf810f0a0 con 0x7f4cf80643a0 2024-03-22T02:55:16.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33134/0 (socket says 172.21.15.103:33134) 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 -- 172.21.15.103:0/1184607050 learned_addr learned my addr 172.21.15.103:0/1184607050 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf659c640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 0x7f4cf8114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 -- 172.21.15.103:0/1184607050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 msgr2=0x7f4cf8114f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 0x7f4cf8114f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 -- 172.21.15.103:0/1184607050 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce8005040 con 0x7f4cf8064cb0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf659c640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 0x7f4cf8114f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cf5d9b640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00eb70 tx=0x7f4cec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cec007c30 con 0x7f4cf8064cb0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf810f320 con 0x7f4cf8064cb0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cec00f040 con 0x7f4cf8064cb0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cec0095b0 con 0x7f4cf8064cb0 2024-03-22T02:55:16.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.716+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf810f840 con 0x7f4cf8064cb0 2024-03-22T02:55:16.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.717+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80fb730 con 0x7f4cf8064cb0 2024-03-22T02:55:16.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.718+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4cec009730 con 0x7f4cf8064cb0 2024-03-22T02:55:16.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.718+0000 7f4ce6ffd640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 0x7f4ce00707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:16.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.718+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4cec08db60 con 0x7f4cf8064cb0 2024-03-22T02:55:16.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.718+0000 7f4cf659c640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 0x7f4ce00707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:16.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.719+0000 7f4cf659c640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 0x7f4ce00707c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:16.723 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:16.723+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cec056d60 con 0x7f4cf8064cb0 2024-03-22T02:55:17.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.030+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f4cf810b550 con 0x7f4cf8064cb0 2024-03-22T02:55:17.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.031+0000 7f4ce6ffd640 1 -- 172.21.15.103:0/1184607050 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v37) v1 ==== 107+0+4078 (secure 0 0 0) 0x7f4cec05a420 con 0x7f4cf8064cb0 2024-03-22T02:55:17.032 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:17.032 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":44253,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/1635524194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1635524194},{"type":"v1","addr":"172.21.15.103:6829","nonce":1635524194}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/615549338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":615549338},{"type":"v1","addr":"172.21.15.122:6825","nonce":615549338}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:17.033 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 msgr2=0x7f4ce00707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 0x7f4ce00707c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 msgr2=0x7f4cf8115450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00eb70 tx=0x7f4cec00b040 comp rx=0 tx=0).stop 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 shutdown_connections 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f4ce006e310 0x7f4ce00707c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf8064cb0 0x7f4cf8115450 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 --2- 172.21.15.103:0/1184607050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4cf80643a0 0x7f4cf8114f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 >> 172.21.15.103:0/1184607050 conn(0x7f4cf80710b0 msgr2=0x7f4cf80f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 shutdown_connections 2024-03-22T02:55:17.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.033+0000 7f4cfcb5d640 1 -- 172.21.15.103:0/1184607050 wait complete. 2024-03-22T02:55:17.045 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:17.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-22T02:55:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- 172.21.15.103:0/1893151735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b40643a0 msgr2=0x7f06b4064770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 --2- 172.21.15.103:0/1893151735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b40643a0 0x7f06b4064770 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f06b00048d0 tx=0x7f06b0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- 172.21.15.103:0/1893151735 shutdown_connections 2024-03-22T02:55:17.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 --2- 172.21.15.103:0/1893151735 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b4064cb0 0x7f06b4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 --2- 172.21.15.103:0/1893151735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b40643a0 0x7f06b4064770 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- 172.21.15.103:0/1893151735 >> 172.21.15.103:0/1893151735 conn(0x7f06b40710b0 msgr2=0x7f06b40714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- 172.21.15.103:0/1893151735 shutdown_connections 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- 172.21.15.103:0/1893151735 wait complete. 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 Processor -- start 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- start start 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40643a0 0x7f06b4114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b4115a50 con 0x7f06b4064cb0 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.147+0000 7f06bc39d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b410f0a0 con 0x7f06b40643a0 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33154/0 (socket says 172.21.15.103:33154) 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 -- 172.21.15.103:0/4151420566 learned_addr learned my addr 172.21.15.103:0/4151420566 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:17.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06ba139640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40643a0 0x7f06b4114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 -- 172.21.15.103:0/4151420566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40643a0 msgr2=0x7f06b4114f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40643a0 0x7f06b4114f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 -- 172.21.15.103:0/4151420566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0005040 con 0x7f06b4064cb0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06b9938640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f06a4002a50 tx=0x7f06a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a4007c30 con 0x7f06b4064cb0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b410f380 con 0x7f06b4064cb0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06a400f040 con 0x7f06b4064cb0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a400db50 con 0x7f06b4064cb0 2024-03-22T02:55:17.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.148+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b410f8a0 con 0x7f06b4064cb0 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.149+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b40fb730 con 0x7f06b4064cb0 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.150+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f06a401c400 con 0x7f06b4064cb0 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.150+0000 7f06aaffd640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 0x7f06a00707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.150+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f06a408d6f0 con 0x7f06b4064cb0 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.150+0000 7f06ba139640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 0x7f06a00707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.151+0000 7f06ba139640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 0x7f06a00707c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f06b0003080 tx=0x7f06b0009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.155+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06a40568f0 con 0x7f06b4064cb0 2024-03-22T02:55:17.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:17 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3025984978' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-22T02:55:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:17 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2377431291' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-22T02:55:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:17.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:17 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1184607050' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-22T02:55:17.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:17 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3025984978' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-22T02:55:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:17 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2377431291' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-22T02:55:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:17.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:17 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1184607050' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-22T02:55:17.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.387+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f06b410b520 con 0x7f06b4064cb0 2024-03-22T02:55:17.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.388+0000 7f06aaffd640 1 -- 172.21.15.103:0/4151420566 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v37) v1 ==== 107+0+4928 (secure 0 0 0) 0x7f06a4059fb0 con 0x7f06b4064cb0 2024-03-22T02:55:17.389 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:17.389 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44253,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/1635524194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1635524194},{"type":"v1","addr":"172.21.15.103:6829","nonce":1635524194}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/615549338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":615549338},{"type":"v1","addr":"172.21.15.122:6825","nonce":615549338}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:17.389 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 25 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 msgr2=0x7f06a00707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 0x7f06a00707c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f06b0003080 tx=0x7f06b0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 msgr2=0x7f06b4115450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f06a4002a50 tx=0x7f06a4004600 comp rx=0 tx=0).stop 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 shutdown_connections 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f06a006e310 0x7f06a00707c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06b4064cb0 0x7f06b4115450 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 --2- 172.21.15.103:0/4151420566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06b40643a0 0x7f06b4114f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 >> 172.21.15.103:0/4151420566 conn(0x7f06b40710b0 msgr2=0x7f06b40f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 shutdown_connections 2024-03-22T02:55:17.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.390+0000 7f06bc39d640 1 -- 172.21.15.103:0/4151420566 wait complete. 2024-03-22T02:55:17.402 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:17.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 -- 172.21.15.103:0/1495932690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 msgr2=0x7f7cac101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 --2- 172.21.15.103:0/1495932690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac101510 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0048d0 tx=0x7f7c9c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 -- 172.21.15.103:0/1495932690 shutdown_connections 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 --2- 172.21.15.103:0/1495932690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 0x7f7cac0fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 --2- 172.21.15.103:0/1495932690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac101510 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 -- 172.21.15.103:0/1495932690 >> 172.21.15.103:0/1495932690 conn(0x7f7cac0710b0 msgr2=0x7f7cac0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 -- 172.21.15.103:0/1495932690 shutdown_connections 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.501+0000 7f7cb1607640 1 -- 172.21.15.103:0/1495932690 wait complete. 2024-03-22T02:55:17.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 Processor -- start 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 -- start start 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 0x7f7cac112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac1138e0 con 0x7f7cac101140 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7cb1607640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac10cea0 con 0x7f7cac0fb110 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33172/0 (socket says 172.21.15.103:33172) 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 -- 172.21.15.103:0/3647118981 learned_addr learned my addr 172.21.15.103:0/3647118981 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7caaffd640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 0x7f7cac112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 -- 172.21.15.103:0/3647118981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 msgr2=0x7f7cac112d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 0x7f7cac112d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 -- 172.21.15.103:0/3647118981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c005040 con 0x7f7cac101140 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.502+0000 7f7ca3fff640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400b790 tx=0x7f7c940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.503+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c9400c700 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.503+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac10d180 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.503+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c9400cd30 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.503+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c94013370 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.503+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac10d6a0 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.504+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac0fc800 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.504+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7c94012070 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.504+0000 7f7ca37fe640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 0x7f7c980707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.505+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c9408da30 con 0x7f7cac101140 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.505+0000 7f7caaffd640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 0x7f7c980707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.505+0000 7f7caaffd640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 0x7f7c980707c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c007ac0 tx=0x7f7c9c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.509+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c9408fe60 con 0x7f7cac101140 2024-03-22T02:55:17.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.730+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f7cac04bcc0 con 0x7f7cac101140 2024-03-22T02:55:17.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.730+0000 7f7ca37fe640 1 -- 172.21.15.103:0/3647118981 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v37) v1 ==== 107+0+4146 (secure 0 0 0) 0x7f7c94056c30 con 0x7f7cac101140 2024-03-22T02:55:17.732 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:17.732 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":44255,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/615549338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":615549338},{"type":"v1","addr":"172.21.15.122:6825","nonce":615549338}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:17.732 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 26 2024-03-22T02:55:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 msgr2=0x7f7c980707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 0x7f7c980707c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c007ac0 tx=0x7f7c9c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 msgr2=0x7f7cac113250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400b790 tx=0x7f7c940091d0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 shutdown_connections 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7c9806e310 0x7f7c980707c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cac101140 0x7f7cac113250 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 --2- 172.21.15.103:0/3647118981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cac0fb110 0x7f7cac112d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.732+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 >> 172.21.15.103:0/3647118981 conn(0x7f7cac0710b0 msgr2=0x7f7cac0643a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.733+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 shutdown_connections 2024-03-22T02:55:17.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.733+0000 7f7cb1607640 1 -- 172.21.15.103:0/3647118981 wait complete. 2024-03-22T02:55:17.745 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:17.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 -- 172.21.15.103:0/2246297689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 msgr2=0x7f49c4102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 --2- 172.21.15.103:0/2246297689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4102610 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f49b40048d0 tx=0x7f49b4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 -- 172.21.15.103:0/2246297689 shutdown_connections 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 --2- 172.21.15.103:0/2246297689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 0x7f49c40647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 --2- 172.21.15.103:0/2246297689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4102610 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 -- 172.21.15.103:0/2246297689 >> 172.21.15.103:0/2246297689 conn(0x7f49c40710b0 msgr2=0x7f49c40714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 -- 172.21.15.103:0/2246297689 shutdown_connections 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.843+0000 7f49c979d640 1 -- 172.21.15.103:0/2246297689 wait complete. 2024-03-22T02:55:17.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 Processor -- start 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 -- start start 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 0x7f49c4112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c4113810 con 0x7f49c4102240 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c979d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c410ce60 con 0x7f49c40643a0 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33188/0 (socket says 172.21.15.103:33188) 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 -- 172.21.15.103:0/1521972589 learned_addr learned my addr 172.21.15.103:0/1521972589 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c2ffd640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 0x7f49c4112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 -- 172.21.15.103:0/1521972589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 msgr2=0x7f49c4112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 0x7f49c4112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 -- 172.21.15.103:0/1521972589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4005040 con 0x7f49c4102240 2024-03-22T02:55:17.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.844+0000 7f49c27fc640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c790 tx=0x7f49b80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.845+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b800a820 con 0x7f49c4102240 2024-03-22T02:55:17.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.845+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49b800ae50 con 0x7f49c4102240 2024-03-22T02:55:17.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.845+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c410d140 con 0x7f49c4102240 2024-03-22T02:55:17.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.845+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b80115e0 con 0x7f49c4102240 2024-03-22T02:55:17.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.845+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c410d660 con 0x7f49c4102240 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.846+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c40fd940 con 0x7f49c4102240 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.846+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f49b8012400 con 0x7f49c4102240 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.846+0000 7f49ab7fe640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 0x7f49ac0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.847+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49b808def0 con 0x7f49c4102240 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.847+0000 7f49c2ffd640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 0x7f49ac0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:17.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.847+0000 7f49c2ffd640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 0x7f49ac0707c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49b4003080 tx=0x7f49b4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:17.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:17.851+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49b8057170 con 0x7f49c4102240 2024-03-22T02:55:18.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.107+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f49c4109320 con 0x7f49c4102240 2024-03-22T02:55:18.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.107+0000 7f49ab7fe640 1 -- 172.21.15.103:0/1521972589 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v37) v1 ==== 107+0+4996 (secure 0 0 0) 0x7f49b805a830 con 0x7f49c4102240 2024-03-22T02:55:18.109 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:18.109 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44255,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/615549338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":615549338},{"type":"v1","addr":"172.21.15.122:6825","nonce":615549338}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:18.109 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 27 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.109+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 msgr2=0x7f49ac0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.109+0000 7f49c979d640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 0x7f49ac0707c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f49b4003080 tx=0x7f49b4009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 msgr2=0x7f49c4113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c790 tx=0x7f49b80043c0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 shutdown_connections 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f49ac06e310 0x7f49ac0707c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f49c4102240 0x7f49c4113210 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 --2- 172.21.15.103:0/1521972589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c40643a0 0x7f49c4112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 >> 172.21.15.103:0/1521972589 conn(0x7f49c40710b0 msgr2=0x7f49c40f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 shutdown_connections 2024-03-22T02:55:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.110+0000 7f49c979d640 1 -- 172.21.15.103:0/1521972589 wait complete. 2024-03-22T02:55:18.122 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:18.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-22T02:55:18.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 -- 172.21.15.103:0/1529412075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 msgr2=0x7fcbf8102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/1529412075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8102610 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80048d0 tx=0x7fcbe8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 -- 172.21.15.103:0/1529412075 shutdown_connections 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/1529412075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 0x7fcbf80647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/1529412075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8102610 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 -- 172.21.15.103:0/1529412075 >> 172.21.15.103:0/1529412075 conn(0x7fcbf80710b0 msgr2=0x7fcbf80714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 -- 172.21.15.103:0/1529412075 shutdown_connections 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.223+0000 7fcbfe11d640 1 -- 172.21.15.103:0/1529412075 wait complete. 2024-03-22T02:55:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 Processor -- start 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 -- start start 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 0x7fcbf8112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8113810 con 0x7fcbf8102240 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbfe11d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf810ce60 con 0x7fcbf80643a0 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33208/0 (socket says 172.21.15.103:33208) 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 -- 172.21.15.103:0/3076341959 learned_addr learned my addr 172.21.15.103:0/3076341959 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf77fe640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 0x7fcbf8112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 -- 172.21.15.103:0/3076341959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 msgr2=0x7fcbf8112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 0x7fcbf8112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.224+0000 7fcbf6ffd640 1 -- 172.21.15.103:0/3076341959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe8005040 con 0x7fcbf8102240 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbf6ffd640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcbec002a50 tx=0x7fcbec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbec007c30 con 0x7fcbf8102240 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbec00f040 con 0x7fcbf8102240 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf810d140 con 0x7fcbf8102240 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbec00db50 con 0x7fcbf8102240 2024-03-22T02:55:18.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.225+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf810d660 con 0x7fcbf8102240 2024-03-22T02:55:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.226+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf80fd940 con 0x7fcbf8102240 2024-03-22T02:55:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.227+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fcbec01c400 con 0x7fcbf8102240 2024-03-22T02:55:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.228+0000 7fcbdffff640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 0x7fcbe00707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.228+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcbec08d570 con 0x7fcbf8102240 2024-03-22T02:55:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.228+0000 7fcbf77fe640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 0x7fcbe00707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.228+0000 7fcbf77fe640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 0x7fcbe00707c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8003080 tx=0x7fcbe8009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:18.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.233+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbec0568a0 con 0x7fcbf8102240 2024-03-22T02:55:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:18 smithi122 ceph-mon[83368]: pgmap v713: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:18 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/4151420566' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-22T02:55:18.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:18 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3647118981' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-22T02:55:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:18 smithi103 ceph-mon[115211]: pgmap v713: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:18 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/4151420566' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-22T02:55:18.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:18 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3647118981' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-22T02:55:18.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.490+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fcbf8109320 con 0x7fcbf8102240 2024-03-22T02:55:18.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.491+0000 7fcbdffff640 1 -- 172.21.15.103:0/3076341959 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v37) v1 ==== 107+0+4217 (secure 0 0 0) 0x7fcbec056020 con 0x7fcbf8102240 2024-03-22T02:55:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:18.492 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:18.492 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 28 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 msgr2=0x7fcbe00707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 0x7fcbe00707c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8003080 tx=0x7fcbe8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 msgr2=0x7fcbf8113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcbec002a50 tx=0x7fcbec004600 comp rx=0 tx=0).stop 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 shutdown_connections 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fcbe006e310 0x7fcbe00707c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcbf8102240 0x7fcbf8113210 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 --2- 172.21.15.103:0/3076341959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbf80643a0 0x7fcbf8112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 >> 172.21.15.103:0/3076341959 conn(0x7fcbf80710b0 msgr2=0x7fcbf80f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:18.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 shutdown_connections 2024-03-22T02:55:18.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.493+0000 7fcbfe11d640 1 -- 172.21.15.103:0/3076341959 wait complete. 2024-03-22T02:55:18.508 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:18.508 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.613+0000 7faeb695d640 1 -- 172.21.15.103:0/4278025068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 msgr2=0x7faeb0101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.613+0000 7faeb695d640 1 --2- 172.21.15.103:0/4278025068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb0101510 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7faea00048d0 tx=0x7faea0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- 172.21.15.103:0/4278025068 shutdown_connections 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 --2- 172.21.15.103:0/4278025068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 0x7faeb00fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 --2- 172.21.15.103:0/4278025068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb0101510 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- 172.21.15.103:0/4278025068 >> 172.21.15.103:0/4278025068 conn(0x7faeb00710b0 msgr2=0x7faeb00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- 172.21.15.103:0/4278025068 shutdown_connections 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- 172.21.15.103:0/4278025068 wait complete. 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 Processor -- start 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- start start 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 0x7faeb010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb0065310 con 0x7faeb0101140 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.614+0000 7faeb695d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb010e900 con 0x7faeb00fb110 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33230/0 (socket says 172.21.15.103:33230) 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 -- 172.21.15.103:0/1283664722 learned_addr learned my addr 172.21.15.103:0/1283664722 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaffff640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 0x7faeb010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 -- 172.21.15.103:0/1283664722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 msgr2=0x7faeb010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 0x7faeb010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 -- 172.21.15.103:0/1283664722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0005040 con 0x7faeb0101140 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeaf7fe640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faea400eb70 tx=0x7faea400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea4007c30 con 0x7faeb0101140 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb010eb80 con 0x7faeb0101140 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faea400f040 con 0x7faeb0101140 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea40095b0 con 0x7faeb0101140 2024-03-22T02:55:18.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.615+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb0117140 con 0x7faeb0101140 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.617+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb00fc800 con 0x7faeb0101140 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.617+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7faea4009730 con 0x7faeb0101140 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.618+0000 7faeacff9640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 0x7fae980707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.618+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faea408daa0 con 0x7faeb0101140 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.618+0000 7faeaffff640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 0x7fae980707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.619+0000 7faeaffff640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 0x7fae980707c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faea0009990 tx=0x7faea0009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:18.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.626+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faea4056d50 con 0x7faeb0101140 2024-03-22T02:55:18.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.888+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7faeb004bcc0 con 0x7faeb0101140 2024-03-22T02:55:18.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.889+0000 7faeacff9640 1 -- 172.21.15.103:0/1283664722 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v37) v1 ==== 107+0+5067 (secure 0 0 0) 0x7faea405a410 con 0x7faeb0101140 2024-03-22T02:55:18.889 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:18.889 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24285,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1723132276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1723132276},{"type":"v1","addr":"172.21.15.122:6827","nonce":1723132276}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:18.890 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 29 2024-03-22T02:55:18.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 msgr2=0x7fae980707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 0x7fae980707c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faea0009990 tx=0x7faea0009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 msgr2=0x7faeb010e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:18.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faea400eb70 tx=0x7faea400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 shutdown_connections 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fae9806e310 0x7fae980707c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeb0101140 0x7faeb010e3c0 secure :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faea400eb70 tx=0x7faea400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 --2- 172.21.15.103:0/1283664722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faeb00fb110 0x7faeb010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 >> 172.21.15.103:0/1283664722 conn(0x7faeb00710b0 msgr2=0x7faeb00f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.891+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 shutdown_connections 2024-03-22T02:55:18.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:18.892+0000 7faeb695d640 1 -- 172.21.15.103:0/1283664722 wait complete. 2024-03-22T02:55:18.907 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:18.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 -- 172.21.15.103:0/1899037240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 msgr2=0x7f88d00fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/1899037240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d00fc1e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f88b80048d0 tx=0x7f88b8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 -- 172.21.15.103:0/1899037240 shutdown_connections 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/1899037240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d00fc1e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/1899037240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 0x7f88d0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.016+0000 7f88d5a6e640 1 -- 172.21.15.103:0/1899037240 >> 172.21.15.103:0/1899037240 conn(0x7f88d00f7dc0 msgr2=0x7f88d00fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:19.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 -- 172.21.15.103:0/1899037240 shutdown_connections 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 -- 172.21.15.103:0/1899037240 wait complete. 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 Processor -- start 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 -- start start 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 0x7f88d0112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0113810 con 0x7f88d0102b50 2024-03-22T02:55:19.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.017+0000 7f88d5a6e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d010ce60 con 0x7f88d01021b0 2024-03-22T02:55:19.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33256/0 (socket says 172.21.15.103:33256) 2024-03-22T02:55:19.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 -- 172.21.15.103:0/3468886608 learned_addr learned my addr 172.21.15.103:0/3468886608 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:19.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88cf7fe640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 0x7f88d0112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 -- 172.21.15.103:0/3468886608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 msgr2=0x7f88d0112cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 0x7f88d0112cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 -- 172.21.15.103:0/3468886608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b8005040 con 0x7f88d0102b50 2024-03-22T02:55:19.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88c7fff640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f88b8004930 tx=0x7f88b8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88b80094c0 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d010d0e0 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88b8032e50 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88b80433b0 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.018+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d010d5a0 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.019+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d00fd920 con 0x7f88d0102b50 2024-03-22T02:55:19.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.020+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f88b802f070 con 0x7f88d0102b50 2024-03-22T02:55:19.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.020+0000 7f88ccff9640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 0x7f88bc0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.020+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f88b80afc80 con 0x7f88d0102b50 2024-03-22T02:55:19.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.020+0000 7f88cf7fe640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 0x7f88bc0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.021+0000 7f88cf7fe640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 0x7f88bc0707c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009470 tx=0x7f88c0009ed0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.025+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88b8078e80 con 0x7f88d0102b50 2024-03-22T02:55:19.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.283+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f88d01092f0 con 0x7f88d0102b50 2024-03-22T02:55:19.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.284+0000 7f88ccff9640 1 -- 172.21.15.103:0/3468886608 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v37) v1 ==== 107+0+4285 (secure 0 0 0) 0x7f88b807c540 con 0x7f88d0102b50 2024-03-22T02:55:19.285 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:19.285 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:19.285 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 30 2024-03-22T02:55:19.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 msgr2=0x7f88bc0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 0x7f88bc0707c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009470 tx=0x7f88c0009ed0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 msgr2=0x7f88d0113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f88b8004930 tx=0x7f88b8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 shutdown_connections 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f88bc06e310 0x7f88bc0707c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88d0102b50 0x7f88d0113210 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 --2- 172.21.15.103:0/3468886608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88d01021b0 0x7f88d0112cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.286+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 >> 172.21.15.103:0/3468886608 conn(0x7f88d00f7dc0 msgr2=0x7f88d00f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.287+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 shutdown_connections 2024-03-22T02:55:19.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.287+0000 7f88d5a6e640 1 -- 172.21.15.103:0/3468886608 wait complete. 2024-03-22T02:55:19.299 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:19 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1521972589' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-22T02:55:19.300 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:19 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3076341959' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-22T02:55:19.300 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:19 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1283664722' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-22T02:55:19.301 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:19.301 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-03-22T02:55:19.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:19 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1521972589' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-22T02:55:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:19 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3076341959' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-22T02:55:19.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:19 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1283664722' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-22T02:55:19.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 -- 172.21.15.103:0/3024931970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 msgr2=0x7f8608101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 --2- 172.21.15.103:0/3024931970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608101520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0048d0 tx=0x7f85fc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 -- 172.21.15.103:0/3024931970 shutdown_connections 2024-03-22T02:55:19.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 --2- 172.21.15.103:0/3024931970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f86080fbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 --2- 172.21.15.103:0/3024931970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608101520 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 -- 172.21.15.103:0/3024931970 >> 172.21.15.103:0/3024931970 conn(0x7f8608070880 msgr2=0x7f8608070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:19.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 -- 172.21.15.103:0/3024931970 shutdown_connections 2024-03-22T02:55:19.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.407+0000 7f860eb9d640 1 -- 172.21.15.103:0/3024931970 wait complete. 2024-03-22T02:55:19.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 Processor -- start 2024-03-22T02:55:19.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 -- start start 2024-03-22T02:55:19.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608113790 con 0x7f8608101150 2024-03-22T02:55:19.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f860eb9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860810cde0 con 0x7f86080fb9b0 2024-03-22T02:55:19.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.408+0000 7f8607fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860c939640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860c939640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.103:60786/0 (socket says 172.21.15.103:60786) 2024-03-22T02:55:19.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860c939640 1 -- 172.21.15.103:0/1021830177 learned_addr learned my addr 172.21.15.103:0/1021830177 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:19.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f8607fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33276/0 (socket says 172.21.15.103:33276) 2024-03-22T02:55:19.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f8607fff640 1 -- 172.21.15.103:0/1021830177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 msgr2=0x7f8608112c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f8607fff640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f8607fff640 1 -- 172.21.15.103:0/1021830177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc005040 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860c939640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f8607fff640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c7e0 tx=0x7f85f4007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f400a820 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85f400ae50 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860810d0c0 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f40113c0 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.409+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860810d5e0 con 0x7f8608101150 2024-03-22T02:55:19.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.410+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86080fc9c0 con 0x7f8608101150 2024-03-22T02:55:19.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.412+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f85f4012400 con 0x7f8608101150 2024-03-22T02:55:19.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.413+0000 7f86057fa640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 0x7f85f80707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.413+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f85f408ea30 con 0x7f8608101150 2024-03-22T02:55:19.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.413+0000 7f860c939640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 0x7f85f80707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.413+0000 7f860c939640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 0x7f85f80707c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009fa0 tx=0x7f85fc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.416+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f85f4057d70 con 0x7f8608101150 2024-03-22T02:55:19.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.696+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f860804bcc0 con 0x7f8608101150 2024-03-22T02:55:19.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.696+0000 7f86057fa640 1 -- 172.21.15.103:0/1021830177 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v37) v1 ==== 107+0+5135 (secure 0 0 0) 0x7f85f405b430 con 0x7f8608101150 2024-03-22T02:55:19.698 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:19.698 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:37:16.024479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:19.698 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 31 2024-03-22T02:55:19.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 msgr2=0x7f85f80707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 0x7f85f80707c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009fa0 tx=0x7f85fc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 msgr2=0x7f8608113190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85f400c7e0 tx=0x7f85f4007f20 comp rx=0 tx=0).stop 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 shutdown_connections 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f85f806e310 0x7f85f80707c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8608101150 0x7f8608113190 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 --2- 172.21.15.103:0/1021830177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86080fb9b0 0x7f8608112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.698+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 >> 172.21.15.103:0/1021830177 conn(0x7f8608070880 msgr2=0x7f8608104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.699+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 shutdown_connections 2024-03-22T02:55:19.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.699+0000 7f860eb9d640 1 -- 172.21.15.103:0/1021830177 wait complete. 2024-03-22T02:55:19.710 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-22T02:55:19.711 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-03-22T02:55:19.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- 172.21.15.103:0/3379450537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6360101a60 msgr2=0x7f63600fb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 --2- 172.21.15.103:0/3379450537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6360101a60 0x7f63600fb0f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f63500048d0 tx=0x7f6350009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- 172.21.15.103:0/3379450537 shutdown_connections 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 --2- 172.21.15.103:0/3379450537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6360101a60 0x7f63600fb0f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 --2- 172.21.15.103:0/3379450537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63601010c0 0x7f6360101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- 172.21.15.103:0/3379450537 >> 172.21.15.103:0/3379450537 conn(0x7f63600f6c90 msgr2=0x7f63600f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:19.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- 172.21.15.103:0/3379450537 shutdown_connections 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- 172.21.15.103:0/3379450537 wait complete. 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 Processor -- start 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.810+0000 7f6366618640 1 -- start start 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6360101a60 0x7f636010e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600664f0 con 0x7f63601010c0 2024-03-22T02:55:19.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636010e950 con 0x7f6360101a60 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33290/0 (socket says 172.21.15.103:33290) 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 -- 172.21.15.103:0/3929125783 learned_addr learned my addr 172.21.15.103:0/3929125783 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635f7fe640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6360101a60 0x7f636010e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 -- 172.21.15.103:0/3929125783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6360101a60 msgr2=0x7f636010e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6360101a60 0x7f636010e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 -- 172.21.15.103:0/3929125783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350005040 con 0x7f63601010c0 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635ffff640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6348002a50 tx=0x7f6348004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6348007c30 con 0x7f63601010c0 2024-03-22T02:55:19.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636010ec30 con 0x7f63601010c0 2024-03-22T02:55:19.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f634800f040 con 0x7f63601010c0 2024-03-22T02:55:19.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634800db50 con 0x7f63601010c0 2024-03-22T02:55:19.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.811+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63601171f0 con 0x7f63601010c0 2024-03-22T02:55:19.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.813+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63600fc830 con 0x7f63601010c0 2024-03-22T02:55:19.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.813+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f634801c400 con 0x7f63601010c0 2024-03-22T02:55:19.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.813+0000 7f635cff9640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 0x7f634c0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:19.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.813+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f634808d640 con 0x7f63601010c0 2024-03-22T02:55:19.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.813+0000 7f635f7fe640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 0x7f634c0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:19.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.814+0000 7f635f7fe640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 0x7f634c0707c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6360065d40 tx=0x7f6350009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:19.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:19.822+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f63480568f0 con 0x7f63601010c0 2024-03-22T02:55:20.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.049+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f636004bcc0 con 0x7f63601010c0 2024-03-22T02:55:20.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.050+0000 7f635cff9640 1 -- 172.21.15.103:0/3929125783 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v37) v1 ==== 107+0+5134 (secure 0 0 0) 0x7f6348059fb0 con 0x7f63601010c0 2024-03-22T02:55:20.051 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:20.051 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:14.627941+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:20.052 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 32 2024-03-22T02:55:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 msgr2=0x7f634c0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 0x7f634c0707c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6360065d40 tx=0x7f6350009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 msgr2=0x7f636010ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6348002a50 tx=0x7f6348004600 comp rx=0 tx=0).stop 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 shutdown_connections 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f634c06e310 0x7f634c0707c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6360101a60 0x7f636010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 --2- 172.21.15.103:0/3929125783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63601010c0 0x7f636010ded0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 >> 172.21.15.103:0/3929125783 conn(0x7f63600f6c90 msgr2=0x7f63600f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:20.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 shutdown_connections 2024-03-22T02:55:20.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.052+0000 7f6366618640 1 -- 172.21.15.103:0/3929125783 wait complete. 2024-03-22T02:55:20.064 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 -- 172.21.15.103:0/842683529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0998064cb0 msgr2=0x7f0998108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 --2- 172.21.15.103:0/842683529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0998064cb0 0x7f0998108780 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f098c0048d0 tx=0x7f098c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 -- 172.21.15.103:0/842683529 shutdown_connections 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 --2- 172.21.15.103:0/842683529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0998064cb0 0x7f0998108780 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 --2- 172.21.15.103:0/842683529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09980643a0 0x7f0998064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.188+0000 7f099e310640 1 -- 172.21.15.103:0/842683529 >> 172.21.15.103:0/842683529 conn(0x7f09980710b0 msgr2=0x7f09980714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 -- 172.21.15.103:0/842683529 shutdown_connections 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 -- 172.21.15.103:0/842683529 wait complete. 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 Processor -- start 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 -- start start 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0998064cb0 0x7f0998115440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998115a40 con 0x7f09980643a0 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f099e310640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099810f090 con 0x7f0998064cb0 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f0997fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f0997fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33314/0 (socket says 172.21.15.103:33314) 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.189+0000 7f0997fff640 1 -- 172.21.15.103:0/1370190621 learned_addr learned my addr 172.21.15.103:0/1370190621 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f09977fe640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0998064cb0 0x7f0998115440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0997fff640 1 -- 172.21.15.103:0/1370190621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0998064cb0 msgr2=0x7f0998115440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0997fff640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0998064cb0 0x7f0998115440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0997fff640 1 -- 172.21.15.103:0/1370190621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c005040 con 0x7f09980643a0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0997fff640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0988002a50 tx=0x7f0988004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0988007c30 con 0x7f09980643a0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099810f370 con 0x7f09980643a0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f098800f040 con 0x7f09980643a0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f098800db50 con 0x7f09980643a0 2024-03-22T02:55:20.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.190+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099810f860 con 0x7f09980643a0 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.191+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09980fb730 con 0x7f09980643a0 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.192+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f098801c400 con 0x7f09980643a0 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.192+0000 7f0994ff9640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 0x7f09800707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.193+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f098808d640 con 0x7f09980643a0 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.193+0000 7f09977fe640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 0x7f09800707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.193+0000 7f09977fe640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 0x7f09800707c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f09980fa530 tx=0x7f098c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:20.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.201+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09880568f0 con 0x7f09980643a0 2024-03-22T02:55:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:20 smithi122 ceph-mon[83368]: pgmap v714: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:20 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3468886608' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-22T02:55:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:20 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1021830177' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-22T02:55:20.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:20 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3929125783' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-22T02:55:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:20 smithi103 ceph-mon[115211]: pgmap v714: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:20 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3468886608' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-22T02:55:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:20 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1021830177' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-22T02:55:20.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:20 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3929125783' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-22T02:55:20.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.509+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f0998110570 con 0x7f09980643a0 2024-03-22T02:55:20.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.510+0000 7f0994ff9640 1 -- 172.21.15.103:0/1370190621 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v37) v1 ==== 107+0+5225 (secure 0 0 0) 0x7f0988059fb0 con 0x7f09980643a0 2024-03-22T02:55:20.511 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:20.511 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:15.629623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34380,"mds_1":34412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34380":{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":1,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:20.511 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 33 2024-03-22T02:55:20.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 msgr2=0x7f09800707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 0x7f09800707c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f09980fa530 tx=0x7f098c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 msgr2=0x7f0998114f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0988002a50 tx=0x7f0988004600 comp rx=0 tx=0).stop 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 shutdown_connections 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f098006e310 0x7f09800707c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0998064cb0 0x7f0998115440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 --2- 172.21.15.103:0/1370190621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09980643a0 0x7f0998114f00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 >> 172.21.15.103:0/1370190621 conn(0x7f09980710b0 msgr2=0x7f09980f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 shutdown_connections 2024-03-22T02:55:20.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.512+0000 7f099e310640 1 -- 172.21.15.103:0/1370190621 wait complete. 2024-03-22T02:55:20.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-03-22T02:55:20.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 -- 172.21.15.103:0/824938145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fd1f0 msgr2=0x7fe5280fd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/824938145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fd1f0 0x7fe5280fd5c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0048d0 tx=0x7fe51c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 -- 172.21.15.103:0/824938145 shutdown_connections 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/824938145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fdb90 0x7fe5281061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/824938145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fd1f0 0x7fe5280fd5c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 -- 172.21.15.103:0/824938145 >> 172.21.15.103:0/824938145 conn(0x7fe5280f6ca0 msgr2=0x7fe5280f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 -- 172.21.15.103:0/824938145 shutdown_connections 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.623+0000 7fe52eb5d640 1 -- 172.21.15.103:0/824938145 wait complete. 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 Processor -- start 2024-03-22T02:55:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 -- start start 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fd1f0 0x7fe528112d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281138a0 con 0x7fe5280fdb90 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe52eb5d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52810cef0 con 0x7fe5280fd1f0 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe527fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe527fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33328/0 (socket says 172.21.15.103:33328) 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.624+0000 7fe527fff640 1 -- 172.21.15.103:0/2192213269 learned_addr learned my addr 172.21.15.103:0/2192213269 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe52c8f9640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fd1f0 0x7fe528112d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe527fff640 1 -- 172.21.15.103:0/2192213269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fd1f0 msgr2=0x7fe528112d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe527fff640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fd1f0 0x7fe528112d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe527fff640 1 -- 172.21.15.103:0/2192213269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c005040 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe527fff640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe51400eb70 tx=0x7fe51400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe514007c30 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe51400f040 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52810d170 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5140095b0 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.625+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52810d690 con 0x7fe5280fdb90 2024-03-22T02:55:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.626+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528065900 con 0x7fe5280fdb90 2024-03-22T02:55:20.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.627+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fe514009730 con 0x7fe5280fdb90 2024-03-22T02:55:20.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.627+0000 7fe5257fa640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 0x7fe5180707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:20.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.628+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe51408dab0 con 0x7fe5280fdb90 2024-03-22T02:55:20.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.628+0000 7fe52c8f9640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 0x7fe5180707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:20.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.628+0000 7fe52c8f9640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 0x7fe5180707c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe51c003080 tx=0x7fe51c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:20.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.632+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe514056d60 con 0x7fe5280fdb90 2024-03-22T02:55:20.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.917+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fe5281093e0 con 0x7fe5280fdb90 2024-03-22T02:55:20.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.917+0000 7fe5257fa640 1 -- 172.21.15.103:0/2192213269 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v37) v1 ==== 107+0+5231 (secure 0 0 0) 0x7fe51405a420 con 0x7fe5280fdb90 2024-03-22T02:55:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:20.918 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:22.743051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34380,"mds_1":34412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34380":{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":0,"incarnation":33,"state":"up:resolve","state_seq":241,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":1,"incarnation":33,"state":"up:resolve","state_seq":237,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:20.919 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 34 2024-03-22T02:55:20.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 msgr2=0x7fe5180707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 0x7fe5180707c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe51c003080 tx=0x7fe51c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 msgr2=0x7fe5281132a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe51400eb70 tx=0x7fe51400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 shutdown_connections 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fe51806e310 0x7fe5180707c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5280fdb90 0x7fe5281132a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 --2- 172.21.15.103:0/2192213269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5280fd1f0 0x7fe528112d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 >> 172.21.15.103:0/2192213269 conn(0x7fe5280f6ca0 msgr2=0x7fe5280f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 shutdown_connections 2024-03-22T02:55:20.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:20.919+0000 7fe52eb5d640 1 -- 172.21.15.103:0/2192213269 wait complete. 2024-03-22T02:55:20.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-03-22T02:55:21.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- 172.21.15.103:0/3523763781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df40fa8d0 msgr2=0x7f7df40fad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/3523763781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df40fa8d0 0x7f7df40fad20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0048d0 tx=0x7f7ddc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- 172.21.15.103:0/3523763781 shutdown_connections 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/3523763781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df40fa8d0 0x7f7df40fad20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/3523763781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df41008c0 0x7f7df4100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- 172.21.15.103:0/3523763781 >> 172.21.15.103:0/3523763781 conn(0x7f7df40708f0 msgr2=0x7f7df4070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- 172.21.15.103:0/3523763781 shutdown_connections 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- 172.21.15.103:0/3523763781 wait complete. 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 Processor -- start 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.030+0000 7f7df8b5d640 1 -- start start 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df40fa8d0 0x7f7df4112d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4113880 con 0x7f7df41008c0 2024-03-22T02:55:21.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df410ced0 con 0x7f7df40fa8d0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33338/0 (socket says 172.21.15.103:33338) 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 -- 172.21.15.103:0/2178471069 learned_addr learned my addr 172.21.15.103:0/2178471069 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df259c640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df40fa8d0 0x7f7df4112d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 -- 172.21.15.103:0/2178471069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df40fa8d0 msgr2=0x7f7df4112d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df40fa8d0 0x7f7df4112d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 -- 172.21.15.103:0/2178471069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc005040 con 0x7f7df41008c0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7debfff640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc004930 tx=0x7f7ddc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ddc0094c0 con 0x7f7df41008c0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df410d150 con 0x7f7df41008c0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ddc032e50 con 0x7f7df41008c0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ddc0433b0 con 0x7f7df41008c0 2024-03-22T02:55:21.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.031+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df410d610 con 0x7f7df41008c0 2024-03-22T02:55:21.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.032+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df40fbfe0 con 0x7f7df41008c0 2024-03-22T02:55:21.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.033+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7ddc02f070 con 0x7f7df41008c0 2024-03-22T02:55:21.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.033+0000 7f7deaffd640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 0x7f7de00707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.033+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ddc0afc20 con 0x7f7df41008c0 2024-03-22T02:55:21.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.033+0000 7f7df259c640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 0x7f7de00707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.034+0000 7f7df259c640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 0x7f7de00707c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7de4004ed0 tx=0x7f7de400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.042+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ddc078ed0 con 0x7f7df41008c0 2024-03-22T02:55:21.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.273+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f7df41079c0 con 0x7f7df41008c0 2024-03-22T02:55:21.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.274+0000 7f7deaffd640 1 -- 172.21.15.103:0/2178471069 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v37) v1 ==== 107+0+5235 (secure 0 0 0) 0x7f7ddc07c590 con 0x7f7df41008c0 2024-03-22T02:55:21.274 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:21.275 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:23.653446+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34380,"mds_1":34412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34380":{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":0,"incarnation":33,"state":"up:reconnect","state_seq":242,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":1,"incarnation":33,"state":"up:reconnect","state_seq":238,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:21.275 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 35 2024-03-22T02:55:21.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 msgr2=0x7f7de00707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 0x7f7de00707c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7de4004ed0 tx=0x7f7de400b040 comp rx=0 tx=0).stop 2024-03-22T02:55:21.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 msgr2=0x7f7df4113280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc004930 tx=0x7f7ddc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 shutdown_connections 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7de006e310 0x7f7de00707c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7df41008c0 0x7f7df4113280 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 --2- 172.21.15.103:0/2178471069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7df40fa8d0 0x7f7df4112d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 >> 172.21.15.103:0/2178471069 conn(0x7f7df40708f0 msgr2=0x7f7df40f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 shutdown_connections 2024-03-22T02:55:21.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.276+0000 7f7df8b5d640 1 -- 172.21.15.103:0/2178471069 wait complete. 2024-03-22T02:55:21.286 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1370190621' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-22T02:55:21.286 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:21 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2192213269' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-22T02:55:21.288 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-03-22T02:55:21.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1370190621' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-22T02:55:21.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:21 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2192213269' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 -- 172.21.15.103:0/2605333103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7780fb110 msgr2=0x7fa7780fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 --2- 172.21.15.103:0/2605333103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7780fb110 0x7fa7780fb560 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0048d0 tx=0x7fa76c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 -- 172.21.15.103:0/2605333103 shutdown_connections 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 --2- 172.21.15.103:0/2605333103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7780fb110 0x7fa7780fb560 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 --2- 172.21.15.103:0/2605333103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa778101140 0x7fa778101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 -- 172.21.15.103:0/2605333103 >> 172.21.15.103:0/2605333103 conn(0x7fa7780710b0 msgr2=0x7fa7780714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 -- 172.21.15.103:0/2605333103 shutdown_connections 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.387+0000 7fa77e314640 1 -- 172.21.15.103:0/2605333103 wait complete. 2024-03-22T02:55:21.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 Processor -- start 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 -- start start 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7780fb110 0x7fa778111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa778112740 con 0x7fa778101140 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa77e314640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77810bd90 con 0x7fa7780fb110 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa7777fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa7777fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33350/0 (socket says 172.21.15.103:33350) 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa7777fe640 1 -- 172.21.15.103:0/920118231 learned_addr learned my addr 172.21.15.103:0/920118231 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.388+0000 7fa777fff640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7780fb110 0x7fa778111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa7777fe640 1 -- 172.21.15.103:0/920118231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7780fb110 msgr2=0x7fa778111c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa7777fe640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7780fb110 0x7fa778111c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa7777fe640 1 -- 172.21.15.103:0/920118231 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c005040 con 0x7fa778101140 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa7777fe640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0041d0 tx=0x7fa76c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa76c0092e0 con 0x7fa778101140 2024-03-22T02:55:21.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77810c010 con 0x7fa778101140 2024-03-22T02:55:21.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77810c530 con 0x7fa778101140 2024-03-22T02:55:21.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa76c009440 con 0x7fa778101140 2024-03-22T02:55:21.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.389+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa76c043400 con 0x7fa778101140 2024-03-22T02:55:21.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.390+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa764005210 con 0x7fa778101140 2024-03-22T02:55:21.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.391+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fa76c03b420 con 0x7fa778101140 2024-03-22T02:55:21.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.391+0000 7fa774ff9640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 0x7fa7600707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.391+0000 7fa777fff640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 0x7fa7600707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.391+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa76c0b0920 con 0x7fa778101140 2024-03-22T02:55:21.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.392+0000 7fa777fff640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 0x7fa7600707c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa7680041d0 tx=0x7fa768004330 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.395+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa76c079c60 con 0x7fa778101140 2024-03-22T02:55:21.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.630+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7fa7640054c0 con 0x7fa778101140 2024-03-22T02:55:21.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.630+0000 7fa774ff9640 1 -- 172.21.15.103:0/920118231 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v37) v1 ==== 107+0+5229 (secure 0 0 0) 0x7fa76c07d320 con 0x7fa778101140 2024-03-22T02:55:21.631 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:21.632 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44291,"name":"cephfs.smithi122.ztrkgp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/2991160252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2991160252},{"type":"v1","addr":"172.21.15.122:6825","nonce":2991160252}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44311,"name":"cephfs.smithi122.lvsegg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/3038924550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3038924550},{"type":"v1","addr":"172.21.15.122:6827","nonce":3038924550}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:25:42.837872+0000","modified":"2024-03-22T02:53:24.627121+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34380,"mds_1":34412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34380":{"gid":34380,"name":"cephfs.smithi103.gtuatp","rank":0,"incarnation":33,"state":"up:rejoin","state_seq":243,"addr":"172.21.15.103:6827/4019539425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":4019539425},{"type":"v1","addr":"172.21.15.103:6827","nonce":4019539425}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34412":{"gid":34412,"name":"cephfs.smithi103.xhmfjp","rank":1,"incarnation":33,"state":"up:rejoin","state_seq":239,"addr":"172.21.15.103:6829/3490746036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3490746036},{"type":"v1","addr":"172.21.15.103:6829","nonce":3490746036}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-03-22T02:55:21.632 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 36 2024-03-22T02:55:21.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 msgr2=0x7fa7600707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 0x7fa7600707c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa7680041d0 tx=0x7fa768004330 comp rx=0 tx=0).stop 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 msgr2=0x7fa778112140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0041d0 tx=0x7fa76c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 shutdown_connections 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fa76006e310 0x7fa7600707c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa778101140 0x7fa778112140 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 --2- 172.21.15.103:0/920118231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7780fb110 0x7fa778111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.632+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 >> 172.21.15.103:0/920118231 conn(0x7fa7780710b0 msgr2=0x7fa7780f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.633+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 shutdown_connections 2024-03-22T02:55:21.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.633+0000 7fa77e314640 1 -- 172.21.15.103:0/920118231 wait complete. 2024-03-22T02:55:21.644 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-22T02:55:21.659 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-22T02:55:21.659 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:55:21.659 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:55:21.675 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:55:21.675 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:55:21.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:55:21.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- 172.21.15.103:0/365263704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f0101140 msgr2=0x7f65f0101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 --2- 172.21.15.103:0/365263704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f0101140 0x7f65f0101510 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0048d0 tx=0x7f65ec009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- 172.21.15.103:0/365263704 shutdown_connections 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 --2- 172.21.15.103:0/365263704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f00fb110 0x7f65f00fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 --2- 172.21.15.103:0/365263704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f0101140 0x7f65f0101510 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- 172.21.15.103:0/365263704 >> 172.21.15.103:0/365263704 conn(0x7f65f0071130 msgr2=0x7f65f0071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- 172.21.15.103:0/365263704 shutdown_connections 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- 172.21.15.103:0/365263704 wait complete. 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 Processor -- start 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- start start 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 0x7f65f010e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0065310 con 0x7f65f00fb110 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.870+0000 7f65f759d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f010e920 con 0x7f65f0101140 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33366/0 (socket says 172.21.15.103:33366) 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 -- 172.21.15.103:0/3958199172 learned_addr learned my addr 172.21.15.103:0/3958199172 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:21.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f4b38640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 0x7f65f010e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 -- 172.21.15.103:0/3958199172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 msgr2=0x7f65f010e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 0x7f65f010e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 -- 172.21.15.103:0/3958199172 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec005040 con 0x7f65f00fb110 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f4b38640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 0x7f65f010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f5339640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0041d0 tx=0x7f65ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65ec030870 con 0x7f65f00fb110 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65ec0092e0 con 0x7f65f00fb110 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f010eba0 con 0x7f65f00fb110 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65ec043400 con 0x7f65f00fb110 2024-03-22T02:55:21.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.871+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f0117160 con 0x7f65f00fb110 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.872+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00fc800 con 0x7f65f00fb110 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.873+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f65ec03b420 con 0x7f65f00fb110 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.873+0000 7f65e5ffb640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 0x7f65dc0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.873+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f65ec0b0b90 con 0x7f65f00fb110 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.873+0000 7f65f4b38640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 0x7f65dc0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:21.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.874+0000 7f65f4b38640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 0x7f65dc0707c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f65f0064b60 tx=0x7f65e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:21.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:21.878+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f65ec079e10 con 0x7f65f00fb110 2024-03-22T02:55:22.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.061+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65f004bcc0 con 0x7f65f00fb110 2024-03-22T02:55:22.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.062+0000 7f65e5ffb640 1 -- 172.21.15.103:0/3958199172 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f65ec07d4d0 con 0x7f65f00fb110 2024-03-22T02:55:22.063 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:22.063 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":84,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:38:09.183439+0000","last_up_change":"2024-03-22T02:37:15.002547+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-22T02:25:41.764005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-22T02:25:42.064180+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6803","nonce":1332641876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6805","nonce":1332641876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6809","nonce":1332641876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6807","nonce":1332641876}]},"public_addr":"172.21.15.103:6803/1332641876","cluster_addr":"172.21.15.103:6805/1332641876","heartbeat_back_addr":"172.21.15.103:6809/1332641876","heartbeat_front_addr":"172.21.15.103:6807/1332641876","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6811","nonce":292218030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6813","nonce":292218030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6817","nonce":292218030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6815","nonce":292218030}]},"public_addr":"172.21.15.103:6811/292218030","cluster_addr":"172.21.15.103:6813/292218030","heartbeat_back_addr":"172.21.15.103:6817/292218030","heartbeat_front_addr":"172.21.15.103:6815/292218030","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6819","nonce":117269184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6821","nonce":117269184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6825","nonce":117269184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6823","nonce":117269184}]},"public_addr":"172.21.15.103:6819/117269184","cluster_addr":"172.21.15.103:6821/117269184","heartbeat_back_addr":"172.21.15.103:6825/117269184","heartbeat_front_addr":"172.21.15.103:6823/117269184","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6801","nonce":2866011991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6803","nonce":2866011991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6807","nonce":2866011991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6805","nonce":2866011991}]},"public_addr":"172.21.15.122:6801/2866011991","cluster_addr":"172.21.15.122:6803/2866011991","heartbeat_back_addr":"172.21.15.122:6807/2866011991","heartbeat_front_addr":"172.21.15.122:6805/2866011991","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6809","nonce":1009596471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6811","nonce":1009596471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6815","nonce":1009596471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6813","nonce":1009596471}]},"public_addr":"172.21.15.122:6809/1009596471","cluster_addr":"172.21.15.122:6811/1009596471","heartbeat_back_addr":"172.21.15.122:6815/1009596471","heartbeat_front_addr":"172.21.15.122:6813/1009596471","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6817","nonce":1621369093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6819","nonce":1621369093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6823","nonce":1621369093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6821","nonce":1621369093}]},"public_addr":"172.21.15.122:6817/1621369093","cluster_addr":"172.21.15.122:6819/1621369093","heartbeat_back_addr":"172.21.15.122:6823/1621369093","heartbeat_front_addr":"172.21.15.122:6821/1621369093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.122:6824/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.103:6829/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.122:0/2086230685":"2024-03-23T02:30:07.935727+0000","172.21.15.103:6827/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6828/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6826/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.122:0/1988149757":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.122:6829/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3955856929":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6800/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.122:0/3639247922":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.122:6828/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3724441621":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6801/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2509522170":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/385301047":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/1309215164":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3243311702":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2788707715":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.122:0/867576006":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/1085098758":"2024-03-23T02:32:46.783649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:55:22.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 msgr2=0x7f65dc0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 0x7f65dc0707c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f65f0064b60 tx=0x7f65e000b040 comp rx=0 tx=0).stop 2024-03-22T02:55:22.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 msgr2=0x7f65f010dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0041d0 tx=0x7f65ec009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 shutdown_connections 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f65dc06e310 0x7f65dc0707c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0101140 0x7f65f010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 --2- 172.21.15.103:0/3958199172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65f00fb110 0x7f65f010dea0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.064+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 >> 172.21.15.103:0/3958199172 conn(0x7f65f0071130 msgr2=0x7f65f00f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.065+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 shutdown_connections 2024-03-22T02:55:22.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.065+0000 7f65f759d640 1 -- 172.21.15.103:0/3958199172 wait complete. 2024-03-22T02:55:22.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:23:06.693135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-22T02:25:41.764005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-22T02:25:42.064180+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:55:22.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 -- 172.21.15.103:0/2940638724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 msgr2=0x7f7b301014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2940638724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b301014a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7b200048d0 tx=0x7f7b20009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 -- 172.21.15.103:0/2940638724 shutdown_connections 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2940638724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b300fb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2940638724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b301014a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 -- 172.21.15.103:0/2940638724 >> 172.21.15.103:0/2940638724 conn(0x7f7b300f6ca0 msgr2=0x7f7b300f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.202+0000 7f7b3742e640 1 -- 172.21.15.103:0/2940638724 shutdown_connections 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 -- 172.21.15.103:0/2940638724 wait complete. 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 Processor -- start 2024-03-22T02:55:22.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 -- start start 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b3010c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30065e90 con 0x7f7b301010d0 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b3742e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30065fd0 con 0x7f7b30101a70 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b351ca640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b351ca640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33390/0 (socket says 172.21.15.103:33390) 2024-03-22T02:55:22.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.203+0000 7f7b351ca640 1 -- 172.21.15.103:0/2090745970 learned_addr learned my addr 172.21.15.103:0/2090745970 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b349c9640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b3010c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b351ca640 1 -- 172.21.15.103:0/2090745970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 msgr2=0x7f7b3010c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b351ca640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b3010c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b351ca640 1 -- 172.21.15.103:0/2090745970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20005040 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b349c9640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b3010c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b351ca640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b20004930 tx=0x7f7b20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b200094c0 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3010c7d0 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b20032e50 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b200433b0 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.204+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b30119470 con 0x7f7b301010d0 2024-03-22T02:55:22.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.206+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b300fc840 con 0x7f7b301010d0 2024-03-22T02:55:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.206+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f7b2002f070 con 0x7f7b301010d0 2024-03-22T02:55:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.206+0000 7f7b25ffb640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 0x7f7b1c0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.206+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7b200afcd0 con 0x7f7b301010d0 2024-03-22T02:55:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.206+0000 7f7b349c9640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 0x7f7b1c0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.207+0000 7f7b349c9640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 0x7f7b1c0707c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009fd0 tx=0x7f7b2c009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.215+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b20078ed0 con 0x7f7b301010d0 2024-03-22T02:55:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:22 smithi122 ceph-mon[83368]: pgmap v715: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:22 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2178471069' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-22T02:55:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:22 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/920118231' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-22T02:55:22.349 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:22 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3958199172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:22 smithi103 ceph-mon[115211]: pgmap v715: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:22 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2178471069' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-22T02:55:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:22 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/920118231' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-22T02:55:22.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:22 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3958199172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:22.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.412+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b3004bcc0 con 0x7f7b301010d0 2024-03-22T02:55:22.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.412+0000 7f7b25ffb640 1 -- 172.21.15.103:0/2090745970 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b3004bcc0 con 0x7f7b301010d0 2024-03-22T02:55:22.413 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-03-22T02:55:22.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.414+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 msgr2=0x7f7b1c0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 0x7f7b1c0707c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009fd0 tx=0x7f7b2c009e50 comp rx=0 tx=0).stop 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 msgr2=0x7f7b3010bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b20004930 tx=0x7f7b20009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 shutdown_connections 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f7b1c06e310 0x7f7b1c0707c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b30101a70 0x7f7b3010c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 --2- 172.21.15.103:0/2090745970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b301010d0 0x7f7b3010bc60 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 >> 172.21.15.103:0/2090745970 conn(0x7f7b300f6ca0 msgr2=0x7f7b300f7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 shutdown_connections 2024-03-22T02:55:22.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.415+0000 7f7b3742e640 1 -- 172.21.15.103:0/2090745970 wait complete. 2024-03-22T02:55:22.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 -- 172.21.15.103:0/3892328595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 msgr2=0x7fc36c0fe420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 --2- 172.21.15.103:0/3892328595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c0fe420 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0048d0 tx=0x7fc35c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 -- 172.21.15.103:0/3892328595 shutdown_connections 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 --2- 172.21.15.103:0/3892328595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c0fe420 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 --2- 172.21.15.103:0/3892328595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 0x7fc36c1047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.527+0000 7fc373914640 1 -- 172.21.15.103:0/3892328595 >> 172.21.15.103:0/3892328595 conn(0x7fc36c0f9fc0 msgr2=0x7fc36c0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 -- 172.21.15.103:0/3892328595 shutdown_connections 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 -- 172.21.15.103:0/3892328595 wait complete. 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 Processor -- start 2024-03-22T02:55:22.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 -- start start 2024-03-22T02:55:22.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 0x7fc36c114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c115a30 con 0x7fc36c104d90 2024-03-22T02:55:22.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.528+0000 7fc373914640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c10f080 con 0x7fc36c1043f0 2024-03-22T02:55:22.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33418/0 (socket says 172.21.15.103:33418) 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 -- 172.21.15.103:0/851267164 learned_addr learned my addr 172.21.15.103:0/851267164 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc3716b0640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 0x7fc36c114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 -- 172.21.15.103:0/851267164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 msgr2=0x7fc36c114ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 0x7fc36c114ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 -- 172.21.15.103:0/851267164 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c005040 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc370eaf640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc35c004930 tx=0x7fc35c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35c0094c0 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c10f300 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc35c032e50 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35c0433b0 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.529+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c10f7c0 con 0x7fc36c104d90 2024-03-22T02:55:22.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.530+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c0ffb60 con 0x7fc36c104d90 2024-03-22T02:55:22.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.531+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fc35c02f070 con 0x7fc36c104d90 2024-03-22T02:55:22.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.532+0000 7fc361ffb640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 0x7fc3580707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.532+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc35c0afe20 con 0x7fc36c104d90 2024-03-22T02:55:22.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.532+0000 7fc3716b0640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 0x7fc3580707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.532+0000 7fc3716b0640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 0x7fc3580707c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc3680041d0 tx=0x7fc368009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.540+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc35c079020 con 0x7fc36c104d90 2024-03-22T02:55:22.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.724+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc36c10b5c0 con 0x7fc36c104d90 2024-03-22T02:55:22.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.724+0000 7fc361ffb640 1 -- 172.21.15.103:0/851267164 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc35c07c6e0 con 0x7fc36c104d90 2024-03-22T02:55:22.725 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 msgr2=0x7fc3580707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 0x7fc3580707c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc3680041d0 tx=0x7fc368009040 comp rx=0 tx=0).stop 2024-03-22T02:55:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 msgr2=0x7fc36c115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc35c004930 tx=0x7fc35c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 shutdown_connections 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fc35806e310 0x7fc3580707c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc36c104d90 0x7fc36c115430 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 --2- 172.21.15.103:0/851267164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc36c1043f0 0x7fc36c114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 >> 172.21.15.103:0/851267164 conn(0x7fc36c0f9fc0 msgr2=0x7fc36c0faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 shutdown_connections 2024-03-22T02:55:22.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.726+0000 7fc373914640 1 -- 172.21.15.103:0/851267164 wait complete. 2024-03-22T02:55:22.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 -- 172.21.15.103:0/3202529883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c101a70 msgr2=0x7f674c0fb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 --2- 172.21.15.103:0/3202529883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c101a70 0x7f674c0fb100 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f67400048d0 tx=0x7f6740009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 -- 172.21.15.103:0/3202529883 shutdown_connections 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 --2- 172.21.15.103:0/3202529883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c101a70 0x7f674c0fb100 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 --2- 172.21.15.103:0/3202529883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c1010d0 0x7f674c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 -- 172.21.15.103:0/3202529883 >> 172.21.15.103:0/3202529883 conn(0x7f674c0f6ca0 msgr2=0x7f674c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 -- 172.21.15.103:0/3202529883 shutdown_connections 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 -- 172.21.15.103:0/3202529883 wait complete. 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.840+0000 7f6751d5d640 1 Processor -- start 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6751d5d640 1 -- start start 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6751d5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6751d5d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 0x7f674c10e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6751d5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c065fc0 con 0x7f674c1010d0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6751d5d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c10e900 con 0x7f674c101a70 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33432/0 (socket says 172.21.15.103:33432) 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 -- 172.21.15.103:0/1180697542 learned_addr learned my addr 172.21.15.103:0/1180697542 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674affd640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 0x7f674c10e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 -- 172.21.15.103:0/1180697542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 msgr2=0x7f674c10e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 0x7f674c10e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 -- 172.21.15.103:0/1180697542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740005040 con 0x7f674c1010d0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674affd640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 0x7f674c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f674b7fe640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f673c00eb70 tx=0x7f673c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.841+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c007c30 con 0x7f674c1010d0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.842+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c10eb80 con 0x7f674c1010d0 2024-03-22T02:55:22.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.842+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f673c00f040 con 0x7f674c1010d0 2024-03-22T02:55:22.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.842+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f673c0095b0 con 0x7f674c1010d0 2024-03-22T02:55:22.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.842+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c117140 con 0x7f674c1010d0 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.843+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c0fc840 con 0x7f674c1010d0 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.843+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f673c009730 con 0x7f674c1010d0 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.843+0000 7f6733fff640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 0x7f67340707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.844+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f673c08db50 con 0x7f674c1010d0 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.844+0000 7f674affd640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 0x7f67340707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:22.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.844+0000 7f674affd640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 0x7f67340707c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f674c065810 tx=0x7f6740009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:22.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:22.848+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f673c056d50 con 0x7f674c1010d0 2024-03-22T02:55:23.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.082+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f674c04bcc0 con 0x7f674c1010d0 2024-03-22T02:55:23.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.082+0000 7f6733fff640 1 -- 172.21.15.103:0/1180697542 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f673c05a410 con 0x7f674c1010d0 2024-03-22T02:55:23.083 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 msgr2=0x7f67340707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 0x7f67340707c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f674c065810 tx=0x7f6740009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 msgr2=0x7f674c10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f673c00eb70 tx=0x7f673c00b040 comp rx=0 tx=0).stop 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 shutdown_connections 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f673406e310 0x7f67340707c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f674c101a70 0x7f674c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 --2- 172.21.15.103:0/1180697542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f674c1010d0 0x7f674c10de80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 >> 172.21.15.103:0/1180697542 conn(0x7f674c0f6ca0 msgr2=0x7f674c0f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 shutdown_connections 2024-03-22T02:55:23.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.085+0000 7f6751d5d640 1 -- 172.21.15.103:0/1180697542 wait complete. 2024-03-22T02:55:23.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T02:55:23.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- 172.21.15.103:0/3351752016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 msgr2=0x7f45340fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 --2- 172.21.15.103:0/3351752016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f45340fb3a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f45240048d0 tx=0x7f4524009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- 172.21.15.103:0/3351752016 shutdown_connections 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 --2- 172.21.15.103:0/3351752016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 --2- 172.21.15.103:0/3351752016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f45340fb3a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- 172.21.15.103:0/3351752016 >> 172.21.15.103:0/3351752016 conn(0x7f45340f6cc0 msgr2=0x7f45340f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- 172.21.15.103:0/3351752016 shutdown_connections 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- 172.21.15.103:0/3351752016 wait complete. 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 Processor -- start 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- start start 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534115680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4534115c80 con 0x7f45340fafd0 2024-03-22T02:55:23.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.196+0000 7f4538c6e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453410f2d0 con 0x7f45340fb8e0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33446/0 (socket says 172.21.15.103:33446) 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 -- 172.21.15.103:0/2982978607 learned_addr learned my addr 172.21.15.103:0/2982978607 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4531d9b640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534115680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 -- 172.21.15.103:0/2982978607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 msgr2=0x7f4534115680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534115680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 -- 172.21.15.103:0/2982978607 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524005040 con 0x7f45340fafd0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4531d9b640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534115680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f453259c640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4524003080 tx=0x7f4524009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f452403b7f0 con 0x7f45340fafd0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4524031780 con 0x7f45340fafd0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45240439a0 con 0x7f45340fafd0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453410f5b0 con 0x7f45340fafd0 2024-03-22T02:55:23.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.197+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453410fb50 con 0x7f45340fafd0 2024-03-22T02:55:23.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.198+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4534065df0 con 0x7f45340fafd0 2024-03-22T02:55:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.199+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f4524043b00 con 0x7f45340fafd0 2024-03-22T02:55:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.199+0000 7f4522ffd640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 0x7f451c0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.200+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f45240b0530 con 0x7f45340fafd0 2024-03-22T02:55:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.199+0000 7f4531d9b640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 0x7f451c0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.200+0000 7f4531d9b640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 0x7f451c0707c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4534110a00 tx=0x7f4528009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.204+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45240797b0 con 0x7f45340fafd0 2024-03-22T02:55:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:23 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2090745970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:23 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/851267164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:23 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1180697542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.439+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f453410b510 con 0x7f45340fafd0 2024-03-22T02:55:23.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.440+0000 7f4522ffd640 1 -- 172.21.15.103:0/2982978607 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2348 (secure 0 0 0) 0x7f452407ce70 con 0x7f45340fafd0 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6825/1237228570 2024-03-23T02:37:16.024221+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6824/1237228570 2024-03-23T02:37:16.024221+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6829/2193581075 2024-03-23T02:37:16.024200+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/2086230685 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6827/3786434277 2024-03-23T02:25:58.328445+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1548567537 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/179622989 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6828/2193581075 2024-03-23T02:37:16.024200+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/2790950462 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/2790950462 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/375109873 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6826/3786434277 2024-03-23T02:25:58.328445+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/1988149757 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/595266923 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6829/2536422926 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3955856929 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3344821418 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2826893442 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/3639247922 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/313561406 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6828/2536422926 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3150495731 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/1197413654 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3724441621 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.442 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/375109873 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2509522170 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/1197413654 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3834849577 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/385301047 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3344821418 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/1160196781 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1309215164 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2699747522 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3243311702 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2788707715 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3672865475 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/1160196781 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/867576006 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:23.443 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1085098758 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:23.444 INFO:teuthology.orchestra.run.smithi103.stderr:listed 39 entries 2024-03-22T02:55:23.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 msgr2=0x7f451c0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 0x7f451c0707c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4534110a00 tx=0x7f4528009040 comp rx=0 tx=0).stop 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 msgr2=0x7f4534115140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4524003080 tx=0x7f4524009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 shutdown_connections 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f451c06e310 0x7f451c0707c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45340fb8e0 0x7f4534115680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 --2- 172.21.15.103:0/2982978607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45340fafd0 0x7f4534115140 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.442+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 >> 172.21.15.103:0/2982978607 conn(0x7f45340f6cc0 msgr2=0x7f45340f7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.443+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 shutdown_connections 2024-03-22T02:55:23.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.443+0000 7f4538c6e640 1 -- 172.21.15.103:0/2982978607 wait complete. 2024-03-22T02:55:23.454 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-22T02:55:23.454 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:55:23.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:55:23.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:23 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2090745970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:23 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/851267164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:23 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1180697542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 -- 172.21.15.103:0/3386434023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 msgr2=0x7f17380fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3386434023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f17380fb560 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f17280048d0 tx=0x7f1728009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 -- 172.21.15.103:0/3386434023 shutdown_connections 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3386434023 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f17380fb560 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3386434023 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 0x7f1738101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 -- 172.21.15.103:0/3386434023 >> 172.21.15.103:0/3386434023 conn(0x7f1738071130 msgr2=0x7f1738071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 -- 172.21.15.103:0/3386434023 shutdown_connections 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.618+0000 7f173fa2e640 1 -- 172.21.15.103:0/3386434023 wait complete. 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.619+0000 7f173fa2e640 1 Processor -- start 2024-03-22T02:55:23.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.619+0000 7f173fa2e640 1 -- start start 2024-03-22T02:55:23.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.619+0000 7f173fa2e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173fa2e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 0x7f173810e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173fa2e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1738065330 con 0x7f17380fb110 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173fa2e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173810e900 con 0x7f1738101140 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33472/0 (socket says 172.21.15.103:33472) 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 -- 172.21.15.103:0/3251261503 learned_addr learned my addr 172.21.15.103:0/3251261503 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173cfc9640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 0x7f173810e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 -- 172.21.15.103:0/3251261503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 msgr2=0x7f173810e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 0x7f173810e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 -- 172.21.15.103:0/3251261503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1728005040 con 0x7f17380fb110 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173d7ca640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1734002a50 tx=0x7f1734004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1734007c30 con 0x7f17380fb110 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f173400f040 con 0x7f17380fb110 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173810ebe0 con 0x7f17380fb110 2024-03-22T02:55:23.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.620+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f173400dbe0 con 0x7f17380fb110 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.621+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1738117170 con 0x7f17380fb110 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.622+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17380fc800 con 0x7f17380fb110 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.622+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f173400dd40 con 0x7f17380fb110 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.622+0000 7f172dffb640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 0x7f17240707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.622+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f173408dd40 con 0x7f17380fb110 2024-03-22T02:55:23.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.622+0000 7f173cfc9640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 0x7f17240707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.623+0000 7f173cfc9640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 0x7f17240707c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1728000c00 tx=0x7f1728009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.627+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1734091030 con 0x7f17380fb110 2024-03-22T02:55:23.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.820+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1738117360 con 0x7f17380fb110 2024-03-22T02:55:23.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.821+0000 7f172dffb640 1 -- 172.21.15.103:0/3251261503 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16716 (secure 0 0 0) 0x7f1734056f40 con 0x7f17380fb110 2024-03-22T02:55:23.822 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-22T02:55:23.823 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":84,"fsid":"67659b4a-e7f2-11ee-95cd-87774f69a715","created":"2024-03-22T02:18:41.818687+0000","modified":"2024-03-22T02:38:09.183439+0000","last_up_change":"2024-03-22T02:37:15.002547+0000","last_in_change":"2024-03-22T02:24:03.466577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:23:06.693135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-22T02:25:41.764005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-22T02:25:42.064180+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"57d1ef5a-9736-4d69-9026-3dc88ac0880d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6803","nonce":1332641876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6805","nonce":1332641876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6809","nonce":1332641876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1332641876},{"type":"v1","addr":"172.21.15.103:6807","nonce":1332641876}]},"public_addr":"172.21.15.103:6803/1332641876","cluster_addr":"172.21.15.103:6805/1332641876","heartbeat_back_addr":"172.21.15.103:6809/1332641876","heartbeat_front_addr":"172.21.15.103:6807/1332641876","state":["exists","up"]},{"osd":1,"uuid":"65887971-4f0f-4bda-a25a-621be3121b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6811","nonce":292218030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6813","nonce":292218030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6817","nonce":292218030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":292218030},{"type":"v1","addr":"172.21.15.103:6815","nonce":292218030}]},"public_addr":"172.21.15.103:6811/292218030","cluster_addr":"172.21.15.103:6813/292218030","heartbeat_back_addr":"172.21.15.103:6817/292218030","heartbeat_front_addr":"172.21.15.103:6815/292218030","state":["exists","up"]},{"osd":2,"uuid":"7d03ea66-4cf1-4f34-a046-709ed8f6f037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6819","nonce":117269184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6821","nonce":117269184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6825","nonce":117269184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":117269184},{"type":"v1","addr":"172.21.15.103:6823","nonce":117269184}]},"public_addr":"172.21.15.103:6819/117269184","cluster_addr":"172.21.15.103:6821/117269184","heartbeat_back_addr":"172.21.15.103:6825/117269184","heartbeat_front_addr":"172.21.15.103:6823/117269184","state":["exists","up"]},{"osd":3,"uuid":"12aeb69e-863c-492d-b243-d1696e37848f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6801","nonce":2866011991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6803","nonce":2866011991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6807","nonce":2866011991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2866011991},{"type":"v1","addr":"172.21.15.122:6805","nonce":2866011991}]},"public_addr":"172.21.15.122:6801/2866011991","cluster_addr":"172.21.15.122:6803/2866011991","heartbeat_back_addr":"172.21.15.122:6807/2866011991","heartbeat_front_addr":"172.21.15.122:6805/2866011991","state":["exists","up"]},{"osd":4,"uuid":"6ede66c5-d812-4f2c-b642-ac6595a340d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6809","nonce":1009596471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6811","nonce":1009596471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6815","nonce":1009596471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":1009596471},{"type":"v1","addr":"172.21.15.122:6813","nonce":1009596471}]},"public_addr":"172.21.15.122:6809/1009596471","cluster_addr":"172.21.15.122:6811/1009596471","heartbeat_back_addr":"172.21.15.122:6815/1009596471","heartbeat_front_addr":"172.21.15.122:6813/1009596471","state":["exists","up"]},{"osd":5,"uuid":"9918a976-1e8c-4fb8-93ec-6600de78f79a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6817","nonce":1621369093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6819","nonce":1621369093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6823","nonce":1621369093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1621369093},{"type":"v1","addr":"172.21.15.122:6821","nonce":1621369093}]},"public_addr":"172.21.15.122:6817/1621369093","cluster_addr":"172.21.15.122:6819/1621369093","heartbeat_back_addr":"172.21.15.122:6823/1621369093","heartbeat_front_addr":"172.21.15.122:6821/1621369093","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:09.116294+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:22:37.142100+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:04.508324+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:28.786608+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:23:54.874153+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:24:20.776915+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.122:6824/1237228570":"2024-03-23T02:37:16.024221+0000","172.21.15.103:6829/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.122:0/2086230685":"2024-03-23T02:30:07.935727+0000","172.21.15.103:6827/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.103:0/1548567537":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/179622989":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6828/2193581075":"2024-03-23T02:37:16.024200+0000","172.21.15.103:6801/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/2790950462":"2024-03-23T02:20:50.358474+0000","172.21.15.103:6800/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6826/3786434277":"2024-03-23T02:25:58.328445+0000","172.21.15.122:0/1988149757":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/595266923":"2024-03-23T02:19:16.742515+0000","172.21.15.122:6829/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3955856929":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6800/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2826893442":"2024-03-23T02:20:50.358474+0000","172.21.15.122:0/3639247922":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/313561406":"2024-03-23T02:20:50.358474+0000","172.21.15.122:6828/2536422926":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/3150495731":"2024-03-23T02:19:47.880360+0000","172.21.15.103:6800/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3724441621":"2024-03-23T02:32:46.783649+0000","172.21.15.103:6801/375109873":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2509522170":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1197413654":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3834849577":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/385301047":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/3344821418":"2024-03-23T02:28:17.965286+0000","172.21.15.103:6801/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.103:0/1309215164":"2024-03-23T02:28:17.965286+0000","172.21.15.103:0/2699747522":"2024-03-23T02:19:47.880360+0000","172.21.15.103:0/3243311702":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/2788707715":"2024-03-23T02:32:46.783649+0000","172.21.15.103:0/3672865475":"2024-03-23T02:19:16.742515+0000","172.21.15.103:6800/1160196781":"2024-03-23T02:19:16.742515+0000","172.21.15.122:0/867576006":"2024-03-23T02:30:07.935727+0000","172.21.15.103:0/1085098758":"2024-03-23T02:32:46.783649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 msgr2=0x7f17240707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 0x7f17240707c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1728000c00 tx=0x7f1728009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 msgr2=0x7f173810de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1734002a50 tx=0x7f1734004600 comp rx=0 tx=0).stop 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 shutdown_connections 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.823+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f172406e310 0x7f17240707c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.824+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1738101140 0x7f173810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.824+0000 7f173fa2e640 1 --2- 172.21.15.103:0/3251261503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17380fb110 0x7f173810de50 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.824+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 >> 172.21.15.103:0/3251261503 conn(0x7f1738071130 msgr2=0x7f17380f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.824+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 shutdown_connections 2024-03-22T02:55:23.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.824+0000 7f173fa2e640 1 -- 172.21.15.103:0/3251261503 wait complete. 2024-03-22T02:55:23.835 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:23:06.693135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-22T02:25:41.764005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-22T02:25:42.064180+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:55:23.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:55:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.940+0000 7f983c39d640 1 -- 172.21.15.103:0/1665177548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98340fb110 msgr2=0x7f98340fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.940+0000 7f983c39d640 1 --2- 172.21.15.103:0/1665177548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98340fb110 0x7f98340fb560 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f98240048d0 tx=0x7f9824009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.940+0000 7f983c39d640 1 -- 172.21.15.103:0/1665177548 shutdown_connections 2024-03-22T02:55:23.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 --2- 172.21.15.103:0/1665177548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98340fb110 0x7f98340fb560 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 --2- 172.21.15.103:0/1665177548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9834101140 0x7f9834101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- 172.21.15.103:0/1665177548 >> 172.21.15.103:0/1665177548 conn(0x7f98340710b0 msgr2=0x7f98340714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- 172.21.15.103:0/1665177548 shutdown_connections 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- 172.21.15.103:0/1665177548 wait complete. 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 Processor -- start 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- start start 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 0x7f9834111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9834112740 con 0x7f9834101140 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f983c39d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983410bd90 con 0x7f98340fb110 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f9839938640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f9839938640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47782/0 (socket says 172.21.15.103:47782) 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.941+0000 7f9839938640 1 -- 172.21.15.103:0/766258385 learned_addr learned my addr 172.21.15.103:0/766258385 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:23.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f983a139640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 0x7f9834111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f9839938640 1 -- 172.21.15.103:0/766258385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 msgr2=0x7f9834111c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f9839938640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 0x7f9834111c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f9839938640 1 -- 172.21.15.103:0/766258385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824005040 con 0x7f9834101140 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f983a139640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 0x7f9834111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f9839938640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9824030640 tx=0x7f9824009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9824030820 con 0x7f9834101140 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98240092e0 con 0x7f9834101140 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983410c010 con 0x7f9834101140 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9824043400 con 0x7f9834101140 2024-03-22T02:55:23.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.942+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983410c530 con 0x7f9834101140 2024-03-22T02:55:23.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.943+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98340fc800 con 0x7f9834101140 2024-03-22T02:55:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.944+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f982403b420 con 0x7f9834101140 2024-03-22T02:55:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.944+0000 7f982affd640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 0x7f98200707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.944+0000 7f983a139640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 0x7f98200707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.944+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f98240b0b70 con 0x7f9834101140 2024-03-22T02:55:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.945+0000 7f983a139640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 0x7f98200707c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9830004ab0 tx=0x7f983000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:23.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:23.953+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9824079df0 con 0x7f9834101140 2024-03-22T02:55:24.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.141+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9834108220 con 0x7f9834101140 2024-03-22T02:55:24.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.141+0000 7f982affd640 1 -- 172.21.15.103:0/766258385 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f982407d4b0 con 0x7f9834101140 2024-03-22T02:55:24.142 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-03-22T02:55:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 msgr2=0x7f98200707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 0x7f98200707c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9830004ab0 tx=0x7f983000b040 comp rx=0 tx=0).stop 2024-03-22T02:55:24.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 msgr2=0x7f9834112140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9824030640 tx=0x7f9824009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 shutdown_connections 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f982006e310 0x7f98200707c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9834101140 0x7f9834112140 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 --2- 172.21.15.103:0/766258385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98340fb110 0x7f9834111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 >> 172.21.15.103:0/766258385 conn(0x7f98340710b0 msgr2=0x7f98340f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 shutdown_connections 2024-03-22T02:55:24.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.143+0000 7f983c39d640 1 -- 172.21.15.103:0/766258385 wait complete. 2024-03-22T02:55:24.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-22T02:55:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 -- 172.21.15.103:0/143624604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 msgr2=0x7f065c101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 --2- 172.21.15.103:0/143624604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c101510 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f064c0048d0 tx=0x7f064c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 -- 172.21.15.103:0/143624604 shutdown_connections 2024-03-22T02:55:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 --2- 172.21.15.103:0/143624604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 0x7f065c0fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 --2- 172.21.15.103:0/143624604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c101510 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.261+0000 7f0660c58640 1 -- 172.21.15.103:0/143624604 >> 172.21.15.103:0/143624604 conn(0x7f065c0710b0 msgr2=0x7f065c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 -- 172.21.15.103:0/143624604 shutdown_connections 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 -- 172.21.15.103:0/143624604 wait complete. 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 Processor -- start 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 -- start start 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 0x7f065c10de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c065310 con 0x7f065c101140 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0660c58640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c10e910 con 0x7f065c0fb110 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0653fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0653fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47798/0 (socket says 172.21.15.103:47798) 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.262+0000 7f0653fff640 1 -- 172.21.15.103:0/2963755420 learned_addr learned my addr 172.21.15.103:0/2963755420 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f065a59c640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 0x7f065c10de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0653fff640 1 -- 172.21.15.103:0/2963755420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 msgr2=0x7f065c10de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0653fff640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 0x7f065c10de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0653fff640 1 -- 172.21.15.103:0/2963755420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c005040 con 0x7f065c101140 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0653fff640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0644002a50 tx=0x7f0644004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0644007c30 con 0x7f065c101140 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f064400f040 con 0x7f065c101140 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c10ebf0 con 0x7f065c101140 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f064400dbe0 con 0x7f065c101140 2024-03-22T02:55:24.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.263+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c1171b0 con 0x7f065c101140 2024-03-22T02:55:24.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.264+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c0fc800 con 0x7f065c101140 2024-03-22T02:55:24.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.265+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f064400dd40 con 0x7f065c101140 2024-03-22T02:55:24.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.265+0000 7f0652ffd640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 0x7f06480707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.265+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f064408ddb0 con 0x7f065c101140 2024-03-22T02:55:24.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.265+0000 7f065a59c640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 0x7f06480707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.266+0000 7f065a59c640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 0x7f06480707c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f064c009990 tx=0x7f064c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:24.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.272+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0644056fb0 con 0x7f065c101140 2024-03-22T02:55:24.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:24 smithi103 ceph-mon[115211]: pgmap v716: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:24.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:24 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2982978607' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:55:24.375 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:24 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/3251261503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.455+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f065c04bcc0 con 0x7f065c101140 2024-03-22T02:55:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.455+0000 7f0652ffd640 1 -- 172.21.15.103:0/2963755420 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f064405a670 con 0x7f065c101140 2024-03-22T02:55:24.456 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 msgr2=0x7f06480707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 0x7f06480707c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f064c009990 tx=0x7f064c009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 msgr2=0x7f065c10e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0644002a50 tx=0x7f0644004600 comp rx=0 tx=0).stop 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 shutdown_connections 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f064806e310 0x7f06480707c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f065c101140 0x7f065c10e3d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 --2- 172.21.15.103:0/2963755420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c0fb110 0x7f065c10de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 >> 172.21.15.103:0/2963755420 conn(0x7f065c0710b0 msgr2=0x7f065c0f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 shutdown_connections 2024-03-22T02:55:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.457+0000 7f0660c58640 1 -- 172.21.15.103:0/2963755420 wait complete. 2024-03-22T02:55:24.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-22T02:55:24.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 -- 172.21.15.103:0/3998728264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 msgr2=0x7f24cc1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 --2- 172.21.15.103:0/3998728264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1063e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0048d0 tx=0x7f24bc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 -- 172.21.15.103:0/3998728264 shutdown_connections 2024-03-22T02:55:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 --2- 172.21.15.103:0/3998728264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1063e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 --2- 172.21.15.103:0/3998728264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc0fd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.568+0000 7f24d202e640 1 -- 172.21.15.103:0/3998728264 >> 172.21.15.103:0/3998728264 conn(0x7f24cc0f6eb0 msgr2=0x7f24cc0f92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.569+0000 7f24d202e640 1 -- 172.21.15.103:0/3998728264 shutdown_connections 2024-03-22T02:55:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.569+0000 7f24d202e640 1 -- 172.21.15.103:0/3998728264 wait complete. 2024-03-22T02:55:24.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.569+0000 7f24d202e640 1 Processor -- start 2024-03-22T02:55:24.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.569+0000 7f24d202e640 1 -- start start 2024-03-22T02:55:24.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.569+0000 7f24d202e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24d202e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24d202e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc113ae0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24d202e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc10d130 con 0x7f24cc0fd400 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47810/0 (socket says 172.21.15.103:47810) 2024-03-22T02:55:24.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 -- 172.21.15.103:0/2722331956 learned_addr learned my addr 172.21.15.103:0/2722331956 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24cb7fe640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 -- 172.21.15.103:0/2722331956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 msgr2=0x7f24cc112fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc112fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 -- 172.21.15.103:0/2722331956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc005040 con 0x7f24cc0fdd10 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24cb7fe640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:55:24.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24caffd640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0041d0 tx=0x7f24bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc0092e0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24bc009440 con 0x7f24cc0fdd10 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc10d3b0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc043640 con 0x7f24cc0fdd10 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.570+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc10d8d0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.572+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc065e50 con 0x7f24cc0fdd10 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.572+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7f24bc03b9f0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.572+0000 7f24b3fff640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 0x7f24b40707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.572+0000 7f24cb7fe640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 0x7f24b40707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.572+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f24bc0b0b70 con 0x7f24cc0fdd10 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.573+0000 7f24cb7fe640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 0x7f24b40707c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24c00041d0 tx=0x7f24c0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:24.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.577+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f24bc079df0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:24 smithi122 ceph-mon[83368]: pgmap v716: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:24.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:24 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2982978607' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:55:24.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:24 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/3251261503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:55:24.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.896+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f24cc109560 con 0x7f24cc0fdd10 2024-03-22T02:55:24.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.897+0000 7f24b3fff640 1 -- 172.21.15.103:0/2722331956 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f24bc07d4b0 con 0x7f24cc0fdd10 2024-03-22T02:55:24.898 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-03-22T02:55:24.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 msgr2=0x7f24b40707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 0x7f24b40707c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24c00041d0 tx=0x7f24c0009040 comp rx=0 tx=0).stop 2024-03-22T02:55:24.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 msgr2=0x7f24cc1134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0041d0 tx=0x7f24bc009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 shutdown_connections 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7f24b406e310 0x7f24b40707c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24cc0fdd10 0x7f24cc1134e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 --2- 172.21.15.103:0/2722331956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24cc0fd400 0x7f24cc112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 >> 172.21.15.103:0/2722331956 conn(0x7f24cc0f6eb0 msgr2=0x7f24cc0f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 shutdown_connections 2024-03-22T02:55:24.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:24.899+0000 7f24d202e640 1 -- 172.21.15.103:0/2722331956 wait complete. 2024-03-22T02:55:24.913 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T02:55:25.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 -- 172.21.15.103:0/25952970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 msgr2=0x7fb410108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:25.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 --2- 172.21.15.103:0/25952970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410108780 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80048d0 tx=0x7fb3f8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 -- 172.21.15.103:0/25952970 shutdown_connections 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 --2- 172.21.15.103:0/25952970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410108780 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 --2- 172.21.15.103:0/25952970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 0x7fb410064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 -- 172.21.15.103:0/25952970 >> 172.21.15.103:0/25952970 conn(0x7fb4100710b0 msgr2=0x7fb4100714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 -- 172.21.15.103:0/25952970 shutdown_connections 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.011+0000 7fb414d1d640 1 -- 172.21.15.103:0/25952970 wait complete. 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 Processor -- start 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 -- start start 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 0x7fb410114f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb410115a40 con 0x7fb410064cb0 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb414d1d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41010f090 con 0x7fb4100643a0 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb405d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:25.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb405d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47838/0 (socket says 172.21.15.103:47838) 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.012+0000 7fb405d9b640 1 -- 172.21.15.103:0/1032384344 learned_addr learned my addr 172.21.15.103:0/1032384344 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb40e59c640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 0x7fb410114f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb405d9b640 1 -- 172.21.15.103:0/1032384344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 msgr2=0x7fb410114f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb405d9b640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 0x7fb410114f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb405d9b640 1 -- 172.21.15.103:0/1032384344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8005040 con 0x7fb410064cb0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb405d9b640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8004930 tx=0x7fb3f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f80094c0 con 0x7fb410064cb0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3f8032e50 con 0x7fb410064cb0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41010f310 con 0x7fb410064cb0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3f80433b0 con 0x7fb410064cb0 2024-03-22T02:55:25.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.013+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41010f7d0 con 0x7fb410064cb0 2024-03-22T02:55:25.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.014+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4100fb730 con 0x7fb410064cb0 2024-03-22T02:55:25.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.015+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 41) v1 ==== 92219+0+0 (secure 0 0 0) 0x7fb3f802f070 con 0x7fb410064cb0 2024-03-22T02:55:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.015+0000 7fb4077fe640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 0x7fb3fc0707c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:55:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.015+0000 7fb40e59c640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 0x7fb3fc0707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:55:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.015+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3f80afc20 con 0x7fb410064cb0 2024-03-22T02:55:25.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.016+0000 7fb40e59c640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 0x7fb3fc0707c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb400004ed0 tx=0x7fb40000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:55:25.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.022+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3f8078ed0 con 0x7fb410064cb0 2024-03-22T02:55:25.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.217+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb41010b510 con 0x7fb410064cb0 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.218+0000 7fb4077fe640 1 -- 172.21.15.103:0/1032384344 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2348 (secure 0 0 0) 0x7fb3f807c590 con 0x7fb410064cb0 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6825/1237228570 2024-03-23T02:37:16.024221+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6824/1237228570 2024-03-23T02:37:16.024221+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6829/2193581075 2024-03-23T02:37:16.024200+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/2086230685 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6827/3786434277 2024-03-23T02:25:58.328445+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1548567537 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/179622989 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:25.219 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6828/2193581075 2024-03-23T02:37:16.024200+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/2790950462 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/2790950462 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/375109873 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6826/3786434277 2024-03-23T02:25:58.328445+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/1988149757 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/595266923 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6829/2536422926 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3955856929 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3344821418 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2826893442 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/3639247922 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/313561406 2024-03-23T02:20:50.358474+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:6828/2536422926 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.220 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3150495731 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/1197413654 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3724441621 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/375109873 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2509522170 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/1197413654 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3834849577 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/385301047 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3344821418 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/1160196781 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1309215164 2024-03-23T02:28:17.965286+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2699747522 2024-03-23T02:19:47.880360+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3243311702 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2788707715 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3672865475 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:25.221 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/1160196781 2024-03-23T02:19:16.742515+0000 2024-03-22T02:55:25.222 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.122:0/867576006 2024-03-23T02:30:07.935727+0000 2024-03-22T02:55:25.222 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1085098758 2024-03-23T02:32:46.783649+0000 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:listed 39 entries 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 msgr2=0x7fb3fc0707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 0x7fb3fc0707c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb400004ed0 tx=0x7fb40000b040 comp rx=0 tx=0).stop 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 msgr2=0x7fb410115440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8004930 tx=0x7fb3f8009da0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 shutdown_connections 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:6800/1355893550,v1:172.21.15.103:6801/1355893550] conn(0x7fb3fc06e310 0x7fb3fc0707c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb410064cb0 0x7fb410115440 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 --2- 172.21.15.103:0/1032384344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4100643a0 0x7fb410114f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 >> 172.21.15.103:0/1032384344 conn(0x7fb4100710b0 msgr2=0x7fb4100f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 shutdown_connections 2024-03-22T02:55:25.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-22T02:55:25.220+0000 7fb414d1d640 1 -- 172.21.15.103:0/1032384344 wait complete. 2024-03-22T02:55:25.231 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:25 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/766258385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:25.231 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:25 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2963755420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:25.231 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:25 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/2722331956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:25.232 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi103.front.sepia.ceph.com... 2024-03-22T02:55:25.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:55:25.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-22T02:55:25.260 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T02:55:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:25 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/766258385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:55:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:25 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2963755420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-22T02:55:25.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:25 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/2722331956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: pgmap v717: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: from='client.? 172.21.15.103:0/1032384344' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:55:26.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:55:26.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: pgmap v717: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:26.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: from='client.? 172.21.15.103:0/1032384344' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:55:26.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:55:26.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:55:26.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:55:26.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:55:28.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:28 smithi103 ceph-mon[115211]: pgmap v718: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:28.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:28 smithi122 ceph-mon[83368]: pgmap v718: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:30.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:30 smithi103 ceph-mon[115211]: pgmap v719: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:30.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:30 smithi122 ceph-mon[83368]: pgmap v719: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:32 smithi103 ceph-mon[115211]: pgmap v720: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:32.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:32.601 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:32 smithi122 ceph-mon[83368]: pgmap v720: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:32.601 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:34.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:34 smithi103 ceph-mon[115211]: pgmap v721: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:34.401 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:34 smithi122 ceph-mon[83368]: pgmap v721: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:36.374 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:36 smithi103 ceph-mon[115211]: pgmap v722: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:36.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:36 smithi122 ceph-mon[83368]: pgmap v722: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:38.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:38 smithi122 ceph-mon[83368]: pgmap v723: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:38.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:38 smithi103 ceph-mon[115211]: pgmap v723: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:40.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:40 smithi103 ceph-mon[115211]: pgmap v724: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:40.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:40 smithi122 ceph-mon[83368]: pgmap v724: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:42.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:42 smithi122 ceph-mon[83368]: pgmap v725: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:42.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:42 smithi103 ceph-mon[115211]: pgmap v725: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:44.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:44 smithi122 ceph-mon[83368]: pgmap v726: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:44.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:44 smithi103 ceph-mon[115211]: pgmap v726: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:46.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:46 smithi122 ceph-mon[83368]: pgmap v727: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:46.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:46 smithi103 ceph-mon[115211]: pgmap v727: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:47.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:47.626 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:55:48.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:48 smithi103 ceph-mon[115211]: pgmap v728: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:48.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:48 smithi122 ceph-mon[83368]: pgmap v728: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:50.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:50 smithi103 ceph-mon[115211]: pgmap v729: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:50.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:50 smithi122 ceph-mon[83368]: pgmap v729: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:52.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:52 smithi122 ceph-mon[83368]: pgmap v730: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:52.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:52 smithi103 ceph-mon[115211]: pgmap v730: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:54.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:54 smithi122 ceph-mon[83368]: pgmap v731: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:54.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:54 smithi103 ceph-mon[115211]: pgmap v731: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:56.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:56 smithi122 ceph-mon[83368]: pgmap v732: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:56.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:56 smithi103 ceph-mon[115211]: pgmap v732: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:58.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:55:58 smithi122 ceph-mon[83368]: pgmap v733: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:55:58.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:55:58 smithi103 ceph-mon[115211]: pgmap v733: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:00.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:00 smithi103 ceph-mon[115211]: pgmap v734: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:00.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:00 smithi122 ceph-mon[83368]: pgmap v734: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:02.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:02 smithi122 ceph-mon[83368]: pgmap v735: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:02.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:02 smithi103 ceph-mon[115211]: pgmap v735: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:02.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:04.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:04 smithi122 ceph-mon[83368]: pgmap v736: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:04.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:04 smithi103 ceph-mon[115211]: pgmap v736: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:05.598 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:05 smithi122 ceph-mon[83368]: pgmap v737: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:05.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:05 smithi103 ceph-mon[115211]: pgmap v737: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:07 smithi122 ceph-mon[83368]: pgmap v738: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:07 smithi103 ceph-mon[115211]: pgmap v738: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:09.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:09 smithi122 ceph-mon[83368]: pgmap v739: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:09.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:09 smithi103 ceph-mon[115211]: pgmap v739: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:11 smithi122 ceph-mon[83368]: pgmap v740: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:11 smithi103 ceph-mon[115211]: pgmap v740: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:13.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:13 smithi122 ceph-mon[83368]: pgmap v741: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:13.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:13 smithi103 ceph-mon[115211]: pgmap v741: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:15 smithi122 ceph-mon[83368]: pgmap v742: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:15 smithi103 ceph-mon[115211]: pgmap v742: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:17 smithi122 ceph-mon[83368]: pgmap v743: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:17.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:17 smithi103 ceph-mon[115211]: pgmap v743: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:19.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:19 smithi122 ceph-mon[83368]: pgmap v744: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:19 smithi103 ceph-mon[115211]: pgmap v744: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:21 smithi122 ceph-mon[83368]: pgmap v745: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:21 smithi103 ceph-mon[115211]: pgmap v745: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:23.850 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:23 smithi122 ceph-mon[83368]: pgmap v746: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:23 smithi103 ceph-mon[115211]: pgmap v746: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:25.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:25 smithi122 ceph-mon[83368]: pgmap v747: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:25 smithi103 ceph-mon[115211]: pgmap v747: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:26 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:56:26.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:26 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:56:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:27 smithi122 ceph-mon[83368]: pgmap v748: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:56:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:56:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:56:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:27 smithi103 ceph-mon[115211]: pgmap v748: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:56:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:56:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:56:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:29 smithi122 ceph-mon[83368]: pgmap v749: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:29 smithi103 ceph-mon[115211]: pgmap v749: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:31.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:31 smithi122 ceph-mon[83368]: pgmap v750: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:31 smithi103 ceph-mon[115211]: pgmap v750: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:32.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:33 smithi122 ceph-mon[83368]: pgmap v751: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:33 smithi103 ceph-mon[115211]: pgmap v751: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:35 smithi122 ceph-mon[83368]: pgmap v752: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:35 smithi103 ceph-mon[115211]: pgmap v752: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:37 smithi122 ceph-mon[83368]: pgmap v753: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:37 smithi103 ceph-mon[115211]: pgmap v753: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:39 smithi122 ceph-mon[83368]: pgmap v754: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:39.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:39 smithi103 ceph-mon[115211]: pgmap v754: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:41 smithi122 ceph-mon[83368]: pgmap v755: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:41 smithi103 ceph-mon[115211]: pgmap v755: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:43.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:43 smithi122 ceph-mon[83368]: pgmap v756: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:43 smithi103 ceph-mon[115211]: pgmap v756: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:45 smithi122 ceph-mon[83368]: pgmap v757: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:45 smithi103 ceph-mon[115211]: pgmap v757: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:47 smithi122 ceph-mon[83368]: pgmap v758: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:47.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:47 smithi103 ceph-mon[115211]: pgmap v758: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:56:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:49 smithi122 ceph-mon[83368]: pgmap v759: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:49 smithi103 ceph-mon[115211]: pgmap v759: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:51 smithi122 ceph-mon[83368]: pgmap v760: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:51 smithi103 ceph-mon[115211]: pgmap v760: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:53 smithi122 ceph-mon[83368]: pgmap v761: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:53 smithi103 ceph-mon[115211]: pgmap v761: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:55 smithi122 ceph-mon[83368]: pgmap v762: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:55 smithi103 ceph-mon[115211]: pgmap v762: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:57 smithi122 ceph-mon[83368]: pgmap v763: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:57 smithi103 ceph-mon[115211]: pgmap v763: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:56:59 smithi122 ceph-mon[83368]: pgmap v764: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:56:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:56:59 smithi103 ceph-mon[115211]: pgmap v764: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:01 smithi122 ceph-mon[83368]: pgmap v765: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:01 smithi103 ceph-mon[115211]: pgmap v765: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:03 smithi122 ceph-mon[83368]: pgmap v766: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:03 smithi103 ceph-mon[115211]: pgmap v766: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:05 smithi122 ceph-mon[83368]: pgmap v767: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:05 smithi103 ceph-mon[115211]: pgmap v767: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:07 smithi122 ceph-mon[83368]: pgmap v768: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:07 smithi103 ceph-mon[115211]: pgmap v768: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:09.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:09 smithi122 ceph-mon[83368]: pgmap v769: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:09.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:09 smithi103 ceph-mon[115211]: pgmap v769: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:11.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:11 smithi122 ceph-mon[83368]: pgmap v770: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:11 smithi103 ceph-mon[115211]: pgmap v770: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:13.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:13 smithi122 ceph-mon[83368]: pgmap v771: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:13.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:13 smithi103 ceph-mon[115211]: pgmap v771: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:15 smithi122 ceph-mon[83368]: pgmap v772: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:15 smithi103 ceph-mon[115211]: pgmap v772: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:17.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:17 smithi122 ceph-mon[83368]: pgmap v773: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:17.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:17 smithi103 ceph-mon[115211]: pgmap v773: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:19.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:19 smithi122 ceph-mon[83368]: pgmap v774: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:19 smithi103 ceph-mon[115211]: pgmap v774: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:21.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:21 smithi122 ceph-mon[83368]: pgmap v775: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:21 smithi103 ceph-mon[115211]: pgmap v775: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:23.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:23 smithi122 ceph-mon[83368]: pgmap v776: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:23.875 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:23 smithi103 ceph-mon[115211]: pgmap v776: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:25.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:25 smithi122 ceph-mon[83368]: pgmap v777: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:25.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:25 smithi103 ceph-mon[115211]: pgmap v777: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:27.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:27 smithi122 ceph-mon[83368]: pgmap v778: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:57:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:57:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:57:27.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:57:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:27 smithi103 ceph-mon[115211]: pgmap v778: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:57:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:57:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:57:27.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:57:29.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:29 smithi122 ceph-mon[83368]: pgmap v779: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:29.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:29 smithi103 ceph-mon[115211]: pgmap v779: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:31.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:31 smithi122 ceph-mon[83368]: pgmap v780: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:31.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:31 smithi103 ceph-mon[115211]: pgmap v780: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:32.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:32.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:33.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:33 smithi122 ceph-mon[83368]: pgmap v781: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:33.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:33 smithi103 ceph-mon[115211]: pgmap v781: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:35.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:35 smithi122 ceph-mon[83368]: pgmap v782: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:35.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:35 smithi103 ceph-mon[115211]: pgmap v782: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:37.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:37 smithi122 ceph-mon[83368]: pgmap v783: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:37.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:37 smithi103 ceph-mon[115211]: pgmap v783: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:39.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:39 smithi122 ceph-mon[83368]: pgmap v784: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:39.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:39 smithi103 ceph-mon[115211]: pgmap v784: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:41.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:41 smithi122 ceph-mon[83368]: pgmap v785: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:41.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:41 smithi103 ceph-mon[115211]: pgmap v785: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:43.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:43 smithi122 ceph-mon[83368]: pgmap v786: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:43.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:43 smithi103 ceph-mon[115211]: pgmap v786: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:45.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:45 smithi122 ceph-mon[83368]: pgmap v787: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:45.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:45 smithi103 ceph-mon[115211]: pgmap v787: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:47.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:47 smithi122 ceph-mon[83368]: pgmap v788: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:47.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:47 smithi103 ceph-mon[115211]: pgmap v788: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:47.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:57:49.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:49 smithi122 ceph-mon[83368]: pgmap v789: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:49.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:49 smithi103 ceph-mon[115211]: pgmap v789: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:51.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:51 smithi122 ceph-mon[83368]: pgmap v790: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:51.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:51 smithi103 ceph-mon[115211]: pgmap v790: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:53.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:53 smithi122 ceph-mon[83368]: pgmap v791: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:53.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:53 smithi103 ceph-mon[115211]: pgmap v791: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:55.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:55 smithi122 ceph-mon[83368]: pgmap v792: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:55.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:55 smithi103 ceph-mon[115211]: pgmap v792: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:57.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:57 smithi122 ceph-mon[83368]: pgmap v793: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:57.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:57 smithi103 ceph-mon[115211]: pgmap v793: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:59.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:57:59 smithi122 ceph-mon[83368]: pgmap v794: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:57:59.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:57:59 smithi103 ceph-mon[115211]: pgmap v794: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:01.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:01 smithi122 ceph-mon[83368]: pgmap v795: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:01.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:01 smithi103 ceph-mon[115211]: pgmap v795: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:02.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:02.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:03.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:03 smithi122 ceph-mon[83368]: pgmap v796: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:03.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:03 smithi103 ceph-mon[115211]: pgmap v796: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:05.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:05 smithi122 ceph-mon[83368]: pgmap v797: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:05.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:05 smithi103 ceph-mon[115211]: pgmap v797: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:07.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:07 smithi122 ceph-mon[83368]: pgmap v798: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:07.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:07 smithi103 ceph-mon[115211]: pgmap v798: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:09.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:09 smithi122 ceph-mon[83368]: pgmap v799: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:09.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:09 smithi103 ceph-mon[115211]: pgmap v799: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:11.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:11 smithi122 ceph-mon[83368]: pgmap v800: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:11.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:11 smithi103 ceph-mon[115211]: pgmap v800: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:13.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:13 smithi122 ceph-mon[83368]: pgmap v801: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:13.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:13 smithi103 ceph-mon[115211]: pgmap v801: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:15.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:15 smithi122 ceph-mon[83368]: pgmap v802: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:15.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:15 smithi103 ceph-mon[115211]: pgmap v802: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:17 smithi103 ceph-mon[115211]: pgmap v803: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:17.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:17 smithi122 ceph-mon[83368]: pgmap v803: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:19.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:19 smithi103 ceph-mon[115211]: pgmap v804: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:20.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:19 smithi122 ceph-mon[83368]: pgmap v804: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:21.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:21 smithi103 ceph-mon[115211]: pgmap v805: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:21 smithi122 ceph-mon[83368]: pgmap v805: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:23.874 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:23 smithi103 ceph-mon[115211]: pgmap v806: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:23 smithi122 ceph-mon[83368]: pgmap v806: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:25 smithi122 ceph-mon[83368]: pgmap v807: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:25 smithi103 ceph-mon[115211]: pgmap v807: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:27 smithi122 ceph-mon[83368]: pgmap v808: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:28.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:27 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:58:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:27 smithi103 ceph-mon[115211]: pgmap v808: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:27 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:58:29.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:58:29.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:58:29.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:58:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:58:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:58:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:58:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:29 smithi122 ceph-mon[83368]: pgmap v809: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:29 smithi103 ceph-mon[115211]: pgmap v809: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:31 smithi122 ceph-mon[83368]: pgmap v810: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:31 smithi103 ceph-mon[115211]: pgmap v810: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:33 smithi122 ceph-mon[83368]: pgmap v811: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:34.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:33 smithi103 ceph-mon[115211]: pgmap v811: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:35 smithi122 ceph-mon[83368]: pgmap v812: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:35 smithi103 ceph-mon[115211]: pgmap v812: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:38.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:37 smithi122 ceph-mon[83368]: pgmap v813: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:38.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:37 smithi103 ceph-mon[115211]: pgmap v813: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:39 smithi122 ceph-mon[83368]: pgmap v814: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:39 smithi103 ceph-mon[115211]: pgmap v814: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:41 smithi122 ceph-mon[83368]: pgmap v815: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:42.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:41 smithi103 ceph-mon[115211]: pgmap v815: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:43 smithi122 ceph-mon[83368]: pgmap v816: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:43 smithi103 ceph-mon[115211]: pgmap v816: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:46.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:45 smithi122 ceph-mon[83368]: pgmap v817: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:46.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:45 smithi103 ceph-mon[115211]: pgmap v817: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:47 smithi122 ceph-mon[83368]: pgmap v818: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:47 smithi103 ceph-mon[115211]: pgmap v818: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:58:50.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:49 smithi122 ceph-mon[83368]: pgmap v819: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:49 smithi103 ceph-mon[115211]: pgmap v819: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:52.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:51 smithi122 ceph-mon[83368]: pgmap v820: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:51 smithi103 ceph-mon[115211]: pgmap v820: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:54.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:53 smithi122 ceph-mon[83368]: pgmap v821: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:54.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:53 smithi103 ceph-mon[115211]: pgmap v821: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:56.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:55 smithi122 ceph-mon[83368]: pgmap v822: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:56.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:55 smithi103 ceph-mon[115211]: pgmap v822: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:58.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:57 smithi122 ceph-mon[83368]: pgmap v823: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:58:58.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:57 smithi103 ceph-mon[115211]: pgmap v823: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:00.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:58:59 smithi122 ceph-mon[83368]: pgmap v824: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:58:59 smithi103 ceph-mon[115211]: pgmap v824: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:01 smithi122 ceph-mon[83368]: pgmap v825: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:01 smithi103 ceph-mon[115211]: pgmap v825: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:03.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:03.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:04.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:03 smithi122 ceph-mon[83368]: pgmap v826: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:03 smithi103 ceph-mon[115211]: pgmap v826: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:05 smithi122 ceph-mon[83368]: pgmap v827: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:05 smithi103 ceph-mon[115211]: pgmap v827: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:07 smithi122 ceph-mon[83368]: pgmap v828: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:07 smithi103 ceph-mon[115211]: pgmap v828: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:09 smithi122 ceph-mon[83368]: pgmap v829: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:09 smithi103 ceph-mon[115211]: pgmap v829: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:12.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:11 smithi122 ceph-mon[83368]: pgmap v830: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:11 smithi103 ceph-mon[115211]: pgmap v830: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:14.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:13 smithi122 ceph-mon[83368]: pgmap v831: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:13 smithi103 ceph-mon[115211]: pgmap v831: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:16.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:15 smithi122 ceph-mon[83368]: pgmap v832: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:15 smithi103 ceph-mon[115211]: pgmap v832: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:18.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:17 smithi122 ceph-mon[83368]: pgmap v833: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:18.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:18.127 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:17 smithi103 ceph-mon[115211]: pgmap v833: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:18.128 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:20.100 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:19 smithi122 ceph-mon[83368]: pgmap v834: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:20.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:19 smithi103 ceph-mon[115211]: pgmap v834: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:22.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:21 smithi122 ceph-mon[83368]: pgmap v835: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:21 smithi103 ceph-mon[115211]: pgmap v835: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:24.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:23 smithi122 ceph-mon[83368]: pgmap v836: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:23 smithi103 ceph-mon[115211]: pgmap v836: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:26.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:25 smithi122 ceph-mon[83368]: pgmap v837: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:26.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:25 smithi103 ceph-mon[115211]: pgmap v837: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:28.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:27 smithi122 ceph-mon[83368]: pgmap v838: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:28.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:27 smithi103 ceph-mon[115211]: pgmap v838: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:29.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:28 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:59:29.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:28 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:59:30.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:29 smithi122 ceph-mon[83368]: pgmap v839: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:30.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:59:30.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:59:30.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:29 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:59:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:29 smithi103 ceph-mon[115211]: pgmap v839: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:59:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:59:30.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:29 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' 2024-03-22T02:59:32.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:31 smithi122 ceph-mon[83368]: pgmap v840: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:32.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:31 smithi103 ceph-mon[115211]: pgmap v840: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:33.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:32 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:33.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:32 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:34.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:33 smithi122 ceph-mon[83368]: pgmap v841: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:34.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:33 smithi103 ceph-mon[115211]: pgmap v841: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:36.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:35 smithi122 ceph-mon[83368]: pgmap v842: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:36.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:35 smithi103 ceph-mon[115211]: pgmap v842: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:38.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:37 smithi122 ceph-mon[83368]: pgmap v843: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:38.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:37 smithi103 ceph-mon[115211]: pgmap v843: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:40.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:39 smithi122 ceph-mon[83368]: pgmap v844: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:40.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:39 smithi103 ceph-mon[115211]: pgmap v844: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:42.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:41 smithi122 ceph-mon[83368]: pgmap v845: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:42.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:41 smithi103 ceph-mon[115211]: pgmap v845: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:44.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:43 smithi122 ceph-mon[83368]: pgmap v846: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:44.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:43 smithi103 ceph-mon[115211]: pgmap v846: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:46.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:45 smithi122 ceph-mon[83368]: pgmap v847: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:46.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:45 smithi103 ceph-mon[115211]: pgmap v847: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:48.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:47 smithi122 ceph-mon[83368]: pgmap v848: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:48.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:47 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:47 smithi103 ceph-mon[115211]: pgmap v848: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:48.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:47 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T02:59:50.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:49 smithi122 ceph-mon[83368]: pgmap v849: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:50.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:49 smithi103 ceph-mon[115211]: pgmap v849: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:52.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:51 smithi122 ceph-mon[83368]: pgmap v850: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:52.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:51 smithi103 ceph-mon[115211]: pgmap v850: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:54.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:53 smithi122 ceph-mon[83368]: pgmap v851: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:54.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:53 smithi103 ceph-mon[115211]: pgmap v851: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:56.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:55 smithi122 ceph-mon[83368]: pgmap v852: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:56.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:55 smithi103 ceph-mon[115211]: pgmap v852: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:58.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:57 smithi122 ceph-mon[83368]: pgmap v853: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T02:59:58.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:57 smithi103 ceph-mon[115211]: pgmap v853: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:00.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 02:59:59 smithi122 ceph-mon[83368]: pgmap v854: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:00.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 02:59:59 smithi103 ceph-mon[115211]: pgmap v854: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:01.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:00 smithi122 ceph-mon[83368]: overall HEALTH_OK 2024-03-22T03:00:01.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:00 smithi103 ceph-mon[115211]: overall HEALTH_OK 2024-03-22T03:00:02.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:01 smithi122 ceph-mon[83368]: pgmap v855: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:02.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:01 smithi103 ceph-mon[115211]: pgmap v855: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:03.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:02 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T03:00:03.125 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:02 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T03:00:04.099 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:03 smithi122 ceph-mon[83368]: pgmap v856: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:04.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:03 smithi103 ceph-mon[115211]: pgmap v856: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:06.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:05 smithi122 ceph-mon[83368]: pgmap v857: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:06.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:05 smithi103 ceph-mon[115211]: pgmap v857: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:08.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:07 smithi122 ceph-mon[83368]: pgmap v858: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:08.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:07 smithi103 ceph-mon[115211]: pgmap v858: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:10.098 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:09 smithi122 ceph-mon[83368]: pgmap v859: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:10.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:09 smithi103 ceph-mon[115211]: pgmap v859: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:12.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:11 smithi103 ceph-mon[115211]: pgmap v860: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:12.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:11 smithi122 ceph-mon[83368]: pgmap v860: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:14.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:13 smithi103 ceph-mon[115211]: pgmap v861: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:14.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:13 smithi122 ceph-mon[83368]: pgmap v861: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:16.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:15 smithi103 ceph-mon[115211]: pgmap v862: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:16.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:15 smithi122 ceph-mon[83368]: pgmap v862: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:17 smithi103 ceph-mon[115211]: pgmap v863: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:18.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:17 smithi103 ceph-mon[115211]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T03:00:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:17 smithi122 ceph-mon[83368]: pgmap v863: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:18.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:17 smithi122 ceph-mon[83368]: from='mgr.34104 172.21.15.103:0/3414603574' entity='mgr.smithi103.pfvmui' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T03:00:20.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:19 smithi103 ceph-mon[115211]: pgmap v864: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:20.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:19 smithi122 ceph-mon[83368]: pgmap v864: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:22.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:21 smithi103 ceph-mon[115211]: pgmap v865: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:22.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:21 smithi122 ceph-mon[83368]: pgmap v865: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:24.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:23 smithi103 ceph-mon[115211]: pgmap v866: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:24.333 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-03-22T03:00:24.333 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-22T03:00:24.335 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-22T03:00:24.346 INFO:tasks.cephadm:Teardown begin 2024-03-22T03:00:24.346 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-22T03:00:24.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T03:00:24.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:23 smithi122 ceph-mon[83368]: pgmap v866: 65 pgs: 65 active+clean; 257 MiB data, 996 MiB used, 535 GiB / 536 GiB avail 2024-03-22T03:00:24.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T03:00:24.414 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-22T03:00:24.414 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T03:00:24.435 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T03:00:24.473 INFO:tasks.cephadm:Stopping all daemons... 2024-03-22T03:00:24.473 INFO:tasks.cephadm.mon.smithi103:Stopping mon.smithi103... 2024-03-22T03:00:24.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103 2024-03-22T03:00:24.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:24 smithi103 systemd[1]: Stopping Ceph mon.smithi103 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:24 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T03:00:24.800+0000 7f4cf6d5a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi103 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:24 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103[115207]: 2024-03-22T03:00:24.800+0000 7f4cf6d5a700 -1 mon.smithi103@0(leader) e3 *** Got Signal Terminated *** 2024-03-22T03:00:25.124 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:24 smithi103 podman[161941]: 2024-03-22 03:00:24.952013426 +0000 UTC m=+0.274770460 container died 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-22T03:00:25.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:25 smithi103 podman[161941]: 2024-03-22 03:00:25.425561824 +0000 UTC m=+0.748318857 container cleanup 73b5f7067b0e26dce9728b33e6627489c6c1d29fa774c1271663aed18326858f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T03:00:25.624 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Mar 22 03:00:25 smithi103 bash[161941]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi103 2024-03-22T03:00:25.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi103.service' 2024-03-22T03:00:25.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:00:25.921 INFO:tasks.cephadm.mon.smithi103:Stopped mon.smithi103 2024-03-22T03:00:25.921 INFO:tasks.cephadm.mon.smithi122:Stopping mon.smithi122... 2024-03-22T03:00:25.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi122 2024-03-22T03:00:26.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:25 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:26.848 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:26 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122[83364]: 2024-03-22T03:00:26.547+0000 7f10a114f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:26 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122[83364]: 2024-03-22T03:00:26.547+0000 7f10a114f700 -1 mon.smithi122@1(peon) e3 *** Got Signal Terminated *** 2024-03-22T03:00:26.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:26 smithi122 podman[115637]: 2024-03-22 03:00:26.73099447 +0000 UTC m=+0.617991444 container died b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:27 smithi122 podman[115637]: 2024-03-22 03:00:27.230845343 +0000 UTC m=+1.117842272 container cleanup b334dc9d4e9fed44e5595a96961d947a287b9ff78dd7bd3a52db543f3ec0f00f (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True) 2024-03-22T03:00:27.348 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Mar 22 03:00:27 smithi122 bash[115637]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-mon-smithi122 2024-03-22T03:00:27.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@mon.smithi122.service' 2024-03-22T03:00:27.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:00:27.694 INFO:tasks.cephadm.mon.smithi122:Stopped mon.smithi122 2024-03-22T03:00:27.694 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-22T03:00:27.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.0 2024-03-22T03:00:28.030 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:27 smithi103 systemd[1]: Stopping Ceph osd.0 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:28.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T03:00:28.029+0000 7f40c94c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:28.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T03:00:28.029+0000 7f40c94c9700 -1 osd.0 84 *** Got signal Terminated *** 2024-03-22T03:00:28.374 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:28 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0[126630]: 2024-03-22T03:00:28.029+0000 7f40c94c9700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:00:33.849 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:33 smithi103 podman[162070]: 2024-03-22 03:00:33.543807318 +0000 UTC m=+5.627310954 container died fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T03:00:34.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:33 smithi103 podman[162070]: 2024-03-22 03:00:33.939223023 +0000 UTC m=+6.022726659 container cleanup fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T03:00:34.124 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:33 smithi103 bash[162070]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0 2024-03-22T03:00:34.573 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:34 smithi103 podman[162085]: 2024-03-22 03:00:34.199532735 +0000 UTC m=+0.658328635 container remove fc16d3339890670ceacb4a0b8d1c60dc9408848813ea352f6a8778453c256503 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-22T03:00:34.874 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:34 smithi103 podman[162174]: 2024-03-22 03:00:34.476014889 +0000 UTC m=+0.020125329 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:00:34.874 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:34 smithi103 podman[162174]: 2024-03-22 03:00:34.600689204 +0000 UTC m=+0.144799648 container create 0091c9f78771a8eb63557557185064eef5f2e954cc5f5e78f1a3f5e336f0df95 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-22T03:00:35.251 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:34 smithi103 podman[162174]: 2024-03-22 03:00:34.922101727 +0000 UTC m=+0.466212160 container init 0091c9f78771a8eb63557557185064eef5f2e954cc5f5e78f1a3f5e336f0df95 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T03:00:35.251 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:34 smithi103 podman[162174]: 2024-03-22 03:00:34.926120172 +0000 UTC m=+0.470230605 container start 0091c9f78771a8eb63557557185064eef5f2e954cc5f5e78f1a3f5e336f0df95 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:35.252 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:35 smithi103 podman[162174]: 2024-03-22 03:00:35.020893814 +0000 UTC m=+0.565004256 container attach 0091c9f78771a8eb63557557185064eef5f2e954cc5f5e78f1a3f5e336f0df95 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-22T03:00:35.529 INFO:journalctl@ceph.osd.0.smithi103.stdout:Mar 22 03:00:35 smithi103 podman[162174]: 2024-03-22 03:00:35.250921157 +0000 UTC m=+0.795031596 container died 0091c9f78771a8eb63557557185064eef5f2e954cc5f5e78f1a3f5e336f0df95 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:36.225 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.0.service' 2024-03-22T03:00:36.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:00:36.273 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-22T03:00:36.273 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-22T03:00:36.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.1 2024-03-22T03:00:36.614 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:36 smithi103 systemd[1]: Stopping Ceph osd.1 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:36.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:36 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T03:00:36.612+0000 7fa7f7b9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:36.875 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:36 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T03:00:36.612+0000 7fa7f7b9e700 -1 osd.1 84 *** Got signal Terminated *** 2024-03-22T03:00:36.875 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:36 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1[132453]: 2024-03-22T03:00:36.612+0000 7fa7f7b9e700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:00:42.306 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:41 smithi103 podman[162292]: 2024-03-22 03:00:41.996200699 +0000 UTC m=+5.510384403 container died 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T03:00:42.623 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:42 smithi103 podman[162292]: 2024-03-22 03:00:42.404193525 +0000 UTC m=+5.918377225 container cleanup 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-22T03:00:42.623 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:42 smithi103 bash[162292]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1 2024-03-22T03:00:42.874 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:42 smithi103 podman[162312]: 2024-03-22 03:00:42.622322984 +0000 UTC m=+0.619326802 container remove 6e232377738367dd4685185dbdf2b3c48b8c11631f6fefe3fa375569c6d49688 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T03:00:43.226 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:42 smithi103 podman[162403]: 2024-03-22 03:00:42.857681908 +0000 UTC m=+0.023030800 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:00:43.226 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:42 smithi103 podman[162403]: 2024-03-22 03:00:42.981833816 +0000 UTC m=+0.147182701 container create 95c03ef6a6c8b0b502228f9aaca95222bdff39187d947e122eacfede0c7f0311 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-22T03:00:43.226 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:43 smithi103 podman[162403]: 2024-03-22 03:00:43.225962129 +0000 UTC m=+0.391311007 container init 95c03ef6a6c8b0b502228f9aaca95222bdff39187d947e122eacfede0c7f0311 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-22T03:00:43.561 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:43 smithi103 podman[162403]: 2024-03-22 03:00:43.229655542 +0000 UTC m=+0.395004423 container start 95c03ef6a6c8b0b502228f9aaca95222bdff39187d947e122eacfede0c7f0311 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-22T03:00:43.561 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:43 smithi103 podman[162403]: 2024-03-22 03:00:43.317749525 +0000 UTC m=+0.483098403 container attach 95c03ef6a6c8b0b502228f9aaca95222bdff39187d947e122eacfede0c7f0311 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T03:00:43.839 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 22 03:00:43 smithi103 podman[162403]: 2024-03-22 03:00:43.560380489 +0000 UTC m=+0.725729365 container died 95c03ef6a6c8b0b502228f9aaca95222bdff39187d947e122eacfede0c7f0311 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T03:00:44.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.1.service' 2024-03-22T03:00:44.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:00:44.198 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-22T03:00:44.198 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-22T03:00:44.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.2 2024-03-22T03:00:44.567 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:44 smithi103 systemd[1]: Stopping Ceph osd.2 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:44.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:44 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T03:00:44.566+0000 7fa0e9cfa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:44.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:44 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T03:00:44.566+0000 7fa0e9cfa700 -1 osd.2 84 *** Got signal Terminated *** 2024-03-22T03:00:44.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:44 smithi103 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2[138264]: 2024-03-22T03:00:44.566+0000 7fa0e9cfa700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:00:50.210 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:49 smithi103 podman[162509]: 2024-03-22 03:00:49.918717022 +0000 UTC m=+5.477153747 container died 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True) 2024-03-22T03:00:50.512 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:50 smithi103 podman[162509]: 2024-03-22 03:00:50.308652439 +0000 UTC m=+5.867089164 container cleanup 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:50.513 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:50 smithi103 bash[162509]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2 2024-03-22T03:00:50.874 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:50 smithi103 podman[162529]: 2024-03-22 03:00:50.577219574 +0000 UTC m=+0.650701470 container remove 2be02d5e6564d9a3499b989a57100e27a8963ccd0d432452a951e6d296692725 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T03:00:51.198 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:50 smithi103 podman[162621]: 2024-03-22 03:00:50.822723012 +0000 UTC m=+0.018254081 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:00:51.198 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:50 smithi103 podman[162621]: 2024-03-22 03:00:50.945136666 +0000 UTC m=+0.140667740 container create 278feeadeb32eb28c2e78973003193858525c0cc6b4f620c6b148efd790bbd07 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-22T03:00:51.198 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:51 smithi103 podman[162621]: 2024-03-22 03:00:51.197587958 +0000 UTC m=+0.393119027 container init 278feeadeb32eb28c2e78973003193858525c0cc6b4f620c6b148efd790bbd07 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-22T03:00:51.542 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:51 smithi103 podman[162621]: 2024-03-22 03:00:51.200820021 +0000 UTC m=+0.396351103 container start 278feeadeb32eb28c2e78973003193858525c0cc6b4f620c6b148efd790bbd07 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T03:00:51.542 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:51 smithi103 podman[162621]: 2024-03-22 03:00:51.281017882 +0000 UTC m=+0.476548955 container attach 278feeadeb32eb28c2e78973003193858525c0cc6b4f620c6b148efd790bbd07 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-22T03:00:51.859 INFO:journalctl@ceph.osd.2.smithi103.stdout:Mar 22 03:00:51 smithi103 podman[162621]: 2024-03-22 03:00:51.539270822 +0000 UTC m=+0.734801900 container died 278feeadeb32eb28c2e78973003193858525c0cc6b4f620c6b148efd790bbd07 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T03:00:52.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.2.service' 2024-03-22T03:00:52.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:00:52.205 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-22T03:00:52.205 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-22T03:00:52.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.3 2024-03-22T03:00:52.509 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:52 smithi122 systemd[1]: Stopping Ceph osd.3 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:00:52.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:52 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T03:00:52.507+0000 7f2dcc272700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:00:52.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:52 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T03:00:52.507+0000 7f2dcc272700 -1 osd.3 84 *** Got signal Terminated *** 2024-03-22T03:00:52.849 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:52 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3[92311]: 2024-03-22T03:00:52.507+0000 7f2dcc272700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:00:55.098 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:54 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:54.863+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:00:56.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:55 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:55.908+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:00:56.848 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:00:56 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:00:56.403+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:00:57.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:56 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:56.957+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:00:57.848 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:00:57 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:00:57.376+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:00:58.320 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:58.007+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:00:58.320 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:58 smithi122 podman[115758]: 2024-03-22 03:00:58.04784286 +0000 UTC m=+5.653417876 container died 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-22T03:00:58.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:00:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:00:58.401+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:00:58.599 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:58 smithi122 podman[115758]: 2024-03-22 03:00:58.418526989 +0000 UTC m=+6.024101996 container cleanup 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:58.599 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:58 smithi122 bash[115758]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3 2024-03-22T03:00:58.986 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:58 smithi122 podman[115789]: 2024-03-22 03:00:58.728448348 +0000 UTC m=+0.675806885 container remove 4357b0d26181adccc79f7d162f521b2475cb57d745334e5fcc81d7d08a992b11 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T03:00:59.249 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:58 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:58.985+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:00:59.250 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:58.980823083 +0000 UTC m=+0.017930333 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:00:59.250 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:59.104564927 +0000 UTC m=+0.141672161 container create 783ee40ed73ff13efc9182611d3528a595fad851379b17b877410a5e61bae219 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T03:00:59.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:00:59 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:00:59.356+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:00:59.599 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:59.357793012 +0000 UTC m=+0.394900263 container init 783ee40ed73ff13efc9182611d3528a595fad851379b17b877410a5e61bae219 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T03:00:59.599 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:59.361354292 +0000 UTC m=+0.398461526 container start 783ee40ed73ff13efc9182611d3528a595fad851379b17b877410a5e61bae219 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-22T03:00:59.599 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:59.449163859 +0000 UTC m=+0.486271110 container attach 783ee40ed73ff13efc9182611d3528a595fad851379b17b877410a5e61bae219 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:00:59.989 INFO:journalctl@ceph.osd.3.smithi122.stdout:Mar 22 03:00:59 smithi122 podman[115883]: 2024-03-22 03:00:59.657975798 +0000 UTC m=+0.695083034 container died 783ee40ed73ff13efc9182611d3528a595fad851379b17b877410a5e61bae219 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-22T03:01:00.310 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:00:59 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:00:59.988+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:00.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.3.service' 2024-03-22T03:01:00.372 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:01:00.372 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-22T03:01:00.372 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-22T03:01:00.373 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.4 2024-03-22T03:01:00.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:00.372+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:00.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:00 smithi122 systemd[1]: Stopping Ceph osd.4 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:01:00.977 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:00.721+0000 7fa432ce7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:01:00.977 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:00.721+0000 7fa432ce7700 -1 osd.4 84 *** Got signal Terminated *** 2024-03-22T03:01:00.977 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:00.721+0000 7fa432ce7700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:01:01.345 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:00 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:00.976+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:01.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:01 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:01.344+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:02.296 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:02.002+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:02.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:02.295+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:03.307 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:02 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:02.956+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:03.598 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:03 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:03.306+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:04.259 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:03 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:03.949+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:04.259 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:03 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:03.949+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:04.601 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:04 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:04.258+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:04.602 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:04 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:04.258+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:39.273225+0000 front 2024-03-22T03:00:39.273243+0000 (oldest deadline 2024-03-22T03:01:03.373243+0000) 2024-03-22T03:01:05.246 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:04 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:04.918+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:05.276 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:04 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:04.918+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:05.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:05 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:05.246+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:31.672297+0000 front 2024-03-22T03:00:31.672364+0000 (oldest deadline 2024-03-22T03:00:55.772337+0000) 2024-03-22T03:01:05.599 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:05 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4[97164]: 2024-03-22T03:01:05.246+0000 7fa42f2db700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:39.273225+0000 front 2024-03-22T03:00:39.273243+0000 (oldest deadline 2024-03-22T03:01:03.373243+0000) 2024-03-22T03:01:06.349 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:06 smithi122 podman[115988]: 2024-03-22 03:01:06.144190054 +0000 UTC m=+5.545976953 container died d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-22T03:01:06.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:05 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:05.938+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:06.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:05 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:05.938+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:06.782 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:06 smithi122 podman[115988]: 2024-03-22 03:01:06.563696538 +0000 UTC m=+5.965483444 container cleanup d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux ) 2024-03-22T03:01:06.782 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:06 smithi122 bash[115988]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4 2024-03-22T03:01:07.098 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:06 smithi122 podman[116023]: 2024-03-22 03:01:06.782055904 +0000 UTC m=+0.627603733 container remove d25f12ea02320453bc37285d6b3b058917b2f5deb7de0f81b415aab0d5ec4531 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:01:07.099 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:06.892+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:07.099 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:06 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:06.892+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:07.403 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.003009962 +0000 UTC m=+0.018002352 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:01:07.403 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.108028485 +0000 UTC m=+0.123020905 container create 35f3c52c66263e210f5beabdeffbd4722d91fc6ed8fe20bf2c3970fffa8c3242 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T03:01:07.403 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.319572522 +0000 UTC m=+0.334564908 container init 35f3c52c66263e210f5beabdeffbd4722d91fc6ed8fe20bf2c3970fffa8c3242 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T03:01:07.403 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.323081719 +0000 UTC m=+0.338074098 container start 35f3c52c66263e210f5beabdeffbd4722d91fc6ed8fe20bf2c3970fffa8c3242 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T03:01:07.403 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.402362333 +0000 UTC m=+0.417354714 container attach 35f3c52c66263e210f5beabdeffbd4722d91fc6ed8fe20bf2c3970fffa8c3242 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-22T03:01:07.848 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 22 03:01:07 smithi122 podman[116115]: 2024-03-22 03:01:07.613175654 +0000 UTC m=+0.628168024 container died 35f3c52c66263e210f5beabdeffbd4722d91fc6ed8fe20bf2c3970fffa8c3242 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-22T03:01:08.207 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:07 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:07.902+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:08.207 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:07 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:07.902+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:08.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.4.service' 2024-03-22T03:01:08.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:01:08.267 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-22T03:01:08.267 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-22T03:01:08.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.5 2024-03-22T03:01:08.594 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 systemd[1]: Stopping Ceph osd.5 for 67659b4a-e7f2-11ee-95cd-87774f69a715... 2024-03-22T03:01:08.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:08.593+0000 7ff1d5c70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T03:01:08.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:08.593+0000 7ff1d5c70700 -1 osd.5 84 *** Got signal Terminated *** 2024-03-22T03:01:08.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:08.593+0000 7ff1d5c70700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T03:01:09.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:08.931+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:09.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:08 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:08.931+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:10.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:09 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:09.930+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:10.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:09 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:09.930+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:11.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:10 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:10.939+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:11.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:10 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:10.939+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:12.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:11 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:11.935+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:12.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:11 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:11.935+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:13.348 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:12 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:12.917+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-03-22T03:00:29.222798+0000 front 2024-03-22T03:00:29.222831+0000 (oldest deadline 2024-03-22T03:00:53.922877+0000) 2024-03-22T03:01:13.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:12 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:12.917+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6814 osd.1 since back 2024-03-22T03:00:40.724765+0000 front 2024-03-22T03:00:40.724804+0000 (oldest deadline 2024-03-22T03:01:03.024852+0000) 2024-03-22T03:01:13.349 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:12 smithi122 ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5[102440]: 2024-03-22T03:01:12.917+0000 7ff1d2264700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.103:6822 osd.2 since back 2024-03-22T03:00:48.126487+0000 front 2024-03-22T03:00:48.126452+0000 (oldest deadline 2024-03-22T03:01:12.826485+0000) 2024-03-22T03:01:14.287 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 podman[116220]: 2024-03-22 03:01:14.027477368 +0000 UTC m=+5.537367739 container died e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T03:01:14.570 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 podman[116220]: 2024-03-22 03:01:14.368542522 +0000 UTC m=+5.878432882 container cleanup e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-22T03:01:14.570 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 bash[116220]: ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5 2024-03-22T03:01:14.848 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 podman[116240]: 2024-03-22 03:01:14.569816445 +0000 UTC m=+0.532626197 container remove e61a4141f08d3081283263bf78afd7f48517d25d200376c0b749caa10bf83c67 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T03:01:15.299 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 podman[116330]: 2024-03-22 03:01:14.876917292 +0000 UTC m=+0.017383809 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T03:01:15.300 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:14 smithi122 podman[116330]: 2024-03-22 03:01:14.979166668 +0000 UTC m=+0.119633184 container create 7901f307aae6495f2226b825d2d114a9b299d8e96a729970c09c154c255ad7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T03:01:15.300 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:15 smithi122 podman[116330]: 2024-03-22 03:01:15.215343261 +0000 UTC m=+0.355809783 container init 7901f307aae6495f2226b825d2d114a9b299d8e96a729970c09c154c255ad7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T03:01:15.300 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:15 smithi122 podman[116330]: 2024-03-22 03:01:15.219050959 +0000 UTC m=+0.359517475 container start 7901f307aae6495f2226b825d2d114a9b299d8e96a729970c09c154c255ad7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-22T03:01:15.300 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:15 smithi122 podman[116330]: 2024-03-22 03:01:15.298705727 +0000 UTC m=+0.439172241 container attach 7901f307aae6495f2226b825d2d114a9b299d8e96a729970c09c154c255ad7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-22T03:01:15.598 INFO:journalctl@ceph.osd.5.smithi122.stdout:Mar 22 03:01:15 smithi122 podman[116330]: 2024-03-22 03:01:15.509828765 +0000 UTC m=+0.650295279 container died 7901f307aae6495f2226b825d2d114a9b299d8e96a729970c09c154c255ad7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-67659b4a-e7f2-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T03:01:16.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67659b4a-e7f2-11ee-95cd-87774f69a715@osd.5.service' 2024-03-22T03:01:16.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T03:01:16.175 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-22T03:01:16.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T03:01:19.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi103.stderr:ceph-fuse[77676]: fuse finished with error 0 and tester_r 0 2024-03-22T03:01:40.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T03:01:55.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T03:01:55.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T03:01:55.535 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-22T03:01:55.537 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi103/crash 2024-03-22T03:01:55.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash -- . 2024-03-22T03:01:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T03:01:55.575 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-03-22T03:01:55.577 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi122/crash 2024-03-22T03:01:55.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash -- . 2024-03-22T03:01:55.604 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T03:01:55.605 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-03-22T03:01:55.605 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-22T03:01:55.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-22T03:01:55.652 INFO:teuthology.orchestra.run.smithi103.stdout:1711074608.780399 mon.smithi103 (mon.0) 922 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T03:01:55.652 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-22T03:01:55.652 INFO:tasks.cephadm:Compressing logs... 2024-03-22T03:01:55.653 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T03:01:55.695 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T03:01:55.723 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.xhmfjp.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi103.pfvmui.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.xhmfjp.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi103.pfvmui.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.gtuatp.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi103.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.gtuatp.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-22T03:01:55.724 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi103.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi122.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log: gzip/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi122.log: -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi122.prgtti.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi122.prgtti.log: gzip -5 92.2% --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.lvsegg.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.ztrkgp.log 2024-03-22T03:01:55.729 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.lvsegg.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T03:01:55.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.ztrkgp.log: 91.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T03:01:55.730 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log 2024-03-22T03:01:55.730 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log: 85.5% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T03:01:55.730 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T03:01:55.731 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.1.log: 91.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T03:01:55.734 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log: 94.4% 89.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T03:01:55.734 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.lvsegg.log.gz 2024-03-22T03:01:55.734 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-03-22T03:01:55.736 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T03:01:55.749 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log 2024-03-22T03:01:55.750 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log: 85.5% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T03:01:55.754 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T03:01:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/ceph-client.1.log: 91.0% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T03:01:55.762 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-22T03:01:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log: 89.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T03:01:55.775 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T03:01:55.796 INFO:teuthology.orchestra.run.smithi122.stderr: 90.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi122.prgtti.log.gz 2024-03-22T03:01:55.843 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.2.log: /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T03:01:56.226 INFO:teuthology.orchestra.run.smithi103.stderr: 89.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mgr.smithi103.pfvmui.log.gz 2024-03-22T03:01:56.362 INFO:teuthology.orchestra.run.smithi122.stderr: 92.5% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi122.log.gz 2024-03-22T03:01:57.054 INFO:teuthology.orchestra.run.smithi103.stderr: 90.2% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mon.smithi103.log.gz 2024-03-22T03:02:00.229 INFO:teuthology.orchestra.run.smithi122.stderr: 93.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-22T03:02:01.184 INFO:teuthology.orchestra.run.smithi103.stderr: 95.1% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.gtuatp.log.gz 2024-03-22T03:02:01.294 INFO:teuthology.orchestra.run.smithi122.stderr: 93.5% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-22T03:02:01.634 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-22T03:02:01.944 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-22T03:02:02.321 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-22T03:02:03.382 INFO:teuthology.orchestra.run.smithi122.stderr: 93.4% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-22T03:02:10.009 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-03-22T03:02:10.009 INFO:teuthology.orchestra.run.smithi122.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-03-22T03:02:10.285 INFO:teuthology.orchestra.run.smithi103.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-22T03:02:41.177 INFO:teuthology.orchestra.run.smithi122.stderr: 93.2% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi122.ztrkgp.log.gz 2024-03-22T03:02:41.178 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-22T03:02:41.179 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m45.469s 2024-03-22T03:02:41.179 INFO:teuthology.orchestra.run.smithi122.stderr:user 1m15.713s 2024-03-22T03:02:41.179 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m2.394s 2024-03-22T03:02:42.840 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/67659b4a-e7f2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi103.xhmfjp.log.gz 2024-03-22T03:02:42.842 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-22T03:02:42.842 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m47.134s 2024-03-22T03:02:42.842 INFO:teuthology.orchestra.run.smithi103.stderr:user 1m24.717s 2024-03-22T03:02:42.842 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m2.776s 2024-03-22T03:02:42.842 INFO:tasks.cephadm:Archiving logs... 2024-03-22T03:02:42.843 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi103/log 2024-03-22T03:02:42.844 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T03:02:55.253 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi122/log 2024-03-22T03:02:55.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T03:03:07.108 INFO:tasks.cephadm:Removing cluster... 2024-03-22T03:03:07.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 --force 2024-03-22T03:03:07.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67659b4a-e7f2-11ee-95cd-87774f69a715 --force 2024-03-22T03:03:08.202 INFO:tasks.cephadm:Removing cephadm ... 2024-03-22T03:03:08.202 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T03:03:08.219 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T03:03:08.234 INFO:tasks.cephadm:Teardown complete 2024-03-22T03:03:08.234 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-22T03:03:08.246 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-22T03:03:08.246 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-22T03:03:08.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T03:03:08.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T03:03:08.353 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-22T03:03:08.364 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-22T03:03:08.365 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T03:03:08.367 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T03:03:08.379 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-22T03:03:08.381 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +2315us[+2305us] +/- 85ms 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1766us[-1776us] +/- 69ms 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -132us[ -142us] +/- 83ms 2024-03-22T03:03:08.385 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T03:03:08.386 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T03:03:08.387 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-22T03:03:08.387 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +2383us[+2383us] +/- 85ms 2024-03-22T03:03:08.387 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1693us[-1685us] +/- 69ms 2024-03-22T03:03:08.387 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -84us[ -84us] +/- 83ms 2024-03-22T03:03:08.387 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T03:03:08.387 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-22T03:03:08.398 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-22T03:03:08.399 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-22T03:03:08.410 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T03:03:08.512 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T03:03:08.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T03:03:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:03:08.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T03:03:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:03:08.653 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-22T03:03:08.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-22T03:03:08.676 INFO:teuthology.task.internal:Duration was 3274.349734 seconds 2024-03-22T03:03:08.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-22T03:03:08.688 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-22T03:03:08.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T03:03:08.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T03:03:08.731 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T03:03:08.736 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T03:03:08.910 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-22T03:03:08.911 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-03-22T03:03:08.911 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T03:03:08.970 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-03-22T03:03:08.970 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T03:03:08.997 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-22T03:03:08.997 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T03:03:09.013 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T03:03:09.066 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-22T03:03:09.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T03:03:09.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T03:03:09.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-22T03:03:09.988 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-22T03:03:09.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T03:03:10.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T03:03:10.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-22T03:03:10.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:03:10.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:03:10.082 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-03-22T03:03:10.112 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-03-22T03:03:10.391 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:03:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:03:10.406 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:03:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:03:10.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-22T03:03:10.460 INFO:teuthology.task.internal:Transferring archived files... 2024-03-22T03:03:10.461 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi103 2024-03-22T03:03:10.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T03:03:10.520 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797/remote/smithi122 2024-03-22T03:03:10.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T03:03:10.562 INFO:teuthology.task.internal:Removing archive directory... 2024-03-22T03:03:10.563 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T03:03:10.565 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T03:03:10.611 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-22T03:03:10.624 INFO:teuthology.task.internal:Not uploading archives. 2024-03-22T03:03:10.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-22T03:03:10.637 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-22T03:03:10.637 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T03:03:10.639 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T03:03:10.652 INFO:teuthology.orchestra.run.smithi103.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 22 03:03 /home/ubuntu/cephtest 2024-03-22T03:03:10.652 INFO:teuthology.orchestra.run.smithi103.stdout: 534468 4 d--------- 2 ubuntu ubuntu 4096 Mar 22 02:26 /home/ubuntu/cephtest/mnt.0 2024-03-22T03:03:10.653 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-03-22T03:03:10.653 INFO:teuthology.orchestra.run.smithi103.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-22T03:03:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:03:10.656 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-22T03:03:10.657 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-22T03:03:10.674 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-22T03:03:10.780 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-22T03:03:10.780 INFO:teuthology.nuke:Checking targets against current locks 2024-03-22T03:03:10.808 DEBUG:teuthology.nuke:shortname: smithi103 2024-03-22T03:03:10.808 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T03:03:10.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:59:08.689238', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9gYB2JLCkhhT7tRIg+YLlsEkMATFzPQQ8SJHSXQFHF6cKHd1tOP43ruhbCek5CtFhzw1V8mRnW5ArGBuEzOtk='} 2024-03-22T03:03:10.835 DEBUG:teuthology.nuke:shortname: smithi122 2024-03-22T03:03:10.835 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T03:03:10.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615797', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:59:08.691214', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI0jM9zBtKu98jQ4JaAW1AjLMKw+ByZWirpv/cqAd7JP3LSLTbheXI3hQoSBW7mH4qnef52ht82Qro/uCBqaaO0='} 2024-03-22T03:03:10.883 INFO:teuthology.orchestra.console.smithi103:Power off 2024-03-22T03:03:10.884 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T03:03:10.911 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-03-22T03:03:10.911 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:10.926 INFO:teuthology.orchestra.console.smithi122:Power off 2024-03-22T03:03:10.927 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T03:03:10.936 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-22T03:03:10.949 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-03-22T03:03:10.950 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:10.974 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-22T03:03:14.937 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:14.975 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:15.048 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-22T03:03:15.086 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-22T03:03:19.050 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:19.086 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:19.161 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-22T03:03:19.196 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-22T03:03:23.163 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:23.197 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:03:23.273 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-03-22T03:03:23.307 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-03-22T03:03:23.374 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-03-22T03:03:23.408 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-03-22T03:03:23.512 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3274.3497343063354 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_pdonnell@teuthology status: fail success: false 2024-03-22T03:03:23.512 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T03:03:23.598 INFO:teuthology.run:FAIL